Floating point from scratch: Hard Mode

Well, that escalated quickly. From skimming the first few screenfuls of the page, I thought it was going to be about a beginner programmer's first time dissecting the low-level bit structure of floating-point numbers and implementing some of the arithmetic and logic from scratch using integer operations.

And then she writes about logic cells, hardware pipeline architectures, RTL (register-transfer level), ASIC floorplans, and tapeout. Building hardware is hard, as iteration cycles are long and fabricating stuff costs real money. That was quite a journey.

Her "about" page adds more context that helps everything make sense:

> Professionally, I have a Masters in Electrical and Computer Engineering, have previously worked as a CPU designer at ARM and an FPGA engineer at Optiver. -- https://essenceia.github.io/about/

Nice write-up.

Let me offer a nitpck: in the "Gradual underflow" section it says this about subnormal numbers:

    Bonus: we have now acquired extra armour against a division by zero:

    if ( x != y ) z = 1.0 / ( x - y );

Think about it this way: the smallest subnormal double is on the order of 10^-324, but the largest double is on the order of 10^308. If `x - y` is smaller than 10^-308, `1.0 / (x - y)` will be larger than 10^308, which can't be represented and must overflow to infinity.

This C program demonstrates this:

    #include <stdio.h>
    #include <float.h>
    #include <math.h>

    // return a (subnormal) number that results in zero when divided by 2:
    double calc_tiny(void)
    {
        double x = DBL_MIN; // the normal number closest to 0.0
        while (1) {
            double next = x / 2.0;
            if (next == 0.0) {
                return x;
            }
            x = next;
        }
    }

    int main(void)
    {
        double y = calc_tiny();
        double x = 2.0 * y;
        if (x != y) {
            double z = 1.0 / (x - y);
            printf("division is safe: %g\n", z);
        } else {
            printf("refusing to divide by zero\n");
        }
    }

I had to do an FP8 adder as my final project for my FPGA lab. It was at least a full page of state machine. And I write small. I ended up just not doing the rounding and truncating instead because I was so done with it.

Consider me educated on the mantissa. That's a nifty pedantry.

Typo in the c++?

> cout << "x =/= x: " << ((x=x) ? "true" : "false") << endl;

Should be x != x?

For the leading 0 counter, I've found it's even better for the tool if I use a for loop. Keeps things scalable and it's even less code. I'm not understanding this takeaway though

> Sometimes a good design is about more than just performance.

The good design (unless the author's note that it's easier to read and maintain makes it worse design?) was better performing. So sometimes a good design creates performance.

Likewise for pipelining: it would have been interesting to know if the tools can reorder operations if you give them some pipeline registers to play with. In Xilinx land it'll move your logic around and utilize the flops as it sees fit.

I think the thing that truly scares me about floating point is not IEEE-754 or even the weird custom floating points that people come up with but the horrible, horrible things that some people think passes for a floating point implementation in hardware, who think that things like infinities or precision in the last place are kind of overrated.

Everyone who has ever had to build a floating point unit has hated it with a passion, I've watched in done from afar, and done it myself

I love this article, the edge cases are where the seeming “simplicity” of floating-point numbers breaks down

Recently wrote a chapter in tiny-vllm course about floats in context of LLM inference, much shorter and not that deep as this one, for anyone interested in topic you might like it too https://github.com/jmaczan/tiny-vllm?tab=readme-ov-file#how-...

Super cool to tape out a nearly 500MHz systolic array!!! I wish sometimes I had gone far enough with my hardware education to be able to do more than simple FOGA designs.

A really cool thing to see would be the newer block scaled fp4/fp8 data types. Maybe for their next asic they can try it haha - https://arxiv.org/abs/2310.10537

I wish I knew more people like this author. Then, maybe, I would have maintained my faith in humanity.