Oh nice, hadn't seen it. It looks good, and they've clearly got some decent stuff I don't (chrome extension, embedding in model cards, family pages).
The main difference appears to be how the graph gets made. From what I can tell they analyze the config on the server (their page says the first request can take a few minutes). I actually build the model on PyTorch's meta device and run a fake forward pass through it, so I get the real execution order and the tensor shapes at every module. That's what the animated replay and the compute/KV cache numbers are based on.
alt Hacker News
Oh nice, hadn't seen it. It looks good, and they've clearly got some decent stuff I don't (chrome extension, embedding in model cards, family pages).
The main difference appears to be how the graph gets made. From what I can tell they analyze the config on the server (their page says the first request can take a few minutes). I actually build the model on PyTorch's meta device and run a fake forward pass through it, so I get the real execution order and the tensor shapes at every module. That's what the animated replay and the compute/KV cache numbers are based on.