logoalt Hacker News

jpgvm • yesterday at 7:05 PM • 3 replies • view on HN

Disclaimer: I was at Groq and now work at NVIDIA on LPX.

Specifically: https://www.nvidia.com/en-us/data-center/lpx/

We ran a pure-LPU demo for HotChips that AA benchmarked: https://developer.nvidia.com/blog/how-nvidia-groq-3-lpx-unlo...

TLDR: LPUs are very efficient, quite a lot more efficient and faster tok/s/u than anything else at this years Hot Chips.

It's my personal opinion that with the way models change you don't want silicon that is too specialised but it needs to be specialised enough. LPUs are designed to strike this balance such that they remain flexible if model architectures change drastically.


Replies

buildbot • yesterday at 8:26 PM

In my personal opinion as well "specialized enough" chips are exactly the right way to go - though disclaimer, also At Nvidia, working on spec dec/external drafts :)

With things like https://developer.nvidia.com/blog/co-designing-ai-models-usi... you can specialize one chip for decode/drafting, and the other for prefill and draft verification.

Melatonic • yesterday at 8:36 PM

Is LPX only going for rack size integration or is the plan to offer an LPX companion chip to something like a 6000 Blackwell ?

Could see a board with something like two SXM slots (maybe or similar) and one being for LPX optionally be cool

justinc-md • yesterday at 7:14 PM

What's your opinion of high bandwidth flash? It seems ideal for storing and retrieving model weights. Do you see that dramatically changing inference chip design over the next couple of years?