I tried running this on a H100 and got 190ms compared to Jev's 170ms. Maybe I set it up wrong?
It seems like caching has a huge contribution towards the low latency numbers they report. With a single request and no cache it appears to be quite slow, roughly in line with what you observed.
alt Hacker News
It seems like caching has a huge contribution towards the low latency numbers they report. With a single request and no cache it appears to be quite slow, roughly in line with what you observed.