> I would still pay ~500 for a chip that runs 10kt/s of a ~100b model on my machine even if the half life is 6months.

....

:T

Considering the 8B model uses 53 billion transistors, that's 6.625 transistors per parameter.

https://taalas.com/products/

Assuming they can get it down to 3 (somehow), that's still 300 transistors, or 5.565 RX 9070s.

https://www.techpowerup.com/gpu-specs/radeon-rx-9070.c4250

You're looking at

1) waiting for another 3-5 generations of transistor improvements before it can fit into a single conventional chip, or

2) another generation before getting a monster of a chip (1000+ mm^2), and prices for flawless etching scale quadraticly (likely $1000+ for manufacturing costs alone).

Could happen, but it's a long shot for a market that could be satiated by specialized accelerators.

10000%

500 what? You’re missing the unit