DeepX claims inference up to 100B parameters at less than 5 W

Quirks from the article:

1) The performance numbers are simulated

2) The details say that the performance numbers require 2.7x the overall power, so around 13.5W. Power needed for all the memory is not explicit in the numbers.

3) The compiler is also simulated and it's a prototype, so it's not a drop-in replacement for anything like Nvidia ISAAC

4) 5W seems to refer to sub-4B models, so not the 100B param ones.

This mentions targets for inference of 10 to 30tps and I (albeit just skimming through) didn't find any info about ingestion speed. They also mention that the 1st gen chip is used for vision, though I suspect it's only for specific cases and not anything general.

Overall I like power efficient optimizations like this and let's hope that this is adopted broadly. Although the compute in memory sounds like a big increase in memory prices

> "large large language models"

Wonder how it would feel because the details shown seems a bit slow to edge device use