So, on a crude calculation, for a 3090 with 936.2 GB/s, a model that has 20 GB of active params would run at 45tps and one with 3GB active params at 300tps?
In practice, I don't think I saw over 100tps on a 3090, for a local 20-30B model, be it MoE or not.
alt Hacker News
So, on a crude calculation, for a 3090 with 936.2 GB/s, a model that has 20 GB of active params would run at 45tps and one with 3GB active params at 300tps?
In practice, I don't think I saw over 100tps on a 3090, for a local 20-30B model, be it MoE or not.