logoalt Hacker News

nicce • today at 12:06 AM • 0 replies • view on HN

Bonsai 2 has a bit different architecture as far I understand and speculative decoding is not so great yet. Before that point, original model is faster if you have the memory. Would be nice to see the speed with proper speculative decoder.