logoalt Hacker News

abraxas • yesterday at 9:27 PM • 2 replies • view on HN

I'm not following the local mdoel scene too closely but this seems quite amazing. Is this able to be run on Apple silicon too?

Replies

Havoc • yesterday at 9:47 PM

Their first 27B bonsai was able to run on an iphone.

kamranjon • yesterday at 9:28 PM

"Ternary Bonsai 2 27B reaches up to 143 tokens/second on NVIDIA GeForce RTX 5090 and 46.8 tokens/second on M5 Max. On an RTX 4090, Ternary Bonsai 2 27B consumes just 0.714 mWh/token, making it 40% more energy-efficient than an 8B model running in full-precision."

➕ show 2 replies