alt Hacker NewsI've been using Qwen3.6 35B A3B, and with reasoning turned on, I'd say 2/3 (give or take) of the tokens for a response are thinking tokens. Which at 70+ tps locally, that isn't that awful. I run an 80k context across 4-10 "agents" for my solo TTRPG, where Qwen is the GM, each NPC at a location, the director, and the narrator.
Each turn is about 45-60 seconds to generate all of the various responses. The GM and director have reasoning on, and the NPCs/Location/Narrator do not.
It's a fairly good "engine" for that. I'm not sure how a denser Qwen would do here regarding speed.
Replies
This sound very interesting, do you have any resource I could look at? Me and my son did a very rudimentary (compared to yours) setup to play Paranoia, but this is at another level.
I'm working on something similar. My biggest annoyance is that the overly-helpful LLM was making every die roll succeed. I ended up building some tooling around rolling dice. Also some tooling around character stats and inventory management, so those don't get lost in context compression.
Is there some sort of dedicated tool for this type of setup, or did you hand-craft it ?
Are you running inference in parallel? 70 tps seems low for parallel execution.
I like the tabletop RPG use case, and wanted to say: If your hardware likes it you should check out Gemma 4 for creative DMing use case. I found it to be much better at holding the plotlines and being creative on gaming turns. My experimental case was an audio-only Zork and Gemma 12B and even E4B were pretty good!