alt Hacker News> On-policy distillation: The expanded student model learns token-specific feedback from teacher distributions generated on self-sampled sequences.
The weird thing is to see this based off the KV-Cache in the diagrams.
Completely different from the Databricks RAFT [1].
My team built "Dreamtime" earlier this year [2], which used a different mechanism to distill out learnings across multiple sessions.
This didn't pull from the token layer, but from user nudges on unsatisfactory responses.
This works like a sawtooth wave where we accumulate context, check for dissonances (a user gets to say "It depends" and fix it), compress it back, retire the old one.
The audit requirements are the real problem for customers, where you need to be able to ask "When did this change? and who changed it?" which was hard to answer from the lowest layer of the model unless you can retain lineage to the original sessions.
A model we train can't serve as an accomplice to a user who wants to hide some financial crimes by nudging the agent repeatedly.