alt Hacker NewsWas really cool to see yous use Engrams to cut down compute!
Given its basically an O(1) lookup with disk space being the main constraint, I was curious if you've tried ablating engram layers and sizes across your setup?
Also, why mHC over attention residuals?
Yes, we ablated Engrams rigorously and found that it returned world knowledge like FFN without without compute expenditure.