Earlier this year I explored exactly this (although only at a small scale) inspired by Lattice Deduction Transformers. https://github.com/emberian/clairnets. It seemed promising at small scales but I have yet to revisit it.