I've been interested in faster attention and smaller models for some time but haven't had the time to do serious research so I can't answer your questions.

However, everything you do sounds very interesting, useful and well thought out, please keep doing it, I'd encourage others to work in the same direction too.

I hope, more of us can find the time for more than best wishes in the near future.