I've been interested in training a transformer from scratch for the same learning reasons. The GPU cost/availability seemed prohibitive to do anything useful but you seem to have flipped that on its head. I love your outside the box approach.