alt Hacker Newswould be nice if the transformers code for one of these frontier LLM models got leaked, HN will have a field day with a reveal like that
would be nice if the transformers code for one of these frontier LLM models got leaked, HN will have a field day with a reveal like that
I doubt there is anything special about the transformer code the frontier labs use. The only thing proprietary in it are probably the infrastructure-specific optimizations for very large scale distributed training and some GPU kernel tricks. The real moat is the training data, especially the RLHF/finetuning data and verifiable reward environments, and the GPU clusters of course.
The open source models are quite close, and they'd probably be just as good with the equivalent amount of compute/data the frontier labs have access to.