alt Hacker NewsDisclaimer, I work on Gemma and open models at Deepmind and the opinions here are my own
There were open models from EleutherAI (GPT-Neo), Google Brain (T5X, Bert), and HuggingFace was promoting open models (and others doing open work I haven't listed here) all prior to 2023 and the big Chatgpt moment.
https://github.com/EleutherAI/gpt-neo/releases
https://github.com/google-research/bert
https://github.com/google-research/t5x
If you're learning about AI models it's still worthwhile to review these models and codebases because they continue to be the basis of the technology that's being produced today! It'll give you a good perspective of how things have changed, similar to say learning about propeller planes before moving onto modern jet engines.
Replies
I honestly think of the T5 model family to sort of be the real beginning of this open model craze - I know BERT was already popular for classification etc, but T5 was the first sort of generally useful model, was exceptionally simple to fine-tune, and is still in use today (t5 base is still averaging over a million downloads a month on huggingface), has tons of variants and sort of kickstarted this whole community. US labs get a lot of flack but Google has been super supportive and open in a lot of ways that has pushed this whole endeavor forward, even if I feel like they've sort of declined in transparency in recent years with their open models.
Thanks!
When is Gemma5 coming out? :-)
Hey, with the changes at Deepmind, is the Gemma project still ongoing?
And yet it's not called eleuther.cpp or bert.cpp. And the famous subreddit isn't called r/localbert but r/locallama.
[dead]
Google had done much to lay the framework for current models (and is currently doing good stuff with Gemma). But after ChatGPT took off I don’t think Google continued to release any big models freely before meta released its models (maybe it was because Google didn’t have any models at all during the time)