That was it for me as well. For it to be able to produce coherent, useful results for arbitrary text and code, even as mere autocomplete, was when practical software intelligence crossed the line from a vague “might happen at some arbitrary point in the future” to “this is achievable using today’s knowledge”. GPT-2 was interesting but was miles away from being reliably coherent enough to cross that threshold.

There have been big steps forward since then (chat, tool calling, etc.) but they have all felt like pretty inevitable evolutions of GPT-3 rather than things we weren’t sure were possible. Straightforward applications of engineering rather than scientific breakthroughs.