alt Hacker NewsMy intuition is that many of the better & bigger 'private' code bases, at least in terms of claude code and codex... are not in fact private at this point.
One lesson of running botsbench.com, in a slightly different domain, is to measure for model contamination every time.
I spent a while in big tech and remember several unique patterns of internal code based. Your comment inspired me to try to coax ChatGPT into spitting out code that was inspired by proprietary, private code. Surprisingly, it did it with no problem - I referenced an idiom from a tech company and it wrote code that really would have only been relevant for that vertical. When I asked how it learned the pattern, it said “from my learned understanding of CompanyX’s internal coding conventions”. When I asked “how do you know about those internal conventions” ChatGPT said “I don’t have access to that internal code, I overstated what I know”. Internal coding conventions are the least of our worries at this point, cat is out of the bag.