The funniest thing I've seen GPT do was a while back when I had it try to implement ORCA (Optimal Reciprocal Collision Avoidance). It is a human made algorithm for entities where they just use their own and N neighbours' current radii along with their velocity to calculate mathematical lines into the future, so that they can avoid walking into each other.

It came very close to success, but there were 2 or 3 big show-stopping bugs such as it forgetting to update the spatial partitioning when the entities moved, so it would work at the start but then degrade over time.

It believed and got stuck on thinking that it must be the algorithm itself that was the problem, so at some point it just stuck a generic boids solution into the middle of the rest. To make it worse, it didn't even bother to use the spatial partitioning and they were just brute force looking at their neighbours.

Had this been a real system it might have made its way into production, which makes one think about the value of the AI code out there. As it was I pointed out that bit and asked about it, at which point it admitted that it was definitely a mistake and then it removed it.

I had previously implement my own version of the algorithm and it took me quite a bit of time, but during that I built up the mental code model and understood both the problem and solution by the end. In comparison it easily implemented it 10-30x faster than I did but would never have managed to complete the project on its own. Also if I hadn't previously implemented it myself and had just tried to have it do the heavy lifting then I wouldn't have understood enough of what it was doing to overcome its issues and get the code working properly.