Yeah this is what they end up doing. The language itself isn't that hard for them, it's separating the language from the traps in the spec. I also tried making a hostile esolang, without all the prompt injections and whatnot, and it was very easy for LLMs. (It was mainly based around tokenizer quirks, like the r in strawberry test, but becomes trivial with scripts).

This depends on them being able to interpret the spec correctly. They could certainly write a Python script, but their implementation would need to match the spec. I haven't seen any model attempt it this way so far.

An LLM could equally decompile the binary, clean up the code, and recode it in one /goal.