I built a brand new language to test this[1]. Not only is the language different to basically any other language but it also tries to be adversarial against LLM understanding.

The best models can still make sense of it[2], though the tasks so far have been pretty basic. But I do think it gives some evidence that languages which aren’t well represented in an LLM’s training can still be reasoned about and written well by LLMs.

1 - https://killswitch-lang.org

2 - https://bench.killswitch-lang.org