alt Hacker NewsI have a hunch (based on using the Kimi models to write some clojure) that the article's AST point is exactly wrong. I had to spend a lot of time cleaning up when it miscounted closing parens, which implies that while the LLM may be operating on the AST, mapping to and from the token stream is harder, not easier, when the individual tokens carry less information.
If the author is finding that it works well, I suspect there's something else (code or comment style, maybe) that's compensating for it which didn't seem worth mentioning.
Replies
Indeed, I use LLMs on some hobby Racket programs, and for Emacs Lisp, and it always messes up parentheses; then burns tokens trying to count them over and over (feels like "the number or rs in strawberry" problem).
I've found https://github.com/shcv/parenmedic to be somewhat helpful, which diagnoses parentheses issues based on when they disagree with indentation, rather than simply counting. The fact this works indicates that LLMs are paying more attention to whitespace than "actual structure".
>mapping to and from the token stream is harder, not easier, when the individual tokens carry less information.
this is exactly what I would expect. Also if you are training on code on the internet, what are the chances that you get these kinds of structural errors, especially on code in blogs etc.?
Lisp is known for being easy to drop a paren on accident so you saying not closing parens jibes with what I expect, that the LLM would predict wrong every now and then about if it should put one in a particular place.
Paren issues with Clojure probably mean your functions are too long? I like to keep mine down to 8-10 lines or less when possible, keep them flat and composable, use threading macro and then transducers for performance. At least that's how I read it. AST might not matter much either way, or in a stranger way, because the LLM's corpus and progression through code will give it a kind of shadow or grooves of an ast, but it isn't making or receiving any ast from this piece of code.
Still, Elixir scores best on the TenCent AutoCodebench, by far actually, "despite" being like Clojure built with an AST and immutability. Clojure wasn't part of those tests but I use both daily with LLMs (mostly Codex) and imagine it's on par with Elixir. The repl is better than Elixir's. Both have serious strengths.