alt Hacker NewsI would say that LLMs have really good understanding of LSPs, but they are not necessarily good at building them. Had I blindly followed the proposed flow, Rust Glancer wouldn't have reached a stage where it is at least remotely usable. At some point the size of the project becomes too big for LLM to fit in its context window, and with the tendence to add code rather than remove, the bloat can explode really quickly. At a pretty early stage, I did not catch a situation where LLM suggested an extremely stupid design (because I wasn't familiar with the scope enough at the moment), and it implemented a whole new parallel hierarchy of functionality that was already implemented but in a _slightly different_ form. When I realized it, it took nearly two weeks to unfuck the situation.
So all in all -- yeah, LLMs can be good _domain experts_ when you build an LSP, but a) I wouldn't trust them blindly, and b) the quality of code is still very much your responsibility.
Replies
> At some point the size of the project becomes too big for LLM to fit in its context window, and with the tendence to add code rather than remove, the bloat can explode really quickly.
While it's certainly worse with LLMs, humans, especially teams of humans, have the same issue. This is where being good with software architecture is a superpower for working with LLMs, since good separation of concerns and modularity help this immensely. LLMs can even often do this themselves if you are very harsh and critical about this in design review stages, but it requires the skill to be able to recognize it and to have a "vision" for the software architecture.
I don't think I disagree with any of that.
TLA+ is a relatively simple language so I think it's a good candidate for this kind of stuff; most of the stuff in the generated LSP also just proxies straight to the official command line tools. It's certainly a simpler language than Rust, so I think it's easier for Claude to keep a higher percentage of stuff in context, and at least using the TLA+ bindings seems to work pretty well.
I haven't done it since my laptop has lots of RAM, but I suspect that I will likely edit the generated code to eat less at some point.
> LLMs can be good _domain experts_ when you build an LSP, but a) I wouldn't trust them blindly, and b) the quality of code is still very much your responsibility.
That's the feeling I have with these models as well. When I read about companies replacing devs due to AI I really hope its not actual coding/design jobs they're replacing. Unless they want to become evolutionary dead ends.