logoalt Hacker News

aleksiy123today at 8:39 PM3 repliesview on HN

Agree with most of these.

One thing I don’t love about codex/sol is I find it tends to overengineer and be overly cautious.

I was using it to do create some scraping + data processing.

It went kind of crazy on the provenance, need at least 3 sources of consensus before promoting facts type bullshit.

defined a bunch of enums and gates.

I just wanted scrape some site data and put it into a SQLite dB. Like chill codex.

I feel like Claude is better at that.


Replies

spudlyotoday at 9:41 PM

I've noticed that too with Sol (xhigh) it starts with pretty good architectural goals, but really get lost in the weeds with certification and validation. Like it won't trust native tools like (in my case packer, Ansible, gcloud) to fail reliably under possible error conditions, and insists on implementing custom verification.

It also doesn't have a clear idea of what the actual threat model is, and builds all kinds of extremely defensive systems to account for imagined hostile actors. I'm like "Dude, it's only our systems that are creating these SVGs, they're never going to be user supplied, so you don't need to write an entire validation and sanitation framework here."

It also seems to treat the desired initial state of something as a permanent invariant and designs elaborate tests to ensure that it remains that way. Then when you make one little change it has to go and update a ton of tests it created.

I've had to rip out a bunch of overengineered jank from several feature implementations, and in doing so I ended up having to create retrospective documents that warn against this kind of behavior that I'll have the model review whenever a plan begins to go sideways.

show 1 reply
aleksiy123today at 8:49 PM

As a follow up.

I feel like codex/sol is better at well scoped hard technical problem.

Where it can sort of run this brute force analytical loop.

Like doing performance optimization or other search type problems. I think the math proofs are good examples of this.

show 1 reply
akssstoday at 8:50 PM

What, you don't like it proposing sha256 checksums and state enums for everything?? :D It does seem to have this predilection, but I find it calms down if I tell it to simplify for "this phase". Sure, we'll pick that up down the road.. I promise. :D