We spent a lot time evaluating which model is the best to use in our atopile agent harness and thought the benchmark and results are interesting enough to share.

Evaluating a model for specific use-cases like this that are bit more broad scope than a lot of typical microbenchmarks turned out to be quite the challenge.

Determining deterministically what a "good" electrical design is non-trivial and most electrical-engineers rely on good ol' intuition and decade long experience, so we tried to formalize it.

The only reason it was somewhat possible within a reasonable amount of effort is because atopile provides most of the complicated infrastructure for the benchmark: constraint solver, simulation, code-first electronics modeling, erc & drc checks, high level models of passive electronics components etc