logoalt Hacker News

Test/verification technique effectiveness with Astra

85 points • by luu • yesterday at 7:33 AM • 15 comments • view on HN

Comments

reedf1 • today at 6:29 AM

The really powerful thing you've done here is begin to create a taxonomy of agent patterns. Given we are pretty much all engineers in this space; I am shocked at the lack of engineering taking place. People are picking patterns they find intuitive; for various invented reasons, this is fine in the beginning, but we need to start objectively measuring their performance. It needs to be said that there are an infinite number of plausible agent patterns, and we should view all (from skills to MCP to Audit, whathaveyou) as something that may go the same way as netscape until there is real justification.

➕ show 1 reply
vessenes • today at 1:43 PM

I like research notes, and I like reading what Dan is up to, always. That said, some helpful context to this post - Dan is working on some engineering questions -- how can we best use the tools we have, and how does that evolve?

His general test methods this year have been to provide a specification document of some sort and then prompt a clean room implementation of the spec, which can be tested against holdout tests.

Things move pretty quickly - this summer, his test was Zstd and results were at like 25-30% of tests passing at best. He now considers Zstd saturated, as is IMAP from RFC; pandoc is one that he's used throughout, and is not yet perfect.

With all that in mind, read away.

kooi • today at 6:19 AM

Nice post. Good motivation to look more at formal verification. There was that Aristotle post today which seems like a prompt driven, plug and play verifier using lean.

Makes me think that LLM assisted formal requirements generation for a roll your own "Aristotle" is possible.

Just thinking about our own projects in the signal processing domain. A nice path could be: human language -> temporal logic specs for black and white box run-time testing.

There are code generators which will turn formal logic into a run time monitor. In the past, I've never used them past toy examples because its so verbose.

> The point of this post is to demonstrate that it’s possible in the era of AI coding to still make decisions based on evidence rather than vibes.

That's the difference between an engineer and a script kiddie.

Domain expertise will always be valuable.

➕ show 2 replies
__mharrison__ • today at 2:43 PM

Are my eyes just failing me or is there no date on this post?

Otherwise, I love this. I was pretty involved in Python testing circles (there was an old Python in tears mailing lists,) and would love to find similar for agentic testing.

➕ show 1 reply
ArtRichards • today at 6:36 AM

I use what I would say is a TDD skill, but I couldn't find anywhere in the article what you meant by TDD? How did you prompt it to use TDD? Why was it less effective? I feel targeted ;)

➕ show 1 reply
polskibus • today at 6:48 AM

Great write up, interesting observations. I wonder where’s the budget for those experiments coming from?

scotty79 • today at 12:41 PM

If your AI codes in Rust telling it to use Proptest sounds like no brainer. Possible quality improvements and cost/time reductions.

scotty79 • today at 7:31 AM

They have another interesting investigation into languages for AI.

https://danluu.com/pl-tokens/

I'd love to see more of that. I'm currently defaulting to Rust in my software building but F# looks super strong there. On par with Python for correctness with Ultra effort.

I wonder how Rust and other languages are doing in Astra 6/Sol 6.1 and on more projects.

patryk_builds • today at 3:22 PM

[flagged]

haukebri • today at 7:16 AM

[flagged]

makercfd • today at 3:03 PM

[flagged]

lubosPetorvic • today at 10:58 AM

[flagged]