Ask HN: How to deal with shipping AI-written code you don't understand?

Formal verification, pick a language that has it integrated and prove relevant post conditions.

AI still does not know how to even write code that makes impossible state unpresentable so testing won’t help.

Product doesn’t care. And most of us will get replaced by agents. I guess around 30% of the engineering workforce will keep their jobs which will be mainly to keep the product up and running, doing whatever it takes (so using agents to fix stuff basically).

The ones who get to keep their jobs will be rebranded as Maintenance engineers with a lower salary. VPs and Heads of will keep their cozy job (bc there are not so many of them), and they are not that dumb to fire themselves.

Code understanding doesn’t matter anymore (or won’t matter soon)

I had same problem, but after a while I just accepted that is inevitable

I love using AI, but any feature it creates I test the hell out of it, i become the QA testers I hated and do all sort of ways to break the app, and after 100 or so hours, I have a bug free and fully working app, now will this app scale as I add more features that to be seen.

That's the neat part, you don't.

Or at least, I don't think people should.

To me this completely goes against the whole point of engineering.

IMO If you don't understand the code, then you're just a manager now. Soon your salary may also reflect as such, assuming you get to keep your job.

This is OT and a cheap shot, but if I were an engineer at Meta I would have felt pretty uncomfortable well before AI showed up...

Claude to Prod. Every day.

if you have to maintain it, you will still have to get an understanding of it when it breaks.

not understanding the detail is quite similar to a coworker writing the code and another coworker approving and merging the changes.

Flat-out, you _need_ to understand the code you're merging. If you don't, ownership then lands on whatever AI agent you're using. Ultimately, you can't fire the AI agent in the same way you can fire the person at the driver seat when shit hits the fan and they've consistently caused incidents that force people to work late hours to resolve it.

I like starting with defining outcomes, determining invariants, and then consider TLA+ or Lean models, or both. It's generally not good to ship stuff you don't understand, but if you work on big legacy code bases from 20-30 years ago, there's probably no one who understands all the code all the time.

It's at least nice to be able to prove something about the code, rather than blasting it with tests which only prove that the tests that are written do not fail (and perhaps get you some amount of code coverage — necessary but insufficient for anything like quality).

How to do all of that in your environment? That can vary greatly!

By taking responsibility of it, maybe by saying "Hey this is not a good idea", and if that doesn't work perhaps by switching jobs if that's an option?

The mental gymnastics developers are willing to go through just so they can avoid standing up for something is quite perplexing.

>How are you handling this?

easy, don't write code with LLMs. If my job forced me to do so I would quit.

If you are a senior engineer then there is no need for you to be shipping code that you don't understand. People can definitely generate code a lot faster than before, but you also have a magic genie that can help you critique and improve the code that you're reviewing, including when that code was written by agents.

An experienced senior engineer with access to a coding agent is the most powerful code-understanding system that has ever existed. There's no excuse for shipping or approving anything you don't fully grok.

An engineer from Meta not knowing what they're doing is exactly how I imagined engineers at Meta. Thank you for confirming it's an actual reality.

On the other hand, not even surprised anymore most people even try to provide serious answers to this. Welcome to the pure insanity.

How do you deal with code other people wrote you don’t understand?

Same method.

When you code, do you understand the asm code being built? What are the cache line access patterns? Most software developers have already given up the details of how things work, and lean on tools to handle them.

Which is fine, usually. Depends on what you are doing.

The things that always mattered were the core system, the trunk of a tree if you will. The branches less so, and the leaves not at all (though that is what people see and appreciate! The irony!).

the guidance / guardrails, agent roles, etc are the critical thing now, it's kind of like meta programming AI to get the outcomes you want. Any problems need addressing at this level. You don't need to understand all the code, but you should understand how the system is building up and adjust as needed. This would be an architectural review process you go through with AI. Features to code that should plug into existing architectural structures are generally no problem. It's just boilerplate code AI is generating. But architecture is only one aspect, you should have lots of guidance for all kinds of thing. There's security, performance, correctness, QA, Ux, Deployment, Process, Monitoring etc and having feedback systems for everything. You keep refining and reviewing your guidance and the impact it is having. As well as this you are molding the product for the things you want via feature development. What I'm not sure of is if you are some cog somewhere in a company like Meta whether whatever slice you become responsible for becomes a bit meh if you are mostly just prompting a feature and not much more. As a developer in a smaller company (we are global with a smallish tech team) I get to do a lot of things from interacting with our customers through to the entire backend and get to make critical decisions about nearly everything. Nearly everything in the company is starting to be driven via guidance and feedback loops (including on the guidance itself).

It's simple: you don't. You slow down. It's essential to understand what you ship, before you ship, as there are real world consequences for not doing so, ranging from wasting the time and energy of yourself and others, up to failures which cause a great deal of harm to people and the world. To knowingly ship code you don't understand is grossly negligent, regardless of how common it becomes.

Peter Naur in 1985 wrote that programming is the act of building a working theory of the system in our minds, and this often takes form as tacit knowledge in the head of programmers. Even if you handed the complete codebase and perfect documentation to a completely new team, they would often need to go back to the old team and ask questions.

And before agents, we often built this working theory of the system by writing the code by hand. It helps us figure out the "grain" of the codebase so we have a better idea of how to extend it.

So what happens when agents are writing the code? The main avenue with which we built the working theory of the system in our mind is now taken away, and we're left feeling like our understanding of the system is blinded.

I ended up more or less changing my entire workflow. Four main principles:

1. Interrogation/interview-based design and specification

2. Make the decisions that matter and leave the rest to the agent.

3. Use agent to generate explainers, quizzes, and simulations.

4. Use verifiers and guardrails in the coding loop.

Interview-based design and spec

When I tell the agent what I want to build or extend, there's naturally ambiguity. But the agent knows what is ambiguous to it, so it can ask me. /grill-me is one way to do this. You'd think this would be commonplace now, but it's actually not. Some people are still wondering why they're getting bad results. It's because they're not resolving the ambiguity.

Making the decisions that matter

Not every decision matters in a product. Agents are fast at writing code and pretty good at some of the nitty gritty details, so I let it make those decisions. Also, if a particular subsystem is commonplace and undifferentiating, such as login, then I also let it make architectural decisions for that subsystem.

For decisions that are product differentiators, where that differentiation is supported by technology, I make decisions on the stack, the architecture, and the approach. This is the design phase. Because agents tend to make the mainstream decisions, they almost always take a midwit approach which doesn't leverage the technology in a way that makes a difference for the customer.

Some try to build this understanding from code reviews. I personally don't think it's the way to go, because it's not motivating. I find that I'm more motivated to learn how the system works today, because I want to change how it works tomorrow--and it sticks in my head better too (no different than pre-agents).

Generating explainers, quizzes, and simulations

As I'm doing design for an improvement and I don't know how a part of the system works, I can ask questions of the agent in chat. If I still don't get it, I can ask for diagrams (either ASCII or mermaid). But if it's an entire subsystem, I have a skill in which I can ask for visual and interactive explainers. It'll generate a markdown page (or html) that has these interactive components that help demonstrate a concept.

To help make the concept stick, there are quizzes after each section and at the end.

With the computational simulations, I generate a throwaway tool in which I can explore the state space. This doesn't happen as often, as I try to keep state spaces small, and for those explainers tend to work well.

With the design simulations, a same-model agent and a cross-model agent and I will talk about an explainer, and we'll name things that we understand or notice about the design and the intended direction we want to extend it. Then, we'll use subagents to generate options and then discuss the pros and cons. I find this is a good way for me to understand the dead-ends of a design.

Using Verifiers and Guardrails

I define guardrails as mechanistic checks that doesn't change with the feature. such as linters, typecheckers, and pre-commit hooks are all guardrails. Verifiers are mechanistic checks that will be different with each feature, such as unit/integration tests.

For more complicated states or concurrent parts, I'll ask the agent to use formal methods to write a proof. Then use the proof to generate a trace (a series of steps that demonstrate a happy or unhappy path). Then I run that trace in unit tests (kinda like property-based testing--so I guess this can be called trace-based testing), and the code has to adhere to the trace.

I also wrote a design verifier, where the requirements in the spec can be converted into a series of mechanically checkable rules. Then during testing, the verifier fires up the headless browser and uses the DOM measurements of the page to see if it matches up to the rules. This was completely impractical before agents.

All of this so a) I can be sure that the agent is doing what it's supposed to with the decisions that it's making b) I can be out of the coding loop, so that I'm not up at 2am begging, "you fixed this, but you broke that. pls fix." The agent can check for itself.

---

If any of this sounds helpful, lemme know. I've been wondering if other engineers care about understanding the code that's being written, or if people just don't care anymore. I've spoken to a few, and it seems to be split. And it may just be a matter of degree and detail. Some like it at a higher level. Some like it nitty gritty.

I don't ship code I don't understand. I fully review every line if it's going to prod.

My dumb little utils in ~/.local/bin? Those get merged after a quick functional test without my having read a single line of it. Still, the skill I maintain for such a job directs agents to write such throwaway scripts in anything but shellscript because I might want to extend some day and fuck shellscript

> How to deal with shipping AI-written code you don't understand?

Stop shipping code you don't understand.

Is there really any other solution? You can either ignore it and pretend like it's the new era of software like half the other commenters are, or you can sit down and understand the code and what you're shipping.

The only answer is to not ship code you don't understand. And I think the easiest way to do that is not use AI for ANYTHING except troubleshooting and doc searching. Those that draw a land in the sand and develop this way will be slower in the short term but their work will last out in the long term because it will be more solid.

if i was working for meta i would be more concerned about my culpability in undermining american democracy.

Generally speaking I brain storm the feature first and produce a dev plan. This includes asking it to reuse code/libraries we’ve already got! After it’s gone off ah done its thing we go back over and do a QA on what it’s built, why that approach, what can we do better etc etc?

I spend time to add extra observability, cross-checking and alerting at runtime to offset the risk.

It feels like another layer of "conceptual compression". I'm not sure who/when that phrase came around but in context of Rails is where I heard it and its things like ActiveRecord or really any ORM. You aren't writing the SQL directly, something else is and for the most part it works. Then it doesn't work or your database grows too much and you need to optimize table structures with indexes, etc.

My point is if you don't understand the fundamentals of the underlying code (SQL in the example above), you're more likely to have issues in the future. Or maybe not if that app doesn't reach a usage pattern or complexity where the nuances under the hood matter.

It really depends on what you're shipping. Meta ships social media apps where in the grand scheme of things if something is broken software wise it largely doesn't matter. What's the worst case? lost ad revenue? Users can't do things? If that were software for MRI machines or aerospace where software issues/bugs can result in lives lost then it matters much more in understanding what is going on at all levels of the software and potentially hardware.

Why not just force the llm to be much more concise and require that every line is worth its code. LoC is a inability after all.

> it all feels like a black box.

I have some work in Lean4 that I'm certainly unqualified to validate; Layer Coding QEC derivation, Positive Grassmanians and the Fresnel-Kerchhoff correction to the Parallel-Axis theorem, Huygens intensity as an alternate model for simulating photonic and electronic interference in a double slit experiment, for example.

It was a black box but I'm letting it tutor me about it so

Visual proofs, Tools for logic and formal methods in distributed systems

Things that compile logic!

> That means you can end up with systems you've built where you don't truly understand how all the code works

Though AI coding and system development can be overwhelming due to information overload and rapid growth in complexity (and concern for unwittingly embedded mistakes, the unsecurability of agent-based development, and number of LOC of code per day), not understanding the whole system sufficiently is normal and expected in engineering.

You are accountable.

You're going to have to understand it one way or another, either at writing or at debugging. Even if you delegate all debugging to AI, you'll still need to determine what is "right".

If you don't understand the code at writing or debugging, you'll pay the architectural price (where it degrades). That price may be worth paying, but that depends on the system you are building.

I deal with it by not doing it.

We just don't do that, I don't know why people talk like that is a given

Is it a company culture thing or a domain specific thing?

Don't.

By doing so, you debase your own profession.

You just stop caring. Everyone else did about three months ago. Time to find something new to care about.

If it's important code (and much isn't), then I spend a lot of time using Matt Pocock's /grilling first, and then I get the agent to kick off peer-review with other agents, and surface any disagreements. Some models are better at it than others. This usually forces me to broadly understand the architecture, and I spend most of my time answering the design questions for which there isn't an obvious solution. I have been pretty happy with the code I've launched to production, as measured by the ways it breaks, and by how agents who weren't part of the original committee describe it.

This is the state that founders and decision makers are usually in at a startup. Sure, sometimes you have the coder founder who wrote the first iteration of the product, but many times the founders have no clue how the product works under the hood. They entrust that to the engineers. That was then. Now, much of that engineering layer has been removed, so the founder just needs a smaller team of people who understand how to structure the software and interact with the LLM. LLMs do the rest.

The problem here is that you're trying to do a job that is already being done for you. Maybe you don't yet trust it to do a good job, but that is quickly changing.

Anyone who trusts the output of LLMs uncritically, even after the impressive level models and harnesses have gotten to, is borrowing against their future.

Models and harnesses have gotten really good, but they still are hard-capped by the operators that use them. The sycophancy of models is perhaps their greatest weakness, as when a user who might be experienced in one area attempts to use AI to build within an area they are less familiar with, the model does not take a step back and examine the "why" of what is being built from an architectural standpoint, it simply performs the expected task in a direct manner, ultimately creating sloppy work that may be easy to ship but difficult to maintain.

A good example of this was a recent interaction I had with a Dev who was very familiar with how to develop embedded and system level applications. He and every other dev are being asked for increasingly more due to AI, and he had to bring something he had built as a single-user desktop application into a distributed cloud environment. He ended up rsyncing a copy of his working directories including multiple credentials and sensitive internal project documents to a virtual machine binding it to a public ip address. Thankfully our tooling detected this before he could get any farther but it illustrates how the AI model isn't doing any real thinking or engineering at the end of the day. Unless the operator knew to instruct the model about proper cloud architecture and understood best practices around deployment of public systems, the model simply did what it was being told to do by a user who didn't know any better.

Another example of this is the sheer amount of unsophisticated users we've been detecting using Javascript injection into their web browser sessions as a form of automation for web services they may use. These users would never know to ask how to request an API token which may allow them to accomplish things in a much easier and approved way, and instead are relying on the frailest possible automations that expose them to risks of credential exposure.

We've got to be careful with how we engineer things with AI models. Just because we've made boilerplate easier to generate doesn't mean that we can avoid the actual engineering of what we do. And I can't help but think that firms which are fully committed to AI code generation, won't just be introducing an avalanche of slop into their product which will explode the costs of maintenance over time.

If you don't understand it, then you haven't built it.

Do not

It's Meta. Shipping crap is par for the course. You're fine.

Replace "AI agents" with "a bunch of human coders" and you'll see the answer. Your product needs decent testing and it's your job as the responsible person in charge to make sure that's in place and working properly. So if anything you probably want to be reviewing the test code. But often that isn't necessary since you can do things like introduce bugs deliberately and see if tests fail; check test logs for information that confirms the tests were run as expected; try some manual testing once in a while; ask for new more advanced tests (e.g. test performance not just functionality); and so on.

no one can force you, take responsibility for what you ship, what you choose to ship, how you choose to work, you signed that contract you go to work every day, your company is composed by people like you

don’t wash your hands, or at least don’t expect us to wash them for you

> How to deal with shipping AI-written code you don't understand?

Recognise you aren't the one shipping it.

[flagged]

The part that makes me uneasy is when the AI writes both the code and the tests. They might agree with each other and still miss the same thing. How are people here checking for that without doing the whole job twice?

[flagged]

[flagged]

[flagged]

[flagged]