2026-09-01 · Eric Crook

Four things that break when agents write your code

Everyone can describe the happy path. You write a spec, the agent writes the code, the tests pass. I have been running agentic development as a standing production system rather than a demo, and the interesting part is not that it works. It is the four ways it degrades once you push on it.

My setup is a skill that autonomously pulls tasks from a work queue and hands subagents the context to execute them, usually implementing a feature. The same orchestration reaps the git worktrees, runs validation, merges, and deploys, and spawns subagents for research. My job is steering, and my steering gets organized back into the queue.

That description makes it sound tidy. Here is what actually goes wrong.

Failure oneBacklog starvation

If you rank the backlog, some tasks never get done. The system keeps generating new work, and the new work consistently outranks the older work. It's fresher, it's more specific, and it was created by an agent that just learned something. So the bottom of your queue isn't a queue at all. It is a graveyard.

That's not a prompting problem. It's a scheduling problem, with the same shape as every other priority-inversion bug: the thing that never runs is invisible precisely because it never runs.

Failure twoUnbounded queue growth

If you generate tasks faster than you complete them, the queue grows without bound and it stops being clear when — or whether — anything will finish. Agents are very good at producing more work. Every research subagent returns with three follow-ups. Every code review surfaces two refactors.

The generation rate and the completion rate are two different numbers, and if you aren't watching both you'll feel productive right up until you notice that nothing has shipped.

Failure threeThe attention ceiling

Tasks that need high human mental feedback get hard past about five in parallel. Not five agents. Five things that require me to actually think. Past that the constraint stops being compute and starts being me.

There's a small, stupid, real failure mode inside this one. You type steering feedback into the wrong chat context. It's unpleasant, it's easy, and it happens more the more parallel work you run.

Failure fourVerification is the bottleneck

This is the big one. Most of the work is not generation. It's the verification and feedback step: given this feature request, and given what the agent did, is it good?

Answering that honestly pulls in computer vision, deterministic quality rules, adversarial testing, and code testing of every kind. Then iterations get slower and slower, and your LLM code-production machine bogs down waiting on tests and feedback. So you split into fast and slow paths and work optimistically ahead. Then the cycle starts again one level up.

SoHow I measure non-deterministic systems

If verification is the bottleneck, verification is where the engineering goes. It's like any other testing. You just have to run more of it.

• Measure everything you can deterministically, and that's a lot if you're clever: latency, correct output on known problems, task completion, recovery success.
• Let the models generate the test cases. They're good at volume and at adversarial variation.
• For everything else, LLM-as-judge, across model families. Don't use an OpenAI model to judge another OpenAI model. Use an Anthropic model to judge an OpenAI one, and prefer the more capable model as judge.
• Keep the agent driving a mechanical, measurable system wherever possible. The less of your system that's non-deterministic, the more of it you can actually assert on.
• Treat every escaped failure as an eval bug. When something breaks in the wild, go back and find out why the evals didn't catch it — not just that specific failure, but the class of failure it exposed.

The part I did not expect

Agents are excellent at orientation: explaining code, pointing at the critical parts, walking a code path, drawing the diagram, doing a whole-language rewrite, adversarially critiquing my own spec before I commit to it. They're weak at explaining genuinely complex concepts. Good for the briefing before the dive. Not a substitute for it.

What works well is a well-defined task with a clear way to measure success. Given that, models work well unsupervised once the spec is written. Everything above is what it takes to keep producing that clarity at volume.