Wait, that doesn't add up. What's left for a human to do

My agent team writes most of the code. I went back through the session transcripts and looked for every time I interrupted an agent and the interruption changed the outcome. What I found wasn't what I expected, and one of the cases is my failure, not theirs.

Image by Tumisu from Pixabay

My development team is a group of specialized AI agents. Each one has a territory; they read each other's work, open pull requests, and argue with me about scope. Most of the code in the product was written by them, and it's good code: better than what I wrote alone, faster than I wrote it, and covered by tests with a consistency I never had.

So the interesting question isn't whether they're good. The question is what's left for me.

I went back through several months of session transcripts and looked for every time I interrupted an agent and the interruption changed the outcome. A few hundred candidates; two or three dozen that actually changed something. What surprised me wasn't that the agents got things wrong. It was where they got them wrong, and in one case, where I did.

And here's the thing I didn't expect at all. Not one of those corrections came from me reading the code better than the agent. Not once.

101, off by a factor of almost 100

An agent was preparing a data migration against production. It ran a dry run, and the dry run reported that 101 records needed handling. The agent checked the script carefully: no exceptions being silently swallowed, no accidental limit, selection logic sound. It was right about every one of those things.

Then it did something worse than missing a bug. It built an argument on top of the number:

It's a coherent, well-argued product insight, and it's made entirely of a bug. My reply was one sentence, without a shred of analysis:

The paging helper set a limit of 200 and asked for a single batch, but never set the batch size. A limit caps the total; it doesn't enlarge the batch. The server returned one batch at the default size, 101 documents, and closed the cursor. The real number was nearly 100 times larger.

The same helper had been copied into four different scripts. One of them had already run in production.

It's worth pausing on that for a moment. There was no exception. The tests passed. The code was internally consistent, the summary said "success," and every step of the agent's reasoning was reasonable on its own. Nothing in the diff, the logs or the test suite could have caught it. The only detector available was someone who knows roughly what the answer is supposed to be.

Sixteen seconds of nothing in particular

After a spike in production response times, an agent produced a genuinely good incident report and recommended ignoring the incident. I accepted the recommendation and refused the number:

That pushed it from "this endpoint is slow" to reading the query itself, and there it found something real and completely unimpressive: a filter no index could serve, and a sort on an unindexed field, so every single call scanned the whole collection and then sorted in memory. One sentence of its has stayed with me ever since, and I'm leaving it exactly as written:

The take bounds the result, not the work — which is the part that makes this look deceptively cheap in code review.

But the second round is the better story. The agent measured against production and found the query takes 72 milliseconds. Not sixteen seconds. Seventy-two milliseconds. It had an elegant explanation ready (cost times concurrency, not a bad query plan) and I would have accepted it, except that the two stories didn't fit together:

The number came from a timer that wrapped the entire client operation against the database. It included the sub-query that ran after it, the time it took to turn both result sets into objects, the wait for a free connection, and an event loop choking under sixty sibling requests doing exactly the same thing. It was never database time. Every conclusion built on that number had been quietly skewed.

The same question also shook loose a second bug nobody was looking for: a field that was supposed to measure how long a request waited for a connection was computed from a timestamp set at exactly the same moment as the start, so it faithfully duplicated the total and measured nothing.

I want to be precise about what I contributed there, because it wasn't expertise. I didn't know how the timer worked. I noticed that two of the agent's own statements couldn't both be true, and I said so. That move costs one sentence, requires nothing but attention, and is the highest-return thing I do all week.

Three things no repository knows

An agent read an old integration flow, concluded from the code that it was live, and carefully scoped a task that took pains not to break the users depending on it. There weren't any:

Nothing in a codebase says nobody can get here. The routes exist, the components render, the tests pass. An agent drawing conclusions from source code will systematically overweight dead code, and spend real effort protecting users who don't exist.

A design agent capped a layout's width, to be safe, in a way that looked perfectly reasonable on the screen it was tested on:

And an agent proposed three analytics additions, each one justified, with a clear owner and a recommendation. I didn't engage with any of them:

It went and counted: about 150 analytics events, and a dozen or so of them don't fire anywhere. In this case the proposal on the table didn't need any new events, so it went ahead as is, but the count stuck, and it changed a convention going forward.

What's worth noticing is that nobody adds a bad event. Each of the 150 was a locally correct decision, by someone competent solving the ticket in front of them. The cost is cumulative and structurally invisible from inside any single task, because the sprawl is never in the diff you're looking at. I'm the only one in the loop who carries that weight from session to session.

Blast radius

An agent needed access to the production database to finish a performance investigation. It asked for it and laid out the options, all reasonable, all sufficient.

What it came back with wasn't just a smaller request; it was better engineering. A role that reads metadata only (it can list collections, inspect indexes and read statistics) with the data-read permission deliberately left out, so it can't run a data query at all. Not by policy. By construction.

That design had been there all along. The agent just didn't go looking for it, because the broad version unblocked it, and the task was the goal. Then, unprompted, it added something I keep coming back to: if you want this guaranteed at the platform level, it wrote, and not on my word, here's the setting to turn on.

And now, where my review failed

Everything so far makes me look good, which is a recipe for a boring essay. Here's the one that doesn't.

Testing on my phone, abroad, I found a feature telling users an album wasn't available when it clearly was: one path in the app linked to it correctly, while another insisted it didn't exist. The root cause was structural, not a slip: we stored an identifier that is region-specific in a single global field, and two different processes wrote to it from two different regions.

Then the agent said the thing that has bothered me ever since:

The design review happened. I did it. I read the document and approved it. And the contradiction wasn't buried in an appendix: two adjacent lines, in perfectly plain language, in a document I had asked for and took seriously.

So why didn't I catch it?

Because both lines were true. That's the whole trap. Design docs written by an agent are long, fluent and locally correct everywhere; your attention naturally goes to whether each paragraph is right, and each paragraph is right. The flaw exists only in the relationship between two true statements forty words apart. Reviewing prose is simply the wrong tool for that. And anyway, nobody re-derives every claim in an eight-page document; you read it to check direction and to check the risky decisions. The direction was right. The risky decisions were flagged.

A human in the loop was necessary. It wasn't sufficient. Where the human sits in the loop turned out to be the real design question, and I never asked it

Give the machine the machine's half

The check that would have caught it is embarrassingly mechanical:

No domain knowledge here. No taste. No product judgment. Just a consistency sweep that doesn't get bored and has no narrative momentum carrying it past line 80, which is exactly the work an agent does better than I do, and exactly the work I did badly by hand without noticing.

So today our design docs go through an adversarial consistency pass before a human reads them. Its only job is to ask whether the document contradicts itself. And note that this is not one more approver in the chain: my review didn't fail because there was too little of it. It failed because it was aimed at the wrong layer. Let the machines proofread. I'll decide whether it's the right product.

And I'm wrong too, regularly

A post claiming the human is always right is worthless, so: I once reported a serious regression in the player, with total confidence, and spelled out exactly which providers were failing and how.

The agent diagnosed a plausible cause within minutes, approved a rollback to unblock me, and took responsibility in writing. And then:

The loop works because both sides are correctable. One line from that same conversation is worth keeping on its own, on the subject of green test suites:

The work that's actually left for me

Reading the code more carefully isn't the job. I'll lose that contest, and so will you. Every one of the interventions above came from somewhere else entirely:

Agents don't fail at things. They fail at the relationships between true things: between a number and reality, between two paragraphs, between a decision and its cumulative cost. And relationships are exactly what never shows up in the diff under review.

So human oversight still matters. It just isn't proofreading. And the moment I mistakenly thought it was, something got into the product that a five-line prompt would have caught six weeks earlier.