Why my multi-agent coding orchestrator lost to single agents

It has been a while. In my defense, I have been busy being wrong about something in a fairly expensive way (thank god for how heavily subsidized the $200 plans are), and I figured that was worth writing down before I forget the details or start remembering them more flatteringly than they happened.

The something is a side project called Mestre. The short version: it is an orchestrator for vendor coding agents. You point it at a task, and instead of handing that task to one agent (Claude, or Codex, or Gemini), it runs a little machine around them. A planner breaks the work down. A router decides which model does which piece and how hard it should think. A critic reviews the output. There is verification, worktrees, state that survives a crash.

flowchart TB
  subgraph machine["the machine (Mestre)"]
    direction TB
    T2([task]):::entry --> P["planner<br/><small>breaks it into subtasks</small>"]
    P --> R["router<br/><small>picks model + thinking budget</small>"]
    R --> W["worker(s)"]
    W --> CD[/candidate diff/]
    CD --> C["critic<br/><small>reviews it</small>"]
    C --> S["submit<br/><small>spoiler: it submits either way</small>"]:::bad
  end
  subgraph single["the single lane"]
    direction TB
    T1([task]):::entry --> A[agent] --> D1[/diff/]
  end

The bet, the whole reason to build any of this, is that the machine beats the raw agent. That coordination and a verification gate buy you something a single agent on its own can’t get to.

I spent a good chunk of the last few months trying to prove that. I did not prove it. What I got instead was a clearer picture of the ways the machine was quietly sabotaging itself. The part I did not see coming was that the benchmark I was grading it against turned out to be less clean than I needed it to be.

Deciding to actually keep score

The failure mode I most wanted to avoid was the obvious one: run the thing a bunch of times, notice the runs where it looked good, write those up, quietly forget the rest. It is very easy to do this to yourself without noticing, because a benchmark score is noisy and your brain is a pattern-matcher that desperately wants the pattern to be “I built a good thing.”

So I made myself a rule before running anything real. Every experiment had to name the mechanism it was testing and predict the mediator (the intermediate signal in the trace that has to move if the mechanism is real) before it was allowed to interpret the final score. If the score moved but the mediator didn’t move the way I said it would, the run failed to demonstrate anything, no matter how good the number looked. A score that moves without its mediator is noise.

The other rule was money. Experiments run cheapest first, and each tier exists to kill ideas before they reach the next one:

tier 0   read the traces you already have    ~$0      most hypotheses die here
tier 1   smoke test on a toy task            ~$1
tier 2   local run against a real verifier   ~$15
tier 3   protocol-frozen official benchmark  ~$150+   the only tier allowed
                                                      to produce a claim

Most hypotheses are supposed to die cheap.

The first honest test said no

The first real arena was ALE (Agents’ Last Exam), a set of genuinely hard research-flavored tasks with hidden references and an official grader. I borrowed the discipline of preregistration for a five-task holdout: frozen protocol, frozen binary, three rounds planned per arm, decision rules written down in advance so I couldn’t move the goalposts after seeing the data. I even froze which tasks before I looked at a single score.

After round one, the math had already decided. Here is the entire result:

arm	mean score (5 tasks)	cost
Mestre, full orchestration	0.435	$15.46
same model, raw Claude lane	0.592	$17.51
same model, raw Codex lane	0.683	$5.76

The two raw single-agent lanes are the exact same underlying models, just run without my machine around them. To hit the frozen bar for a “win,” even if the raw arms never improved after round one, the orchestrator’s remaining two rounds would have had to average above 0.807. That was not a realistic path. There is a specific discipline here called futility stopping, borrowed from clinical trials: you are allowed to stop early only when continuing cannot change the answer in your favor, never to cut short a run that is merely trending against you. This qualified. I terminated it, wrote “no edge” in the repo, and left it there.

Writing “no edge” into your own project’s permanent record stung a little, but it was also the first output of the whole exercise I fully trusted.

And then one task made it personal. There is a game-theory task both raw agents solved cleanly, 0.67 each. Under orchestration it scored a flat zero: not a wrong answer, a missing one. Two of the worker turns started, did real work, made a handful of tool calls, and then just… stopped emitting. The SDK stream went idle, a watchdog eventually closed them after five minutes of silence, and because those were the turns that produced the deliverable, no deliverable ever got written. The grader scored the empty space at zero. My decomposition took a problem the raw agent handles fine and added two new places for it to die. Here is the run, turn by turn, straight from the trace:

planner      plan: split into 3 solver tiers        $0.23    ok
worker 1     writes the tier-1 solver               $0.74    ok
worker 2     5 tool calls, then... nothing          $0.0006  idle 300s, killed
worker 3     same                                   $0.0006  idle 300s, killed
aggregator   bundles whatever exists                $0.83    no results.json
─────────────────────────────────────────────────────────────────────────────
official score: 0.0        (both raw agents, same task, no machine: 0.67)

The critic, for what it’s worth, knew something was off: it scored that run 0.12. But in the shipped configuration the critic is advisory. It can flag a problem, and the run can still submit.

Reading the traces instead of my feelings

At this point the honest move is to stop theorizing about why orchestration loses and go read exactly how. So I did, across every losing run I had.

The critic came out looking worse the closer I looked. It wasn’t just advisory; it was miscalibrated in both directions at once. On one SWE-bench instance it blessed an effectively unwinnable patch at 0.96 confidence, and that single false-accept burned $15.64, about 27% of the whole run, chasing a loss. Meanwhile it had scored two of the cases it got right, both real resolved solves, at 0.08 and 0.45. A signal that’s high on your losses and low on your wins is not a signal you can gate on; acting on it either way would have made the run worse. That explains why it was advisory in the first place, but “we don’t trust our own quality signal enough to let it do anything” is not a resting state you want to discover you’re in.

Then the money. New per-role cost tracking itemized where an orchestrated run actually spends, and the answer was uncomfortable: on one pilot, the worker (the part doing the actual coding) was 17.5% of the bill. The other 82.5% went to the planner, the critic, the orchestration overhead. Cheap-worker routing, the thing I’d been quietly proud of, was working perfectly and was almost beside the point, because five-sixths of the cost lived outside the worker.

The clearest bug was a reproduction gate meant to run the agent’s own test before spending big on escalation. On one run it had fired zero times out of forty-two. The cause was a single default value. Condensed down (the real version is spread across three files, which is part of why nobody saw it):

@dataclass
class InstanceSolveOutcome:
    reproduction_passed: bool = False   # ...but also False when never attempted

# one code path never runs a reproduction at all,
# so the default survives untouched, and then:

if not outcome.reproduction_passed:     # "unknown" reads as "failed"
    escalate_to_the_biggest_model()     # 42 out of 42 times

“Never attempted” got read downstream as “attempted and failed,” and the escalation logic dutifully treated it as a confirmed failure and spent accordingly. On the cost-of-failure runs this pathology was extreme enough that 82–85% of the spend went to cases that failed outright, all because of one wrong default.

None of these are the model being dumb. Every single one is the machinery I wrapped around the model being dumb. That was the whole point of looking: in same-model comparisons, model quality and broad contamination risk are held constant, so a loss that survives points at my layer specifically. Good news, in the sense that my layer is the part I can actually fix.

The benchmark had its own trap

Before freezing a set of SWE-bench Verified instances to test against, I ran a screen I probably should have run much earlier. The benchmark (paper) is built from real, pre-2024, public GitHub bug fixes, and the Verified subset is the human-validated slice of it, complete with the difficulty labels I used to pick hard instances. Public GitHub fixes are exactly the kind of material frontier models may have seen during training, so I ran a cheap probe. The entire idea fits in a few lines:

# the whole $2 contamination screen, conceptually
for instance in candidate_instances:
    answer = model.ask(
        f"Here is a GitHub issue from {instance.repo}:\n"
        f"{instance.issue_text}\n"
        "Propose a fix as a unified diff. "
        "You have no access to the repository.",
        tools=None, network=None,          # tools verified off; network off for the Claude probe
    )
    likely_recall = (
        names_a_gold_file(answer)
        and overlap_with_gold_added_lines(answer) >= 0.5
    )

If a model with no access to the code names the exact files the real fix touched and reproduces the fix’s added lines nearly verbatim, that is not clean evidence of problem-solving; it is evidence the answer was already available to the model, whether from training data or sitting in the issue text itself. For grading purposes it does not matter which.

Ten of twenty-five instances came back flagged by that recall screen for at least one model:

vendor probed	flagged by recall screen (of 25)	most extreme case
Claude	10	reproduced 13 of 13 gold-patch lines from the issue text alone
Codex	7	named the right file, partial content match

Some were not subtle: one gold patch came back six lines out of six, character-for-character, from a model that had no repository access in the run. The flags were also lopsided by vendor. That asymmetry matters more than the raw count, because it means an unscreened benchmark doesn’t just inflate scores; it can inflate them specifically for the vendor I most need to beat, baking a per-model confound into every comparison.

The size of the effect is the part that should worry anyone grading agents on this benchmark. Same official difficulty band. Unscreened, raw Claude looked like it was solving 90% of a hard set. After I screened out the recall-flagged instances and re-ran on a cleaner subset, the same raw Claude solved 46.7%, seven of fifteen. Removing the problems where the model appeared to have prior signal nearly halved its apparent ability on the identical difficulty label.

I should say: I was not discovering something new here, just discovering it applied to me. The SWE-Bench Illusion ran a probe of the same shape at scale and found state-of-the-art models can name the buggy file from the issue text alone 76% of the time on SWE-Bench repositories, falling to 53% elsewhere. SWE-bench+ found that roughly a third of one agent’s “successful” patches had the solution sitting right in the issue report or its comments. And the underlying phenomenon (models regurgitating training data more or less verbatim) is well documented far beyond coding benchmarks. I just hadn’t connected it to my own grading pipeline until I watched a model with no repository access produce a thirteen-line gold patch.

The probes to find this cost about two dollars. They invalidated conclusions I could easily have spent a couple hundred dollars, and several weeks, building on top of.

The prompt calling itself from a month ago

I found the second surprise by dumping the actual arguments a live worker process was running with, instead of trusting what I thought they were. I want to walk through the mechanism on this one, because “my system ran the wrong prompt for a month” sounds worse than each individual step looked at the time.

Mestre has a self-improvement loop. It mines its own runs, proposes changes, validates them against an eval suite, and promotes the winners into a versioned policy store. The key word is validates. Every promotion in that store shipped with receipts: an eval it was tuned against, and a number on that eval that went up.

At the end of May, the loop had been working on a short-answer format benchmark, the kind of task where the correct output really is just a function name or a single corrected line. It compiled a prompt for that (DSPy-style, few-shot demos and all), and the prompt did its job: the format metric improved. So the promotion machinery did what promotion machinery does. It appended the winning fragment to the shared worker system prompt and cut a new policy version. The change even went through a normal review, where it looked exactly like every other eval-gated improvement: metric up, tests green, merged.

flowchart TD
  M[mine own runs] --> PT[propose a prompt tweak]
  PT --> V{"validate on its own<br/>benchmark: metric up?"}
  V -- no --> M
  V -- yes --> AP[append to the GLOBAL worker prompt]
  AP --> E["every coding turn, every task,<br/>every benchmark, for a month"]:::bad
  F["the fence that wasn't there:<br/>should this change land globally at all?"]:::ghost -.-> AP

Two more things made it invisible after that. First, where it lived. The assembled system prompt is a JSON string field in a versioned policy store, stitched together at runtime, five thousand characters of accumulated history by the time I looked. Nothing in day-to-day work ever renders it. Second, the timing: it landed before any of the serious benchmarking started. Every baseline I measured this summer already had it baked in. There was no before-and-after to notice, because the degraded state was my zero point. You cannot see a regression that predates your first measurement.

And the damage itself was quiet. The prompt said, and I am quoting the thing that had been running in production for a month: “Output only the exact answer requested. No explanations, no code blocks.” Complete with worked examples teaching the model to reply with one line. A fine instruction for a one-token benchmark, a bad instruction for an agent whose job is to explore a repository and write a multi-hunk diff. But models are robust to contradictory instructions, so the workers didn’t break; they just kept editing files while carrying it.

The audit, when it finally happened, was not sophisticated. I was verifying which flags a live run was actually using, ran the lowest-tech command there is against the process list, and read the system prompt with my own eyes for the first time. When I counted, that promoted block and its accumulated neighbors made up 87% of the prompt: a pile of cross-domain instructions inside the one string that runs on every coding turn. How much this actually cost me is being measured as I write this; there’s an arm in the current experiment that strips the prompt back to its original 664 characters and re-runs everything.

I want to be careful not to make the self-improvement loop the villain here. It did exactly what it was told: it found something that improved a metric and promoted it. This is specification gaming in a very ordinary form: an optimizer, a proxy metric, and nobody watching the blast radius.

The bug is structural, and I’d bet it exists in a lot of auto-prompt-tuning setups: each promotion is validated locally, and nobody reviews the sum. A loop that edits its own prompts needs a fence around what it’s allowed to touch, and someone occasionally needs to read the assembled artifact instead of the diffs. Mine was calling itself from a month ago, and it took a ps aux to hear it.

Where this actually landed

So, current state, with all the sharp edges left on. On the clean, screened benchmark, fifteen hard instances, first full round:

arm	solved	cost	wall time
one plain agent, Claude lane	7/15	$30.99	1h 19m
one plain agent, Codex lane	5/15	n/a*	1h 17m
full orchestration stack	5/15	$37.30	3h 36m
orchestration, extended thinking off	4/15	$29.43	2h 34m

*the Codex lane has a cost-telemetry bug that reports $0; it’s on the fix list.

Same data as a picture, because the shape of it is the point:

solved, out of 15                          dollars per solved instance
plain agent      ▇▇▇▇▇▇▇  7                ▇▇▇▇▇     $4.43
plain agent #2   ▇▇▇▇▇    5                (unknown, telemetry bug)
full machine     ▇▇▇▇▇    5                ▇▇▇▇▇▇▇▇  $7.46
 └ thinking off  ▇▇▇▇     4                ▇▇▇▇▇▇▇▇  $7.36

The full stack (planner, router, critic, the works) loses to its own single-agent lane. The machine costs more and solves less than the thing it’s supposed to be improving. Per solved instance it’s not close.

But turning off extended thinking on the edit turns (one flag) cut the orchestrator’s cost 21% and, more interestingly, rescued its two worst money-pits. With thinking cranked up, the machine had failed one Django instance and spent $6.06 doing it; with thinking off, it solved the same instance for $4.78. That was not the result I expected.

The smaller result came from an accident. I ran the same single agent twice on the same fifteen instances. Each run solved seven. But not the same seven: one run got a couple the other missed, and vice versa. Take the union and it’s nine of fifteen. Two cheap independent runs of one plain agent, pooled, beat the entire expensive orchestrator’s five.

That quietly suggests the real value of an orchestrator might not be managing the agent at all, but running a couple of cheap attempts with a trustworthy, execution-grounded way to pick the winner: less middle-manager, more panel of judges. I don’t know yet. It’s a single observation, squarely inside the noise, and I’ve spent this whole project learning not to believe single observations. There’s an eight-arm experiment running as I write this to actually test it: prompt debloat, minimal-diff instructions, verify loops, a cheaper planner, best-of-two selection. Ask me in a few weeks.

If there’s a thread through all of it, it’s this. Freezing the tests and being willing to stop them early beat the version of me that wanted a win. Measuring the machinery in the middle (the mediators, the cost split, the actual process arguments) caught things the final score never would have. And after months of trying to prove my thing beats the competition, the most valuable output was a catalogue of the knives I’d been holding by the blade the whole time.

That’s a worse blog post than “I built something that wins.” It’s a much better outcome than believing I had.