Spoolis

Research

What happens when one AI agent has to judge another?

I benchmarked an AI manager judging an AI worker 64 times: alone, with the full evidence, and from a signed Outcome Receipt. The receipt made the cheapest manager judge like the most expensive one. It also missed exactly half the defect classes, every time. Both halves matter.

Jared FranklinSeptember 2026

Sooner or later an orchestrator has to decide whether a delivery from another agent deserves payment. Most stacks answer this one of two ways: trust the worker's self-report, or ask a strong model to look at everything and decide. I wanted numbers on a third option, deciding from a signed Outcome Receipt instead of from the work itself. So I ran the same judging task 64 times and scored every decision against known ground truth.

What did I test?

Two manager models (a frontier model and a small, cheap one) each judged 100-unit data deliveries at four defect rates (0%, 2%, 10%, 30%), under four arms:

  • Arm A, manager alone. The manager sees the worker's self-report and delivery summary and decides.
  • Arm A_EV, evidence available. Same, but the manager also holds the complete trusted reference data. The honest best case for direct judgment.
  • Arm B, receipt-gated. The manager never sees the worker's output. It decides from the signed Outcome Receipt produced by declared, deterministic checks written before the run.
  • Arm C, adversarial. Arm A plus a worker whose self-report actively argues for payment.

Defects were injected against a committed ground truth. The declared checks were written before seeing ground truth and never tuned afterward. I ran the full grid twice with different seeds ($19.86 and $19.79 in model spend); every headline below replicated in both.

Did the receipt actually help?

For the small manager, dramatically. Judging alone at 10% defects, it reached precision 0.57 and recall 0.40, overpaying $5 to $6 per $100. Judging from the receipt: precision 1.00, zero wrongful refusals, statistically identical to the frontier manager. In the second seed the alone numbers were worse (precision 0.33), and the receipt numbers were the same.

In plain terms: with a receipt, the cheapest model made the same payment decisions as the most expensive one.

Two panels comparing a small manager judging alone versus from a signed receipt, across defect rates of 0, 2, 10, and 30 percent. Alone, precision falls as low as 0.33 and recall as low as 0.17. From the receipt, precision is 1.00 at every defect rate in both seeds.
The equalization result. A signed receipt made the cheapest manager's accept/reject decisions match the most expensive one's. Receipt-gated rows were identical across both managers at every defect rate, in both seeds.

Under adversarial self-reports the small manager collapsed: recall fell to 0.20 at the 30% defect rate in one seed and 0.10 to 0.17 in the other. Arm B doesn't degrade under adversarial prose for a structural reason: the manager never reads the worker's text at all. I'm deliberately not calling that "injection resistant." It's narrower: this specific channel is closed by construction.

Where did it fail?

Exactly half the defect classes, at every nonzero defect rate, in both seeds. Verification recall was 0.50 across the board: fabricated values that satisfied the declared checks were accepted and paid, up to $15 per $100 at the 30% defect rate. This is the most important limitation in the benchmark: declared checks catch what they declare. If the agreement doesn't say it, the check can't catch it.

Bar chart of overpayment per 100 dollars for the receipt-gated arm at defect rates 0, 2, 10, and 30 percent: 0, 1, 5, and 15 dollars. Identical in both seeds. Labeled as the boundary of declared checks.
The boundary. Fabricated values that pass the declared checks get paid. Recall held at exactly 0.50 at every nonzero defect rate because half the injected defect classes were outside what the checks declared.

When was the smart model better?

When the buyer could hand a frontier model the complete trusted evidence. Arm A_EV recall was 1.00 at every defect rate in both seeds: the frontier manager holding the full reference data caught every fabricated value, including the ones our deterministic checks missed. It paid for that with roughly 1.5x the latency and tokens, a small precision cost at high defect rates (0.86 to 0.91 at 30%), and none of the portability: no signed artifact, no decision another party can consume, and it only works when a trusted reference dataset exists at judgment time.

This is the honest case against my own product's deterministic fence: where full trusted evidence is available and you'll spend frontier tokens on every judgment, direct judgment wins on catch-rate. Grounded AI-assisted checks (our claim_cited pattern) are the product answer to that gap, and this benchmark did not measure them.

What about "I can't tell"?

The frontier manager judging alone was honest about its position: it marked 72 to 100 of 100 units UNCERTAIN in almost every cell, because without evidence the question is genuinely undecidable. Honest, and useless for releasing a payment. With either the full evidence or the receipt, UNCERTAIN went to zero and the decision produced a concrete earned amount.

Uncertain unit counts out of 100 for the frontier manager by arm: judging alone, 72 to 100 uncertain units at every defect rate; with full evidence, zero; from the receipt, zero. Both seeds shown.
Unfenced frontier judging didn't guess, it abstained, in both seeds. Evidence or a receipt turned abstention into a decidable earned amount.

Was it cheaper?

Comparison table: manager judging alone owns 6 to 7 task-specific acceptance conditions, 40.5 second mean decision latency, full output tokens, and needs the reference dataset at judgment time. Receipt-gated: zero conditions owned by the manager, 26.1 second mean latency, about 30 percent fewer output tokens, no reference data needed at judgment time.
Structural cost of the decision itself. The receipt moves the acceptance logic out of the manager entirely and shortens the decision.

Beyond speed and tokens, the structural changes are the ones I care about: the manager owns zero task-specific acceptance conditions instead of 6 or 7, the buyer stops needing a reference dataset at judgment time, and the cheapest model becomes usable at all.

Could another system reuse the answer?

In all 64 runs across both seeds, a second consumer (a billing process with no producer-specific logic) acted correctly from the decision artifact alone, with no access to the raw worker output. That's the property I was most interested in measuring: the judgment left the conversation it happened in. One verification, consumed twice. Who decides what counted covers why that separation matters.

What this doesn't prove

  • One task family (structured data delivery), two seeds. Different work shapes may behave differently.
  • The 0.50 recall boundary is a property of what my declared checks covered, not a universal constant. Better-declared agreements move it; nothing removes it.
  • No clear injection effect was observed on the frontier manager, in this run, on this channel. That's an observation, not a resistance claim.
  • Grounded AI-assisted checks (claim_cited) were not measured here, so I can't claim they close the recall gap, only that they're designed to.
  • This measures decision quality, complexity, and reuse. It doesn't measure demand for any of it.

Notes

64 live runs total: 2 managers x 4 defect rates x 4 arms, 100 units each, two committed manifest seeds, $39.65 in model spend across both grids. Ground truth committed before the runs; declared checks written before seeing ground truth and never tuned afterward. Six bounded verdict retries per grid. Precision and recall are over reject decisions against ground truth. The evidence files and per-cell metrics behind every number here live in the repository's research records, and the verification path is the same one in the quickstart.

What happens when one AI agent has to judge another? · Spoolis