Agent benchmark stability

What this is

One fixed agent scaffold run across a grid of (model x task family x perturbation) cells. Three families are generated procedurally and scored programmatically with no human grading: tabular (output bytes vs. the known transform), filesys (final simulated file-system state vs. the goal condition), query (answer vs. the value computed from the generated SQLite database). Six seeded axes: prompt paraphrase, tool renaming, distractor tools, instance resampling, temperature, task order. Directional hypotheses were fixed in HYPOTHESES.md before any result existed and the analysis reports the commit they were pinned to. Everything below loads live from the artifacts the pipeline wrote.

How it was built

A human-gated pipeline with two agents in separated roles: one plans, implements and runs the automated checks; a second writes none of the code, attacks the plan before implementation, writes the manual checklist afterwards, and runs the final audit. Neither agent grades its own work. Each approval is stored as the sha256 of the exact document signed off, so an approved plan cannot change quietly. The chain below is read from the files on disk.

Where this runs

No GPU anywhere, by design. Models are evaluated over their APIs; the laptop only generates tasks, drives the tool loop, verifies results and runs the statistics. The full grid is reproducible offline from a clean checkout with no keys and no spend — that is what these timings are, a cold run into an empty output root measured with /usr/bin/time.

The full pipeline is 6.5 seconds cold, end to end, peaking at 215 MB in analyze, the only stage holding the whole result set in memory. The server does no pipeline work: it serves these static pages and a read-only API over the artifacts the laptop produced.

Power: how many instances the grid would need

Pre-registered target is 80% power at alpha 0.05, two-sided. Paired axes use a McNemar normal approximation over within-lineage differences; the unpaired resample axis uses a two-proportion test with an exact binomial cross-check. UNDERPOWERED means the observed instances fall short of what the smallest declared effect would need.

What this grid can resolve

The power table is usually read as a caveat. Read forwards instead it is the result: it prices what this eval would cost to make conclusive. Each row below takes the largest requirement across axes at that effect size and works back to the grid that would meet it. Requirements and the observed count come from the table above; the rest is arithmetic over the shipped four-model grid in experiments.yaml, costed with the repository's own worst-case liability function at declared pricing.

The shipped grid is 1,560 episodes at $176 against a $200 cap, so the cheapest conclusive version of this study costs roughly ten times its own budget, and resolving the 3-point difference the power table headlines costs about a hundred times. That is a real finding about perturbation benchmarks rather than a defect in this one: paired designs help, and they do not help nearly enough. Risk K-2 in the approved plan called this outcome in advance and ruled out the tempting response — widening the declared margins or switching estimator after seeing the result would be post-hoc tuning, which invariant I-110 forbids. The grid was left as it is.

Pre-registered hypotheses against their intervals

Each row is a directional claim fixed before the run: no_change means the interval must sit inside the stated margin, decrease means the upper bound must clear zero. Underpowered or empty axes are forced to inconclusive by a rule recorded in methods.json, not applied by hand.

Score movement by axis

Paired axes are differenced within an instance lineage, so instance difficulty cancels. Bars are the mean absolute paired delta and are descriptive only; the interval in the table carries the claim. resample is unpaired by construction and its interval is not comparable to the paired axes.

Mean absolute delta by axis

pass-rate points, averaged over cells

Delta by axis

figure emitted by the analyze stage

Pass rate by model

figure emitted by the analyze stage

Where the variance actually lives

Variance in the pass indicator, split across model, axis level, and instance lineage. When the instance term dominates the model term, which tasks got sampled matters more than which model ran. Negative components are truncated at zero and flagged.

Rank stability across perturbations

Kendall tau-b between the baseline model ordering and the ordering under each perturbation level. Only emitted at four or more models; below that the raw rank-flip count is reported and rank-dependent hypotheses are forced inconclusive. This run has one model, so every row is unsupported.

Per-cell pass rates

One row per (model, family, difficulty, axis, level) cell. Intervals are cluster percentile bootstrap resampled over instance lineages, not episodes, because episodes sharing a lineage are not independent. Infrastructure errors are excluded from both numerator and denominator and counted separately.

Run health

Planned against observed episodes per cell. A cell that silently produced fewer episodes than planned would bias its own pass rate, so the count is reconciled against the planned grid rather than assumed.

Independent review and next steps

Findings from the reviewing agent, reproduced unedited. Next, in order: real-provider integration (AUD-007), empty-dataset handling in analyze (AUD-002), paraphrase-template sign-off (AUD-003).

Limitations

Written by the analyze stage from the same run that produced the tables above.

Provenance