What is an hour of audio data worth?
Conclusion, up front
The pipeline is finished; the experiment is not. Five stages, 35 runs, four quality axes and nine pre-registered predictions are built, tested and reproducible. Nothing has been trained yet, so there are no WER numbers on this page. The remaining work is compute, not code: one 16 GB CUDA card and the real corpus.
What is demonstrated today is the machinery and the process. The whole chain runs end to end on a laptop in 30 s on a fixture corpus; 213 of 215 tests pass, with ruff and strict mypy clean; of the independently written manual checklist, 12 checks were executed and passed — 3 of them since re-run by hand rather than by the agent — 1 defect was found, and 17 are waiting on hardware.
The workflow is the other half of the project. Claude plans, implements and verifies; Codex reviews the plan adversarially and writes the manual checklist without seeing the implementation notes; I set requirements and approve each gate, recorded as the SHA-256 of the exact document approved. The review raised 15 findings before a line of code existed, 8 of them blocking — but re-running that same review shows 15 was not the whole set, which is measured below rather than glossed over.
The question is the size of each data-quality effect, since that is what turns a data decision into a budget. The intended output is an exchange rate: how many noisy hours it takes to match one clean hour, with a bootstrap confidence interval, for one model and one corpus, rerunnable by someone else. Predictions are committed and hashed first, so the results read as a test of the design rather than a description of whatever came out.
Scope: LibriSpeech train-clean-100, whisper-small, one GPU, four quality axes, 35 runs. Single-device by choice, no hyperparameter search, one frozen recipe for every cell.
How this was built
Two agents in fixed, non-overlapping roles, with a human gate between stages. The separation is what makes the review worth anything: the agent that wrote the plan is not the one that attacks it, and the agent that wrote the code is not the one that decides how it is tested by hand. Gates are enforced by a harness, not by good intentions — approving stores the document's SHA-256, every later stage rechecks it, and editing an approved plan reopens the gate.
| Stage | Artifact | Owner | What happens |
|---|---|---|---|
| 1 | REQUIREMENTS_INTERPRETATION.md | human | What the project is for, read back and approved before any planning. |
| 2 | PROJECT_PLAN.md | Claude | Behaviors, invariants, failure modes, verification strategy. No code yet. |
| 3 | ADVERSARIAL_REVIEW.md | Codex | An independent agent breaks the plan on paper: 15 findings, 8 blocking. |
| 4 | UPDATED_PROJECT_PLAN.md | Claude | Written disposition per finding: 13 accepted, 2 partially. No silent acceptance or dismissal. |
| 5 | source + IMPLEMENTATION_NOTES.md | Claude | 3 defects fixed along the way, 7 deviations recorded rather than quietly made. |
| 6 | AUTOMATED_TEST_REPORT.md | Claude | Every check with its command, exit status and output. Nothing reported that was not executed. |
| 7 | MANUAL_CHECKLIST.md | Codex | Manual checks written from requirements and source, not from the implementer's account. |
| 8 | VERIFICATION_REPORT.md + DEFECTS.md | Claude | Each item lands PASS, FAIL, BLOCKED or NOT RUN with evidence. Blocked is never converted to passed. |
Three findings that changed the design before any code existed. AR-002: the plan promised clip-level CIs while storing one aggregate WER per run; evaluation now writes per-clip S/D/I/H counts as the authoritative analysis input. AR-006: mtime and read-only bits prove nothing about ordering, so pre-registration is now a blob hash bound into every result row, with a written statement of what it does and does not guarantee. AR-008: shrinking batch size on OOM while stamping the same recipe hash would make one run incomparable and label it conformant; there is now no runtime override.
What the review missed
A single review pass is easy to present as if it were complete. To find out, the same Stage-3 review was run three more times against the byte-identical approved plan — once with the prompt verbatim, once with the documents in the opposite order, once with the review categories reversed. The approved review found 15 findings. The three re-runs found 26, 24 and 22. Since the verbatim replica alone found 26, run-to-run variance swamps any ordering effect, so this measures the reviewer's stability rather than its bias.
The good half: the findings the project acted on are not artifacts of one sampling run. Per-clip counts, the hash-bound pre-registration and the no-runtime-override rule all survive replication. The bad half: five themes were raised by all three re-runs and appear nowhere in the approved review — the fixed-epoch confound below, resource budgets stated as assertions rather than acceptance criteria, archive-extraction risk, model identity in the offline contract, and tests that can pass while the behavior is scientifically wrong. That last one is category 8 of the 12 the review prompt explicitly asks for, and the approved review returned nothing under it.
The finding that matters: fixed epochs confound data with compute
The plan nominates its own most attackable choice — every cell trains for 2 epochs, so a 50 h pool receives ten times the optimizer steps of a 5 h pool, and the quantity curve therefore measures data and compute rather than data alone. The plan accepts this and mitigates it by recording the step count and printing a caveat. The approved review did not raise it at all; the word "epoch" does not appear in it. All three re-runs rate it Critical and reject the mitigation, on the grounds that recording a confound does not remove it. This matters because the quantity curve is what every clean-hour equivalence on this page is priced against.
It does not invalidate the design, and the fix costs no compute: the result has to be stated as the value of an hour under a fixed-epoch training policy, which is what it measures. A fixed-step arm stays out of scope; naming the estimand accurately does not.
Two honest limits on this exercise. More findings is not automatically better — the re-runs were not adjudicated, and only the fixed-epoch finding was checked back against the plan text. And this is still one model reviewing another: there is no human-reviewer baseline, so the reviewer's recall against a human has not been measured, only its recall against itself.
What the experiment varies
Four axes, one at a time, each against a clean baseline of identical size and composition. Quantity and label noise carry enough hour levels to fit a power law; the other two get pointwise contrasts.
| Axis | Levels | Hours | Runs | Fit | What it isolates |
|---|---|---|---|---|---|
| quantity | clean | 5 / 10 / 25 / 50 | 12 | power law | The reference curve. Inverting it is what turns a WER penalty into a clean-hour equivalent. Trained at a fixed 2 epochs, so it prices data and optimizer steps together — see above. |
| label noise | ln05, ln10, ln20 5%, 10%, 20% of words corrupted |
5 / 10 / 25 | 15 | power law | Transcript quality, audio fixed. Word-level corruption, every edit logged so it replays per clip. |
| acoustic | snr20, snr10, snr5, mp3_32k, ds8k | 25 | 5 | pointwise | Audio quality, transcript fixed: pink noise to a target SNR, 32 kbps MP3, 16k→8k→16k. |
| diversity | spk10, spk_all | 4 | 2 | pointwise | Speaker count at a fixed budget. 4 h is what 10 speakers can supply, so the data sets the budget. |
| baseline | zeroshot | 0 | 1 | excluded | Off-the-shelf whisper-small. Gives every other cell something to be measured against. |
Pre-registered hypotheses
Written before the first result, hashed into the manifest and every result row; the
analysis refuses to emit verdicts against rows carrying a different hash unless --post-hoc
is passed. That binds an analysis to one version of the file, but it does not prove the predictions
came first — that needs an external timestamping authority, and the hypotheses file says so.
Supported when the 95% bootstrap CI excludes zero with the predicted sign, refuted
with the opposite sign, inconclusive otherwise; effects under 0.5 WER points are also
flagged negligible.
| ID | Effect | Test set | Sign | Prediction |
|---|---|---|---|---|
| H-QTY-01 | slope of quantity / clean | test-clean | negative | More clean audio lowers WER, close enough to a power law over 5–50 h for a reliably negative exponent. |
| H-QTY-02 | slope of quantity / clean | test-other | negative | Same on the harder split, so the benefit is not an artifact of matched-condition clean speech. |
| H-LN-01 | ln20 @ 25 h vs clean @ 25 h | test-clean | positive | The load-bearing prediction: 20% corrupted words raises WER, making the exchange rate measurable at this scale. |
| H-LN-02 | ln05 @ 25 h vs clean @ 25 h | test-clean | positive | Even 5% corruption degrades WER measurably. May come back negligible, which is itself useful. |
| H-LN-03 | slope of label_noise / ln20 | test-clean | negative | Imperfectly labeled hours still help; this slope against the clean slope is the quantity-for-quality rate. |
| H-AC-01 | snr5 @ 25 h vs clean @ 25 h | test-clean | positive | Training at 5 dB SNR raises clean-speech WER against a clean pool of the same size. |
| H-AC-02 | snr5 @ 25 h vs clean @ 25 h | stress-snr5 | negative | Sign flips on the degraded set. This separates a domain-match effect from plain capacity loss. |
| H-AC-03 | mp3_32k @ 25 h vs clean @ 25 h | test-clean | positive | 32 kbps MP3 raises WER, by less than 5 dB mixing does. A negligible flag would be reasonable. |
| H-DIV-01 | spk10 @ 4 h vs spk_all @ 4 h | test-clean | positive | At a fixed 4 h budget, 10 speakers is worse than all of them, so diversity has value beyond quantity. |
Invariants the code enforces
Each has an enforcement point and a test, so the guarantee lives in the pipeline rather than in a convention someone has to remember.
| ID | Invariant | What it protects |
|---|---|---|
| I-001 | Test and dev sets are never perturbed, subsetted, or trained on. The stress set is the one derived copy, built read-only from test-clean. | clean evaluation |
| I-002 | Every cell trains under byte-identical hyperparameters; only data and seed differ. No runtime override, including on OOM. | cross-cell comparability |
| I-003 | For a fixed manifest and seed, prepare and perturb are bit-reproducible on the same platform, regardless of worker count. | reproducibility and resume |
| I-004 | The hypotheses file is fixed before the first analysis output exists, and the analysis stage never edits it. | an honest pre-registration |
| I-005 | A result row is written only after its evaluation completes. No partial rows, and each key is write-once. | trustworthy curves |
| I-006 | Pools are immutable after prepare. Perturb only writes new cell directories, and pool clips are digest-checked on read. | isolation between cells |
Pipeline
Five subcommands over one manifest. Every unit is guarded by a _COMPLETE
marker written last, so re-running a stage skips finished units and picks up interrupted ones. A
marker recording a different configuration is drift: the pipeline stops and names both hashes rather
than rebuilding silently, and --force is the deliberate way through.
prepare
splits, pools, test sets, QC
- Splits checked against the MD5s OpenSLR publishes, then built into hour pools by deterministic speaker round-robin, so smaller pools are exact prefixes of larger ones.
- Per-clip SNR, duration and silence ratio written beside every unit. Undecodable clips are quarantined with a reason; impossible hour/speaker requests fail before anything is written.
perturb
one dataset per cell and replicate
- Only acoustic cells write new audio; the rest store digest-bearing references into the pool, which keeps the disk budget reasonable.
- Label-noise cells emit a word-level alignment log that replays the corruption exactly. A digest mismatch stops the stage and changes nothing.
train
fine-tune under the frozen recipe
- AdamW, lr 1e-5, linear schedule, 10% warmup, 2 epochs, effective batch 16, grad checkpointing, bf16 on CUDA and fp32 elsewhere — every value from the manifest, none changed at runtime.
- Re-running the same command resumes from the last valid checkpoint or restarts that run cleanly.
eval
score every test set
- Greedy decoding under a frozen protocol whose normalization rules live in the manifest and are hashed into every row.
- Per-clip S/D/I/H counts are the authoritative analysis input, which is what makes clip-level bootstrap possible; the aggregate row carries WER, CER, seeds, four hashes, git SHA, versions and device.
analyze
tables, fits, figures, verdicts
- Dose-response tables, power-law fits with 2000-resample bootstrap CIs, the clean-hour equivalence table, and figures as PNG and SVG with backing Parquet. Equivalences outside the fitted range are marked out of support, not extrapolated.
- A partial grid still analyzes: missing keys are listed, nothing interpolated, exit status 0. Everything consumed is copied into a published snapshot.
--smoke
the whole thing in 30 seconds
- All five stages on 20 vendored utterances and a tiny fixture model, offline on CPU. Same code path, separate manifest: a smaller experiment, not an override.
- Last measured chain: 29.7 s, 30 result rows over 30 keys,
SMOKE OKfrom every stage.
How it is checked
Every command below was executed, with exit statuses and output recorded in the test report in the repository.
| Check | Command | Exit | Result |
|---|---|---|---|
| Formatting | ruff format --check . | 0 | PASS 74 files |
| Linting | ruff check . | 0 | PASS no rule disabled, no noqa added |
| Type checking | mypy (strict) | 0 | PASS 39 source files, two pre-existing ignores removed |
| Full test suite | pytest -q | 0 | PASS 213 passed, 2 skipped, 123 s |
| Fast subset | pytest -q -m "not slow" | 0 | PASS 156 passed, 1 skipped |
| Startup smoke | five --smoke stages, chained | 0 | PASS SMOKE OK x5, 33.7 s |
On top of that, an independent reviewer wrote a manual checklist against the requirements and the source, which was then executed against the working tree: 12 checks passed, including byte-identical output under 1 versus 4 workers and two force-rebuilds of the same acoustic cell. 17 more name a CUDA GPU or the full corpus as their own prerequisite and are listed as waiting rather than dropped.
Who executed those checks
The checklist was written independently, but those 12 checks were executed by the same agent that wrote the implementation. That is the weakest link in the verification story: an implementer checking its own work is the arrangement the role separation exists to avoid. So read those 12 as agent-attested.
Three of them have since been re-run by hand. Only one of the twelve P0 checks is feasible without a GPU or the real corpus, so the laptop-runnable subset — the smoke chain (P0), worker-count determinism and force-rebuild reproducibility (both P1) — was packaged as a script and executed by me directly, not by the agent. All three passed: the chain in 27 s with 30 result rows, 98 data files byte-identical between 1 and 4 workers, and both acoustic cells identical across two force-rebuilds. The caveat that keeps this honest is that the script was written by the implementing agent, so independent execution is a real improvement but not the same as independent judgement about what to run.
The one defect it found
Two concurrent analyze commands against the same workspace: the second correctly exits with a precondition failure, so no duplicate output is produced. But the message says an unidentified process holds the lock, and in one trial reported a stale PID from an unrelated run. It reproduces reliably and is filed with a fix planned; the per-unit train lock already names PID and host, so this is the results lock catching up.
What has run, and on what
Every number on this page came off one machine: a MacBook Pro with an Apple M1 Pro, macOS 26.5.2, arm64, no CUDA device.
Ran on the laptop
Apple M1 Pro · CPU, and Metal where the device resolver picked it
- Static checks and tests. ruff, ruff format and strict mypy over 39 files, all clean; 213 of 215 tests passed on CPU in 123 s.
- Smoke chain. 33.7 s as shipped with
--device autoresolving tomps; 29.7 s independently rerun from a clean workspace on--device cpu. - 12 manual checks. Boundary conditions, invalid input, failure paths, worker-count determinism, force-rebuilds.
- Scale. 20 vendored utterances and a tiny fixture model. No real corpus and no whisper-small training happened here.
Waiting on a GPU
one 16 GB CUDA card · 120 GB disk · the four real splits
- prepare on the real corpus, then 34 fine-tunes — 673 hours of training audio, twice through whisper-small, plus the zero-shot baseline.
- 105 result rows across test-clean, test-other and the 5 dB stress set.
- The analysis that needs real WER. The code paths run today on fixture data, but fixture WER is degenerate, so those outputs are structural rather than meaningful.
- 17 manual checks, including the VRAM ceiling and real out-of-memory behavior.
One caveat on Metal: it ran the fixture smoke chain and is not a stand-in for the CUDA target the recipe is written against, since the recipe specifies bf16 on CUDA and fp32 elsewhere. That is why the device is recorded in every result row.
What comes next
Compute, not code: download the four splits and run 34 fine-tunes on a 16 GB card inside a 120 GB disk budget. The grid runs without touching the source and partial results are analyzable as they land, so the curves fill in axis by axis. Then the power-law fits, the clean-hour equivalence table, the verdict table against the nine predictions, and a write-up of what the exchange rate turns out to be. Out of scope throughout: generative audio and TTS, multi-GPU training, hyperparameter search, new architectures, state-of-the-art claims, and license-gated datasets.