Failure Taxonomy of an Agent-Run Research Lab: 105 Documented Process Failures, Who Caught Them, and an Open Dataset

Kiyoshi Casey

doi:10.5281/zenodo.23458363

← 321 AI Labs/321 AI Labs Technical Report TR#5

Failure Taxonomy of an Agent-Run Research Lab: 105 Documented Process Failures, Who Caught Them, and an Open Dataset

A census of one lab's documented process failures: where they lived, why they happened, which governance layer caught them, and the audited open dataset behind all of it.

Kiyoshi Casey · 321 AI Labs · 2026-10-17

In one paragraph

A failure taxonomy built from a research fleet's own operational record: N = 105 documented failure records drawn from four source families of one agent-run lab (flashnext 33, sprint-review 60, timesfm 7, job-killer 5), September 11-15 2026. Each record classifies one documented failure by pipeline phase, cause class, and detection layer. Failures concentrate in the build and verify phases, and one cause class dominates: verification_gap accounts for 46 of 105 records (44%). Detection resolves to self_caught 72/105 (69%) and reviewer_caught 33/105 (31%), with owner_caught 0 and escaped 0, reported as measured. The classification itself is audited: two independent blind classifiers reached Cohen's kappa 0.965/1.000/1.000 after two logged taxonomy revisions.

105

documented process failures in the sealed v1.1 dataset, four source projects

46/105

verification_gap cause class: the check existed but was never run (44%)

escaped failures in the window (owner_caught 0 too), with the census caveat attached

What did the audit find?

Failures concentrate where work is built and checked: build 35 and verify 27 together hold 62 of 105 records, with ops 25 and spec 18 behind them.
One cause class dominates: verification_gap 46/105 (44%), failures whose check existed but was never run or never wired; then environment_drift 17 and spec_drift 16.
Detection layering caught everything documented: self_caught 72/105 (69%), reviewer_caught 33/105 (31%), owner_caught 0, escaped 0, reported as measured with the dataset card's survivorship caveat.
The taxonomy was audited into shape: two blind classifiers on a 40-record stratified sample, per-field Cohen's kappa rising 0.728/0.458/0.946 to 0.965/1.000/1.000 across two logged revisions.
The Day-1 five (W2 D1-D5) resolve to corpus records P5-F-060 through P5-F-064; four of five catches came from the reviewer layer.
Cost coverage is a kept negative result: 3 measured / 1 estimate / 101 null, so no cost distribution is reported.

What does this not claim?

A census of documented failures in four deliberately governance-heavy projects; biased toward well-logged, well-caught failures, and not suitable for per-agent-hour rates.
escaped = 0 is a definition effect: late detections are recorded with their delay (e.g. the TimesFM OOM spiral, 8 days), not as escapes.
The kappa audit sample is 40 of 105 records: stratified, but a sample.
No external literature survey was performed; the report makes no comparative claims against agent-benchmark failure literature.
7 cluster records cover roughly 40 steward rejections, so per-record counts understate per-artifact event counts.

Where is everything hosted?

Paper (PDF) ↓Code + Data (GitHub) ↗DOI: 10.5281/zenodo.23458363 ↗

Sealed Artifact

PDF sha256: b2627f4def3e342b70a6d46d1afdf069e312fe5090130699f69700c3f2d60c06