September 12–30, 2026 · recorded project activity · repo

FaultLab

Fault injection and evidence gates for tool-using agents

A team-built, AI-assisted lab that tests agents in isolated synthetic shop worlds and keeps incomplete evidence from becoming an accepted recovery policy.

When the tool succeeds but its response disappears

The task sounds simple: upgrade an order and send one truthful confirmation. But an upgrade can commit while its response is lost. The agent cannot tell whether to retry, wait or report success. A retry risks a duplicate effect; a confident answer without a receipt risks inventing evidence.

FaultLab makes that ambiguity happen in a synthetic shop. Delay, lost responses, stale reads and temporary errors are controlled faults, with no real orders or customers involved. The question is whether an agent can recover safely, and whether a proposed recovery rule survives a harder case.

Separate the proposal from the verdict

The Actor performs the shop task. The Explorer chooses fault schedules and later tries to break a candidate repair. The Mechanic proposes bounded recovery hooks from the recorded evidence. The Referee grades tool effects and the final report with fixed-code checks, using the simulator's private truth. It is fixed code: the model proposing a repair does not grade its own work.

Python and FastAPI expose the lab, while each simulated world has its own SQLite database. Transactions serialize clock changes and effects; protected receipt, projection and event histories preserve what actually happened. The React dashboard presents that evidence and the backend's decisions.

Make each stage earn the next

The loop first reproduces a failure in fresh worlds, then reduces the fault to the smallest recipe that still breaks the agent. A controlled experiment tests the explanation before the Mechanic proposes a rule. Source validation checks that rule against the original failure; an adversarial challenge tests whether it survives different fault schedules. Promotion is a separate gate.

These stages retain different outcomes. An agent violation, a lab error and a scheduled fault that never fired cannot be treated as the same result. A source validation pass alone cannot replace the active policy.

How a candidate repair reaches the promotion gate
Actor runs fresh shop worlds. The fixed-code Referee grades against private simulator truth throughout.
Diagnose
01 · Reproduce & reduce
Find the smallest fault recipe that still reproduces the failure.
02 · Controlled diagnosis
Test the explanation with a controlled experiment.
Propose
03 · Mechanic proposes
Build a bounded recovery hook from recorded evidence.
04 · Source validation
Check the candidate against the original failure.
Challenge
05 · Explorer challenge
Try to break the candidate with other fault schedules.
06 · Promotion decision
A separate gate decides whether a candidate may replace the active policy.
A pass at one stage is not promotion
Failed or incomplete evidence → retain baseline
A failed challenge, lab error or scheduled fault that never fired does not activate a repair.
Recorded continuation through September 30, 2026: no generated policy was accepted; policy-v0 remained active. Historical evidence imports are a separate review workflow.

The historical campaign rejected its own repair

In the September 14, 2026 campaign, a delayed upgrade response left the Actor polling for status without filing its final report. The failure reproduced in three fresh worlds; removing the fault let the agent complete all three.

The Mechanic proposed re-reading the order, waiting, replaying the request once and never confirming without a receipt. One validation batch passed all three trials, while other batches passed one and two: the results disagreed. The Explorer then combined a longer delay with a slow response, and the rule broke in all three challenge trials. Promotion was blocked. After receiving the counterexample, the original Mechanic proposed the same rule again; the run ended no change.

That campaign also read back historical Weave traces by exact trace ID and matched them to local records. This verifies its trace evidence, not repair effectiveness, later source code or a production agent.

Reviewing history must not execute it

The continuation added a regression-review workflow. Importing a ZIP saves historical source evidence without a provider request, new trial or policy activation. Its original decision remains historical rather than becoming a local acceptance claim.

A fresh imported regression run is a separate action. The dashboard shows its task, target and spending bound before confirmation. The backend binds the store, manifest and target configuration to a saved request ID and charges model requests, including transport retries, under the declared ceiling. If the response times out, the UI retains the unresolved ID: checking or retrying that same request recovers its execution rather than starting another. Runtime fingerprints keep older evidence attached to the code that produced it.

What remains unproven

Through September 30, 2026, some native-agent candidates passed source validation and completed challenges with no failed checks. A scheduled status-read fault still did not fire, so those challenges remained inconclusive. Generated accepted policies: ; recorded active pointers remained at policy-v0. Full product acceptance and accepted-repair transfer remain unproven.

The diagnosis step also exposed a failure in the lab itself: an overlength model hypothesis failed schema validation, and code stored an inconclusive default. The repair preserves attempts and validation errors and distinguishes a valid model request from a code default. Recorded offline checks support that software change; they do not establish improved end-to-end repair effectiveness.

An additional diagnosis guide was built but is not offered in normal Learn runs. Its first request-only probe did not meet the declared acceptance rule; the second was incomplete after provider errors. Provider availability was not rechecked for this page.

Co-development and evidence

I co-developed the original project with Aryan Bhusari as members of Team Gatekeeper, a two-person team at CoreWeave Hacks. Implementation and review continued with Claude and Cursor Agent. The capabilities described here belong to the team project; the records do not establish sole authorship of every later component.

The public repository includes the historical campaign README. The résumé corpus also cites dated local continuation records through September 30. This page does not publish those records or imply that all later changes are present in the public repository.

Back to the FaultLab project

How it was built

Time:
Original September hackathon, followed by recorded product continuation through September 30, 2026.
Tools:
Claude, Cursor Agent
My contribution:
Co-developed with Aryan Bhusari as members of Team Gatekeeper, a two-person team at CoreWeave Hacks. The capabilities here belong to the team project; sole authorship of each later component is not established.
Validation:
Recorded offline software checks, saved evidence integrity and fixed experiment gates. Passing software tests does not establish a generated accepted repair or transfer to another agent.