Zero Downtime Hackathon · Port · Bright Data · SigNoz · Claude Code
Agents already write the fix. The failure mode isn't "the agent can't fix it" — it's "the fix is subtly wrong and ships." Canary Refinery canaries every agent-authored change against production data, lets SigNoz decide rollback or promote, and makes Port hold the human gate. Users never see a bad number — even when the agent is wrong.
Product · PricePulse wall · replay of a real run
The real PricePulse projector UI, fed by telemetry recorded from an actual run of the factory (57 frames, one per second). It auto-plays here; open it full-screen to drive it yourself — BREAK injects the bait, the loop repairs, the canary catches the wrong fix, and it waits for you to tap APPROVE.
Mission profile · T+ from BREAK
Four minutes, one button, one tile that never moves. Everything below happens live; the only thing seeded is the bait that makes the agent's first fix subtly wrong — and we say exactly how.
A judge presses the Port self-service action. The store's markup changes: cents in data-price, a was price, split digits. SigNoz freshness goes red → FreshnessBreach webhook → the Mayor opens bead REP-001 → a polecat spins up in a git worktree. PricePulse still shows $1,299 with a STALE badge. Bad-data-minutes: 0.
until ./verify.sh …; do claude -p "$(cat PROMPT.md)"; done. Iteration 1: 4/7 checks. Iteration 2: 7/7 — candidate v2. Each iteration is a span.
The Refinery ships v2 on 5 of 50 pages. SigNoz split dashboard: canary parse_ok 100 % … canary price median $129,900 vs prod $1,299. By every check the agent had, this fix was perfect.
CanaryDiverged → Port automation → rollback. Evidence attached to the Release. Tile: still 0. "The agent was wrong, and nobody got hurt."
The canary diff lands in PROMPT.md — "page P07 expected ≈1299, got 129900 — check units". v3 passes in one iteration. The caught case is banked as a regression fixture and verify.sh grows a magnitude check.
The Witness dropped the site's autonomy tier to approve-required after the rollback. v3 waits in Port with the canary snapshot, diff and trace link. A judge taps Approve → promote → 50/50 pages on v3.
The SigNoz timeline reads like a release log; Port's audit reads v1 → v2 rolled back (evidence) → v3 promoted by <judge>. Run it again: same brief, same sequence.
Vehicle · Gas Town roles · one Ralph loop
You can collapse the processes; you shouldn't collapse the permissions.
Reads SigNoz alerts and Port actions, opens beads, dispatches polecats, runs the production scrape on the promoted config.
One repair job in a throwaway git worktree. Runs the Ralph loop with an allow-listed tool set until verify.sh is green.
Canaries the candidate on 10 % of pages, compares to last-good, promotes or rolls back — only on production evidence. Banks regressions; tightens verify.
Watches for stuck loops and cost, keeps the autonomy tier (auto → approve after a rollback), escalates to a human through Port.
# factory/ralph.sh — dumb, relentless, bounded until ./verify.sh "$RULES"; do i=$((i+1)); [ "$i" -gt "$MAX_ITER" ] && exit 2 python -m factory.agent --rules "$RULES" --prompt PROMPT.md --iteration "$i" done
# brief/constraints.yaml — what the factory is held to
slo: { freshness_seconds: 20, bad_data_minutes: 0 }
canary: { fraction: 0.10, drift_band_pct: 20, healthy_after_seconds: 15 }
repair: { max_iterations: 8, max_cost_usd: 2.00 }
autonomy: { default_tier: auto, drop_to_approve_after_rollback: true }
Payload · what the sponsors' tools carry
Remove any one of them and the loop stops being safe.
inject_site_change is BREAK; approve_release has requiredApproval — the human decision on stage.bdata scraper heal is just another candidate — it still has to survive the canary.config_version and lane; gauges for freshness, price median by lane, canary drift, bad-data-minutes.FreshnessBreach, CanaryDiverged, CanaryHealthy, RepairStuck — webhook the Mayor. SigNoz decides promote vs rollback.Launch · 60 seconds · no accounts needed
Fully offline by default: a deterministic scripted agent and a local alerter that speaks SigNoz's webhook format. Flip env switches to go live with Claude Code, SigNoz Cloud, Port and Bright Data.
git clone https://github.com/vnmoorthy/canary-refinery && cd canary-refinery make setup # uv venv + deps make up # store :8701 · PricePulse :8702/wall · Mayor :8703 make break # VARIANT=3 — the bait make approve # when the tier asks for a human make test # incl. the scripted end-to-end loop
The wrong fix is seeded: verify.sh v1 checks parse rate, nulls, types and schema — deliberately not magnitude — and the bait puts the price in cents in a clean data-price attribute. The canary catching it is real; the rollback, evidence injection, regression banking and tier drop are real; the approval is a real human tap.
With AGENT=claude the live agent may get it right first time. Then the canary promotes cleanly and make replay re-runs the banked wrong-fix path — and we say so.
Telemetry · one trace, one audit trail
Every transition is a canonical event in SigNoz, Port and the local mirror. This is the sequence the factory emits, every run.
Rubric map for judgesOffice hours
Those canary on 5xx and latency. We canary on data correctness — parse rate, null rate, distribution drift vs last-good — for a change a human never wrote, and the rollback feeds the evidence back into the agent loop and tightens the next verify. Nothing today does evidence-gated promotion for agent-authored commits.
We do — every caught case becomes a check (you watch the magnitude rule land). But you can't enumerate every way a fix can be subtly wrong before you've seen it. Verify catches what you knew; canary catches what you didn't.
Scrapers are the wedge because the output is objectively measurable and agent churn is highest. The buyer is the data-platform team; the unit is a protected pipeline; the metric is bad-data-minutes. The same gate generalizes to ETL transforms, prompt changes, feature-flag rules.
The roles are separation of duties: the thing that writes the fix can't promote it; the thing that promotes only acts on production evidence; the thing that watches for stuck loops can't edit code. Beads make every run replayable — that's how we show repeatability.
Drift band and canary size are per-site policy in the brief; a diverged canary escalates to a human with evidence rather than blocking forever; canary pages are stratified by price band. We'd rather false-block and ask than false-promote.