Personal · open source
Agent Harness: overnight software delivery with verification gates
A driver that runs coding agents overnight, one fresh session per feature, and merges only work an independent session has verified.
Designer and operator · Sep – Oct 2026
- features passing in one day
- 55
- verified and merged, 12 PRs
- 52
- tests added
- +374
- per passing feature
- ≈ $2.53
The Problem
I have about 10 hours a week and several personal projects. I wanted coding agents to build overnight while I slept, and to steer them from my phone. Every approach that lived inside one session hit one of three walls: the model's context limit, my plan's usage limit, or the auto-mode permission classifier.
One run
Around it
The Method
Diagnose
Every single-session approach hit one of three walls: the context limit, the usage limit, or the permission classifier. One night three loops shared one plan and every repo went silent from 00:12 to 07:19; a long thread ran out of context; the classifier blocked merges.
Decide
I gave each feature a fresh session and put all state on disk: a feature list, a progress log and git. Agents do the judgment; a deterministic bash driver does the bookkeeping.
Build
Harness v1 was a ~60-line script inside one project. v2 became its own repo: a worktree per run, repair and continue, an independent verify per milestone, steering from my phone. v3 merged verified milestones itself, waited out usage limits and queued runs.
Verify
A red suite un-passes a feature. Each finished milestone gets an independent verify session that re-runs every step and writes one evidence row per step. Nothing merges unless the tree that lands on main is byte-for-byte the tree that was verified.
Simplify
I went from three parallel loops to one serial queue under one usage limit for the whole machine, and from a human tap on every merge to a gated merge. A PR-only mode stays for projects where I still merge, like this site.
Key Decisions
- D-H1
A fresh session per feature
Chose a new headless session for every feature, with its state on disk, over one long loop session or an orchestrator with subagents because every single-session approach hit the context limit eventually; subagents only delayed it.
Cost: Every piece of state has to be written down, which is also what makes it inspectable.
- D-H2
Models by role
Chose the most capable model for the one-time initializer, Opus to build and review, and Sonnet to verify, over one model for everything because the initializer runs once and decides the quality of the whole night, while verification can run on a cheaper model.
- D-H4
Guard against false passes
Chose a red suite that un-passes the feature, plus an independent verify session per milestone, over trusting the session's own "passes" because the expensive failure is a feature marked done that isn't, and a second agent re-running the steps cold is the cheapest independent check.
Cost: One more session per milestone, capped at two rounds after a stub test showed the coder and verifier ping-ponging.
- D-H5
The driver does the git bookkeeping
Chose a deterministic driver for sync, push, tag, PR and cleanup over letting agents merge and clean up because these steps must happen every time and the same way: a script does that; a prompt might not.
- D-H11
Detect limits from structure, test against the real message
Chose reading the structured error fields first, with a regex only as a fallback, and never counting a $0 session as an attempt, over a wider regex because structure doesn't change its wording between releases, and the $0 rule catches the next unknown error too.
- D-H17
The harness merges, under one invariant
Chose one PR per verified milestone, opened and merged by the driver in milestone order, over a human tap on every merge because mornings shouldn't be a merge queue, and the tree that lands on main is byte-for-byte a tree that was verified, after its own review, an independent verify and a green suite.
Cost: Human review moved from every merge to policy and permissions; a PR-only setting restores the tap, and this site's run uses it.
How It Evolved
Sep 2026 · Long sessions
Long interactive sessions, with spec, research and decisions files for each idea.
It worked, but it needed me at the keyboard.
Oct 13–15 · Runbook + loop
A written runbook as the agent's memory and an overnight loop; later one fresh thread per milestone, each ending with a handoff file.
Three loops ran on one plan and every repo went silent from 00:12 to 07:19 on usage limits. A long thread hit "context usage too high".
Oct 15–16 · Orchestrator + subagents
One session orchestrated by role: Sonnet wrote failing tests, Opus implemented, Opus reviewed.
It delivered (one app gained 154 tests in about 2.5 hours), then hit a new wall: the permission classifier blocked a rebuild as a "production deploy", and blocked merges.
Oct 16 eve · Harness v1
After Anthropic's "Effective harnesses for long-running agents": an initializer turned the spec into a feature list, then one fresh session ran per feature, from a ~60-line script.
It lived inside one project, ran in place, and one guard trip stopped the whole night.
Oct 16 night · Harness v2
Its own repo. "Agents do judgment; the driver does bookkeeping." A worktree per run, repair and continue, an independent verify per milestone, tag + PR, steering from my phone through INBOX.md, metrics.
Merges still needed a human, and each run handled usage limits on its own.
Oct 17 · Harness v3
Merges verified milestones itself, waits out usage limits for the whole machine, queues runs, and writes a retro and a ledger entry per run.
The current version. The run that built this site update is a v3 run.
- Sep 2026Long sessionsNeeded me at the keyboard
- Oct 13–15Runbook + loopHit usage and context limits
- Oct 15–16Orchestrator + subagentsBlocked by the permission classifier
- Oct 16 eveHarness v1One guard trip stopped the night
- Oct 16 nightHarness v2Merges still needed a human
- Oct 17Harness v3Current: it built this update
What Broke
What broke The harness broke its own rule. I had tested the usage-limit detector against a message I made up; the real one said "You've hit your session limit", so 45 sessions were counted as failed attempts in 6 minutes, with 5 hours of budget left.
The rule Fakes mirror the live API: read the structured error first, and test against output captured from the real CLI.
What broke Two runs in parallel used up the 5-hour usage window in 2 h 49 min.
The rule One usage limit for the whole machine, and runs go one at a time by default.
What broke On one project, verify sessions took 29–40 s, and review found the real defects later.
The rule A verify writes one evidence row per step; with rows missing, it runs again on Opus.
What broke An independent review of v3 before its first run found 2 critical and 8 important bugs, including a verify refused by a usage limit that counted as "holds" and would have merged code with zero evidence.
The rule The harness gets reviewed before it drives a real night, like any other code.
What broke One feature waiting on a human stopped a whole run: 14 independent features never ran behind it.
The rule Independent work has to be declared. The initializer can give a feature an after list so it doesn't wait on a blocked one, but by default a feature still waits on everything before it; blocking only on declared dependencies is the next proposed change.
Outcome
On Oct 17 the harness ran two projects, starting overnight on v2 and finishing on v3 that day: 71 features planned, 55 passing, 52 verified and merged through 12 PRs, and 374 tests added (1,242 → 1,616). It cost $139.34 in total, about $2.53 per passing feature, across 127 sessions and 13.9 hours of run time. The update to this site is a v3 run too: its pull requests are linked below.
In the Code
# Merge one verified milestone (its tag) into main through a PR. The invariant: the tree that lands on main is the
# tree that was verified. 0 = merged or proposed · 1 = needs Michael (conflict, refusal) · 3 = held: main has content
# this milestone wasn't verified with (the next milestone's verify includes it, and its merge carries this one)
merge_one() {
local m=$1 tag=run/$RUN_ID/$1 c head url tree
head=harness-merge/$RUN_ID-$m
c=$(git rev-list -n1 "$tag") || return 1
git fetch -q origin "$MAIN_BRANCH" || return 1
if git merge-base --is-ancestor "$c" "origin/$MAIN_BRANCH"; then touch "$STATE/merged-$m"; note "$m merged into $MAIN_BRANCH (already there)"; return 0; fi
if ! tree=$(git merge-tree --write-tree "origin/$MAIN_BRANCH" "$c" 2>/dev/null | head -1); then
note "$m conflicts with $MAIN_BRANCH; not merged (needs Michael)"; notify "$m needs Michael: conflicts with $MAIN_BRANCH"; return 1
fi
[[ $tree == "$(git rev-parse "$c^{tree}")" ]] || return 3Links
How it was builtI designed the process, the specs, the decision log and the verification gates, and I ran each night and reviewed its results. Claude Code agents wrote much of the harness code under that direction.