AI BENCHNOTES
Agent loopsBENCH NOTE 007Coding agentsHarnesses
AI BENCHNOTES
THE COMPLETE WORKBENCHPick a machine part.
12 NOTES / 12 LABS / ZERO INCENSE

Start anywhere. The numbers are a useful route, not homework police.

00Animated bench map
CONCEPT 007steel-toe mode

Coding agents:repo repairwith receipts.

A coding agent is not autocomplete with hands. It inspects a repository, forms a bounded plan, uses tools, reads what happened, revises, and hands back evidence.

  • SEARCHbefore changing
  • PATCHthe smallest surface
  • PROVEbefore handing back
01 / THE SHORT VERSION

Autocomplete proposes code. An agent negotiates with reality.

The useful unit is not “a clever completion.” It is a closed loop: observe the repo, make a bounded change, run something that can disagree, then react to the result.

CODE COMPLETION

Nearby text goes in.

const total = items.reduce(…)

A plausible continuation comes out. Useful—and then it stops.

NO FEEDBACK LOOP
CODING AGENT

A task meets a living repository.

  1. 01Read rules + current state
  2. 02Search for the real change surface
  3. 03Patch, execute, inspect, revise
  4. 04Return the diff + evidence + caveats
REALITY GETS A VOTE

The output is not the diff. It is the diff plus a reason to believe the diff belongs there.

02 / INTERACTIVE REPAIR BAY

Turn down the evidence. Watch confidence outrun correctness.

Same task. Same model. One dial changes how much repository evidence the loop may collect. The dial does not add intelligence; it gives reality more chances to disagree.

ACTIVE WORK ORDERAdd owner_name to the CSV export.

Keep column order stable. Update relevant tests. Do not edit generated output or unrelated dirty files.

JOB #CSV-42

Lower settings cut off observations and checks—not merely “extra polish.”

  1. 01
    git status + AGENTS.md

    Scope the bay

    Find local rules, allowed files, and three unrelated user edits.

  2. 02
    rg "owner|csv"

    Locate the source

    Search connects the schema, exporter, and targeted test.

  3. 03
    src/export/csv.ts

    Make one small patch

    Change the header and source serializer; leave nearby cleanup alone.

  4. 04
    pnpm test csv

    Let code object

    FAIL: row serializer omitted owner. Revise the patch. PASS.

  5. 05
    typecheck + git diff

    Close the loop

    Types pass. Diff matches scope. Hand back checks and caveats.

HANDOFF READY
The change arrives with evidence.

Targeted tests and typecheck pass; the final diff is inspected; pre-existing dirty files stay untouched.

03 / THE ACTUAL MACHINE

The loop is the product.

Models propose the next action. The runtime supplies tools, permissions, files, command output, and stopping conditions. Each observation changes what action makes sense next.

STANDARD OPERATING CYCLEFAILURE IS ROUTING INFORMATION
  1. 01
    INSPECT

    Read the shop manual.

    Load the task, nearest repository instructions, current worktree state, and relevant project map.

  2. 02
    LOCATE

    Search before edit.

    Trace names, call sites, tests, and ownership until the change surface is evidence—not a filename-shaped hunch.

  3. 03
    BOUND

    Plan the smallest move.

    Name the files, constraints, expected behavior, and checks. Ask before crossing a permission boundary.

  4. 04
    PATCH

    Change source, not exhaust fumes.

    Edit in reviewable increments. Generated files, build output, and vendored code usually have a source upstream.

  5. 05
    EXERCISE

    Run the closest disagreement.

    Start with a targeted test, typecheck, lint, or build that can expose the specific mistake quickly.

  6. 06
    HAND BACK

    Return proof, not posture.

    Inspect the diff. Report what changed, what passed, what was preserved, and what remains uncertain.

A failed check is not the loop failing. It is the loop finally receiving useful information.

04 / WRITE A FINISH LINE

A good work order has edges.

The agent can discover implementation detail. It should not have to invent the desired outcome, acceptable blast radius, or definition of done.

VAGUE TICKETNO. ???
“Improve CSV exports.”
  • No observable result
  • No boundary around adjacent cleanup
  • No check that defines “improved”
INFINITE SIDE QUEST
BOUNDED WORK ORDERNO. CSV-42
OUTCOME

CSV exports include owner_name after project_name.

ACCEPTANCE

Existing column order stays stable; blank owners serialize as empty.

SCOPE

Exporter source + focused tests. Preserve unrelated dirty files.

BOUNDARY

Do not edit generated clients or change the public schema.

EVIDENCE

Run export tests and TypeScript checks; report anything not run.

AGENT-COMPLETABLE

Useful freedom: let the agent discover where the change belongs. Expensive ambiguity: make it guess what success means.

05 / SHOP RULES

Before touching a bolt.

OBSERVE→ PATCH SMALL→ CHECK→ WIDEN IF NEEDED
  1. 01READ

    Read the nearest instructions.

    Repository guidance, directory-specific rules, conventions, and requested commands define the local physics.

  2. 02STATUS

    Take a worktree baseline.

    Check status before editing. Pre-existing modifications belong to their author; preserve them and avoid accidental bundling.

  3. 03TRACE

    Search through the behavior.

    Find definitions, references, tests, schemas, and generators. The first matching file is a clue, not permission to edit.

  4. 04SMALL

    Patch at checkpoint size.

    Make one coherent change, inspect the diff, run the closest check, then widen only when evidence requires it.

  5. 05ASK

    Respect the cage.

    Sandboxes and approval gates are part of the system. If a command needs more access, explain why and request it—do not route around it.

  6. 06PROVE

    Verify in proportion to risk.

    A copy change may need a build. A data migration needs considerably more. “Looks right” is not an executable check.

CHECKPOINT RHYTHM
  1. observe
  2. edit one idea
  3. inspect diff
  4. run closest check
  5. record evidence
06 / GREASE ON THE FLOOR

Six ways the wrench slips.

Most ugly agent failures are not exotic model mysteries. They are ordinary software-process mistakes performed quickly and at scale.

HAZARD 01

Filename clairvoyance

Smell: Opens the obvious file and edits immediately.

Countermeasure: Search references, tests, and ownership before choosing the change surface.

HAZARD 02

Generated-file surgery

Smell: Fixes dist/, a generated client, lock output, or compiled assets.

Countermeasure: Find the generator or source-of-truth; regenerate only when the workflow calls for it.

HAZARD 03

Dirty-tree amnesia

Smell: Treats every existing diff as agent-created—or disposable.

Countermeasure: Baseline status, isolate the requested patch, and name what was intentionally preserved.

HAZARD 04

The mega-patch

Smell: Mixes the fix, a refactor, formatting, and three acts of personal growth.

Countermeasure: Use small coherent patches. Review and verify before expanding scope.

HAZARD 05

Permission parkour

Smell: Works around a sandbox, secret, network, or approval boundary.

Countermeasure: Pause, state the blocked action and reason, then request the narrow authority needed.

HAZARD 06

Green by assumption

Smell: Reports success because the edit looks plausible.

Countermeasure: Run the nearest meaningful check, inspect its output, and disclose checks that could not run.

07 / IMPLEMENTATION RECEIPT

The model chooses. The runtime makes it real.

“Agent” describes a system, not just a model call. The surrounding runtime decides which files and tools are visible, executes actions, returns observations, enforces permissions, and stops the loop.

PROVIDER-NEUTRAL PSEUDOCODErepair-loop.txt
function repair(task, repo, policy):
  rules    = read_nearest_guidance(repo)
  baseline = inspect_status(repo)
  evidence = search(task, repo)
  plan     = bound_change(task, rules, evidence)

  while plan.has_next_step():
    require_permission(plan.next_step, policy)
    patch  = edit_smallest_source(plan)
    result = run_closest_check(patch)

    if result.failed:
      plan = revise_from(result)
      continue

    return handoff(
      diff=inspect_diff(baseline),
      checks=result.passed,
      caveats=result.not_run
    )

  return blocked(reason, next_evidence_needed)
08 / CALIBRATE YOUR ROBOT COSTUME

Four myths to leave at reception.

01

“It reads the whole repository.”

Usually it sees selected files, search results, instructions, diffs, and command output inside a bounded context. Good retrieval and inspection still matter.

02

“More autonomy means fewer checkpoints.”

Longer loops create more opportunities to drift. Small patches and explicit checkpoints make recovery cheaper.

03

“Passing tests proves correctness.”

Tests prove only the expectations they encode. Review the diff, consider missing coverage, and state caveats.

04

“The fastest agent makes the biggest patch.”

A smaller change is easier to inspect, exercise, revert, and trust. Throughput without recoverability is expensive theater.

AI BENCHNOTES

For builders who prefer understanding the machine to worshipping it.

Back to top ↑