The Harness Was the Hard Part

#ai#kodr2#local-models#agents#harness#cli

The whole argument for kodr2 is that the interesting engineering in a coding agent isn’t the prompt and it isn’t the model - it’s the harness around them. I’ve written that a few ways now, but it’s easy to assert and hard to feel. So I ran an experiment: I had kodr build a smaller version of itself, phase by phase, and stayed out of the way.

The result is kodr-jr - a teaching-sized one-shot coding agent: LM Studio only, path-jailed file tools, an agent loop, a CLI. Every source file was written by kodr, driving a local gemma-4-26b-a4b. And then - the recursion I was after - kodr-jr, on the same model, built a working todo API. A harness wrote a smaller harness, on a laptop; what I want to write down is everything around that.

The frame, not the code

I didn’t write any of kodr-jr’s code - I wrote the frame: an AGENTS.md contract, four phase docs, and the load-bearing piece, a Stop hook. kodr’s hooks bind a shell command to “the agent thinks it’s finished”; a non-zero exit blocks the stop and kicks the model back into its heal loop. Mine ran biome format and node --check - kodr doesn’t walk away until the code is formatted and parses. Aim the model, gate the result.

The gate lied. Twice.

A gate that fails for the wrong reason is worse than no gate at all.

My hook ran find bin src .... In Phase 1 there was no bin/ yet, so it errored and blocked a finished build. kodr healed it the only way it could - mkdir -p bin, an empty directory to satisfy a bogus check. I hardened the hook to skip missing directories - and two phases later hit the same bug in a different coat. Now bin/ existed but was empty, and biome format on an empty directory also exits non-zero; my “fixed” hook only checked that the directory existed, so it sailed through and failed anyway. It burned all three heal turns; the transcript shows a 26-billion-parameter model talking itself into denial:

“The user is reporting a failure, but based on the output of npm run check, it seems that nothing was wrong… The ‘failure’ message looks like it might be from a previous attempt or a misinterpretation.”

Its code was fine; it just couldn’t know the test was broken - so it thrashed, formatting the repo over and over. That’s the lesson, and it’s about small models. A frontier model might have asked “is this check even valid?” This one took the gate as reality - so the harness doesn’t get to be approximately right. The fix was to assert about files, not directories: enumerate the .mjs files, run the tools only when the list is non-empty.

--continue continues a conversation, not a project

Continuation flipped on me too. My instinct was to chain the phases with kodr’s --continue. In Phase 2 that was a straight loss: it dragged Phase 1’s ninety-thousand-token conversation into a task that needed a few hundred tokens off disk, and blew its time budget. Phase 3, run fresh, was leaner for identical output, because the filesystem already held what the new phase needed. Yet --continue was right for a different job: when the agent loop shipped aborting on the first tool error, I fixed it by continuing that very run - “you just wrote this loop; change how it handles errors.” The question is never “is this related work?” - it’s “does the thing I need next live in the transcript, or on the disk?”

What the gate never caught

The hook gated formatting and syntax - so the repo had perfect formatting and valid syntax, and also three unused imports and a format:check script quietly linting kodr’s own run transcripts. A hook is a contract, not a conscience: it enforces exactly what you wrote and not one thing more. Want the guarantee? Write the gate.

A harder job, and a bug the easy one hid

The todo API came out in three turns - reassuring, but three turns barely exercises anything. So I gave kodr-jr something harder, still file-only: a library evaluating arithmetic expressions - evaluate('2 + 3 * (4 - 1)') → 11 - tokenizer, recursive-descent parser, evaluator.

It crashed on the first run. A few writes in, the model fumbled a tool call and left off the path argument entirely. The jail was airtight against a malicious path but had no answer for a missing one - it went straight into path.resolve(cwd, undefined), which throws. And the loop didn’t catch throws: the Phase 3 fix taught it to recover from error values, but an exception sailed past and killed the process. One dropped argument took the whole run down.

My reflex was to hand the bug back to kodr with --continue - but this time it didn’t work: kodr fumbled an edit_file, then timed out without returning. So I made the two-line fixes by hand. “Have the agent fix its own bug” is itself a harness, and it can fail; on a local 26B the self-repair loop sometimes stalls, and the human takes the keyboard. The re-run, hardened, ran eleven turns with no crash, 9/9.

The point

Martin Fowler calls this harness engineering: an agent gets better because a human iterates on the harness around it, not because the agent improves itself. I’ve been building kodr2 on that premise, and watching kodr build kodr-jr was the first time I saw it from the outside: the model was competent throughout, and everything that went sideways was a seam I’d gotten slightly wrong.

kodr built kodr-jr. The model was the easy part. The harness was the hard part - which is, more or less, the whole thing I’ve been trying to say.

Links: