Skip to content

the harness · evidence

Measured, not asserted.

On eight coding tasks it had never seen, the same 27B model on a Mac scored 100 of 100 inside the Skena harness and 75 outside it; the 35B, 100 and 38. Our tasks, our scorer, one Mac, one run — and this page is where those words are unpacked.

held out · run once

Eight tasks it had never seen

Every earlier number the harness produced was post-tuning: the harness was shaped by traces of the tasks it was then measured on. So eight new tasks were written, proven solvable, and committed before any model saw them. Every lane ran them once. The rows are reported as they came, and the harness was not changed in between.

Servant · 12 points

A capability that reads a CSV and aggregates it, with the error path

Qwen 35B (A3B)
0→12
Qwen 27B
12→12

Servant · 13 points

A workflow that fans two independent fetches into one merge step

Qwen 35B (A3B)
13→13
Qwen 27B
13→13

Servant · 12 points

An existing capability repaired in place, its manifest protected

Qwen 35B (A3B)
0→12
Qwen 27B
0→12

Servant · 13 points

A new capability plus its weekday-morning schedule

Qwen 35B (A3B)
0→13
Qwen 27B
13→13

Builder · 13 points

A changelog page over ordered content, routed and linked

Qwen 35B (A3B)
0→13
Qwen 27B
13→13

Builder · 12 points

An optional badge prop on an existing component, one card using it

Qwen 35B (A3B)
0→12
Qwen 27B
12→12

Builder · 12 points

A team page and the component it is built from

Qwen 35B (A3B)
12→12
Qwen 27B
12→12

Builder · 13 points

A new private settings page, and an existing page made private

Qwen 35B (A3B)
13→13
Qwen 27B
0→13

total · 100 points

Qwen 35B (A3B)
38→100
Qwen 27B
75→100

With the harness, both models finished all eight with the project's own check green. Without it, the 35B lost five to the turn cap after one-read-per-turn discovery or to a claim made without running the check; the 27B lost the repair (claimed after hand probes) and the private page (a component check instead of the structural flag).

The harness did visible work on the way: on the 35B it withheld three hand probes of the CSV task before the model ran the project's check, and it reminded the model once, near the turn ceiling, that the repair was still unverified.

One artefact of the test bed is disclosed: a probe that named the workspace by its virtual path saw a missing file. Four runs hit it, all still passed through the project's check, and it was fixed only after these rows were written.

tuned corpora · five runs

The same shape, five times

The two corpora the harness was tuned on were then run five times per lane. These numbers are post-tuning and say so; what they add is repeatability.

Shipping corpus (18 scenarios)

Qwen 35B

without 69 · 84 · 72 · 72 · 66

with 100 · 94 · 96 · 100 · 96

Shipping corpus (18 scenarios)

Qwen 27B

without 74 · 77 · 84 · 77 · 80

with 100 · 100 · 94 · 94 · 100

Authoring corpus (8 tasks)

Qwen 35B

without 12 · 37 · 25 · 25 · 24

with 100 · 100 · 100 · 100 · 100

Authoring corpus (8 tasks)

Qwen 27B

without 76 · 87 · 88 · 75 · 87

with 100 · 100 · 100 · 100 · 100

On the shipping corpus the harness's worst run beats the model's best run without it, on both weights. Every miss with the harness is a single scenario and none is a missing check: a finish summary past its word bound, a summary without its anchor words, a correct edit on the last turn of a four-turn cap. Without the harness the misses are the same shapes every time — the job never resumes after asking the user, an uncertain click is guessed at, a private page gets a component check instead of the structural flag, a repair is claimed without its test.

the mechanics

What the harness actually does

Nothing here trains, tunes or replaces the model. The harness constrains a stock model with rules that live in the product — the same rules run in Servant's agent, in Builder's agent, and in the test bed that produced these numbers.

the contract
Each turn opens with the project's own check and the files that define it. Read the spec before writing; run the check before finishing.
the gate
A "done" with an unverified change is refused, and the refusal names the exact command to run next.
the steer
The first change of a phase carries the check's name in the result the model reads next.
the probe budget
A hand-run script that fails is steered to the check once; after two failures the next probe is withheld until the check runs.
the turn budget
At three quarters of the turn ceiling, one reminder: turns remain, and the change is unverified.
the continuation
Asking the user ends the turn. The answer starts a fresh worker with the objective, the answer and a new observation.
the brakes
Repeated or failed inspections are capped per phase; repeated calls, results or intent stop the run before it burns the budget.
the scope
A checked-in file is changed by a splice, never rewritten; protected files stay protected; structural rules are enforced, not suggested.

× Method and bounds

Weights
Qwen3.6 35B-A3B and Qwen3.8 27B, both 8-bit, served by Servant's own local runtime on one Mac.
Without the harness
The same model in a bare agent loop over the same tools — read, grep, edit, write, a shell, finish — with a two-line system prompt and no rules.
With the harness
The same loop with the mechanics above, exactly as Servant ships them.
Tasks
Real files in a disposable workspace, each with the project's own test. A task counts only when that test is green and the claim was made after it ran.
Held-out
The eight tasks were committed before any run, run once per lane, and reported as they came. A fixture would have been repaired only if no correct answer could pass it; none was.
Bounds
Our tasks, our scorer, one Mac, one run of the held-out set, five of the tuned sets. Two weights, one model family. Wall-clock is not compared. A small local model is not a frontier model; the claim is about the floor a harness puts under one, not the ceiling.
The full report
The technical report and every trace live in the Skena repository (docs/skena-small-model-harness-report.md and docs/evidence/). The same weights through other coding agents are compared there, not on this page.
Download for Mac

Free · Apple Silicon Mac · v0.1.13