Harbor benchmark
Agents do better work with Nous AI: +14 points.
Same agent, same task, with and without Nous AI. With Nous AI, agents met 88% of each task's checks; without it, 74%. Every model tested improved.
Harbor standard v2: 38 paired runs, run by Nous AI on three small, fast models, closed 1 November 2026. Switch between With and Without to see the gain. Round v3, on the rewritten skills, is running; these figures change when it closes. Scores are the share of each task's automated checks met.
+14 pts
average gain with Nous AI (88% vs 74%)
28 of 38
runs won by Nous AI; 3 lost, 7 tied
3 of 3
models improved, +13 to +15 pts each
+9.4 to +18.5
95% interval of the gain: above zero throughout
Current round · harbor-standard-v3
Every skill, every model, as the runs come in.
The skills agents get today, each run one model doing one work task twice, with the skill and without it. Target: 15 runs per skill and model.
75 runs · with the skill 86% vs 80% without (+6 pts; 95% interval +3.9 pts to +9.7 pts) · 43 wins, 12 losses, 20 ties · last run 5 Nov, 17:52 UTC
| Skill | Model | Runs | With | Without | Gain | Won · lost · tied |
|---|---|---|---|---|---|---|
| Teaching (45 of 15) | Claude | 15 | 98% | 96% | +2 pts | 5 · 1 · 9 |
| Gemma | 15 | 87% | 82% | +5 pts | 9 · 3 · 3 | |
| gpt-oss | 15 | 90% | 79% | +11 pts | 10 · 3 · 2 | |
| Curriculum design (6 of 15) | Claude | 2 | 100% | 100% | 0 pts | 0 · 0 · 2 |
| Gemma | 2 | 92% | 83% | +9 pts | 1 · 0 · 1 | |
| gpt-oss | 2 | 88% | 100% | -12 pts | 0 · 2 · 0 | |
| Design (6 of 15) | Claude | 2 | 96% | 85% | +11 pts | 2 · 0 · 0 |
| Gemma | 2 | 71% | 56% | +15 pts | 2 · 0 · 0 | |
| gpt-oss | 2 | 78% | 74% | +4 pts | 1 · 1 · 0 | |
| Finance (6 of 15) | Claude | 2 | 96% | 69% | +27 pts | 2 · 0 · 0 |
| Gemma | 2 | 68% | 61% | +7 pts | 1 · 0 · 1 | |
| gpt-oss | 2 | 64% | 69% | -5 pts | 1 · 1 · 0 | |
| Marketing (6 of 15) | Claude | 2 | 90% | 82% | +8 pts | 2 · 0 · 0 |
| Gemma | 2 | 75% | 68% | +7 pts | 1 · 0 · 1 | |
| gpt-oss | 2 | 61% | 57% | +4 pts | 1 · 0 · 1 | |
| Product (6 of 15) | Claude | 2 | 82% | 64% | +18 pts | 2 · 0 · 0 |
| Gemma | 2 | 50% | 54% | -4 pts | 1 · 1 · 0 | |
| gpt-oss | 2 | 65% | 39% | +26 pts | 2 · 0 · 0 |
Small samples: a skill needs 15 runs per model before its result means much. A skill that loses keeps serving while a better version is written.
Previous round · harbor-standard-v2
The skills before their latest rewrite.
The same tasks, checks and harness, measuring the previous version of the skills. Kept on its own, never pooled with the current round.
38 runs · with the skill 88% vs 74% without (+14 pts; 95% interval +9.4 pts to +18.5 pts) · 28 wins, 3 losses, 7 ties · last run 1 Nov, 15:53 UTC
| Skill | Model | Runs | With | Without | Gain | Won · lost · tied |
|---|---|---|---|---|---|---|
| Teaching (10 of 15) | Claude | 4 | 100% | 94% | +6 pts | 2 · 0 · 2 |
| Gemma | 3 | 88% | 77% | +11 pts | 2 · 0 · 1 | |
| gpt-oss | 3 | 85% | 71% | +14 pts | 2 · 0 · 1 | |
| Curriculum design (6 of 15) | Claude | 2 | 100% | 100% | 0 pts | 0 · 0 · 2 |
| Gemma | 2 | 96% | 79% | +17 pts | 2 · 0 · 0 | |
| gpt-oss | 2 | 92% | 79% | +13 pts | 2 · 0 · 0 | |
| Finance (6 of 15) | Claude | 2 | 100% | 76% | +24 pts | 2 · 0 · 0 |
| Gemma | 2 | 96% | 56% | +40 pts | 2 · 0 · 0 | |
| gpt-oss | 2 | 89% | 60% | +29 pts | 2 · 0 · 0 | |
| Marketing (6 of 15) | Claude | 2 | 100% | 86% | +14 pts | 2 · 0 · 0 |
| Gemma | 2 | 75% | 75% | 0 pts | 1 · 1 · 0 | |
| gpt-oss | 2 | 82% | 61% | +21 pts | 2 · 0 · 0 | |
| Product (6 of 15) | Claude | 2 | 93% | 57% | +36 pts | 2 · 0 · 0 |
| Gemma | 2 | 64% | 47% | +17 pts | 2 · 0 · 0 | |
| gpt-oss | 2 | 54% | 50% | +4 pts | 1 · 0 · 1 | |
| Design (4 of 15) | Claude | 2 | 93% | 89% | +4 pts | 1 · 1 · 0 |
| Gemma | 1 | 57% | 71% | -14 pts | 0 · 1 · 0 | |
| gpt-oss | 1 | 93% | 86% | +7 pts | 1 · 0 · 0 |
Small samples: a skill needs 15 runs per model before its result means much. A skill that loses keeps serving while a better version is written.
Pick a task
A containerized job with a written brief and fixed checks.
Run both sides
The Nous AI side gets skills; the baseline gets none.
Score the file
Code checks the deliverable. No model judges it.
Compare the pair
The result is the difference Nous AI made on that task.
Repeat and record
Runs repeat across models and attempts, all recorded.
Methodology
How the Harbor benchmark works.
The Harbor benchmark is Nous AI's own. It uses Harbor, an open-source framework that runs AI agents such as Claude Code, Codex, Gemini CLI and OpenHands on tasks inside containers, and scores each result with a verifier.
A container per task
Each task ships a written instruction, a starting environment and a verifier, so every run starts from the same files.
Two variants of each task
The Nous AI variant has Nous AI's skills installed where agents look for them, with an AGENTS.md. The baseline gets the same instruction and environment, without skills.
One skill pipeline per round
Each round measures the skills one version of Nous AI's skill pipeline writes. A skill may be rewritten within a round as its data changes; a new pipeline starts a new round.
Scored by code
The verifier checks the deliverable against the task's checks. The reward is the share of checks met. No model judges the output.
Checked before it runs
Each task has a reference solution that scores 1.0, and an empty deliverable scores 0, so the harness is proven before any model is tested.
Matched comparison
Each model runs both variants of the same tasks. The gain is the difference between them; fleet means average the matched runs.
Harbor needs Docker. Nous AI's own worker runs the same tasks with the models it already uses (Claude, Gemma and gpt-oss), giving each agent the task's files through file tools and scoring the result with the same checks. The standard (tasks, checks, scoring and harness) is fingerprinted and frozen per round; a change starts a new round.
Earlier run · October 2026
An earlier, smaller run on a different setup.
Harbor in Docker, fewer tasks, four to thirteen runs per model. Kept as history; the rounds above are the measure.
| Model (repository default) | Without Nous AI | With Nous AI | Gain | Runs |
|---|---|---|---|---|
| gpt-oss 20B | 63.2% | 76.7% | +13.5 | 13 |
| Claude Haiku 4.5 | 87.3% | 91.9% | +4.6 | 4 |
| Gemma 4 | 67.3% | 74.3% | +7.0 | matched |
Matched comparison, means across the matched runs: 81.9% with Nous AI, 77.8% without (+4.1 pts). A small sample on a different harness: it does not carry over to the current skills.
Next areas
Sales, customer success and operations.
Not benchmarked yet. Results for them will be published once their tasks run.
1 of 3
Sales
Prospecting, pipeline management and outreach workflows.
Example: "Build a prospecting plan for 50 target companies."
Not yet benchmarked.