Building the Eval Harness for Your Claude Code Tool

01Throughline

02Secret Sauce

Seven things that, if you internalize them, put your harness ahead of 80% of teams shipping agentic tools right now.

03TL;DR

If you read only one section, read this.

• You don't need another framework. Adopt Inspect AI (UK AISI; safety-grade, sandbox-native, daily commits, the de facto agent-eval framework) or Promptfoo (YAML, weekly cadence, built-in red-team) as your runner. Don't roll your own; nothing useful will come of it.
• Steal the architecture from Anthropic's harness blog (2026-03): Planner → Generator → Adversarial Evaluator (different model family). The evaluator tries to break the generator. Single-agent self-eval is the failure mode; the field has converged on this.
• Seed prompts come from production, not your imagination. Hamel Husain + Shreya Shankar's pipeline: open-code 100 production traces → axial-code into failure modes → mine seeds from real failures. Synthetic seeds via Anthropic's "analyze the production distribution then sample" template fill in coverage gaps.
• Judge prompts are tag-structured CoT-then-verdict. Anthropic's <answer> / <rubric> / <thinking> / <correctness> template parses trivially and beats free-form scoring. Bake in the principal expert's actual critiques as few-shot anchors (Hamel's "Critique Shadowing").
• Your long-term metric is time-horizon, not pass-rate. METR-style "what's the longest human task our wrapper completes at 50% / 80% reliability." Pass-rate saturates; horizon-time doesn't.
• Statistical floor: ≥300 cases, paired bootstrap with 10k resamples, Cohen's κ for judges, Benjamini-Hochberg if scanning >10 rubrics. Sub-3pp deltas without the paired-bootstrap protocol are noise. T=0 isn't deterministic — run N≥3.

04The Eval Harness Landscape

There are roughly two dozen credible eval harnesses in 2026. Most teams don't need all of them — they need to pick one of three primary patterns and steal liberally from the rest.

The split: model-grading harnesses vs application-engineering platforms

The eval landscape has bifurcated. Academic / model-grading harnesses — lm-evaluation-harness (EleutherAI), HELM (Stanford CRFM, v0.5.15 released 2026-04-22), Lighteval (HuggingFace), Inspect AI (UK AISI), AgentBench (Tsinghua) — score models against fixed task suites. Application-engineering platforms — Braintrust, Langfuse, Arize Phoenix, LangSmith, MLflow GenAI, W&B Weave, Maxim, Galileo, Comet Opik, HoneyHive — wrap traces + LLM-as-judge + dataset versioning around production agents. Both camps are converging on OpenTelemetry as the trace substrate; lock-in is dissolving at the trace layer, and differentiation is moving to scoring + workflow + dataset management.

Pick exactly one of these as your runner

If you must pick a hosted observability platform on top: Braintrust (highest mindshare among AI-native startups; Notion, Stripe, Vercel, Airtable, Zapier, Coda are public users) or Langfuse (strongest OSS + self-host story). Avoid rolling your own dataset versioning + CI gate layer — that is what platforms now do well.

What's coming in 2026

• Agent-as-judge — Patronus Percival (memory + human-guided rubrics for multi-step workflows), Microsoft AgentEval (Critic + Quantifier auto-rubric generation), DeepEval's task-completion metrics. Will be the dominant scoring pattern by end of 2026.
• Multi-turn / session evals are now table stakes. LangSmith (Oct 2025), MLflow 3.8 (session-level scorers), Maxim, Galileo, Phoenix all shipped session/thread-level eval primitives in late-2025/early-2026. The single-prompt eval is fading.
• Enterprise APM consolidation. Cisco → Galileo (Apr 2026), Snowflake → TruEra (2024) → MLflow integration (Feb 2026), CoreWeave → W&B (2025). Expect Datadog / New Relic / Splunk to absorb more of this category in the next 12 months.

05Tool-Use Benchmarks Worth Knowing

What the field measures when it says "this model can use tools" — and where every published benchmark falls short for production agentic eval.

The 2026 trinity

What every published benchmark fails to capture

• No public benchmark for wrapper quality. Nobody measures CLAUDE.md design, sub-agent topology, hook architecture. Closest is langchain-ai/claude-code-evals (59 stars) — the gap is enormous and is where your harness can actually be novel.
• No public benchmark for cost-quality frontier. Aider's leaderboard reports $/correct-answer (GPT-5 high 88%/$29; o3-pro 84.9%/$146; DeepSeek-V3.2 Reasoner 74.2%/$1.30). Most others ignore cost entirely. Yours shouldn't.
• Most benchmarks are single-domain. Real agentic tools cross domains; cross-domain trajectory scoring is unsolved.

06LLM-as-Judge: The Failure Modes You Have to Design Around

Your measurement system is only as honest as your judge. Position bias, length bias, self-preference, and verbosity bias will quietly poison results unless you architect against them.

The catalog of judge failure modes (with mitigations)

The architecture that beats single-judge

Panel-of-LLMs (PoLL) + a small (≥100) human-graded calibration set tracked by Cohen's κ. This is what crystallizes across Cohere's "Replacing Judges with Juries," Hamel Husain's eval guides, and Eugene Yan's writeups. The calibration set is the load-bearing piece — without an expert-graded ground-truth, no judge architecture is trustworthy. Track κ over time; if it drifts below 0.6, the judge is broken and you've stopped knowing what your scores mean.

07Seed Prompt Patterns

Concrete patterns for the test cases your harness runs through your tool. Concrete examples and templates pulled verbatim from sources where available.

The pipeline (Hamel Husain + Shreya Shankar)

1. Open-code 100 production traces. Read them. Tag failure modes inline as you read. Don't predefine categories. (This is the work nobody wants to do; it's also what separates teams with real evals from teams running G-Eval over toy datasets.)
2. Axial-code into ~10 stable failure-mode buckets. Cluster the inline tags. The buckets become your eval dimensions.
3. Mine seeds from real failures — production traces are your highest-signal seeds. Anonymize, version, lock.
4. Synthesize seeds for coverage gaps. Use Anthropic's cookbook generator pattern: tell the LLM to first analyze the production distribution of each variable, then sample from it. Don't just say "give me 100 examples" — that produces same-shaped slop.
5. Stratify difficulty. Easy / medium / hard. Top frontier models should hit ~95% on easy, ~70% on medium, ~30% on hard. If your suite is monomodal, it's not discriminating.

Patterns to reach for

• Features × Scenarios × Personas dimension matrix (Hamel) — anchors all later synthesis on real product structure rather than generic prompts.
• Tool-call ground truth dual annotation (Google ADK + τ-bench): annotate both the expected tool sequence (for diagnostics) AND the expected end-state (for outcome scoring). Lets multiple valid trajectories pass.
• Counterfactual seeds — same task with one twist that changes the correct tool selection. Tests whether your tool actually chose right or just memorized.
• Mutation testing for prompts — small perturbations (typos, reordering, synonyms) on each seed; flag any seed whose pass-rate swings >3pp under mutation.
• Adversarial seeds — Promptfoo's red-team module is worth stealing wholesale. Prompt injection, jailbreak attempts, capability misuse.
• Negative seeds — tasks the agent should not attempt or should refuse. Easy to forget; critical for production.
• Curriculum stratification — easy / medium / hard explicit; reported separately.

Seed JSONL schema (steal this)

{
  "id": "seed-0001",
  "released_at": "2026-04-25",
  "category": "tool-selection",
  "difficulty": "medium",
  "task": "...",
  "expected_tools": ["Read", "Grep"],
  "expected_end_state": {"file_modified": "src/foo.py", "tests_pass": true},
  "rubric_tags": ["tool-correctness", "task-completion"],
  "source": "production-trace-2026-03-14"
}

08Judge Rubric Templates

Concrete rubric prompt structures, with verbatim templates pulled from Anthropic, OpenAI, Promptfoo, Prometheus, and applied teams.

Anthropic cookbook grader (binary, parses trivially)

The cleanest published binary judge template. Tag-structured CoT-then-verdict; downstream code regex-extracts <correctness> and is done.

You are a grader. Compare the candidate answer to the rubric.

<rubric>
{rubric}
</rubric>

<answer>
{candidate_answer}
</answer>

Think step-by-step in <thinking> tags about whether the candidate satisfies
the rubric. Then output your verdict in <correctness> tags as exactly one
of: correct | incorrect.

<thinking>...</thinking>
<correctness>...</correctness>

Hamel critique-aligned judge (highest expert agreement)

Bake the principal domain expert's actual critiques in as few-shot. Hamel reports ~90% expert agreement in 3 iterations. The critique field carries the actionable signal — that's what feeds back into seed creation and prompt fixes.

You are evaluating an agent's response. Use the same standard the principal
domain expert uses. Examples of expert critiques are below.

<expert-critiques>
Example 1 — response: "..."
Expert critique: "Used Read tool when Glob would have been correct.
Wasted 2 turns on file location before opening."
Verdict: incorrect

Example 2 — response: "..."
Expert critique: "Correct tool, but skipped the dependency check."
Verdict: incorrect

Example 3 — response: "..."
Expert critique: "Clean trajectory, correct end state."
Verdict: correct
</expert-critiques>

Now evaluate:
<response>{candidate}</response>

Output:
<critique>...</critique>
<verdict>correct | incorrect</verdict>

Prometheus absolute grading (when binary loses information)

5-point Likert only with verbatim per-level rubric anchors. The reference answer is the score-5 anchor. Compresses cleanly to a Pearson 0.897 vs experts when fine-tuned (Prometheus 2).

### Task Description:
Evaluate the response against the rubric. Use the score-5 reference as anchor.

### Score Rubric:
{rubric_dimension}

Score 1: {anchor_1_verbatim}
Score 2: {anchor_2_verbatim}
Score 3: {anchor_3_verbatim}
Score 4: {anchor_4_verbatim}
Score 5: {anchor_5_verbatim_=_reference_answer}

### Reference Answer (score 5):
{reference}

### Response to Evaluate:
{candidate}

### Output:
Feedback: ...
[RESULT] {1 | 2 | 3 | 4 | 5}

Trajectory eval (six-layer composite for agentic tools)

1. End-state diff (highest weight) — did the world end up correct? File written, test passed, DB row inserted.
2. Exact tool sequence match (diagnostic) — did it call the expected tools in expected order? Useful for debugging, not for scoring alone.
3. Step-level diagnostics — per-tool-call error rate, latency, retry count.
4. LLM trajectory judge — rubric grader over the full trajectory (Anthropic four-axis: design quality, originality, craft, functionality).
5. Pass^k consistency (τ-bench style) — run the same seed N times, score all-N-pass.
6. Promptfoo trajectory:goal-success — built-in trajectory assertion if you're using Promptfoo.

09Building a Suite That Survives 18 Months

Contamination, drift, saturation, model upgrades. The harness you ship today will be lying to you in 6 months unless you build for longevity from day one.

Four properties of a suite that survives

The contamination war story you need to know

Time-horizon: the metric that survives saturation

Pass-rate percentages saturate as models improve. METR's 50%-time horizon — the length of human-expert task an agent reliably completes 50% of the time — keeps growing. As of Time Horizon 1.1 (Jan 2026): doubling rate ~131 days, ~10×/year at the recent margin. Latest published numbers:

Build your benchmark to express results in horizon-time, not pass-rate, and you'll still have a meaningful metric two model generations from now.

10What Production Teams Actually Do

Stripping out the marketing. What Anthropic, OpenAI, Cursor, Aider, Devin, and Sierra publish about their real eval practice — and where the labs and applied teams diverge.

What the labs publish (the methodology stack)

• Anthropic — three-agent harness blog (March 2026). Planner → Generator → Evaluator. Evaluator uses Playwright to grade artifacts on four axes: design quality, originality, craft, functionality. Solo agent: 20 min, $9, broken. Full harness: 6h, $200, functional. Steal those four axes verbatim.
• Anthropic — multi-agent research system (June 2025). "Token usage alone explains 80% of performance variance in browsing evaluations." Eval methodology: ~20 queries small-sample, LLM-as-judge with rubric (factual accuracy / citation / completeness / source / efficiency), human validation for SEO-bias edge cases.
• Anthropic — Opus 4.5 published recipe. 64K thinking budget, 200K context, default high effort, averaged across 5 trials, default temp/top-p. Closest thing to a canonical Anthropic eval method.
• METR — Time Horizon methodology. Human baselines from ~5-year experienced pros; logistic curve fit; 6 independent runs per task; 95% CIs published; explicit canary strings per task; partially private suite.
• Sierra — τ²-Bench. Full architecture, judge prompts, ablations, contamination disclosure all published. The most transparent applied team in the field.

What applied teams publish (the outcomes, not the methodology)

This is where the biggest gap lives. Cursor, Cognition (Devin), Sourcegraph Amp, Replit, Vercel publish almost no methodology specifics. They describe outcomes — PR merge rates, "trace-first" — but withhold judge prompts, suite composition, validation procedures. The asymmetry suggests applied eval is now competitive moat; the methodology is the product.

The exceptions worth knowing:

• Cognition's adversarial human reward-hardening. They pay senior engineers to actively try to circumvent their own graders before training. Treats eval design itself as a red-team exercise. Combined with their 3-parallel-judge stack (classical tests + rubrics + browser-use agent for E2E), this is the only public methodology that explicitly assumes the judge will be gamed.
• Cline-Bench (open-sourced, commit-hash-as-ground-truth). Rare published applied-team methodology.
• Continue.dev framework — published eval framework with structure documented.
• Aider — Paul Gauthier's Polyglot leaderboard. 225 Exercism exercises across 6 languages. Two-attempt retry. Edit-format accuracy reported separately from solution correctness (64.4–100% across models). Cost reported per correct answer. Determinism: Paul explicitly notes API non-determinism at temp=0 (5–10 variations per identical request). Most rigorous applied-team eval published.
• Shopify Sidekick — Ground Truth Sets, LLM-powered merchant simulator, judges calibrated to ~0.75 correlation with humans (Cohen's Kappa, Kendall Tau, Pearson tracked), gated rewards (procedural + semantic) to prevent reward hacking.

The METR finding everyone should internalize

11Claude Code-Specific Eval Surfaces

Anthropic gave you eval injection points most teams haven't noticed. Hooks, sub-agents, MCP, and the Agent SDK each open a different door for measurement.

The four eval-injection surfaces unique to Claude Code

claude --bare \
  --mcp-config eval.mcp.json \
  --strict-mcp-config \
  --output-format stream-json \
  --include-hook-events \
  --json-schema "$SCHEMA" \
  --max-budget-usd 2.00 \
  --max-turns 20 \
  --session-id "$UUID" \
  --exclude-dynamic-system-prompt-sections \
  -p "$PROMPT"

Community references worth stealing

• langchain-ai/claude-code-evals — the cleanest published Claude-Code-specific ablation harness. Per-task input_prompt.py + llm_as_a_judge.py + per-task CLAUDE.md. Pattern: same prompt, swap CLAUDE.md content, score with LLM judge. Copy this directory shape.
• sjnims/cc-plugin-eval — 4-stage Analysis → Generation → Execution → Evaluation. Programmatic detection via Agent SDK hooks (Skill / Task / SlashCommand call capture with 100% confidence; MCP via mcp__<server>__<tool> pattern). LLM judgment as secondary 0–10 score.
• jimmc414 SWEbench harness — two-tier scoring: Generation Score (patch produced) vs Evaluation Score (Docker test execution actually passes). "100% generation, 20% eval = 20% real."
• musistudio/claude-code-router (32.9k stars) — routes Claude Code traffic to DeepSeek/Gemini/Ollama/OpenRouter. Free model-ablation harness — same Claude Code surface, different backend. If you want to A/B model swaps under your wrapper, this is the lever.
• VoltAgent/awesome-claude-code-subagents (18.3k stars, 130+ subagents) — every entry uses the same frontmatter pattern. Reviewers get read-only tools; developers get write tools. Tool boundary IS the eval boundary.

Big 2026 caveat to cite anywhere you reference SWE-bench: Verified is contaminated. OpenAI retired it. Claude Opus 4.5 = 80.9% Verified, 45.9% Pro. The 35-point gap is benchmark design, not capability. Always cite Pro alongside Verified.

12The Statistical Floor

Concrete numbers — sample size, variance handling, confidence intervals — so your eval signal isn't actually noise.

Most published LLM "improvements" are within noise. Frontier-lab leaderboard deltas of 1–3pp routinely fail honest significance tests once you account for paired correlation, cluster SEs over related questions, acknowledge nondeterminism at T=0, and correct for cherry-picking from 200 evals. Build the discipline in from week one.

The non-negotiable floor

1. Sample size: ≥300 eval items per task, target 500–1000. Below ~300, use Wilson or Clopper-Pearson intervals — not bare ± std. The CLT does not save you (Bowyer/Brand/Aitchison, arXiv:2503.01747).
2. N ≥ 5 runs at T=0.7, or N ≥ 3 at T=0 — because T=0 is not deterministic. Thinking Machines Lab (Sep 2025) found 1,000 identical T=0 calls produced 80 unique outputs on a frontier model. Anthropic's published Opus 4.5 recipe averages 5 trials.
3. Always paired, always bootstrap. Same prompt → both models. 10,000 bootstrap resamples on Δ. If 0 ∈ 95% CI, it's noise. This single discipline kills 80% of false claims.
4. Treat |Δ| < 1pp as noise. Treat |Δ| < 3pp as below the harness/infra noise floor (Anthropic infrastructure-noise paper) unless you've run the paired-bootstrap protocol (arXiv:2511.19794).
5. McNemar for binary pass/fail with ≥10 discordant pairs. Wilcoxon signed-rank for paired non-normal scores.
6. LLM-judge: ≥100 human-labeled pairs to validate. Refuse to deploy below Cohen's κ = 0.6. Report κ and Pearson — never Pearson alone (it's invariant to additive bias; a systematically harsh judge looks great).
7. Scanning > 10 rubrics? Apply Benjamini-Hochberg at Q=0.10. Holm-Bonferroni for safety/regulatory dimensions where false positives are expensive.
8. Run ≥ 3 diverse judge models (Cohere PoLL pattern) for subjective evals. Disjoint vendor families.
9. Pre-commit primary metric in eval-config.yaml before running new model. Pawel et al. arXiv:2311.18807 — the difference between research and p-hacking.
10. Production A/B: sequential testing (mSPRT) or Bayesian — never fixed-N frequentist. You will peek; design for that.

The biggest mistake everyone makes

Using unpaired tests when both models ran on the same prompts. You throw away the within-prompt correlation that makes the test sensitive. Anthropic's "Adding Error Bars to Evals" (arXiv:2411.00640) shows this routinely flips significance findings; clustered SEs (for evals with shared passages) can legitimately be > 3× larger than naive SEs, retroactively erasing many leaderboard claims.

13Recommended Blueprint for Your Harness

Synthesizing everything above into a single concrete recommendation. This is the answer to "what do I actually build."

The stack, top to bottom

The starter scaffold (the first PR you write)

your-eval-harness/
├── inspect/
│   ├── tasks/
│   │   ├── tool-selection/
│   │   │   ├── seeds.jsonl              # production-mined + synthetic
│   │   │   ├── solver.py                # Inspect AI Solver (calls your wrapper)
│   │   │   └── scorer.py                # end-state + LLM-judge composite
│   │   └── trajectory-quality/
│   │       └── ...
│   ├── judges/
│   │   ├── poll.py                      # 3-model panel, majority vote
│   │   ├── prompts/
│   │   │   ├── binary.md                # Anthropic cookbook tag-structured
│   │   │   └── critique.md              # Hamel critique-shadowing
│   │   └── calibration/
│   │       ├── golden_pairs.jsonl       # 100+ expert-labeled
│   │       └── kappa_history.csv        # tracked over time
│   └── stats/
│       └── paired_bootstrap.py          # arXiv 2511.19794 protocol
├── hooks/
│   ├── settings.json                    # PreToolUse + PostToolUse + Stop HTTP hooks
│   └── handler.py                       # local FastAPI receiving hook events
├── traces/
│   ├── prod-samples/                    # rotating sample of production traces
│   └── ingestion.py                     # nightly +50 cases into regression suite
├── ci/
│   ├── pr-gate.yaml                     # block on regression of fixed-bug suite
│   └── nightly.yaml                     # full run + paired-bootstrap report
├── eval-config.yaml                     # PRE-COMMITTED metrics, frozen pre-run
└── README.md

14Action Items

Ordered by urgency.

This week

• Set up the hook bus. Project-level .claude/settings.json with PreToolUse + PostToolUse + Stop + SubagentStop HTTP hooks pointing at a local FastAPI service. Capture every event to a SQLite trace store. This is the foundation everything else sits on; do it first.
• Pick your runner. Inspect AI. Don't deliberate. Install, walk through the agent-loop tutorial, write one trivial task to confirm sandboxing works on your machine.
• Read three things end-to-end: Anthropic's Effective Harnesses for Long-Running Agents, Anthropic's Eval Awareness BrowseComp, and Hamel Husain's LLM Evals FAQ. Together: ~3 hours. Will save you a month.
• Write your first 30 seeds by hand — from your own production traces if you have them, otherwise from real tasks you'd actually use the tool for. Hand-author beats synthetic at this size. You'll learn what your eval is actually measuring.
• Pre-commit your primary metric in eval-config.yaml before you run any A/B. Decide now what counts as success; write it down; don't let yourself edit it after seeing results.

This month

• Build the PoLL judge. Three models from disjoint vendor families (e.g. Sonnet + Gemini Pro + GPT-5 or any open-weights you trust). Majority vote. Test against your hand-written seeds; pick the rubric template (cookbook binary or Prometheus 5-pt) that matches your eval shape.
• Build the calibration set. 100 expert-labeled pairs. You're the principal expert. Track Cohen's κ vs your judge weekly. If κ ever falls below 0.6, judge is broken — stop deploying.
• Wire the paired-bootstrap protocol (arXiv:2511.19794). 10,000 resamples on Δ. Refuse to ship any "improvement" claim that doesn't pass it.
• Add the PR-blocking CI gate. Regression of any fixed-bug seed = build fails. This single discipline catches more real bugs than any other layer.
• Adopt the JSONL seed schema with released_at field. Even if you don't enforce date-cutoff yet, you'll have the data when you do.
• Pick one auto-refreshing benchmark as your contamination canary. SWE-bench Live (monthly) or BFCL v4 (weekly). Run it weekly; alert if your wrapper's score drifts more than 5pp without a known cause.

Long-term (6–18 months)

• Switch headline metric from pass-rate to time-horizon. METR-style. Pass-rates will saturate; horizon-time won't. Future-proofs your benchmark against the next two model generations.
• Build the production-drift loop. Sample +50 production traces/month into the regression suite. Failures the suite missed → fixed regression tests. This is what separates a benchmark from a living suite.
• Public/held-out split with quarterly rotation. Once you have ~500+ seeds, freeze 50% private. Rotate the public half quarterly. Publish only aggregate metrics on private.
• Treat your harness as a versioned artifact. Every harness change = re-run prior baselines, publish deltas. Don't pretend the harness is invisible — Anthropic's August 2025 incident proved it isn't.
• Add agent-as-judge experimentally in 2H 2026 once Patronus Percival / Microsoft AgentEval patterns mature. Don't lead with it; the field hasn't crystallized yet.

15Gaps in the Field

What no one is doing well yet — i.e. where the opportunity is to build something better than what's published.

• Wrapper-quality benchmarking is wide open. Nobody publishes evals over CLAUDE.md design, sub-agent topology, or hook architecture. langchain-ai/claude-code-evals is closest (59 stars). If your harness produces a published "wrapper-quality" leaderboard with a coherent rubric, you'd be first. The only reason this gap exists is that applied teams treat their wrapper as moat — but there's room for an open community benchmark.
• Cost-quality frontier reporting is rare. Aider does it well; almost no one else. Cursor / Cognition / Replit / Sourcegraph publish quality numbers without dollars. A harness that reports $/correct on every run, by default, would surface the truth most leaderboards hide: the "best" model by quality is rarely the "best" by cost-quality frontier.
• Cross-domain trajectory scoring is unsolved. Real agentic tools cross domains within a single task (read code → query DB → call API → write file → run test). Every published trajectory benchmark is single-domain (τ-bench is customer-service; SWE-bench is code; OSWorld is computer-use). Cross-domain scoring with a unified trajectory grader is research-frontier.
• Eval-awareness defenses are underdeveloped. Anthropic's BrowseComp paper is the only published case of an agent autonomously decrypting an eval canary. The defenses are hand-rolled; there's no toolkit, no skill, no shared playbook. Whoever ships "eval-canary best practices for agentic systems" leads.
• Long-horizon eval is gated by cost. METR's Time Horizon 1.1 hits saturation on tasks where only 5/31 long tasks have human baselines — because each costs hours of expert time to baseline. Cheap or self-supervised long-horizon ground-truthing (synthetic tasks with verifiable end-states) would unblock the field.

16Source Bibliography

All sources consulted across the eight research lanes. Tier 1 = primary (paper, repo, official docs), Tier 2 = credible secondary (validated trade press, recognized practitioners), Tier 3 = community / unverified.

1. Anthropic — Effective Harnesses for Long-Running Agents — anthropic.com/engineering/effective-harnesses-for-long-running-agents T1
2. Anthropic — Harness Design for Long-Running App Dev — anthropic.com/engineering/harness-design-long-running-apps T1
3. Anthropic — Building Effective Agents — anthropic.com/engineering/building-effective-agents T1
4. Anthropic — Multi-Agent Research System — anthropic.com/engineering/multi-agent-research-system T1
5. Anthropic — Eval Awareness in BrowseComp — anthropic.com/engineering/eval-awareness-browsecomp T1
6. Anthropic — Quantifying Infrastructure Noise — anthropic.com/engineering/infrastructure-noise T1
7. Anthropic — SWE-bench Sonnet methodology — anthropic.com/research/swe-bench-sonnet T1
8. Anthropic — Statistical approach to model evals — anthropic.com/research/statistical-approach-to-model-evals T1
9. Anthropic — The "Think" Tool — anthropic.com/engineering/claude-think-tool T1
10. Anthropic — Claude Opus 4.5 release — anthropic.com/news/claude-opus-4-5 T1
11. Claude Code — Hooks reference — code.claude.com/docs/en/hooks T1
12. Claude Code — Sub-agents — code.claude.com/docs/en/sub-agents T1
13. Claude Code — CLI Reference — code.claude.com/docs/en/cli-reference T1
14. Claude Code — Best Practices — code.claude.com/docs/en/best-practices T1
15. Claude Code — Headless / programmatic — code.claude.com/docs/en/headless T1
16. Claude Code — MCP docs — code.claude.com/docs/en/mcp T1
17. Claude Code — Permission modes — code.claude.com/docs/en/permission-modes T1
18. Claude Code — CHANGELOG — github.com/anthropics/claude-code/blob/main/CHANGELOG.md T1
19. Claude Agent SDK — code.claude.com/docs/en/agent-sdk/overview T1
20. Anthropic — Tool Use overview — platform.claude.com/docs/en/build-with-claude/tool-use/overview T1
21. Anthropic — Prompt Caching — platform.claude.com/docs/en/build-with-claude/prompt-caching T1
22. Inspect AI (UK AISI) — github.com/UKGovernmentBEIS/inspect_ai T1
23. AISI 2025 year-in-review (ControlArena) — aisi.gov.uk/blog/our-2025-year-in-review T1
24. OpenAI Evals repo — github.com/openai/evals T1
25. OpenAI Evals dashboard — evals.openai.com T1
26. OpenAI — Testing Agent Skills with Evals — developers.openai.com/blog/eval-skills T1
27. OpenAI — Why we no longer evaluate SWE-bench Verified — openai.com/index/why-we-no-longer-evaluate-swe-bench-verified T1
28. Promptfoo repo — github.com/promptfoo/promptfoo T1
29. Promptfoo — Evaluate Coding Agents — promptfoo.dev/docs/guides/evaluate-coding-agents T1
30. HELM (Stanford CRFM) — github.com/stanford-crfm/helm T1
31. HELM leaderboard — crfm.stanford.edu/helm T1
32. lm-evaluation-harness (EleutherAI) — github.com/EleutherAI/lm-evaluation-harness T1
33. Lighteval (HuggingFace) — github.com/huggingface/lighteval T1
34. AgentBench (THUDM) — github.com/THUDM/AgentBench T1
35. AgentBench paper — arXiv:2308.03688 T1
36. DeepEval — github.com/confident-ai/deepeval T1
37. DeepEval AI Agent Evaluation — deepeval.com/guides/guides-ai-agent-evaluation T1
38. Ragas paper — arXiv:2309.15217 T1
39. LangSmith — langchain.com/langsmith-platform T1
40. LangSmith — Insights Agent + Multi-turn Evals — blog.langchain.com/insights-agent-multiturn-evals-langsmith T1
41. Braintrust — best LLM eval platforms 2025 — braintrust.dev/articles/best-llm-evaluation-platforms-2025 T2
42. W&B Weave — wandb.ai/site/weave T1
43. Arize Phoenix — github.com/Arize-ai/phoenix T1
44. Langfuse — github.com/langfuse/langfuse T1
45. MLflow GenAI evaluate — docs.databricks.com/aws/en/mlflow3/genai/eval-monitor T1
46. TruLens — github.com/truera/trulens T1
47. Comet Opik — github.com/comet-ml/opik T1
48. Microsoft AutoGen / Agent Framework eval — learn.microsoft.com/agent-framework/agents/evaluation T1
49. Cisco intent to acquire Galileo — blogs.cisco.com/news/cisco-announces-the-intent-to-acquire-galileo T1
50. Patronus AI — Generative Simulators (Percival) — patronus.ai/blog/introducing-generative-simulators T1
51. BFCL leaderboard — gorilla.cs.berkeley.edu/leaderboard.html T1
52. Gorilla — github.com/ShishirPatil/gorilla T1
53. τ-bench paper — arXiv:2406.12045 T1
54. taubench.com (τ²-Bench leaderboard) — taubench.com T1
55. ToolBench / ToolLLM — arXiv:2307.16789 T1
56. StableToolBench — arXiv:2403.07714 T1
57. API-Bank — arXiv:2304.08244 T1
58. Apple ToolSandbox — github.com/apple/ToolSandbox T1
59. NESTFUL (nested function calls) — arXiv:2409.03797 T1
60. ToolHop — arXiv:2501.02506 T1
61. MINT — arXiv:2309.10691 T1
62. ToolEmu (DeepMind) — arXiv:2309.15817 T1
63. AgentHarm — arXiv:2410.09024 T1
64. MCP-Bench — arXiv:2508.20453 T1
65. WebArena — webarena.dev T1
66. OSWorld — arXiv:2404.07972 T1
67. AppWorld — arXiv:2407.18901 T1
68. SWE-bench leaderboards — swebench.com T1
69. SWE-bench Pro (Scale AI) — morphllm.com/swe-bench-pro T1
70. SWE-bench Live — arXiv:2505.23419 T1
71. Terminal-Bench 2.0 — tbench.ai T1
72. LiveCodeBench — arXiv:2403.07974 T1
73. GAIA (Meta) — arXiv:2311.12983 T1
74. Zheng et al. — Judging LLM-as-a-Judge with MT-Bench — foundational LLM-judge paper T1
75. G-Eval (Liu et al.) — NLG evaluation with GPT-4 better human alignment T1
76. Prometheus 2 (KAIST) — fine-tuned judge model with 0.897 expert correlation T1
77. JudgeBench — meta-benchmark for evaluating LLM judges T1
78. Verga et al. (Cohere) — Replacing Judges with Juries — arXiv:2404.18796 T1
79. Shankar et al. — Who Validates the Validators? — arXiv:2404.12272 T1
80. Judge's Verdict (Pearson invariance) — arXiv:2510.09738 T1
81. Self-preference / low-perplexity bias — arXiv:2410.21819 T1
82. Adversarial judge fragility (73.8% injection) — arXiv:2504.18333 T1
83. Score compression in 1–10 Likert — arXiv:2506.22316 T1
84. Hamel Husain — LLM Evals FAQ (Jan 2026) — hamel.dev/blog/posts/evals-faq T2
85. Hamel Husain — Field Guide to Rapidly Improving AI Products — hamel.dev/blog/posts/field-guide T2
86. Hamel Husain — LLM-as-a-Judge / Critique Shadowing — hamel.dev/blog/posts/llm-judge T2
87. Hamel Husain — Your AI Product Needs Evals — hamel.dev/blog/posts/evals T2
88. Eugene Yan — Task-Specific LLM Evals — eugeneyan.com/writing/evals T2
89. Eugene Yan — LLM-Evaluators (Judge) — eugeneyan.com/writing/llm-evaluators T2
90. Cameron Wolfe — Applying Statistics to LLM Evaluations — cameronrwolfe.substack.com/p/stats-llm-evals T2
91. Pragmatic Engineer — Pragmatic Guide to LLM Evals — newsletter.pragmaticengineer.com/p/evals T2
92. Shopify Engineering — Sidekick Production-Ready Agents — shopify.engineering/building-production-ready-agentic-systems T1
93. Statsig — Significance Misleads in LLM A/B — statsig.com/perspectives/abtesting-llms-misleading T2
94. Aider — leaderboards — aider.chat/docs/leaderboards T1
95. Aider — benchmark methodology — aider.chat/docs/benchmarks.html T1
96. METR — Measuring AI Ability to Complete Long Tasks (Mar 2025) — metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks T1
97. METR — Time Horizon 1.1 (Jan 2026) — metr.org/blog/2026-1-29-time-horizon-1-1 T1
98. METR — GPT-5 Evaluation Report (Aug 2025) — evaluations.metr.org/gpt-5-report T1
99. METR — Domain Variation (Jul 2025) — metr.org/blog/2025-07-14-how-does-time-horizon-vary-across-domains T1
100. Miller et al. — Adding Error Bars to Evals — arXiv:2411.00640 T1
101. Bowyer/Brand/Aitchison — Don't Use the CLT in LLM Evals — arXiv:2503.01747 T1
102. Heineman et al. (AI2) — Signal and Noise — arXiv:2508.13144 T1
103. Bowman & Dahl — What Will it Take to Fix Benchmarking — NAACL 2021 T1
104. Demšar — Statistical Comparisons of Classifiers — JMLR 2006 T1
105. Wang et al. — Self-Consistency Improves CoT — arXiv:2203.11171 T1
106. He/Atkinson — Non-Determinism of "Deterministic" LLM Settings — arXiv:2408.04667 T1
107. Thinking Machines Lab — Defeating Nondeterminism in LLM Inference — thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference T2
108. Pawel et al. — Pre-registration for Predictive Modeling — arXiv:2311.18807 T1
109. Johari/Koomen/Pekelis/Walsh — Always Valid Inference (mSPRT) — arXiv:1512.04922 T1
110. Russo/Van Roy — Tutorial on Thompson Sampling — stanford.edu/~bvr/pubs/TS_Tutorial.pdf T1
111. Chiang et al. — Chatbot Arena — arXiv:2403.04132 (ICML 2024) T1
112. Argyle et al. — Statistical Framework for Ranking LLM Chatbots — arXiv:2412.18407 (ICLR 2025) T1
113. Cohere — The Leaderboard Illusion — arXiv:2504.20879 T1
114. Stanford — How Is ChatGPT's Behavior Changing — arXiv:2307.09009 T1
115. Paired Bootstrap Protocol — arXiv:2511.19794 T1
116. Tracking the Moving Target — arXiv:2504.18985 T1
117. MathArena — matharena.ai + arXiv:2505.23281 T1
118. K-Prize / Konwinski Prize (TechCrunch coverage) — techcrunch.com/2025/07/23/a-new-ai-coding-challenge T2
119. ArxivRoll — How Much Do LLMs Cheat — arXiv:2507.19219 T1
120. Skywork-13B (test-set leakage detection) — arXiv:2310.19341 T1
121. lmgame-Bench — arXiv:2505.15146 T1
122. DynaBench (Kiela et al., 2021) — methodological ancestor of dynamic benchmarks T1
123. langchain-ai/claude-code-evals — github.com/langchain-ai/claude-code-evals T1
124. sjnims/cc-plugin-eval — github.com/sjnims/cc-plugin-eval T1
125. jimmc414 SWEbench harness — github.com/jimmc414/claudecode_gemini_and_codex_swebench T1
126. musistudio/claude-code-router — github.com/musistudio/claude-code-router T1
127. VoltAgent/awesome-claude-code-subagents — github.com/VoltAgent/awesome-claude-code-subagents T1
128. anthropics/claude-code-action — github.com/anthropics/claude-code-action T1
129. Wasserman — All of Statistics — stat.cmu.edu (textbook) T1
130. Casella & Berger — Statistical Inference 2e — textbook reference T1

Bibliography is deduplicated to ~120 entries from 327 raw sources surfaced across 8 research lanes. Full per-lane source indexes (with all 327) are available on request — they live in the working vault at ~/vault/raw/intel/2026-04-25-ai-tooling-eval-harness/lane-*/findings.md. Confidence tags (KNOWN / ASSUMED / UNVERIFIED) and tier ratings (T1 = primary; T2 = credible secondary; T3 = community / unverified) follow the EP strategic-intelligence-loop discipline.