A coding agent can produce a patch that looks reasonable, passes one visible test, and still leaves the repository in a worse state. It may edit the wrong boundary, skip a required check, hide a failure behind a mock, introduce a security defect, or report completion without running the command that would prove it.
Coding agent evaluation therefore needs more than a benchmark score or a reviewer reading the final answer. A useful program combines controlled repository tasks, fail-to-pass and regression checks, final-state inspection, trajectory evidence, and calibrated software-engineering judgment. The purpose is not to generate one impressive number. It is to support a specific product, release, procurement, or model-improvement decision.
Need engineering reviewers rather than generic annotators? Datrick supports coding-task design, reference solutions, model-output review, rubric calibration, adjudication, and managed technical evaluation delivery.
Start with the decision, not the leaderboard
Define what the evaluation must decide: whether to ship a release, compare two models, select an agent harness, qualify a contributor pool, improve a grader, or determine whether a task family is ready for scaled evaluation. Then define the repositories, languages, tools, permissions, time limits, expected behaviors, failure costs, and evidence that the decision owner will accept.
Public benchmarks can provide external context, but they are not a substitute for tasks that represent the intended product. SWE-bench uses real software issues and repositories, while OpenAI's recent coding-evaluation audit illustrates why task quality and contamination can affect the meaning of a benchmark result. A private evaluation set should be inspected, versioned, and protected with the same care as the grader.
| Evaluation layer | What it verifies | Evidence to retain |
|---|---|---|
| Task contract | The issue, constraints, allowed tools, prohibited changes, and success criteria are unambiguous enough to judge. | Versioned task, repository commit, environment definition, author decisions, and known limitations. |
| Required behavior | The patch creates the intended user-visible or system-visible outcome. | Fail-to-pass tests, state checks, command output, and reproducible steps. |
| Regression safety | Existing behavior and unrelated surfaces remain intact. | Pass-to-pass tests, type checks, lint, integration checks, and changed-file review. |
| Trajectory | The agent used tools and evidence responsibly rather than arriving at a plausible result by accident. | Tool calls, command output, retries, errors, file reads, and final completion claim. |
| Engineering quality | The implementation is maintainable, appropriately scoped, secure, and consistent with the codebase. | Rubric decision with citations to the patch, repository, tests, or trajectory. |
| Program quality | Review decisions stay consistent as tasks, reviewers, and models change. | Calibration results, disagreement classes, adjudication, rework, and rubric versions. |
Use executable graders for claims that software can prove
Executable checks should verify required behavior, regressions, repository state, file boundaries, schema changes, build output, security rules, and other deterministic claims. New tests should fail before the intended fix and pass afterward. Existing tests should remain green. A test that passes before any change does not demonstrate that the patch solved the issue.
Do not optimize the evaluation around tests alone. An agent can overfit visible tests, modify fixtures, weaken assertions, hard-code a sample, or introduce a second defect outside the checked path. Inspect the patch and the final repository state, and use hidden or protected checks where exposure would change the task.
Review the coding-agent trajectory
For an agent, the process is part of the product. Review whether it found the relevant context, respected file and tool boundaries, interpreted errors correctly, validated its own work, recovered from failures, and reported what actually happened. The final response should be checked against the final repository state and command evidence.
Anthropic's agent-evaluation guidance separates code-based, model-based, and human graders. That separation is useful for coding programs: deterministic checks establish what can be proven, model-based graders can scale explicit criteria, and human engineers handle ambiguity, tradeoffs, and consequential judgment.
Use human software engineers where correctness is contextual
Human review is necessary when several implementations may be valid, repository conventions matter, tests are incomplete, the correct scope is debatable, or security and operational consequences cannot be reduced to one assertion. The reviewer should not assign a vague quality score. Each decision should cite observable evidence and use explicit severity and escalation rules.
| Rubric dimension | Reviewer question | Typical critical failure |
|---|---|---|
| Correctness | Does the patch satisfy the task across normal, boundary, and failure cases? | The visible example works but a required branch remains wrong. |
| Scope control | Did the agent change only what was necessary and authorized? | Unrelated behavior, configuration, tests, or dependencies were modified. |
| Security | Does the change preserve trust boundaries, validation, secrets handling, and permissions? | The patch bypasses authorization or introduces unsafe input handling. |
| Maintainability | Does the implementation fit existing architecture and remain understandable? | A brittle workaround passes today but creates hidden ownership cost. |
| Evidence integrity | Does the completion claim match tests, logs, and repository state? | The agent claims checks passed when they were not run or failed. |
Calibrate reviewers before increasing volume
Give qualified reviewers the same bounded sample and require independent decisions. Classify disagreements into reviewer error, rubric ambiguity, missing context, multiple valid solutions, environment instability, and genuinely borderline cases. An authorized technical owner then adjudicates the disagreement and updates the task, examples, or rubric.
Track acceptance, rework, disagreement, escalation, error category, turnaround distribution, and rubric changes. Reviewer agreement alone is not proof of correctness; reviewers can consistently apply a weak standard. Pair agreement data with executable evidence, challenge tasks, and periodic expert review.
Protect private repositories and evaluation sets
Use least-privilege access, isolated environments, approved tooling, named repository owners, contributor confidentiality, logging, and explicit retention and removal rules. Separate public training examples from protected release tasks. Do not place credentials, customer data, production secrets, or unrestricted private repositories into an unmanaged evaluation queue.
What a managed coding-agent evaluation pilot should deliver
- A written decision, bounded task family, repository and environment inventory, and named quality owner.
- Representative tasks with versioned instructions, reference evidence, constraints, and protected checks.
- An executable grading plan covering required behavior, regressions, state, and policy boundaries.
- A technical human-review rubric with examples, severity definitions, and escalation rules.
- An independent calibration sample, disagreement analysis, adjudication record, and revised guidance.
- A controlled pilot with delivery evidence, error taxonomy, limitations, and an expand, revise, or stop recommendation.
Frequently asked questions
What should a coding agent evaluation measure?
A coding agent evaluation should measure required behavior, regressions, repository and environment state, constraint compliance, tool use, security, maintainability, and whether the agent's completion claim matches the evidence. The exact dimensions depend on the release decision and task family.
Why are unit tests not enough to evaluate coding agents?
Unit tests verify selected behavior but can miss unsafe shortcuts, unrelated regressions, inappropriate file changes, poor architecture, misleading completion claims, and failures in the agent's reasoning or tool-use process. Tests should be combined with state checks and bounded technical review.
When does coding agent evaluation need human software engineers?
Human software engineers are needed when correctness depends on repository context, several implementations may be valid, tests are incomplete, engineering tradeoffs matter, errors carry operational risk, or a rubric and automated grader have not yet been calibrated against trusted technical decisions.
Can a coding agent evaluation use private repositories?
Yes, if the evaluation uses explicit authorization, least-privilege access, approved environments, controlled data handling, contributor confidentiality, logging, and a defined retention and removal process. Sensitive repositories should not be copied into an unmanaged benchmark workflow.
How does a managed coding agent evaluation pilot work?
A managed pilot selects one bounded task family, prepares representative repository tasks, defines executable checks and a technical review rubric, calibrates reviewers on a small sample, runs controlled evaluations, adjudicates disagreement, and closes with findings and a recommendation to expand, revise, or stop.
Start with one repository task family. Datrick can review the task, environment, contributor profile, graders, rubric, calibration method, security controls, and pilot evidence before recurring delivery is proposed.