An LLM judge can review more code outputs than a senior engineering team can inspect manually. It can apply a structured rubric, identify likely scope violations, summarize trajectory evidence, and route suspicious cases for review. That scale is useful only when the organization knows how the judge behaves relative to trusted engineering decisions.
Calibration is not asking a few engineers whether the judge's explanations look reasonable. It is a controlled comparison between a versioned judge system and independently established reference decisions across representative and difficult cases. The result should define where the judge can operate, where it must abstain, and where executable checks or human engineers remain authoritative.
Need to scale code evaluation without replacing expert judgment with an untested model? Datrick supports rubric design, expert labeling, calibration sets, judge prompts, error analysis, adjudication, monitoring, and managed technical review.
Use the judge only where deterministic evidence is insufficient
Start by removing criteria that can be tested directly. Required behavior, regressions, repository state, builds, type checks, lint, security rules, schema changes, and command outcomes belong in executable graders. An LLM judge should not estimate whether tests passed when the harness can run them.
Reserve model judgment for explicit contextual criteria: whether a patch is appropriately scoped, whether an explanation matches the change, whether an agent's trajectory used evidence responsibly, whether the implementation fits repository conventions, or whether a tradeoff is acceptable under the task. Anthropic's agent evaluation guidance recommends combining code-based, model-based, and human graders and closely calibrating LLM judges with human experts.
| Criterion | Primary grader | Judge role |
|---|---|---|
| Required behavior and regressions | Fail-to-pass, pass-to-pass, integration, state, and protected tests. | None unless the task requires interpretation that tests cannot encode. |
| Build, type, lint, and static security | Versioned commands and approved scanners. | Summarize evidence or classify known findings, not replace the tools. |
| Scope and repository fit | Human-engineered rubric with repository context. | Apply bounded criteria after calibration and escalate uncertain cases. |
| Maintainability and architecture | Qualified software-engineering review for consequential decisions. | Triage, extract evidence, or score low-risk cases within a validated domain. |
| Trajectory behavior | Tool logs, command output, state changes, and completion claim. | Classify evidence use, recovery, instruction following, and unsupported claims. |
| Security consequence | Security checks and authorized human owner. | Flag and route; do not certify a consequential change alone. |
Version the complete judge, not only the model name
A judge system includes the model and exact version, system and user prompts, rubric, examples, context-selection logic, output schema, temperature or sampling controls, retry policy, aggregation, preprocessing, postprocessing, and abstention rules. Changing any component can change the measurement.
Store a judge manifest with each result. Keep the evaluated model output separate from reference patches and human labels unless the rubric explicitly permits those inputs. A judge that sees the gold solution measures similarity to that solution, which may differ from measuring whether the submitted code is valid.
Create an independent human anchor set
Select a stratified calibration set from the actual task population. Include common tasks, boundary cases, ambiguous requirements, multiple valid solutions, subtle regressions, security-sensitive changes, over-engineered patches, misleading completion claims, and examples that pass tests while remaining technically unacceptable.
Two or more qualified engineers should review the same items independently before seeing the judge decision. Give them the task, repository context, patch, executable evidence, trajectory, and versioned rubric. Require criterion-level decisions with evidence citations, severity, confidence, and an option to mark the task or rubric unjudgeable.
Adjudicate disagreements through an authorized technical owner. Do not force agreement merely to produce a clean gold label. Classify whether the disagreement came from reviewer error, rubric ambiguity, missing context, a broken task, multiple valid interpretations, or a genuinely borderline engineering decision. Fix the evaluation system before using those cases to optimize the judge.
Write criteria that can be falsified
Avoid rubrics such as "good code" or "production quality." Define observable failure and pass conditions. For scope control, state which files, behaviors, dependencies, and interfaces may change. For maintainability, identify the repository conventions and complexity signals that matter. For evidence integrity, require the completion claim to match commands actually run and final repository state.
| Weak criterion | Calibratable criterion | Required evidence |
|---|---|---|
| The code is clean. | The patch follows the named module boundary, avoids duplicated logic, and adds no unexplained abstraction. | Changed files, repository pattern, duplicated branch, or added dependency. |
| The solution is correct. | The patch satisfies the written requirement and the supplied normal, boundary, and failure evidence without a critical regression. | Task clauses, tests, state checks, and cited code path. |
| The agent reasoned well. | The trajectory inspected relevant context, responded to tool errors, ran required checks, and did not claim evidence it lacked. | Tool calls, errors, retries, command output, and final claim. |
| The change is secure. | The patch preserves the specified trust boundary, validation, authorization, secret handling, and approved data flow. | Security rule, scanner result, changed control, and reviewer citation. |
| The patch is appropriately scoped. | No unrelated behavior, test expectation, configuration, dependency, or public interface changed without task support. | Diff, task requirement, changed tests, and affected interfaces. |
Measure the errors that matter to the decision
Start with a confusion matrix for each criterion and severity class. Measure precision when false accusations create unnecessary review or reject valid work. Measure recall when missed critical failures create release risk. Use severity-weighted error so confusing two low-risk categories does not count the same as accepting an authorization bypass.
Agreement adjusted for chance can help when labels are categorical; rank or ordinal measures may fit ordered scores. Track abstention coverage and accuracy among non-abstained cases. Report confidence intervals and sample composition. One aggregate agreement number can look strong while the judge consistently misses a rare but unacceptable class.
OpenAI's GDPval grading documentation uses pairwise human expert preferences as its standard while describing an LLM judge as a rough estimate. That distinction is useful operationally: the judge can provide scalable signal, but its role must follow demonstrated alignment for the actual domain and decision.
Test judge bias and robustness directly
Run controlled perturbations: reverse candidate order, remove style cues, equalize verbosity, rename irrelevant identifiers, vary explanation length, change author identity, and present equivalent implementations. Test whether the judge favors outputs associated with its own model family, longer answers, familiar patterns, or the first option.
Build a challenge set from historical disagreements and known failure classes. Include patches that pass visible tests through unsafe shortcuts, plausible explanations that contradict the diff, unnecessary rewrites, weakened tests, subtle permission errors, and functionally valid alternatives to the reference patch. Keep a held-out set for validating judge changes.
Research continues to find that automated judges can diverge systematically from people. The 2026 ACL industry paper SAJA treats human alignment as a calibration problem rather than assuming an uncalibrated judge is sufficient. For code, the practical lesson is to preserve the human-labeled anchor and re-evaluate every judge change against it.
Set abstention and escalation rules
The judge should be able to return insufficient context, rubric conflict, multiple valid interpretations, environment failure, or human review required. Route cases based on severity, uncertainty, protected areas, disagreement with executable evidence, novel task types, and out-of-distribution signals.
Do not tune away abstention merely to improve automation coverage. Coverage is a business variable, not a quality metric by itself. A judge that confidently decides every case may be less useful than one that reliably handles routine outputs and escalates the decisions that require engineering judgment.
Monitor judge drift as part of evaluation operations
Revalidate when the judge model, prompt, rubric, context package, output schema, task distribution, codebase, agent harness, or acceptance policy changes. Run old and new judge versions on the same held-out set. Compare criterion distributions, class-level errors, abstention, cost, latency, and disagreement with human decisions.
Sample production decisions for blind human audit. Feed confirmed misses into error analysis, but do not train and test on the same examples. Maintain separate development, calibration, and held-out audit sets. Record why every rubric and threshold changed, who approved it, and which historical results remain comparable.
What a code-judge calibration pilot should deliver
- A written decision, judge role, excluded decisions, task population, severity policy, and named quality owner.
- A versioned rubric and judge manifest covering model, prompts, context, examples, output schema, aggregation, and abstention.
- A stratified human anchor set with independent expert labels, evidence citations, adjudication, and task-quality findings.
- Deterministic grader integration so the judge does not estimate claims that tools can prove.
- Class-level metrics, severity-weighted error, uncertainty, abstention coverage, and disagreement analysis.
- A challenge set covering order, verbosity, self-preference, unsafe shortcuts, misleading claims, and valid alternatives.
- Approved thresholds, escalation routes, monitoring cadence, drift triggers, limitations, and an expand, revise, or stop recommendation.
Frequently asked questions
What does LLM-as-a-judge calibration mean for code evaluation?
Calibration means testing and adjusting a specific judge model, prompt, rubric, context package, and output schema against trusted software-engineering decisions. The goal is to understand where the judge agrees, where it fails, and which decisions still require executable checks or human review.
How many human-reviewed examples are needed to calibrate a code judge?
There is no universal sample size. It depends on task diversity, failure prevalence, decision severity, reviewer consistency, and the precision required for each class. Start with a stratified set large enough to include important normal, boundary, ambiguous, adversarial, and critical-failure cases, then calculate uncertainty and add examples where evidence is weak.
Should an LLM judge replace unit tests for generated code?
No. Unit, integration, state, build, type, lint, and security checks should verify claims that software can prove. An LLM judge is more useful for bounded contextual criteria such as maintainability, scope, explanation quality, trajectory behavior, or repository conventions, and those judgments still need human calibration.
Which metrics should be used for LLM judge calibration?
Use class-level confusion matrices, precision and recall for consequential failures, agreement adjusted for chance where appropriate, ordinal or rank measures for ordered ratings, severity-weighted error, abstention coverage, and confidence intervals. Always inspect disagreement examples; one aggregate agreement score can hide unsafe failure patterns.
How often should a code-evaluation judge be recalibrated?
Revalidate whenever the judge model, prompt, rubric, context, output schema, code domain, agent harness, task distribution, or acceptance policy changes. Also run periodic human audits and trigger review when disagreement, severity-weighted error, score distribution, or production escape patterns move beyond approved limits.
Start with one bounded coding criterion and an independent human anchor set. Datrick can help define the rubric, build calibration evidence, analyze judge errors, establish escalation, and operate recurring technical review.