Public coding benchmarks answer a useful but limited question: how an agent performs on a shared task suite under a particular harness and environment. They do not show whether that agent can work safely and effectively inside a specific organization's architecture, conventions, languages, tooling, security boundaries, or maintenance workload.
A private-repository benchmark can provide that missing evidence, but it introduces two linked responsibilities. The benchmark must protect source code, credentials, customer data, hidden checks, and internal context. It must also produce a valid measurement. A secure benchmark with broken tasks is not useful, and a rigorous benchmark that leaks protected assets is not acceptable.
Need repository-level evidence before selecting or releasing a coding agent? Datrick supports private task design, reference solutions, executable graders, technical review, calibration, error analysis, and managed evaluation delivery under agreed access controls.
Start with the decision the benchmark must support
State whether the result will select a model, compare agent harnesses, approve a release, qualify a new workflow, set permission boundaries, or measure regression over time. Then name the decision owner and the consequences of false confidence. A benchmark for autocomplete, autonomous issue resolution, security remediation, and migration work should not use the same tasks or acceptance thresholds.
Record the exact model, version, agent harness, prompts, tools, permissions, repository commit, dependency state, compute limits, network policy, retries, time budget, and grader versions. Anthropic's agent evaluation guidance distinguishes the model from the agent harness and the evaluation harness. That distinction matters: the result belongs to the complete tested system, not to the model name alone.
Build representative tasks without turning private history into a leak
Mine candidate tasks from authorized issue types, maintenance work, incidents, code review findings, migrations, feature changes, and support patterns. Do not copy production secrets, customer records, credentials, restricted logs, or sensitive business rules into prompts. Where a real task contains protected information that is irrelevant to the capability, redact, replace, or reconstruct it in a controlled fixture.
Each task needs a fixed baseline commit, visible requirements, allowed tools, prohibited changes, environment definition, setup steps, reference evidence, fail-to-pass checks, pass-to-pass checks, expected state, and known limitations. A task author should demonstrate that required checks fail before the intended change and pass after a trusted solution.
| Task component | Required evidence | Failure to avoid |
|---|---|---|
| Baseline | Immutable repository commit, dependencies, configuration, fixtures, and tool versions. | The same task behaves differently because the environment moved. |
| Prompt | Observable requirement, context reasonably available in the repository, constraints, and acceptance language. | Hidden tests enforce behavior the prompt and codebase do not imply. |
| Reference | Trusted patch or outcome, author notes, alternatives, and known edge cases. | One implementation is treated as the only valid solution without justification. |
| Required checks | Tests or state checks that fail before the fix and pass after it. | A visible happy-path test passes while the required behavior remains incomplete. |
| Regression checks | Existing tests, build, types, lint, integration checks, and protected behavior. | The task is solved by breaking an adjacent workflow or weakening a test. |
| Security class | Repository sensitivity, data restrictions, allowed providers, network rules, artifacts, and retention. | A low-risk task silently exposes high-risk repository material or secrets. |
Audit tasks before trusting a score
Review prompt quality, test coverage, reference behavior, repository context, environment stability, and whether the task measures the intended capability. OpenAI's coding-evaluation audit identifies overly strict tests, underspecified prompts, low test coverage, and misleading prompts as benchmark failure classes. A private suite needs the same skepticism, even when tasks came from real engineering work.
Use independent technical review for a sample of tasks before scaling. Classify defects separately from model failures. If an experienced engineer can reasonably satisfy the written task and still fail protected checks, the benchmark may be measuring author assumptions rather than coding ability.
Run the agent in an isolated, least-privilege environment
The benchmark runner should receive only the repository material, tools, network destinations, credentials, and time required for the task. Start from an approved baseline, create an ephemeral writable workspace, execute under a non-production identity, capture defined evidence, and destroy the environment after the retention step completes.
| Control | Purpose | Implementation question |
|---|---|---|
| Repository scope | Limit exposure to the minimum code and history required by the task. | Does the agent need the full monorepo, commit history, sibling services, or only a bounded checkout? |
| Ephemeral compute | Prevent residue, cross-run influence, and persistence after evaluation. | Is each run isolated, reset from a trusted image, and destroyed after artifacts are collected? |
| Least privilege | Reduce the consequence of agent or workflow failure. | Are tokens read-only by default, scoped to the task, short-lived, attributable, and revocable? |
| Secret isolation | Keep credentials and protected configuration out of prompts, logs, patches, and model context. | Can the task use fixtures or brokered capabilities instead of raw production secrets? |
| Network control | Restrict data movement and uncontrolled dependency or tool access. | Which endpoints are required, logged, denied, or replaced with internal mirrors? |
| Artifact policy | Control patches, traces, test output, screenshots, caches, and generated files. | Who can access artifacts, what is redacted, and when are they deleted? |
| Human access | Apply the same protection to authors, reviewers, operators, and vendors. | Are roles separated, access approved, activity logged, and removal verified? |
GitHub's secure-use guidance for Actions recommends least privilege and warns about untrusted code and secret exposure. Those principles apply directly to coding-agent benchmarks: generated code and task-controlled text should be treated as untrusted, and a runner must not inherit broad repository or production authority merely because it lives inside an engineering environment.
Separate the model connection from repository custody
Document where prompts and code context are processed, which provider or deployment receives them, whether data may be retained or used for training, which regions and subprocessors apply, how logs are handled, and who can inspect requests. Do not claim that source code never leaves the organization unless the architecture and contracts actually establish that fact.
Several deployment patterns may be valid: an organization-controlled environment calling an approved model API, a cloud deployment inside an approved tenant, a provider-managed coding agent with enterprise controls, or a fully internal model. The correct choice depends on source sensitivity, contractual terms, threat model, performance, model capability, and operational ownership. Benchmark the deployment pattern that could actually be approved for production.
Combine executable evidence with bounded technical review
Use deterministic checks for claims software can prove: required behavior, regressions, repository state, file boundaries, builds, types, lint, security rules, migrations, and policy constraints. Preserve command output and final state. Run checks outside the agent's control where weakening or replacing the grader is a plausible shortcut.
Use qualified engineers where correctness is contextual: architecture, maintainability, security consequences, appropriate scope, repository conventions, ambiguous requirements, and whether the completion claim matches the evidence. Reviewers should cite the patch, task, tests, codebase, or trajectory rather than assign an unexplained quality score.
Control infrastructure noise before comparing small score differences
Repeat a sample of runs and track environment failures separately from agent failures. Record dependency downloads, timeouts, resource limits, flaky tests, network errors, runner initialization, tool failures, and retry policy. Anthropic's research on infrastructure noise in agentic coding evals shows that configuration can materially change benchmark outcomes. A private comparison needs matched environments and uncertainty, not one run presented as precision.
Publish task-level outcomes, error categories, repeated-run behavior, latency, cost, intervention, and limitations alongside aggregate scores. Averages can hide that one agent succeeds on routine edits but fails security-sensitive or cross-module tasks.
Protect the holdout set from contamination
Separate development tasks from protected release tasks. Restrict access to prompts, hidden checks, reference patches, and results; track which people, systems, and model workflows have seen them. Do not use release solutions as few-shot examples, reviewer training material, or debugging context for the agent being measured. Rotate tasks when exposure can no longer be bounded.
Contamination is not only a public-benchmark problem. Internal agents may retain conversation history, developers may tune prompts against the same tasks repeatedly, and task authors may unintentionally reveal expected implementation details. Report the suite version and exposure status with every result.
What a private repository benchmark pilot should deliver
- An approved decision statement, repository scope, threat model, provider review, data classification, and named owners.
- A versioned task specification and a small representative suite with baseline commits, reference evidence, and author validation.
- An isolated runner design covering identity, repository access, network, secrets, compute, artifacts, logging, retention, and destruction.
- Executable graders for required behavior, regressions, state, build, types, security, and task-specific constraints.
- A technical review rubric, independent calibration sample, disagreement classes, adjudication, and escalation rules.
- Repeated baseline runs, infrastructure-failure accounting, task audit, contamination status, error analysis, and limitations.
- A release, selection, revise, or stop recommendation tied to the original decision rather than a leaderboard claim.
Frequently asked questions
Can coding agents be benchmarked on private repositories without sharing the source code publicly?
Yes. A private benchmark can run inside an authorized, isolated environment where the repository, task data, credentials, tests, traces, and artifacts remain under defined access, retention, and deletion controls. The organization must still review the selected model provider, deployment route, data terms, network path, logging, and personnel access.
How do you create coding-agent tasks from a private repository?
Use representative issue types, incidents, maintenance work, and planned changes, then convert them into versioned tasks with a fixed baseline commit, clear requirements, allowed tools, prohibited changes, reference evidence, executable checks, and known limitations. Remove secrets and customer data before authoring or running tasks.
Which graders should a private coding-agent benchmark use?
Use deterministic checks for required behavior, regressions, repository state, builds, types, lint, security rules, and policy boundaries. Add technical human review for ambiguous requirements, architecture, maintainability, security consequences, and completion claims. Model-based graders can assist only after calibration against trusted decisions.
How do you prevent contamination in a private coding benchmark?
Keep release tasks and hidden checks access-controlled, separate authoring from evaluation roles, avoid reusing benchmark solutions in prompts or training examples, version task exposure, rotate compromised tasks, monitor unusual memorization signals, and report results with the exact task-suite version.
What should a private repository coding-agent pilot deliver?
A pilot should deliver an approved threat model, task-suite specification, isolated runner design, representative tasks, executable and human graders, reviewer calibration, repeated baseline runs, error analysis, limitations, and a recommendation to expand, revise, or stop.
Start with one repository and one representative task family. Datrick can review the security boundary, task quality, runner design, graders, expert review, and evidence needed for a useful model or release decision.