A managed AI evaluation RFP should do more than request reviewers, hourly rates, and capacity. It should tell suppliers which decision the evaluation supports, which technical judgments matter, what evidence must be produced, how confidential materials will be handled, and how the buyer will determine whether a pilot is good enough to expand.
Weak RFPs reward confident staffing claims and low unit prices. Strong RFPs expose whether a supplier can inspect task quality, match technical experts, calibrate decisions, adjudicate ambiguity, protect data, and operate a review system that remains visible as models, rubrics, and workload change.
Preparing a technical evaluation program? Datrick can review one task family, contributor profile, rubric, security boundary, quality process, and pilot decision before responding with fit or qualifying questions.
Begin with the decision and problem statement
State whether the program will compare models, improve training data, qualify a release, evaluate an agent, build a benchmark, calibrate an automated grader, or provide recurring human review. Name the accountable decision owner and the consequence of a false pass or false failure. Suppliers need this context to propose an appropriate combination of executable, model-based, and human evaluation.
Describe the use case rather than prescribing every implementation detail. The UK government's AI procurement guidance recommends clear problem statements, output-based requirements, data assessment, multidisciplinary evaluation, ongoing lifecycle management, and risk allocation to the parties able to manage it. Those principles translate directly to managed evaluation procurement.
Define the technical work packages
List the task families and distinguish work that may require different specialists. Coding, SQL, data analysis, mathematics, legal reasoning, support workflows, safety review, and general preference ranking are not interchangeable queues. For each family, provide representative examples, expected reasoning depth, tools, languages, systems, output format, known ambiguity, current volume, and likely change.
| RFP section | Information the buyer should provide | Supplier response required |
|---|---|---|
| Decision and use case | Product, model, agent, training, release, research, or procurement decision and failure consequence. | Proposed evaluation method, limitations, dependencies, and decision evidence. |
| Task families | Domains, tools, languages, complexity, examples, context, output types, and expected changes. | Relevant reviewer profiles, matching method, coverage gaps, and substitution rules. |
| Task assets | Instructions, rubrics, reference answers, tests, datasets, repositories, and current quality concerns. | Inspection method, version control, defect escalation, and improvement ownership. |
| Volume and timing | Baseline, forecast range, burst pattern, priority classes, time zones, and decision dates. | Named management capacity, ramp assumptions, scheduling, and constraint disclosure. |
| Tooling | Required platform, APIs, repositories, execution environments, reporting, and integration constraints. | Operating model, supported evidence, access needs, failure handling, and technical dependencies. |
| Acceptance | Pilot decision, quality evidence, review process, exclusions, and authorized approvers. | Pilot plan, outputs, measurement method, risks, and expand, revise, or stop criteria. |
Require evidence of domain expertise, not only credentials
Ask how the supplier determines whether a reviewer can perform the actual task. Require role profiles, practical qualification samples, calibration method, supervision, substitution controls, and escalation access to senior experts. Specify identity, location, employment or subcontractor status, language, and availability requirements only where they are relevant and lawful.
For technical work, a polished resume is not sufficient. A coding reviewer may need to inspect repositories, run tests, reason about security, and distinguish an alternative valid patch from a hidden-test mismatch. A SQL reviewer may need to identify join-grain errors, business-definition conflicts, or permission risk even when a query executes.
Make task and rubric quality part of the service
State whether the supplier will only execute existing instructions or also inspect tasks, references, rubrics, graders, examples, and edge cases. Require a route for reporting underspecified prompts, contradictory reference answers, narrow tests, missing context, and multiple valid interpretations. Do not penalize suppliers for surfacing defects the program needs to fix.
Ask who owns changes to the rubric, who approves them, how versions are recorded, and whether historical results remain comparable. Define how buyer feedback becomes contributor guidance and how changed instructions are propagated to active work.
Specify calibration, review, and adjudication
The response should explain how reviewers are onboarded, how independent calibration is run, how disagreement is classified, who adjudicates, and what happens when the buyer itself has not established a trusted answer. Require examples of the records produced by calibration and quality review, with confidential details removed.
| Quality control | Question for suppliers | Evidence expected |
|---|---|---|
| Qualification | How do you test practical domain fit before queue access? | Representative assessment design, acceptance authority, and remediation path. |
| Calibration | How do reviewers develop a shared interpretation before volume? | Independent sample, comparison, disagreement classes, adjudication, and revised guidance. |
| Review | Which outputs receive second review, sampling, or targeted review? | Risk-based review plan, selection rules, reviewer independence, and findings. |
| Adjudication | Who decides ambiguous or disputed cases? | Escalation levels, response ownership, decision record, and turnaround treatment. |
| Feedback | How do corrections reach contributors and task owners? | Feedback loop, instruction versions, retraining trigger, and recurring error tracking. |
| Audit | Can the buyer reconstruct why an output was accepted? | Task, rubric, reviewer, evidence, decision, version, and change traceability. |
Choose metrics after the baseline
Ask suppliers to propose metrics and explain what each one can and cannot prove. Relevant measures may include acceptance, rework, reviewer disagreement, adjudication, class-level error, severity-weighted error, rubric ambiguity, escalation, turnaround distribution, queue age, calibration drift, and audit findings. Throughput without quality and task complexity context is not a useful success measure.
Define numerator, denominator, exclusions, sampling, measurement owner, reporting cadence, and change control. Avoid inserting an arbitrary agreement or acceptance target before task inspection. The RFP can require a baseline period and an evidence-based target proposal after the pilot.
Describe data, security, and permitted use
Classify prompts, outputs, reference answers, repositories, datasets, logs, screenshots, credentials, customer information, and benchmark tasks. State approved countries, devices, networks, storage, tools, model providers, subcontractors, and communication routes. Require identity verification, least privilege, access approval, logging, incident response, retention, deletion, offboarding, and evidence of removal.
Ask whether materials may be used for supplier training, model training, service improvement, portfolio examples, or any purpose outside delivery. The default should not be assumed. NIST's Generative AI Profile recommends due diligence for AI acquisition and supplier assessments that includes privacy, security, intellectual property, third-party risk, ongoing monitoring, and value-chain concerns.
Make commercial assumptions comparable
Require suppliers to separate contributor execution, technical review, program management, calibration, adjudication, tooling, security onboarding, reporting, rework, and surge capacity. State whether pricing should be hourly, per accepted unit, per workstream, per managed pod, fixed pilot, or a hybrid. Ask what happens when task complexity or quality requirements change.
Lowest unit price can hide omitted management and quality work. Conversely, a managed fee is not justified unless the response names the operating responsibility and evidence it buys. Require minimum commitments, ramp time, cancellation, unused capacity, overtime, currency, taxes, expenses, rate changes, and payment assumptions.
Use a paid representative pilot
A pilot should test one real task family, not a supplier-created showcase. Provide the same bounded materials and decision criteria to shortlisted vendors where procurement rules allow. Include task inspection, contributor qualification, independent calibration, managed delivery, review, adjudication, reporting, security execution, and a written retrospective.
Judge the pilot on task insight, decision quality, evidence, communication, issue escalation, repeatability, and fit with the operating environment. A supplier that identifies a broken rubric may create more value than one that follows it quickly and produces a misleading acceptance rate.
Score the operating system, not the sales presentation
| Category | Suggested emphasis | What strong evidence looks like |
|---|---|---|
| Domain and task fit | High | Relevant practical reviewers, explicit coverage gaps, and representative qualification. |
| Evaluation method | High | Appropriate grader mix, task inspection, limitations, and decision-linked evidence. |
| Quality operations | High | Independent calibration, risk-based review, adjudication, feedback, and traceability. |
| Security and governance | High | Specific controls for the proposed materials, people, tools, access, and lifecycle. |
| Delivery management | Medium-High | Named ownership, realistic capacity, issue handling, reporting, and change control. |
| Commercial fit | Medium | Comparable assumptions, visible management cost, flexible pilot, and no hidden dependencies. |
| References and evidence | Medium | Verifiable relevant work with confidentiality boundaries and no invented performance claims. |
Proposal red flags
- Guaranteed acceptance, agreement, or turnaround before inspecting the task and baseline.
- Generic reviewer pools presented as sufficient for coding, SQL, data, or other specialist work.
- No independent calibration, adjudication, task-defect route, or authorized quality owner.
- Unclear data reuse, model-provider use, subcontracting, access, retention, or deletion.
- A quality dashboard with metrics that cannot be reconstructed from task-level evidence.
- Pricing that excludes review, rework, management, security onboarding, or changing task complexity.
- Refusal to disclose limitations, capacity constraints, domain gaps, or conditions that would stop delivery.
Frequently asked questions
What should an AI model evaluation RFP include?
Include the business and technical decision, task families, model and system context, domain expertise, data and access rules, evaluation methods, rubric and reference-answer ownership, calibration, quality assurance, adjudication, metrics, security, deliverables, pilot acceptance, capacity assumptions, pricing, intellectual property, support, and exit requirements.
How should buyers compare managed AI evaluation vendors?
Score vendors against demonstrated domain fit, task-quality method, reviewer qualification, calibration and adjudication, evidence traceability, security controls, management capacity, transparent limitations, and performance on a paid representative pilot. Do not select only on hourly rate, claimed contributor count, or a polished sample.
Should an AI evaluation RFP require a pilot?
Usually yes. A bounded paid pilot using one representative task family can test instructions, reviewer fit, calibration, quality controls, security, communication, turnaround, and evidence before either party commits to recurring volume. The RFP should state the pilot decision and acceptance method.
Which quality metrics belong in a managed evaluation contract?
Choose metrics after establishing a baseline. Depending on the work, use acceptance, rework, reviewer disagreement, adjudication, class-level error, severity-weighted error, rubric ambiguity, escalation, turnaround distribution, calibration drift, and audit findings. Define measurement ownership and exclusions rather than inserting unsupported targets.
What are red flags in an AI evaluation vendor proposal?
Red flags include guaranteed quality before task inspection, generic reviewers for specialist work, no independent calibration, unclear data reuse, unrestricted subcontracting, weak identity and access controls, no adjudication route, opaque quality metrics, no task-version traceability, and pricing that omits management, review, rework, or security requirements.
Use one representative task family to make the RFP concrete. Datrick can review technical fit, contributor requirements, task quality, calibration, security, management, and the smallest responsible pilot shape.