Manual chat testing can prove that one prompt worked once. It cannot show whether a schema change broke another domain, an instruction moved route selection, a runtime update changed generated queries, a permission difference altered the answer, or a polished explanation concealed a wrong number. Without a versioned baseline, every release begins from memory.
Microsoft's preview Evaluation SDK can run question and expected-answer pairs against a sandbox or production Fabric Data Agent, save summary and step-level output, and classify answers as true, false, or unclear. That capability becomes a production control only when the dataset represents real decisions, expected answers are independently derived, the critic is calibrated, failure evidence is retained, and thresholds reflect business severity rather than one aggregate percentage.
Can you identify which critical questions regressed between the last approved agent and today's draft? If not, establish the baseline before changing another instruction or example.
Design ground truth around decisions and failure risk
Start with questions users already ask analysts, support teams, or reports. Assign each case an owner and record the intended user, source, metric or relationship, filters, time interpretation, expected answer, deterministic derivation, acceptable tolerance, prohibited behavior, severity, and review date. Store a stable data-snapshot reference so a source refresh is not mistaken for an agent regression.
| Dataset field | Why it matters | Weak implementation | Production control |
|---|---|---|---|
| Question and intent | Connects natural language to one decision and supported agent capability. | Generic demo prompts with no real owner. | Observed user wording, domain label, owner, and decision context. |
| Expected answer | Provides the SDK comparison target. | Answer copied from the agent under test. | Independent SQL, DAX, KQL, GQL, policy evidence, or signed business control. |
| Snapshot and tolerance | Separates data movement and formatting from reasoning failure. | One timeless string compared after every refresh. | Source version, timestamp, unit, ordering rule, numeric tolerance, and acceptable variants. |
| Persona and permissions | Exposes RLS, CLS, item, source, and route-specific differences. | Every test runs as the author. | Positive, restricted, denied, and service-principal cohorts where supported. |
| Severity and gate | Prevents minor phrasing from hiding a critical financial or access failure. | All questions contribute equally to one accuracy score. | Critical zero-tolerance cases plus weighted domain thresholds. |
| Expected route and query | Shows whether a correct-looking result came from the authoritative source and logic. | Only the final prose is evaluated. | Expected tool, source, query properties, result, evidence, and response rubric. |
Partition the suite into smoke, critical, domain, permission, ambiguity, no-answer, adversarial, latency, and long-tail sets. Keep canonical questions stable for trend comparison and add paraphrases as a separate robustness layer. Review stale cases, but never silently rewrite a failed expected answer to make the current agent pass.
Make the SDK run reproducible and diagnosable
Microsoft documents a notebook workflow using the current `fabric-data-agent-sdk`, a pandas DataFrame or CSV with `question` and `expected_answer`, and `evaluate_data_agent`. The call can target the `sandbox` or `production` stage and writes a summary table plus a companion steps table. Preserve the returned evaluation ID, agent and source versions, workspace, stage, runtime, identity, dataset hash, critic version, and execution timestamp with every run.
Use `get_evaluation_summary` for total, true, false, unclear, and accuracy trends. Use `get_evaluation_details` for question, expected answer, actual answer, evaluation result, diagnostics, and the per-run thread link. Export configuration and execution diagnostics for failures so remediation can be assigned to routing, schema, instructions, examples, query generation, permissions, source behavior, response synthesis, or the critic.
| Stage | Automated action | Required evidence | Gate decision |
|---|---|---|---|
| Dataset validation | Check required fields, duplicates, stale snapshots, unsupported personas, and missing owners. | Versioned dataset manifest and deterministic control references. | Reject invalid or unowned cases before agent execution. |
| Sandbox baseline | Run smoke, critical, and changed-domain suites against the draft agent. | Evaluation ID, summary, detail, step table, configuration, and diagnostics. | Stop on critical false/unclear or unexplained route/query change. |
| Human adjudication | Review critic disagreements, ambiguous questions, security cases, and high-impact explanations. | Reviewer decision, rationale, defect owner, and approved expected-answer change. | Fix agent, data, test, or critic; never auto-waive silently. |
| Production comparison | Run the same safe suite against the published baseline and compare case-level outcomes. | Baseline delta by domain, severity, route, source, and failure layer. | Promote only when new gains exceed accepted regressions. |
| Release and canary | Publish through controlled ALM, rerun smoke tests, and monitor real feedback and failures. | Release version, post-publish evaluation, rollback target, and canary observations. | Continue, limit, or roll back using pre-agreed thresholds. |
| Trend review | Track quality, unclear rate, latency, coverage, judge disagreement, and defect recurrence. | Time series with dataset and platform version changes. | Expand, consolidate, or retire cases and configurations deliberately. |
Calibrate the critic instead of treating it as ground truth
The SDK's default evaluator decides whether the actual answer matches the expected answer. Microsoft also supports a custom `critic_prompt` with query, expected answer, and actual answer placeholders. That flexibility is useful for numeric tolerance, reordered rows, equivalent labels, required caveats, citations, and domain-specific semantics, but a more detailed prompt is not automatically a more reliable judge.
Create a human-labeled calibration set containing clear passes, clear failures, formatting-only differences, partial answers, wrong units, stale values, excessive confidence, missing qualifiers, and permission failures. Measure critic false-pass, false-fail, and unclear rates by severity. Keep deterministic numeric and structural checks outside the LLM critic when possible, and require human adjudication for critical or disputed cases.
Version the critic prompt and rerun the calibration set whenever it changes. Do not compare accuracy trends across different critics without a bridge run. Track judge disagreement as its own operational metric; a rising unclear or disagreement rate can signal question drift, expected-answer decay, or output-format changes before headline accuracy collapses.
Connect evaluation to release gates without overstating preview tooling
The evaluation capability is preview and documented as a Fabric notebook workflow. Treat the notebook and its output tables as the evidence-producing test harness, then connect approvals and promotion to your existing Fabric Git integration, deployment pipelines, or controlled release process. Avoid presenting a notebook accuracy number as an infallible CI signal.
Require zero regressions for critical security, financial, compliance, and source-authority cases. Set domain thresholds for lower-severity cases, cap unclear answers, enforce latency and failure-rate limits, and require an owner and expiry for every waiver. Compare sandbox and production under the same dataset, snapshot, identity, runtime, and critic before publishing.
After publication, rerun a safe smoke suite and monitor real question distributions, failure feedback, authorization errors, route and query changes, latency, capacity, source freshness, and platform releases. Preserve the last approved agent definition, source bindings, test dataset, critic, output tables, diagnostics, and rollback procedure.
Run a three-to-five-week evaluation implementation
- Select one Data Agent, target stages, critical domains, release owners, source owners, consumer personas, and current failure history.
- Inventory agent configuration, source versions, runtime, permissions, ALM path, existing questions, deterministic controls, and observability.
- Build and review the versioned ground-truth dataset with snapshots, tolerances, routes, queries, severities, owners, and prohibited outcomes.
- Implement the SDK notebook, output tables, run manifest, summaries, failure extraction, diagnostics retention, and baseline comparison.
- Calibrate the default or custom critic against human-labeled cases and add deterministic numeric, structural, and security checks.
- Run sandbox and production baselines, classify defects, remediate the responsible layer, and prove case-level regression reporting.
- Deliver release thresholds, approval and waiver flow, post-publish smoke tests, dashboards, runbooks, rollback evidence, and expansion roadmap.
Frequently asked questions
Can Microsoft Fabric Data Agent evaluations be automated?
Yes. Microsoft's preview Fabric Data Agent Python SDK can run a pandas or CSV ground-truth dataset of questions and expected answers against a sandbox or production Data Agent, store summary and step-level results, and return an evaluation run ID for analysis.
What metrics does the Fabric Data Agent Evaluation SDK return?
The documented summary includes total questions, true, false, and unclear counts, and accuracy. Detailed results include question, expected answer, actual answer, evaluation result, diagnostic metadata, and a thread URL available to the user who ran the evaluation.
Can the Fabric Data Agent evaluation judge be customized?
Yes. The evaluate_data_agent function accepts a custom critic prompt containing query, expected_answer, and actual_answer placeholders. A custom critic should be calibrated against human-labeled cases before it is trusted as a release gate.
Does an automated Data Agent evaluation replace human review?
No. Automated comparison is useful for repeatable coverage and regression detection, but business ambiguity, security behavior, source authority, misleading explanations, and high-impact decisions still require deterministic controls and qualified human review.
How long does a Fabric Data Agent automated evaluation implementation take?
A focused implementation commonly takes three to five weeks for one agent, one to three domains, and an initial production-shaped ground-truth suite. It includes dataset design, deterministic controls, SDK notebook, critic calibration, baseline results, release thresholds, reporting, ownership, and runbooks.
Official implementation references
- Microsoft guide to evaluating a Fabric Data Agent
- Fabric Data Agent Python SDK
- Example-query validation and conflict detection
- Fabric Data Agent source control, CI/CD, and ALM
- Semantic model automated evaluation guidance
Start with the releases your team cannot explain or compare today. Datrick can turn real business questions into a versioned evaluation asset, calibrate the judge, produce case-level evidence, and connect quality thresholds to an accountable release and rollback process.