A runtime change can alter how a Fabric Data Agent plans, routes, interprets examples, generates queries, handles implied filters, and responds to ambiguity even when the attached data, instructions, and user question remain unchanged. Those changes may improve difficult NL2SQL cases and still regress a production question that users already trust.
Microsoft provides a standard runtime and a preview runtime. The standard runtime is generally available and optimized for stability. The preview runtime receives newer orchestration and query-generation behavior more frequently. This is not a model selector: Microsoft states that underlying model upgrades apply across both runtimes. Treat the choice as a release decision for the agent's core execution behavior.
Which production questions are wrong today, and which correct answers must not change? A runtime assessment needs both an improvement target and a regression boundary.
Choose runtime by production objective
Use the standard runtime when predictable behavior and lower change frequency are more important than early access to query-generation improvements. Use the preview runtime in an isolated development or test path when you need to evaluate upcoming behavior, validate a known failure mode, or provide evidence before improvements reach the standard channel. Preview should not mean unmeasured production experimentation.
Microsoft currently documents Advanced NL2SQL improvements in the preview runtime for SQL sources. These include closer adherence to example-query patterns, reasoning over implied categorical or boolean filters, and asking a clarifying question when a request is ambiguous rather than committing to an assumption. The current list can change as features enter preview and later graduate, so record the documentation date and observed behavior in every assessment.
| Decision area | Standard runtime | Preview runtime | Required evidence |
|---|---|---|---|
| Release posture | GA default with infrequent updates after validation. | Earlier and more frequent changes to orchestration and built-in query tools. | Business criticality, support policy, risk acceptance, and owner approval. |
| SQL generation | GA NL2SQL behavior for supported relational Fabric sources. | Advanced NL2SQL behavior currently includes example adherence, filter substitution, and ambiguity handling improvements. | Paired generated-query and result comparison against ground truth. |
| Behavior stability | Preferred when consistency and minimal churn are production priorities. | Behavior can change more often as improvements are evaluated. | Regression suite, monitoring, release window, and rollback readiness. |
| Underlying model | Runtime selection does not choose the underlying LLM; Microsoft applies model upgrades across both runtimes. | Do not attribute every answer change to the runtime without controlled reproduction. | |
| Published state | The published agent uses the runtime configured at publication time until it is republished with a different selection. | Published version, runtime flag, definition hash, data snapshot, and release record. | |
Build a controlled standard-runtime baseline
Start from the exact production definition: selected sources and schema, agent instructions, source instructions, example queries, published description, and runtime configuration. Record source IDs, workspace, schema versions, data refresh cutoff, identity personas, permissions, capacity, consumption channel, and current published version. If any of those change during the comparison, the result is no longer a runtime-only test.
Construct the evaluation set from production questions and known failure cases. Include straightforward metrics, multi-table joins, time windows, categorical filters, boolean filters, business synonyms, example-query patterns, ambiguous wording, unsupported requests, security boundaries, empty results, and high-cardinality output. For each case, store expected source, interpretation, query constraints, deterministic result, acceptable clarification, prohibited claims, and severity.
Capture more than the final prose. Preserve source selection, generated SQL where available, validation outcome, result rows, answer, latency, consumption, and any clarification. A fluent answer can hide a wrong join or filter. A different query can still be correct, but it must produce the approved business meaning under the same data snapshot.
Run a paired standard-versus-preview evaluation
Use isolated Data Agent candidates or controlled workspace stages so the standard and preview versions do not overwrite each other. Apply the same definition and source bindings, change only the runtime selection, publish both candidates in restricted environments, and execute the same test set with the same identity personas. Randomize or repeat runs when variation matters, and keep the production agent out of the experiment until a release decision is approved.
| Dimension | How to score | Preview gain | Regression gate |
|---|---|---|---|
| Example adherence | Compare generated SQL structure, joins, filters, and business logic with approved example patterns. | Fewer invented constraints and closer use of relevant examples. | No loss on existing high-severity patterns. |
| Filter interpretation | Test implied categories, boolean states, synonyms, exclusions, and multiple simultaneous filters. | Correct substitutions without requiring unnatural prompt wording. | No silent broadening, narrowing, or omitted literal. |
| Ambiguity handling | Define cases that require clarification and cases clear enough to answer directly. | Clarifies consequential ambiguity before query generation. | Does not over-clarify routine questions or confidently assume high-risk meaning. |
| Result correctness | Compare deterministic query results, grain, units, date context, null handling, and totals. | Higher pass rate on targeted failures. | Zero critical security or material financial regressions. |
| Response quality | Score completeness, attribution, uncertainty, limitations, and user utility. | Clearer answers with less correction effort. | No unsupported claim or loss of important qualifier. |
| Operations | Measure latency, errors, retries, capacity consumption, stability, and support burden. | Acceptable cost per correct answer and user experience. | Stays within SLO, capacity, and incident thresholds. |
Use severity-weighted decisions. A small average accuracy improvement does not justify one new cross-customer exposure, incorrect executive KPI, or silent failure to apply a mandatory filter. Separate must-pass questions from exploratory questions and report wins, ties, regressions, and unresolved variance. Review failures with domain owners and engineers rather than allowing an automated judge to make the only release decision.
Investigate each difference at the correct layer. Runtime behavior may explain a changed plan or query, but source data, schema, examples, instructions, identity, prior conversation, capacity, or an underlying model update can also affect output. Reproduce with a new session, fixed data, and preserved configuration before assigning cause.
Republish through a controlled release and rollback path
The runtime used by a published Data Agent is the runtime selected when that version is published. Changing a draft configuration does not silently move current consumers. To release preview behavior, update the controlled candidate, run the complete gate, publish from the approved production path, and record the runtime selection with the definition and test evidence. Confirm the exact consumption channels after publication because MCP, Foundry, Copilot Studio, and Microsoft 365 Copilot can add their own orchestration behavior.
Roll out to a named cohort or restricted route first. Monitor wrong-answer reports, clarification rate, query failures, authorization errors, latency, capacity, consumption, user corrections, and questions that changed from the baseline. Keep the standard-runtime known-good definition and republish procedure ready. Rolling back means selecting the standard runtime in the approved configuration and republishing, followed by verification; it is not enough to change an unpublished draft.
Define automatic stop conditions for critical wrong answers, permission leakage, material metric drift, sustained SLO breach, abnormal capacity pressure, or a sharp increase in clarification and support demand. Communicate the affected version, known limitations, and restoration state. Preserve evidence for Microsoft support when behavior appears to be a runtime defect.
Run a three-to-four-week runtime assessment
- Define the production agent, supported decisions, current runtime, target preview improvements, must-not-regress questions, owners, and release threshold.
- Capture the published definition, source bindings, data snapshot, identity personas, permissions, channels, capacity baseline, and known-good results.
- Create isolated standard and preview candidates that differ only in runtime selection and publish them to restricted test environments.
- Build paired example-adherence, filter, ambiguity, correctness, security, latency, consumption, unsupported, and failure tests.
- Execute the same suite against both candidates, inspect generated queries and results, classify differences, and reproduce material failures.
- Review the severity-weighted scorecard with business, data, security, operations, and platform owners and approve go, limit, or remain on standard.
- Republish through the controlled production path, monitor the cohort, exercise rollback, and retain evidence for the next runtime change.
Frequently asked questions
What is the difference between the Fabric Data Agent standard and preview runtimes?
The standard runtime is the generally available default and is optimized for stable, predictable behavior. The preview runtime receives newer orchestration, planning, routing, and built-in query-generation improvements more frequently. Microsoft currently lists Advanced NL2SQL improvements such as better example-query following, implied filter substitution, and ambiguity clarification in the preview runtime.
Does the Fabric Data Agent runtime control the underlying language model?
No. Microsoft states that model upgrades are applied across the standard and preview runtimes. Runtime selection controls the agent's core orchestration and built-in query-generation components, not which underlying model is used.
Can we switch a published Fabric Data Agent back to the standard runtime?
Yes, but the published agent uses the runtime selected in its configuration at publication time. To change production from preview back to standard, update the configuration, validate the candidate, and republish. Keep a known-good definition, evaluation baseline, and rollback runbook rather than treating the setting as an instant production toggle.
How should we test the Fabric Data Agent preview runtime?
Run standard and preview candidates against the same agent definition, source schema, data snapshot, identity personas, question set, and scoring rubric. Compare source selection, generated query, example adherence, filter interpretation, clarification behavior, result correctness, security, latency, consumption, and answer quality. Include regressions, not only questions expected to improve.
How long does a Fabric Data Agent runtime evaluation take?
A focused assessment commonly takes three to four weeks for one SQL-focused Data Agent and a representative production question set. The work includes baseline capture, isolated candidate configuration, paired evaluation, security and failure tests, release gates, controlled republishing, monitoring, and rollback evidence.
Official implementation references
- Fabric Data Agent standard and preview runtimes
- NL2SQL and Advanced NL2SQL for Fabric SQL sources
- Fabric Data Agent example-query design and evaluation
- Fabric Data Agent Python SDK configuration and publishing
- Fabric Data Agent source control, deployment, and published-state guidance
- Draft refinement and published Data Agent behavior
Start with the questions users already trust and the failures they already report. Datrick can run the paired evaluation, define severity-weighted release gates, and prove the republish and rollback path.