A Data Agent incident is not limited to an error screen. The more dangerous failure is a fluent answer that routes to the wrong source, applies the wrong business logic, omits a security boundary, or uses stale context while appearing credible. If operational or financial decisions depend on the answer, answer quality is a production reliability concern.
Incident response must preserve evidence before edits erase it. Datrick separates the agent definition from its draft and published state, underlying sources, requesting identity, policy controls, consuming channel, generated query, source result, integration behavior, and Fabric capacity. Recovery is proven only when critical questions, affected personas, and production channels work again.
Is a production agent returning answers you can no longer defend? Send the failing question, affected user or channel, first observed time, recent change, and business impact.
Classify the incident by user impact, not by error message
Start with the decisions at risk, affected users, channels, sources, and time window. A visible authorization error can be lower risk than a plausible but incorrect executive metric. Assign one incident owner, preserve the current state, freeze unrelated changes, and maintain a timestamped decision log.
| Production symptom | Evidence to capture | Immediate action |
|---|---|---|
| Wrong or inconsistent answer | Exact question, expected result, actual answer, conversation state, run steps, selected source, generated query, direct source result, and affected decisions. | Mark the answer path unreliable, retest in a fresh conversation, and route critical decisions to an approved report or human owner. |
| Wrong source or no source selected | Source names and descriptions, selected schema, example queries, routing instructions, routing-tool step, and similar questions that behave differently. | Limit the affected question class, preserve routing evidence, and avoid adding broad instructions before the source ambiguity is understood. |
| Unauthorized or empty result | Affected identity, sharing grant, effective source permissions, RLS or CLS role, Purview policy, selected source, tenant, region, and channel. | Reproduce as the affected user; do not grant broad workspace access as a diagnostic shortcut. |
| Item or configuration not found | Workspace and artifact IDs, published state, current source bindings, deployment history, channel connection, and integration logs. | Confirm the agent exists, is published, has valid sources, and the consumer references the intended production item. |
| Timeout, active-run, or intermittent failure | Timestamp, request ID, thread state, client timeout, retry behavior, service health, capacity metrics, query duration, concurrency, and downstream source latency. | Use bounded retries only for transient failures, start a new conversation when thread state is blocked, and protect capacity from retry amplification. |
| Regression after release or source change | Approved commit, deployed definition, draft and published versions, source schema or model diff, permission change, evaluation baseline, and release timestamp. | Stop promotion, compare against the last known-good release, and prepare a dependency-aware restore. |
Use the first 60 minutes to preserve a diagnosable state
- Record severity, decisions at risk, blast radius, owner, responders, communication channel, and the next update time.
- Freeze agent edits, publishing, source-model changes, permission changes, and deployment activity unless required for containment.
- Export or record the current draft and published definitions, source bindings, sharing, permissions, tenant settings, integrations, and recent deployment history.
- Reproduce the exact question with the affected identity and channel, then repeat in a fresh conversation to separate conversation-state effects.
- Expand run steps. Capture source routing, context, generated SQL, DAX, or KQL, validation, execution result, and final narrative.
- Run the generated query directly against the governed source and compare the raw result with the expected business definition.
- Check Fabric service health, capacity pressure, throttling, source latency, integration timeouts, and retry volume without assuming infrastructure is the cause.
Contain unreliable answers without destroying evidence
Containment depends on scope. Restrict or pause the affected consumption path, remove an impacted group from distribution, warn users not to rely on a question class, or direct them to an authoritative report. Preserve a minimal diagnostic path for responders. For a permission incident, narrow access first; do not widen permissions to make the error disappear.
Published and draft versions have different operational roles. Microsoft documents that a published read-only version can continue serving users while builders refine the draft. That separation can reduce change risk, but it also means a production workspace may contain a fixed draft while consumers still use an older published version. Record both states before deciding what to restore.
Isolate the failing layer before applying a fix
| Layer | Diagnostic question | Correction pattern |
|---|---|---|
| Question and conversation | Is the request ambiguous, multi-intent, influenced by previous limited context, or different from the tested question? | Start a fresh conversation, make the business definition explicit, and add a regression case rather than hiding ambiguity. |
| Routing | Did the orchestrator choose the expected source, and which description, schema, examples, or instructions influenced it? | Reduce noisy schema, clarify source descriptions, add distinct examples, then use concise routing rules only when necessary. |
| Query generation | Does generated SQL, DAX, or KQL implement the requested metric, filters, joins, grain, time window, and security context? | Correct semantic metadata, source instructions, examples, model design, or unsupported expectations; add deterministic evaluation. |
| Data and semantic model | Does the direct query return the expected governed result, and did schema, measures, relationships, refresh, or source data change? | Repair the source contract or semantic model, validate refresh and lineage, then rerun agent tests. |
| Identity and policy | Does the affected principal have minimum effective source access, and are RLS, CLS, DLP, or access restriction policies changing the result? | Restore intentional least-privilege access or policy behavior; test allowed and denied personas separately. |
| Published state and integration | Is the intended version published, are source references valid, and does the channel point to the right workspace and artifact IDs? | Restore bindings, publish the approved version, refresh the integration, and run channel-specific smoke tests. |
| Capacity and dependency | Are AI Query work, generated-engine queries, concurrency, retries, or source latency causing timeouts or throttling? | Stop retry amplification, optimize the source path, isolate workloads, adjust capacity deliberately, and validate latency under representative load. |
Microsoft's routing guidance recommends inspecting run steps before changing descriptions, schema, examples, or instructions. Example queries must validate and should map questions clearly to query logic. For authorization and empty-result incidents, sharing the agent is not enough: the requesting identity also needs effective access to the selected source, subject to RLS, CLS, and policy.
Restore a known-good service path, then prove it
A Data Agent rollback is a controlled forward deployment of a known-good state. Restore the approved item definition from source control, apply the correct production bindings, verify dependencies and permissions, deploy to the production workspace, publish the restored version, and run critical acceptance questions through every affected channel and persona.
Do not rely on deployment-pipeline backward deployment as a universal rollback. Microsoft documents backward deployment only when the target stage is empty. Existing production content needs a release procedure that can redeploy a prior definition without losing environment mappings or publishing the wrong state.
An agent-only restore is insufficient when the cause is a source schema, semantic model, data refresh, permission, policy, connection, integration, or capacity change. Choose the smallest coherent recovery unit. Preserve the failed version and evidence for root-cause analysis after service is stable.
Use a production acceptance gate after recovery
- Run critical golden questions and compare selected source, generated query, raw result, final answer, and prior baseline.
- Test allowed, restricted, and denied personas across RLS, CLS, Purview, sharing, and underlying source permissions.
- Verify the published version through Fabric and every connected channel, not only in the builder experience.
- Measure latency, failures, retries, AI Query consumption, generated-engine workload, and shared-capacity impact.
- Confirm that monitoring, ownership, escalation, rollback artifacts, and user communication are active before closing containment.
Structure urgent support around evidence and recovery
Datrick begins with written scoping so a senior lead can review the incident without delaying on calendar coordination. Provide the exact symptom, first observed time, affected users and channels, business impact, recent changes, workspace and agent identifiers, and any captured run steps or error text. Do not include credentials, tokens, or sensitive production data in the inquiry.
The initial engagement establishes severity, containment, access requirements, evidence preservation, and a diagnostic sequence. Deliverables can include a restored production path, tested rollback, root-cause record, question-quality regression suite, persona permission matrix, release and publishing controls, monitoring plan, and a prioritized prevention backlog.
Frequently asked questions
Why did our Fabric Data Agent suddenly start returning wrong answers?
A regression can come from the agent definition, instructions, example queries, source routing, a changed semantic model or schema, source data, permissions, policy, published state, an integration, or capacity conditions. Preserve the failing question, identity, channel, run steps, generated query, source result, and current definitions before changing the configuration. Comparing that evidence with the last known-good release narrows the failing layer.
Can we roll back a Fabric Data Agent deployment?
A practical rollback restores the last known-good item definition and environment bindings, publishes the restored version, and reruns critical answer, permission, and channel tests. Do not treat deployment-pipeline backward deployment as a universal rollback: Microsoft documents backward deployment only when the target stage is empty. Recovery must also account for changed source schemas, semantic models, permissions, and integrations.
Why can a user open a Fabric Data Agent but get unauthorized or empty results?
Sharing the Data Agent does not automatically grant access to every underlying source. Microsoft documents minimum source permissions for semantic models, lakehouses, warehouses, KQL databases, and ontologies. The investigation should reproduce the issue as the affected user and verify effective source access, RLS or CLS, Purview policy, published state, and the source selected for the failing question.
What should we capture before changing a failing Fabric Data Agent?
Capture the exact question, expected and actual answer, affected identity and channel, timestamp, conversation state, run steps, selected source, generated SQL DAX or KQL, direct source result, draft and published definitions, recent deployment and source changes, permissions, policy, integration logs, capacity state, and a list of impacted decisions. This evidence protects the diagnosis and the rollback path.
How does Datrick handle an urgent Fabric Data Agent incident?
Datrick starts with written scoping and an evidence checklist, then establishes severity, containment, ownership, and a diagnostic timeline. The response separates configuration, routing, query generation, data, identity, policy, publishing, integration, and capacity failures; restores a known-good service path; validates critical questions and personas; and delivers a root-cause record plus prevention backlog.
Official implementation references
- Improve data source routing in a Fabric Data Agent
- Fabric Data Agent sharing and permission management
- Fabric Data Agent concept, governance, and limitations
- Fabric Data Agent integration troubleshooting
- Fabric Data Agent example queries and validation
- Source control, CI/CD, and ALM for Fabric Data Agent
Start with the question or error that made production trust uncertain. Datrick can isolate the failing layer, restore a known-good path, validate the recovery, and leave the team with a repeatable incident and rollback runbook.