An Operations Agent incident can appear as no Teams message, a rule that never fires, repeated alerts, a stuck or failed recommendation, the wrong parameter, a Power Automate failure, an inactive agent, unexpected capacity load, or an action that changed the wrong object. These symptoms cross data, AI, identity, messaging, and automation boundaries.
Do not begin by rewriting goals and instructions. Preserve current state, determine business impact, contain unsafe actions, establish the last known-good operation, and trace one expected event through every stage of the control loop. A conversational message is not the system of record; reconcile source rows, queries, rules, operation records, approvals, downstream runs, and actual postconditions.
What was the last event that should have produced a known operation, message, approval, and downstream result? Use that event as the first end-to-end trace.
Contain impact and classify the first failed boundary
Record incident start, affected process and objects, severity, current agent state, last known-good operation, recent configuration or source change, expected and actual outcome, downstream impact, temporary manual control, owner, and recovery objective. Stop the agent or remove a risky action when continued execution could create harm; preserve evidence before editing configuration.
Check platform and tenant prerequisites, supported region, workspace capacity state, source availability and freshness, creator account and permissions, recipient access, Teams app availability, action and flow health, recent direct UI/API/MCP changes, and current Fabric service health. Keep observations separate from hypotheses.
Trace source, detection, reasoning, delivery, approval, and execution
| Boundary | Evidence to collect | Likely failure | Containment or repair |
|---|---|---|---|
| Source and freshness | Expected record, entity ID, value, event and ingestion time, schema, source watermark, duplicates, nulls, and access. | No current matching data, late ingestion, schema drift, wrong source, broken ontology binding, or permission loss. | Restore ingestion or binding, correct source, fix access, backfill carefully, and validate ground truth. |
| Generated query | Rule query, Query insights, property binding, time window, result rows, duration, errors, and five-minute evaluation timing. | Wrong field, threshold, grouping, window, identifier, unsupported table, or query failure. | Run independently, correct data or instruction, regenerate, and replay positive and negative cases. |
| Condition and state | State or transition type, prior value, boundary, recovery, current agent state, activation time, and rule version. | Agent inactive, persistent-state duplication, missed transition, ambiguous threshold, or stale playbook. | Stop or start deliberately, correct condition, define recovery, regenerate, and verify state. |
| Reasoning and operation | Operation ID, detected context, recommendation, reasoning, status, creation and expiry, response, and capacity evidence. | Reasoning failure, simplified message under throttling, operation expiration, or no operation created. | Preserve operation, reduce load, correct rule evidence, restore capacity, or use deterministic fallback. |
| Teams delivery | Teams app installation and tenant allow, recipient, organization membership, item write permission, chat or channel, timestamp, and message status. | App blocked or missing, invalid recipient, insufficient item permission, missing channel, or message throttling. | Restore approved app and recipient access, verify destination, send canary, and add alternate escalation. |
| Approval | Recipient, displayed context and parameters, response, responder, timestamp, operation age, and creator identity. | Wrong approver, parameter edit, no response, three-day expiration, or misunderstanding of execution identity. | Cancel or reject, correct recipient and context, recreate only after evidence review, and train owner. |
| Action and flow | Action ID, Activator connection, flow run, parameters, creator permissions, downstream logs, retry, and error. | Broken connection, wrong flow, invalid parameter, authorization, outbound protection, license, or downstream outage. | Disable action, repair connection or authority, validate contract, reconcile retries, and test nonproduction. |
| Postcondition | Target object before and after, external transaction ID, duplicate or partial effects, compensation path, and owner confirmation. | Action reported success but target state is wrong, partial, delayed, or duplicated. | Contain repeats, compensate through approved process, restore state, and add postcondition checks. |
| Change and capacity | Definition or playbook diff, API/MCP/UI audit, deployment, role change, capacity metrics, throttling state, and service incident. | Unreviewed drift, bad release, creator offboarding, sustained overload, or platform dependency failure. | Restore known-good version, revoke change path, resize or isolate load, and invoke fallback control. |
Microsoft documents that active agents run rule queries every five minutes, operations expire after three days without approval, heavy use can throttle messages and produce simplified non-LLM Teams output, and execution uses the creator's delegated identity even when another recipient approves. Build the incident timeline around those behaviors rather than assuming immediate event delivery or approver identity execution.
Prove recovery with deterministic and failure cases
| Recovery test | Method | Pass condition | Do not reactivate when |
|---|---|---|---|
| Known positive | Replay the incident and representative matching records with exact object, value, time, query, rule, and expected operation. | Every required case is detected within the approved end-to-end objective. | Any material known case remains missed, late, or mapped to the wrong object. |
| Known negative | Replay normal periods, nearby values, excluded entities, maintenance windows, and known exceptions. | No unnecessary operation or message is created. | False positives exceed owner tolerance or create operational burden. |
| Transition and duplicate | Move false to true, remain true, recover, re-enter, duplicate source records, and repeat evaluation cycles. | Signals and repeats match the approved state or transition contract. | Persistent-state messages spam owners or a valid re-entry is missed. |
| Data and permission failure | Pause ingestion, delay records, change schema, remove source access, revoke creator, and restore each dependency. | Failure is visible, contained, escalated, and recoverable without false normality. | The agent fails silently or continues using unintended authority. |
| Teams and approval | Test direct and channel recipient, app block, missing write permission, parameter edit, reject, approve, timeout, and expiration. | Only approved recipients receive clear context and every response has correct status and attribution. | Delivery, context, authority, or expiry behavior is ambiguous. |
| Action failure | Test invalid parameter, broken connection, downstream 4xx/5xx, timeout, duplicate submission, partial success, and retry. | Effects are idempotent, errors visible, retries bounded, and postconditions verified. | Downstream state cannot be reconciled or safely compensated. |
| Capacity pressure | Burst multiple objects and rules while monitoring Capacity Metrics, Eventhouse, reasoning, Teams, and action behavior. | Critical control SLOs and escalation survive expected peak load. | Background work rejects, delivery simplifies without safe fallback, or cost becomes uncontrolled. |
| Rollback and stop | Stop the agent, restore prior definition or configuration, verify dependencies, reactivate in a canary window, and compare evidence. | Known-good behavior returns within the recovery objective. | Current state differs from the approved rollback artifact or audit is incomplete. |
Use one labeled incident replay set as the permanent regression suite. A repair is not complete when a new Teams message appears; it is complete when source, query, rule, operation, message, approval, action, and postcondition all reconcile for positive, negative, and failed paths.
Prevent recurrence with ownership and observable control states
Monitor source freshness, query success and duration, agent active state, operation count and status, rule hit rate, false positives, message delivery and simplification, approvals and expirations, action runs and failures, postconditions, creator access, recipient permission, definition and playbook drift, capacity, and manual fallback availability.
Maintain a runbook with severity criteria, stop authority, source and query checks, rule and state diagnosis, Teams and identity owners, action and downstream contacts, evidence locations, emergency compensation, known-good definition, release and rollback procedure, vendor escalation, communication templates, and reactivation gates.
Exercise the runbook after source, schema, ontology, instruction, playbook, identity, recipient, action, flow, tenant, region, capacity, MCP, REST API, or Fabric release changes. Assign explicit ownership for creator offboarding and direct configuration drift; both can break the control loop outside a normal deployment.
Run urgent triage followed by a one-to-two-week recovery assessment
- Establish impact, severity, affected objects, unsafe actions, current agent state, last known-good operation, recent changes, manual fallback, and containment owner.
- Preserve source, query, playbook, operation, Teams, approval, identity, action, flow, downstream, capacity, deployment, API, MCP, and audit evidence.
- Trace one expected event end to end and identify the first boundary where expected and actual state diverge.
- Contain the failing rule, action, identity, recipient, configuration path, or agent while maintaining the approved manual process.
- Repair the root cause and build a labeled regression set covering the incident, normal data, boundaries, transitions, failures, permissions, load, and recovery.
- Replay every control path, reconcile operation and downstream state, validate Teams delivery and approvals, and exercise stop and rollback.
- Reactivate through a monitored canary window only after owners approve source, rule, recommendation, action, capacity, and recovery evidence.
- Deliver the incident timeline, root cause, impact and reconciliation, repaired configuration, regression suite, monitoring, runbook, ownership changes, and prevention backlog.
Frequently asked questions
Why is Microsoft Fabric Operations Agent not sending Teams messages?
Check whether the agent is active, source data is current, the generated rule query returns matching records, the condition and five-minute evaluation window behave as expected, capacity isn't rejecting background work, the Fabric Operations Agent Teams app is installed and allowed, the recipient belongs to the organization and has write permission on the agent item, and messaging isn't simplified or throttled under heavy use.
Why did Operations Agent playbook generation fail?
Common causes include no connected knowledge source, unsupported or inaccessible source, insufficient permissions, incompatible schema or bindings, ambiguous goals and instructions, capacity or regional prerequisites, and a failed or incomplete asynchronous generation operation. Inspect the actual source, identity, status, and returned error before changing the prompt.
How do you troubleshoot a failed Operations Agent action?
Trace the operation from the matching source record and generated rule through the recommendation, Teams recipient and approval, creator delegated identity, action ID and parameter values, Activator connection, Power Automate run, downstream authorization, idempotency, postcondition, and error. Contain retries until the downstream effect is reconciled.
What should we do if an Operations Agent takes the wrong action?
Stop the agent or remove the affected action, preserve operation and downstream evidence, revoke or narrow authority where needed, identify every impacted object, reverse or compensate changes through an approved process, notify owners, restore a known-good configuration, and rerun the rule and action regression suite before reactivation.
How long does an Operations Agent incident assessment take?
Initial containment and fault-domain triage can often be completed in a focused incident session when access and evidence are available. A durable one-to-two-week recovery assessment typically covers root cause, configuration and data repair, regression replay, monitoring, runbooks, identity and action controls, rollback, and a safe reactivation decision.
Official implementation references
- Operations Agent setup, generated playbook, rules, state, Teams, and actions
- Operations Agent source, cadence, identity, expiry, messaging, region, and capacity limitations
- Operations Agent Teams recommendations, approvals, and downstream actions
- Operations Agent responsible use and probabilistic behavior
- Operations Agent capacity meters, background processing, and reporting
Send the last event that should have completed the control loop and the first observed failure. Datrick can isolate the fault domain, contain risk, repair the root cause, replay the evidence, and establish safe reactivation gates.
For workspace-wide pipeline failures, long-running jobs, and missing schedules, build a monitored ItemJobEventLogs contract with deterministic controls, Operations Agent context, Teams escalation, and NOC runbooks.