Fabric provides several pipeline alert paths: activity-level Outlook or Teams steps, scheduled-run failure notifications, deterministic Activator rules over job events or KQL, workspace-wide monitoring through an Eventhouse, and an Operations Agent that interprets a playbook and sends contextual recommendations. The right design depends on whether the requirement is a guaranteed failure signal, a deterministic threshold, or an AI-assisted diagnosis and response.
Microsoft's Operations Agent example monitors the workspace `ItemJobEventLogs` table, identifies Data Pipeline records, compares `DurationMs` with a 60-minute threshold, and includes run context in Teams. Production design must go further: distinguish completed from currently running jobs, detect missing expected runs, account for ingestion delay, prevent duplicate alerts, preserve critical deterministic controls, and connect each message to an owner and runbook.
Which condition requires guaranteed notification, and which one benefits from contextual AI reasoning? Keep deterministic controls for critical failures and measure the agent's incremental value.
Choose failure notification, Activator, Operations Agent, or a layered design
| Alert path | Best fit | Primary limitation | Production control |
|---|---|---|---|
| Pipeline activity step | Send a direct message or email after a specific activity outcome within one pipeline. | Alert logic is embedded in each pipeline and can fail with or complicate the workflow. | Standard component, failure path tests, centralized ownership, and deployment validation. |
| Scheduled failure notification | Simple built-in email notification when a scheduled pipeline run fails. | Limited context and condition flexibility; doesn't cover every trigger or operational pattern. | Use as a minimum deterministic signal for supported critical schedules. |
| Job event Activator | Deterministic create, update, run-status, email, or Teams actions over selected job events. | Per-rule maintenance and limited contextual reasoning. | Versioned rules, owner, test events, action monitoring, and fallback recipient. |
| KQL plus Activator | Workspace-wide deterministic conditions over monitoring Eventhouse records. | Polling window, ingestion lag, duplicate and missed-event tradeoffs require careful design. | Poll more often than the query window, test overlap, deduplicate by run ID, and monitor freshness. |
| Operations Agent | Natural-language playbook, generated rules, contextual comparison, recommendation, and response guidance. | Probabilistic generation and reasoning, five-minute cadence, identity, Teams, capacity, and preview risks. | Review generated queries, replay ground truth, shadow deterministic controls, and measure added value. |
| Layered NOC design | Critical deterministic failure signal plus AI-assisted duration anomaly, context, triage, and recommendation. | More components and possible duplicate messages without shared incident correlation. | One run ID, severity model, suppression policy, escalation owner, and reconciled incident state. |
Define run identity, expected schedule, threshold, baseline, and escalation
For each pipeline, record workspace, item ID and name, criticality, owner, schedule and trigger type, timezone, expected start window, expected frequency, maximum duration, warning and critical thresholds, historical percentiles, source and target SLAs, dependencies, concurrency, retry policy, maintenance windows, business blackout dates, completion and failure semantics, escalation, and manual recovery.
Map the contract to `ItemJobEventLogs` fields including `Timestamp`, `ItemKind`, `ItemName`, `JobStatus`, `DurationMs`, `JobStartTime`, `JobEndTime`, `JobInvokeType`, `JobInstanceId`, and `WorkspaceName`. Confirm actual values and ingestion behavior in the target workspace rather than assuming display names or status strings.
A completed run above 60 minutes is different from an in-progress run that has already exceeded 60 minutes. The first uses final `DurationMs`; the second needs current elapsed time from `JobStartTime`, a status filter, and a policy for repeated evaluations. Missing expected runs require schedule-aware absence detection, not a duration rule.
Replay normal, failed, slow, missing, repeated, and recovery cases
| Test | Method | Pass condition | Remediation |
|---|---|---|---|
| Completed long run | Replay a completed pipeline immediately below, equal to, and above the duration threshold. | Boundary behavior matches the written rule and message shows correct minutes, excess, and timestamps. | Unit, threshold, comparison, rounding, or field-mapping fix. |
| Currently running breach | Keep a run in progress beyond warning and critical thresholds across multiple evaluations. | Alert arrives before completion and repeats only under the approved reminder policy. | Elapsed-time query, state or transition rule, suppression, or escalation correction. |
| Failed run | Trigger known activity and pipeline failures across scheduled and on-demand invocation. | Critical deterministic signal arrives and AI context includes pipeline plus JobInstanceId for investigation. | Status mapping, job-type filter, recipient, Teams, or fallback control. |
| Missing expected run | Skip a scheduled trigger, disable it, delay start, and create a holiday or maintenance exception. | Unexplained absence alerts after the approved grace period; valid exceptions remain quiet. | Schedule registry, heartbeat logic, timezone, exception calendar, or ownership fix. |
| Historical anomaly | Compare current duration with at least seven days of representative history, weekday patterns, volume, and known change windows. | Baseline comparison is statistically and operationally meaningful for the pipeline. | Segment baseline, minimum history, robust percentile, volume normalization, or remove comparison. |
| Duplicate and retry | Repeat job events, rerun the same pipeline, retry failed activities, overlap query windows, and sustain a breach. | Messages correlate by JobInstanceId and don't create uncontrolled duplicate incidents. | Deduplication, state transition, cooldown, aggregation, or incident-key correction. |
| Monitoring data lag | Delay or stop workspace monitoring ingestion and compare Monitor Hub with Eventhouse evidence. | Freshness breach is visible and absence of logs is never interpreted as healthy pipelines. | Monitoring heartbeat, source alert, fallback API or Monitor Hub check, and escalation. |
| Permissions and Teams | Remove creator source access, recipient write permission, Teams app, channel, and creator account. | Failures are visible, containable, and routed to an alternate operational owner. | Identity ownership, group recipient, permission, app policy, or deterministic fallback. |
| Load and capacity | Burst job events across many pipelines while Eventhouse, Operations Agent, Teams, and capacity are observed. | Critical messages and deterministic controls meet SLO without unacceptable CU or recipient burden. | Scope, aggregation, severity routing, capacity, sampling, or separate agents. |
Run the Operations Agent beside Monitor Hub and the deterministic alert path. Measure failure recall, long-run detection recall, false positives, event-to-message latency, duplicate incidents per run, missing-run detection, context completeness, recommendation utility, acknowledgment, time to triage, capacity, and operator load.
Operate pipeline monitoring as a NOC service
Maintain a monitored-pipeline registry with criticality, owner, schedule, threshold, baseline, dependencies, deterministic alert, agent rule, message destination, escalation, runbook, manual recovery, and retirement date. Reconcile registry scope with actual Fabric pipelines and workspace monitoring logs.
Dashboard expected versus actual runs, active long runs, failures, recoveries, source-log freshness, rule queries, operations, Teams delivery, acknowledgments, repeated alerts, action status, capacity, and unresolved incidents. Link every alert to the Monitoring Hub or run evidence by workspace, item, and JobInstanceId.
Use the agent to improve context, prioritize incidents, compare history, and recommend investigation steps—not to replace a deterministic critical-failure signal before its recall is proven. Review generated rules and KQL after pipeline, schema, status, schedule, workspace monitoring, instruction, runtime, identity, recipient, or Fabric changes.
Run a two-to-four-week pipeline monitoring assessment
- Select one workspace, representative critical pipelines, NOC owner, current alerts, incident history, response objectives, Teams destination, and measurable baseline.
- Enable and validate workspace monitoring; profile ItemJobEventLogs coverage, status values, run identity, timestamps, duration, trigger types, ingestion lag, retention, and permissions.
- Define the pipeline registry, severity model, schedules, duration thresholds, historical baselines, missing-run grace, duplicate policy, escalation, and manual controls.
- Choose deterministic notifications, Activator, Operations Agent, or a layered architecture for each condition based on criticality and reasoning value.
- Configure the Operations Agent, generate and inspect the playbook and queries, verify Teams, and build a labeled run-history evaluation set.
- Replay normal, failed, slow-completed, slow-in-progress, missing, retried, duplicate, stale-log, permission, delivery, load, and recovery cases.
- Run shadow mode, measure recall, false positives, latency, duplicates, triage utility, operator load, and capacity, then exercise incident and rollback runbooks.
- Deliver the architecture, pipeline registry, verified KQL and playbook, test suite, scorecard, dashboards, alerts, escalation matrix, runbooks, and scale, limited pilot, deterministic-only, or stop recommendation.
Frequently asked questions
Can Microsoft Fabric Operations Agent monitor data pipelines?
Yes. Microsoft documents an Operations Agent pattern that uses the workspace monitoring Eventhouse and ItemJobEventLogs table to detect pipeline conditions such as runs exceeding a duration threshold, then sends context through the Fabric Operations Agent Teams app.
How do I alert when a Fabric pipeline runs longer than 60 minutes?
Enable workspace monitoring, connect an Operations Agent to the monitoring Eventhouse, define an explicit 3,600,000 millisecond threshold against DurationMs, identify Data Pipeline records, include ItemName and run timestamps, generate and review the playbook, start the agent, install the Teams app, and validate with a controlled long run. Also test in-progress runs because completed-duration examples alone can alert only after completion.
Should I use failure notifications, Activator, or Operations Agent for pipeline alerts?
Use scheduled-run failure notifications for simple built-in failure email, Activator for deterministic event or KQL conditions and direct actions, and Operations Agent when monitored conditions need natural-language configuration, generated rules, contextual reasoning, or recommendations. Keep a deterministic control for critical failures and use the agent where its additional reasoning has measurable value.
What is ItemJobEventLogs in Microsoft Fabric?
ItemJobEventLogs is a workspace monitoring Eventhouse table containing job execution evidence. Microsoft documents fields used for pipeline monitoring such as Timestamp, ItemKind, ItemName, JobStatus, DurationMs, JobStartTime, JobEndTime, JobInvokeType, JobInstanceId, and WorkspaceName.
How long does a Fabric pipeline Operations Agent assessment take?
A focused assessment commonly takes two to four weeks for one workspace, representative pipeline portfolio, Operations Agent, Teams path, and NOC owner. It covers monitoring architecture, log completeness, thresholds and baselines, generated rules, replay, delivery, escalation, capacity, failure behavior, and operating runbooks.
Official implementation references
- Fabric pipeline failure, Activator, workspace, and Operations Agent alert patterns
- Enable workspace monitoring and its Eventhouse
- Workspace monitoring ItemJobEventLogs schema
- Operations Agent setup, playbook, rules, state, Teams, and actions
- Monitor Fabric pipeline runs in Monitoring Hub
Start with the pipelines whose late or failed completion affects a client, SLA, or downstream business process. Datrick can build the monitoring evidence layer, implement the right alert architecture, evaluate the agent, and operate the escalation path.