Microsoft Foundry can host agents, connect models and tools, capture OpenTelemetry traces in Azure Monitor Application Insights, run evaluations, and surface operational and quality signals in an Agent Monitoring Dashboard. It can also centralize compatible custom agents running outside Foundry through registration, AI Gateway, shared telemetry, and continuous evaluation. These capabilities create an AgentOps foundation; they do not assign an owner to a wrong answer, failed tool, missing span, unsafe action, cost anomaly, or production release.
Datrick provides an ongoing operating layer for an agreed Foundry and custom-agent estate. Named engineers reconcile runtime signals, evaluator scores, user reports, identities, retrieval, tools, dependencies, changes, cost, and business outcomes. The service can work directly with an enterprise team or as an L2/L3 back line under an MSP's brand. Microsoft product support remains part of the escalation path for platform defects; managed operations owns client-specific diagnosis, change, validation, communication, and prevention.
Do you have traces and dashboards but no team accountable for turning them into safe, verified production action? Start with one project and the agents users already depend on.
Define a service boundary across platform, application, model, data, and business outcome
Foundry is one layer in a production system. The complete path can include channels or applications, API Management or AI Gateway, agent orchestration, models, retrieval, vector or search services, MCP servers and functions, application services, identities, Key Vault, databases, queues, Azure Monitor, and downstream systems. The managed service must name which layers it owns, observes, changes, coordinates, or excludes.
Document supported subscriptions, projects, agents, versions, models, regions, applications, business hours, severity, response and update targets, monitoring sources, access, data handling, change authority, vendor escalation, client dependencies, and exit criteria. Keep response, containment, restoration, permanent resolution, quality improvement, and business acceptance separate.
Operate the complete Foundry agent production surface
| Service area | Managed responsibility | Boundary to define |
|---|---|---|
| Runs and dependencies | Invocation success, latency, timeout, queue, exception, retry, dependency, regional, quota, and degraded-fallback behavior. | Supported agents and services, SLO, traffic profile, hours, vendor dependency, and restoration authority. |
| Quality and safety | Continuous and scheduled evaluation, sampled human review, evaluator calibration, groundedness, task adherence, safety, and regression. | Use cases, ground truth, metrics, thresholds, sample rate, reviewer, red-team scope, and acceptable residual risk. |
| Retrieval and data | Source freshness, ingestion, permissions, chunking, index, retrieval, citation, conflict, data cutoff, and answer evidence. | Authoritative sources, owner, region, retention, access model, refresh objective, and deletion. |
| Tools, MCP, and actions | Selection, schema, input, output, authentication, authorization, timeout, side effect, approval, idempotency, rollback, and audit. | Allowed tools, least privilege, action owner, human approval, monetary or data impact, and emergency disable path. |
| Identity and security | User, workload, managed identity, secrets, RBAC, connections, network, prompt injection evidence, anomalous behavior, and access lifecycle. | Security authority, segregation, private connectivity, credential rotation, incident route, and evidence retention. |
| Release and lifecycle | Agent, prompt, model, tool, retrieval, evaluator, policy, code, infrastructure, and dependency change; promotion; rollback; retirement. | Source of truth, environments, approval, test gate, freeze window, emergency path, and acceptance evidence. |
| Cost and capacity | Token, model, tool, search, compute, storage, gateway, Application Insights, Log Analytics, evaluator, and support consumption. | Attribution, budget owner, rate and quota, unit economics, anomaly threshold, retention, and optimization authority. |
Build observability that can detect behavior, not only infrastructure failure
Instrument the complete request path with a correlation ID and OpenTelemetry semantic conventions. Capture the agent and model version, timing, tokens, exceptions, retrieval operations, tool calls, dependencies, and outcome evidence needed for diagnosis. Microsoft Foundry stores traces in the connected Application Insights and Log Analytics configuration; permissions, retention, sampling, ingestion delay, and billing therefore belong in the service design.
Treat trace data as production customer data. Microsoft states that it can include prompts, model inputs and outputs, tool arguments, intermediate steps, timestamps, latency, token usage, and errors. Do not place secrets or credentials in prompts, tool arguments, or span attributes. Minimize or redact personal and sensitive data before collection, restrict access, and test deletion and retention.
Use continuous evaluations on sampled production responses for quality and safety metrics that match the business task. Add deterministic evaluators where the requirement can be expressed exactly and calibrated model-based or human review where judgment is required. Monitor evaluator coverage, version, failure, cost, false positives, and drift. A dashboard score is not trustworthy when the sampled traffic, ground truth, or evaluator changed unnoticed.
Reconcile traces, evaluations, tools, identity, deployment, and outcome
| Symptom | Evidence to reconcile | Safe containment | Permanent control |
|---|---|---|---|
| No trace or missing spans | Application Insights connection, instrumentation, framework support, project and resource RBAC, traffic, sampling, exporter, network, ingestion delay, and semantic conventions. | Preserve application logs and correlation IDs; route critical workload to a known observable path. | Telemetry health check, synthetic run, exporter alert, schema test, and completeness SLO. |
| Wrong or unsafe output | Request, identity, trace, model, prompt, retrieval, tool results, evaluator scores, version, data cutoff, expected result, and recent change. | Disable affected capability, narrow users, add human review, switch model or version, or rollback. | Ground-truth regression suite, continuous evaluation, release gate, source ownership, and sampled human audit. |
| Tool or MCP failure | Selection, schema, arguments, OAuth or managed identity, network, endpoint, permission, response, timeout, retry, side effect, and target state. | Disable tool, prevent unsafe replay, use manual runbook, revoke access, or require approval. | Contract tests, least privilege, idempotency, circuit breaker, tool SLO, expiry alert, and rollback. |
| Latency or token spike | Run topology, model, prompt and context, retrieval, tool count, retries, loops, cache, queue, quota, dependencies, and evaluator load. | Apply budget and step limits, stop loop, route to fallback, reduce traffic, or disable noncritical evaluations. | Per-stage budgets, anomaly alert, representative load test, model routing, cache, and unit economics. |
| Unauthorized or overprivileged action | User and workload identity, token scopes, RBAC, connection, tool policy, network, approval, target audit, trace data, and recent access change. | Disable action, revoke identity or secret, isolate agent, preserve evidence, and invoke security response. | Least privilege, user attribution, policy enforcement, negative tests, access review, and credential lifecycle. |
| Release regression | Agent, prompt, model, evaluator, tool, retrieval, code, infrastructure, dependency, deployment record, canary traffic, and rollback state. | Stop rollout, restore last known good version, protect affected users, and hold further changes. | Immutable version record, evaluation and load gates, canary, approval, compatibility tests, and rehearsed rollback. |
Preserve trace and span IDs, user or workload identity, timing, input, output, tool results, data version, evaluator version, cost, and deployment state before remediation. Apply read-only diagnostics first. Validate recovery against the original failure, representative normal cases, denied identities, relevant safety tests, latency, cost, and downstream state.
Control releases and costs as one operating decision
Version agent definitions, prompts, models, tool and MCP contracts, retrieval configuration, evaluators, policies, code, infrastructure, and dependencies. Promote through controlled environments with automated and human evaluation, security tests, load evidence, approval, canary traffic, monitoring, and rollback. A model alias or external tool can change behavior without an application-code change, so dependency state belongs in the release record.
Attribute model tokens, cached tokens where applicable, tools, search, compute, storage, gateways, tracing, Log Analytics ingestion and retention, continuous evaluations, red-team runs, and support effort to agent, version, client, workload, and successful outcome. Reducing telemetry or evaluation can lower cost while increasing incident and regression risk; test the operational consequence before changing sampling or retention.
Onboard through inventory, observability proof, failure exercises, and shadow operations
- Define direct, co-managed, or white-label responsibilities; stakeholders; service levels; communication; escalation; and commercial boundaries.
- Inventory subscriptions, projects, agents, registrations, applications, models, deployments, knowledge, tools, MCP servers, identities, networks, telemetry, evaluators, and owners.
- Validate least-privilege access, connections, secrets, private paths, trace completeness, redaction, retention, sampling, dashboards, alerts, and cost attribution.
- Baseline representative tasks, quality, safety, latency, tokens, tool success, retrieval, evaluator behavior, incident history, and unresolved risks.
- Build the service map, RACI, severity model, runbooks, evaluation suite, release path, rollback, communication templates, dashboard, and monthly report.
- Exercise missing telemetry, wrong output, tool failure, access violation, latency or token spike, release rollback, vendor escalation, and client update.
- Resolve or explicitly accept critical gaps such as no correlation, sensitive traces, unowned credentials, uncalibrated evaluators, or no rollback.
- Run a shadow period and accept only the steady-state responsibilities supported by access, evidence, authority, and staffing.
Frequently asked questions
What is included in Microsoft Foundry AI agent managed services?
A managed service can include trace and metric monitoring, continuous and scheduled evaluation, quality and safety review, tool and MCP support, identity and connection operations, incident response, model and dependency changes, release and rollback, token and telemetry cost governance, runbooks, reporting, and an improvement backlog. Exact coverage depends on the architecture, service tier, available telemetry, access, and accepted responsibility boundary.
Can Microsoft Foundry monitor agents running outside Foundry?
Microsoft documents a path for registering custom agents, routing them through AI Gateway, and sending OpenTelemetry traces to the Application Insights resource connected to a Foundry project. This can centralize monitoring and continuous evaluation for compatible agents running elsewhere. The integration, semantic conventions, data handling, completeness, latency, and cost still require validation.
How do you monitor a Microsoft Foundry agent in production?
Instrument end-to-end agent runs with correlation IDs and OpenTelemetry, send traces to an approved Application Insights resource, monitor latency, tokens, errors, tool calls, dependencies, run success, and evaluator scores, configure quality and security checks, and reconcile telemetry with user reports and business outcomes. Monitor the monitor itself for ingestion, retention, permissions, sampling, and missing spans.
How do you troubleshoot a Microsoft Foundry agent incident?
Preserve the request, user or workload identity, agent and model version, trace and span IDs, tool arguments and results, retrieval evidence, dependency state, timing, cost, error, and recent changes. Check service health, authentication, authorization, quota, telemetry, tools, data sources, deployment, model behavior, and downstream outcomes. Contain unsafe actions before changing production and validate recovery against representative tasks.
How long does Foundry agent managed-service onboarding take?
A focused onboarding commonly takes two to four weeks for one project and a representative agent portfolio. It covers inventory, responsibility, access, identity, telemetry, evaluation and cost baselines, open incidents, security, release paths, runbooks, controlled failure exercises, a shadow period, and explicit acceptance of the steady-state scope.
Official implementation references
- Microsoft Foundry Agent Monitoring Dashboard
- Set up tracing in Microsoft Foundry
- Microsoft Foundry tracing and data handling
- Configure tracing for AI agent frameworks
- Monitor, evaluate, and operate multi-agent AI solutions in Azure
Begin with the Foundry project where agents already have production users, tools, data access, or business consequences. Datrick can stabilize the highest risks, prove observability and evaluation, and accept a defined recurring support scope.