Databricks can package an agent as an MLflow model, register it in Unity Catalog, deploy it to Model Serving, capture MLflow traces, run scorers, expose endpoint health metrics, govern data and tools, and attribute usage through system tables. Those controls do not decide who owns a wrong answer after a semantic-table change, a tool that returned success while the target system remained wrong, a Unity Catalog permission regression, a retrieval-quality decline, or an endpoint that is healthy while the business workflow is failing.
Datrick provides the ongoing operating layer for an agreed Databricks agent estate. Named engineers correlate endpoint metrics, service and build logs, MLflow traces, scorers, user feedback, models, prompts, tools, Vector Search, data pipelines, Unity Catalog identity and permissions, external-system audits, releases, cost, and business outcomes. Databricks Support remains the escalation path for platform defects. Datrick owns the client-specific diagnosis, containment, validation, communication, change, and prevention accepted in the service boundary.
Do you have Mosaic AI agents in production but no team accountable for turning an error spike, weak evaluation cohort, failed tool, stale retrieval index, permission denial, or cost anomaly into a verified business outcome? Start with one representative portfolio.
Define ownership across serving, traces, data, tools, identity, and outcomes
A production path can include a web or business application, API gateway, Model Serving endpoint, MLflow model, agent framework, foundation or external model, prompt, Unity Catalog function or external tool, Vector Search index, embedding endpoint, Delta tables, data pipelines, service credentials, user identity, AI Gateway policy, and a target system that creates the actual outcome. Name which layers the managed service owns, observes, changes, coordinates, or excludes.
Document workspaces and regions, applications, experiments, registered models, serving endpoints, tools, indexes, source tables and pipelines, identities, permissions, network controls, telemetry storage, data classes, support hours, severity, response and update targets, quality bars, change authority, budgets, fallback, and Databricks escalation.
Operate the complete Databricks AI agent production surface
| Service area | Managed responsibility | Boundary to define |
|---|---|---|
| Agent and endpoint | Availability, latency, request rate, errors, CPU and memory, scale behavior, service logs, build logs, dependencies, client failures, and fallback. | Supported endpoints and versions, traffic, concurrency, SLO, regions, client owner, capacity, and maintenance. |
| Tracing and evaluation | Trace coverage, spans, session IDs, inputs and outputs, scorers, evaluation cohorts, sampled production monitoring, human feedback, regressions, and alerting. | Experiments, retention, sensitive fields, sampling, quality thresholds, evaluator budget, reviewer, and business KPI. |
| Retrieval and data | Source freshness, pipeline health, schema, chunks, embeddings, Vector Search index sync, retrieval relevance, permissions, lineage, and answer grounding. | Authoritative sources, freshness SLO, index type, data owner, quality bar, masking, deletion, and recovery. |
| Tools and actions | Unity Catalog functions and external tools, contracts, authentication, inputs and outputs, timeout, retry, idempotency, side effects, approvals, and target reconciliation. | Allowed tools, principal, scopes, action authority, approval, target SLO, rollback, and emergency disable path. |
| Identity and governance | Endpoint ACLs, service principals and service credentials, Unity Catalog privileges, network policy, AI Gateway permissions, rate limits, payload logging, and audit evidence. | Run-as model, least privilege, secret owner, on-behalf-of-user behavior, data policy, retention, access review, and incident route. |
| Release and change | Agent model, code, prompt, dependencies, model endpoint, tools, indexes, data, policies, test gates, rollout, traffic, rollback, and platform change. | Source of truth, environments, change authority, freeze, approval, canary, compatibility, and acceptance evidence. |
| Cost and value | Model Serving DBUs, launch costs, token usage where enabled, evaluator load, Vector Search, pipelines, retries, loops, support effort, attribution, anomaly, and outcome. | Commercial owner, tags, budget, alert threshold, business KPI, forecast, unit economics, and optimization authority. |
Separate endpoint health, trace completeness, and business quality
Model Serving exposes infrastructure signals such as latency, request rate, error rate, CPU, and memory. Ephemeral service logs help immediate diagnosis; build logs explain environment creation and dependency failures; OpenTelemetry can persist logs, metrics, and spans into Unity Catalog tables in supported regions; and AI Gateway inference tables can retain requests and responses for supported endpoint types. These are different evidence sources with different retention and coverage. Validate them before an incident.
MLflow Tracing records inputs, outputs, intermediate steps, and metadata. Production monitoring can run built-in or custom scorers against a configurable sample of incoming traces, and the same scorers can be reused from development. Sampling controls cost but creates blind spots. Multi-turn judges need consistent session IDs, and production monitoring is currently a beta feature that workspace administrators can control. Define deterministic checks for consequential actions, targeted cohorts for critical use cases, and human review where an LLM judge is not sufficient.
A green endpoint does not prove retrieval freshness, tool correctness, or the final business state. Connect technical alerts to an operating SLO: the user received an accepted answer, the agent used an authorized source, the tool acted as the intended principal, the target system holds the expected state, and the cost remained within the agreed unit economics.
Distinguish serving, quality, retrieval, tool, permission, release, and cost failures
| Symptom | Evidence to reconcile | Safe containment | Permanent control |
|---|---|---|---|
| Endpoint errors or latency spike | Endpoint health, request volume, service and build logs, deployment status, dependencies, autoscaling, client timeout, network policy, and recent change. | Route to fallback or human path, reduce traffic, restore accepted endpoint configuration, preserve evidence, and communicate impact. | SLO alerts, load test, capacity baseline, dependency lock, canary, fallback, and release rollback. |
| Wrong or degraded answer | MLflow trace and spans, prompt and model, scorer results, evaluation cohort, user feedback, retrieved context, source freshness, tool output, and expected answer. | Narrow use cases, route affected cohort to human review, restore accepted version or source, and correct impacted decisions. | Representative evaluation, deterministic assertions, human feedback loop, source-quality gate, cohort monitoring, and release threshold. |
| Retrieval misses or cites stale data | Source tables and pipelines, schema, freshness, index sync, embeddings, filters, permissions, retrieved chunks, relevance scores, lineage, and deletion state. | Disable affected source or path, use an accepted snapshot, route to manual lookup, rebuild or resync safely, and label uncertainty. | Freshness SLO, pipeline and index alerts, retrieval evaluation, schema contract, source ownership, and recovery test. |
| Tool fails or changes the wrong state | Trace span, function or API version, inputs, outputs, principal, credential, scopes, retries, external audit log, idempotency key, and target state. | Disable tool, block automatic replay, revoke or narrow credential if needed, isolate records, use manual fallback, and reconcile side effects. | Contract and negative tests, least privilege, approvals, idempotency, target-state verification, rate limit, and emergency disable path. |
| 403 or Unity Catalog access regression | Endpoint identity, service principal or credential, model EXECUTE access, USE CATALOG and USE SCHEMA, function and table grants, policy, network event, and audit log. | Stop sensitive action, restore only the approved grant or principal, preserve denied-access evidence, and use a controlled manual route. | Permission-as-code, least privilege, pre-release authorization tests, access review, owner mapping, and drift alert. |
| Unexpected spend or agent loop | system.billing.usage, endpoint and custom tags, token usage where enabled, requests, scorer sampling, repeated spans, retries, Vector Search and pipeline use, and outcomes. | Stop loops, cap traffic and iterations, reduce evaluator sampling if safe, scale down noncritical paths, and notify the commercial owner. | Per-endpoint attribution, budgets and alerts, iteration limits, anomaly detection, load forecast, quality-cost review, and unit economics. |
Safe replay is a business decision, not merely resending an inference request. Before retrying, determine whether an external tool already created, changed, approved, sent, paid, assigned, or closed a record. Use idempotency, target-state reconciliation, and explicit approval for consequential actions.
Govern identity and telemetry as production data
Agent deployment requires access to registered models and dependent resources. Unity Catalog privileges such as USE CATALOG, USE SCHEMA, EXECUTE, CREATE MODEL, or CREATE MODEL VERSION apply at different lifecycle steps. Model Serving endpoint ACLs, service principals, service credentials, external OAuth, network policies, and tool-specific authorization add further boundaries. Record the effective runtime principal instead of assuming the notebook author, deployer, and serving endpoint act as the same identity.
Traces, inference payloads, feedback, and OpenTelemetry records can contain prompts, retrieved documents, user identifiers, tool arguments, and business outputs. Set storage location, Unity Catalog access, masking, retention, deletion, and export rules before enabling broad logging. Do not turn observability into an uncontrolled copy of sensitive production data.
Release the complete agent system, not only the registered model
Log and register the agent model in Unity Catalog, but preserve a release manifest that also identifies code and dependencies, prompt versions, foundation or external model, tools and schemas, Vector Search indexes, source tables and pipelines, identities and grants, network and AI Gateway policies, evaluator versions, telemetry configuration, client version, approvals, rollout, and rollback. Databricks notes that existing Model Serving images are not automatically security-patched because changing them can destabilize production; a new model image is created from a new model version. Patch posture therefore belongs in the release process.
Before production, run representative and negative evaluations, test permissions and tool side effects, verify trace and alert coverage, load test the endpoint, canary a limited cohort, reconcile target outcomes, and preserve the accepted prior configuration. Traffic splitting and fallback availability depend on endpoint type, so verify the actual deployment rather than assuming every AI Gateway control applies.
Onboard through inventory, baselines, controlled failures, and shadow operations
- Inventory: workspaces, regions, applications, experiments, registered models, endpoints, prompts, tools, indexes, sources, pipelines, identities, policies, clients, and outcomes.
- Responsibility: define supported layers, SLOs, severity, access, data handling, quality, change authority, budget, dependencies, fallback, Databricks escalation, and exclusions.
- Baseline: measure endpoint health, trace coverage, quality scores, retrieval relevance and freshness, tool and target-state success, access denials, cost, and incidents.
- Controls: validate telemetry retention, scorers and sampling, session IDs, permissions, credentials, safe replay, release evidence, rollback, cost attribution, and alerts.
- Exercise: rehearse an endpoint outage, wrong answer, stale index, duplicate side effect, denied grant, missing trace, version regression, loop, cost spike, and platform incident.
- Transition: operate in shadow, close or accept material gaps, publish runbooks and escalation routes, and accept the steady-state support scope.
Start with the Databricks agents that already influence customer, financial, operational, compliance, or workforce decisions. Datrick can define the operating boundary, close material control gaps, and transition one portfolio into managed support.
Request a Databricks AgentOps reviewOfficial references and adjacent operating guides
- Evaluate and monitor AI agents with MLflow
- Trace agents deployed on Databricks
- Monitor GenAI applications in production
- Monitor Model Serving quality and endpoint health
- Persist Model Serving telemetry to Unity Catalog
- Monitor Model Serving costs
- AI Gateway controls for serving endpoints
- White-label AI agent managed support for MSPs
- Managed human evaluation for AI agents
Frequently asked questions
What is included in Databricks Mosaic AI Agent Framework managed services?
A defined service can include MLflow trace coverage, quality scorers and evaluation, Model Serving endpoint health, service and build logs, OpenTelemetry or inference-table evidence, retrieval and tool behavior, Unity Catalog permissions, incidents, release control, cost attribution, runbooks, and reporting. The exact scope depends on the applications, endpoints, data, tools, integrations, access, support hours, and accepted responsibility boundary.
How do you monitor a Databricks AI agent in production?
Monitor both infrastructure and application quality. Endpoint metrics cover latency, request rate, errors, CPU, and memory. MLflow Tracing records inputs, outputs, intermediate steps, and metadata. Production monitoring can sample traces and run tested scorers, while durable telemetry can be persisted to Unity Catalog tables where supported. Alerts must connect those signals to user impact and the expected business outcome.
Does Databricks production monitoring evaluate every agent conversation?
Not necessarily. MLflow production monitoring uses configurable sampling, so coverage and evaluator cost must be selected deliberately. Multi-turn judges also require consistent session IDs so traces can be grouped into conversations. Critical workflows may need deterministic checks, targeted evaluation cohorts, human review, and business-state reconciliation in addition to sampled LLM judges.
How do you control Databricks AI agent releases and rollback?
Register the agent as an MLflow model in Unity Catalog, preserve a release manifest, test representative and negative cases, validate permissions and dependent resources, deploy through a controlled endpoint change, canary traffic, and keep an accepted prior version and rollback procedure. A model version alone is not enough because prompts, tools, vector indexes, data, policies, dependencies, and external APIs can also change behavior.
How long does Databricks AI agent support onboarding take?
A focused onboarding commonly takes two to four weeks for a representative production portfolio. It covers application and endpoint inventory, trace and evaluation baselines, tools and retrieval, Unity Catalog identity and permissions, incidents, releases, cost attribution, runbooks, controlled failure exercises, and acceptance of the steady-state operating scope.
Operating AI agents directly on governed enterprise data?
Compare the Snowflake Cortex Agents operating boundary