A database performance incident rarely arrives with one clean cause. Application latency can rise while CPU, storage latency, connection count, lock waits, replication lag, and one changed execution plan move at the same time. Static alerts report each symptom independently and leave the operator to reconstruct the sequence under pressure.
AI can accelerate that reconstruction, but it must remain grounded in database-engine evidence. The workflow should rank hypotheses, show supporting and conflicting observations, request the next discriminating check, and abstain when telemetry is incomplete. Diagnosis stays read-only until a DBA chooses an approved response.
Are senior DBAs rebuilding the same evidence bundle during every performance incident? Datrick can assess one recurring incident class, replay historical events, and build a supervised triage workflow around the current monitoring and database estate.
Define the database triage contract
| Triage element | Required evidence | Output |
|---|---|---|
| Impact and scope | Affected service, tenant, operation, region, database, latency, errors, throughput, start time, and customer impact. | Incident boundary, severity evidence, and affected workload. |
| Baseline delta | Comparable healthy period by time, traffic, release, season, maintenance, and resource shape. | What changed beyond expected workload variation. |
| Contention | Active sessions, wait classes, blockers, lock chain, deadlocks, transactions, queues, and connection pools. | Who is waiting on what, for how long, and with what blast radius. |
| Query and plan | Normalized fingerprint, runtime distribution, calls, rows, I/O, CPU, memory, plan, plan history, estimates, and statistics. | Regressed or newly dominant workload with plan evidence. |
| Platform | CPU, memory, cache, storage latency, IOPS, network, replication, checkpoint, log, temp, quota, and capacity. | Resource bottleneck, saturation, or downstream constraint. |
| Change and hypothesis | Deployment, schema, index, statistics, configuration, maintenance, failover, scaling, and traffic change. | Ranked hypotheses, confidence, alternatives, missing evidence, and next diagnostic. |
Database engines expose the raw evidence differently. PostgreSQL provides activity, wait-event, lock, and cumulative statistics views, with pg_stat_statements for normalized planning and execution statistics. MySQL Performance Schema records current, historical, and summarized statement and wait activity. SQL Server Query Store retains query, plan, and runtime history by time window to identify plan-related changes. Oracle AWR and ASH provide historical workload and active-session evidence. The triage layer must understand each engine rather than flatten every signal into generic CPU and latency.
Time-align symptoms, workload, and changes
A performance snapshot taken after the incident may miss the blocking session or plan transition that caused it. Preserve event time, collection time, timezone, sampling interval, aggregation window, and source freshness. Query-level data, infrastructure metrics, application traces, deployments, and incident actions should be aligned to one timeline.
Use query fingerprints and low-cardinality operation names where possible. Raw SQL can contain customer identifiers, literals, secrets, or regulated data. OpenTelemetry database semantic conventions explicitly address query-text sanitization. Collect only what the diagnostic requires, parameterize or redact literals, restrict access, and keep tenant boundaries intact.
Build a read-only evidence and hypothesis workflow
| Component | Responsibility | Production control |
|---|---|---|
| Engine collector | Captures activity, waits, locks, queries, plans, errors, replication, maintenance, and engine state. | Read-only identity, bounded queries, timeout, sampling budget, and version-aware adapters. |
| Platform collector | Captures host or service CPU, memory, storage, network, connection, quota, and scaling evidence. | Time synchronization, source freshness, and no uncontrolled high-cardinality labels. |
| Change and trace collector | Maps deployments, schema, configuration, application traces, service ownership, and customer impact. | Sanitized query context, tenant isolation, access control, and source links. |
| Baseline and delta engine | Compares the incident window with a valid healthy period and ranks abnormal workload components. | Comparable-period rules, seasonality, confidence, and explicit absence of baseline. |
| Hypothesis engine | Connects evidence into ranked causes, alternatives, contradictions, and next discriminating checks. | Evidence citations, abstention, model version, and no remediation authority. |
| DBA workspace | Presents timeline, impact, blockers, top queries, plan changes, platform limits, and safe diagnostics. | Human confirmation, immutable notes, role-based redaction, and incident-system integration. |
| Outcome register | Records accepted diagnosis, tests, action, verification, rollback, root cause, and recurrence. | Reviewed labels, audit history, and feedback separated from model-generated text. |
Separate diagnosis from remediation authority
Terminating a blocker, forcing a plan, adding an index, updating statistics, changing memory, scaling a tier, failing over, or restarting a service can restore performance or worsen the incident. The initial AI workflow should collect and interpret evidence without write access.
If remediation is later automated, route it through a versioned runbook. Check engine and workload eligibility, business impact, transaction state, replication health, resource headroom, change window, and rollback. Use a scoped identity, named approver, execution timeout, post-action verification, and automatic stop when observed behavior diverges from the runbook.
Evaluate whether triage reaches a correct and useful next step
- Evidence: incident-window coverage, source freshness, missing critical signal, timeline accuracy, and query-text redaction.
- Diagnosis: correct hypothesis in top one and top three, false root cause, calibrated confidence, abstention, and discriminating-test quality.
- Operations: time to first evidence bundle, time to viable hypothesis, DBA acknowledgement, duplicate investigation, handoff, and MTTR.
- Safety: unsupported recommendation, dangerous diagnostic, excessive collection load, tenant leakage, credential misuse, and unauthorized action.
- Learning: outcome-label completeness, recurring signature coverage, runbook reuse, post-change verification, and incident recurrence.
Replay historical incidents with the evidence that existed at each point in time. Do not give the model the final postmortem, later plan capture, or after-the-fact query sample while asking it to reproduce an earlier diagnosis. Include false alarms, workload spikes without incidents, maintenance periods, and incidents with overlapping causes.
A useful pilot should beat the current process on time to an evidence-backed next step without increasing unsafe recommendations. Root-cause agreement alone is insufficient if the workflow cannot tell the operator which observation would confirm or reject the hypothesis.
Pilot one engine and one recurring incident class
- Select one database engine and one painful incident class such as blocking, plan regression, connection exhaustion, storage latency, or replication lag.
- Define service impact, severity, evidence window, healthy baseline, diagnostic boundary, owner, and escalation path.
- Inventory engine views, monitoring history, query and plan retention, platform metrics, deployments, traces, incidents, and runbooks.
- Create a reviewed replay set with timelines, accepted hypotheses, discriminating checks, actions, outcomes, and known missing evidence.
- Build read-only collection, normalization, time alignment, baseline comparison, hypothesis ranking, and source-linked summaries.
- Evaluate historical incidents and non-incidents for diagnosis, evidence, safety, redaction, and operational usefulness.
- Run in shadow mode beside the DBA, capture disagreements and missing telemetry, and improve the evidence contract.
- Enable supervised incident notes or diagnostic tasks, monitor outcomes, and add remediation only through separately approved runbooks.
A bounded pilot can often reach supervised operation in two to six weeks when engine telemetry, query and plan history, platform metrics, changes, incidents, access, and DBA owners are available. Instrumentation gaps should become explicit readiness work rather than hallucinated certainty.
Frequently asked questions
What is AI database performance incident triage?
AI database performance incident triage assembles workload, latency, throughput, wait, lock, query, plan, connection, storage, replication, infrastructure, deployment, and business-impact evidence to rank plausible hypotheses and prepare safe diagnostic actions for a DBA. It accelerates investigation without replacing engine-native evidence or accountable judgment.
Can AI identify the root cause of a slow database automatically?
AI can rank likely causes and connect evidence, but a single symptom can support several explanations. High CPU might result from a plan regression, traffic growth, missing statistics, a deployment, or concurrent maintenance. The workflow should present alternatives, confidence, missing evidence, and discriminating tests rather than declare an unsupported root cause.
Which database signals should an incident triage workflow collect?
Collect service impact, workload and throughput, latency distributions, active sessions, waits, blockers and lock chains, query fingerprints, plans and plan history, errors, connection pools, CPU, memory, storage and I/O, replication, maintenance, configuration, schema and deployment changes, and a comparable healthy baseline. Collection must be engine-aware and time-aligned.
Should AI automatically terminate sessions or force query plans?
Not during initial triage. Session termination, plan forcing, index changes, configuration changes, failover, and scaling can have broad side effects. Keep diagnosis read-only first. Any remediation should use an approved runbook with eligibility checks, scoped identity, human authorization, verification, rollback, and a full audit trail.
How long does an AI database triage pilot take?
A pilot for one database engine and one recurring incident class can often reach supervised operation in two to six weeks when telemetry, query and plan history, changes, incidents, runbooks, access, and DBA owners are available. Missing historical baselines, unsafe query-text exposure, multi-tenant constraints, and sparse incident labels can extend the schedule.
Official implementation references
- PostgreSQL monitoring database activity
- PostgreSQL pg_stat_statements
- MySQL Performance Schema summary tables
- SQL Server Query Store
- Oracle Database performance tuning guide
- OpenTelemetry database span conventions
Start with one engine and the incident class that consumes the most senior DBA time. Datrick can assess telemetry, access, historical evidence, evaluation, workflow integration, safety boundaries, and operating ownership before proposing a pilot.