Runbooks reduce operational variance by turning repeated work into an explicit procedure. The procedure may diagnose a service, collect evidence, restart a component, rotate capacity, restore a backup, update a resource, or coordinate a multi-step recovery. Traditional automation executes predetermined steps; AI can help interpret context and choose among approved procedures.
The risk appears when interpretation and execution are treated as one opaque action. A plausible diagnosis does not authorize a production change. A correct runbook does not guarantee that its parameters, target, identity, timing, or current version are safe. The implementation needs separate decision, policy, approval, execution, and verification boundaries.
Do operators repeatedly gather the same evidence and manually launch the same approved procedure? Datrick can assess one runbook, define its control contract, build a supervised workflow, evaluate failures, and hand over production operations.
Grant autonomy one action class at a time
| Level | AI-assisted behavior | Execution boundary |
|---|---|---|
| Retrieve | Find the relevant approved runbook and explain why it matches the event. | No infrastructure write access. |
| Prepare | Collect evidence, resolve identifiers, fill typed parameters, and identify missing context. | Operator reviews inputs and proposed target. |
| Simulate | Run preflight, policy, dependency, impact, and dry-run checks where supported. | No consequential production change. |
| Approve and execute | Route the plan to authorized people and invoke the versioned automation after approval. | Policy and identity, not the model, grant permission. |
| Bounded auto-remediation | Execute a tested, reversible action when explicit conditions and thresholds match. | Allowlist per service, action, environment, tenant, and severity. |
Start at retrieval or preparation. Move an action to automatic execution only after its deterministic runbook, permissions, preflight, verification, rollback, and incident response have been tested independently. A model should be able to abstain without pressuring the operator into a guess.
Define an executable runbook contract
A production runbook needs an owner, version, purpose, supported environments, trigger conditions, prerequisites, typed parameters, target selector, required identity, permissions, protected actions, approval rule, concurrency rule, timeout, steps, expected observations, success criteria, rollback, verification, fallback, escalation, evidence retention, and review date.
Natural-language instructions can orient people, but executable steps should call versioned scripts, APIs, infrastructure automation, or platform-native actions. Do not let a model invent shell commands and run them with broad credentials because the intent sounded familiar.
Separate reasoning from execution
| Component | Responsibility | Control |
|---|---|---|
| Event and context collector | Receives approved alerts, tickets, service state, changes, and operator requests. | Resolve tenant and environment first; reject stale or conflicting context. |
| Runbook catalog | Stores approved versions, metadata, owners, parameters, policies, and tests. | Only published versions are executable. |
| AI decision support | Ranks runbooks, summarizes evidence, proposes parameters, and states uncertainty. | No direct production credential or unrestricted tool access. |
| Policy and preflight | Validates target, identity, permission, environment, severity, maintenance window, and impact. | Deterministic deny rules override any model recommendation. |
| Approval workflow | Shows plan, evidence, diff, risk, rollback, and approver requirements. | Record named approval or rejection; expire stale approvals. |
| Execution engine | Runs the approved, versioned automation with scoped identity. | Idempotency, concurrency lock, timeout, secret isolation, and cancellation. |
| Verifier and recorder | Checks expected state, triggers rollback or escalation, and records outputs. | Independent success criteria and immutable audit evidence. |
AWS Systems Manager Automation supports explicit approval steps that pause a runbook until designated principals approve or reject it. Azure Automation recommends managed identities for protected resources and warns that webhook URLs contain security tokens that must be treated like passwords. These are baseline platform controls, not optional additions after the AI layer is built.
Protect credentials, tenants, and targets
The orchestrator should pass short-lived, least-privilege credentials to a bounded execution environment. The model does not need to see secrets. Resolve client, subscription, account, cluster, namespace, region, service, and resource identifiers deterministically, then show them clearly to reviewers.
For MSP delivery, isolate client context, logs, runbook catalogs, credentials, approval groups, and execution destinations. A valid action in one tenant can be catastrophic in another. Maintain client-specific policies and test that retrieval, memory, and error messages cannot leak operational data across accounts.
Evaluate trajectories and side effects
Test more than the final state. Record which evidence the agent retrieved, which runbook and version it selected, proposed parameters, confidence, policy results, approvals, tool calls, outputs, retries, verification, and rollback. Create cases with stale alerts, duplicate events, missing targets, conflicting changes, revoked access, unavailable tools, partial execution, timeout, and misleading log text.
- Decision quality: correct runbook, parameter accuracy, abstention, protected-action handling, and escalation.
- Execution: success, idempotency, duplicate suppression, concurrency safety, timeout, and cancellation.
- Risk: unauthorized attempts, wrong target, cross-tenant access, destructive side effects, secret exposure, and policy bypass.
- Recovery: verification accuracy, rollback success, manual fallback, and evidence preserved after failure.
- Value: operator preparation time, mean time to restore, repeat incidents, review burden, and cost per execution.
Evaluate each runbook version before release and after material changes to infrastructure, APIs, permissions, or policies. Production monitoring should detect drift in selection, overrides, failures, execution time, and downstream health.
Pilot one mature, reversible runbook
- Select one frequently used procedure with an owner, stable steps, observable success, and a tested rollback.
- Baseline preparation time, execution time, failure, escalation, recovery, and repeat work.
- Convert the procedure into a versioned contract with typed parameters and explicit policy.
- Create evaluation cases across valid, ambiguous, stale, unauthorized, and failure conditions.
- Implement retrieval and preparation without execution, then test operator review and abstention.
- Add preflight, approval, scoped execution, verification, logs, rollback, and manual fallback.
- Run in a test or low-risk environment before supervised production use.
- Review outcomes and authorize any increase in autonomy per action and environment.
A bounded pilot can often reach supervised testing in two to six weeks when a mature runbook, test environment, identity, APIs, and owners are available. If the current procedure is tribal knowledge or cannot verify and reverse its own effects, stabilize the runbook before adding AI.
Frequently asked questions
What is AI runbook automation for IT operations?
AI runbook automation uses operational context to recommend an approved runbook, prepare or validate parameters, collect evidence, and coordinate review and execution. The reliable version keeps executable actions deterministic, restricts identity and scope, requires policy and approval where needed, records every step, and verifies the result.
Should an AI agent execute infrastructure changes automatically?
Only narrowly defined, tested, reversible, and authorized actions should be candidates for automatic execution. High-impact, destructive, cross-tenant, security-sensitive, ambiguous, or novel actions should require human approval or remain advisory. Autonomy should be granted per action and environment, not as one global permission.
How do you make AI runbook execution safe?
Use an approved runbook catalog, typed parameter schemas, environment and tenant validation, least-privilege identities, allowlists, protected-action approval, dry-run or preflight checks, concurrency controls, timeouts, idempotency, rollback, result verification, immutable logs, and a tested manual fallback.
How is AI runbook automation evaluated?
Evaluate runbook selection, parameter accuracy, abstention, protected-action handling, execution success, verification quality, rollback, unauthorized attempts, side effects, recovery, operator overrides, time saved, mean time to restore, repeat incidents, cost, and behavior under stale data or tool failures.
How long does an AI runbook automation pilot take?
A pilot for one mature runbook and one non-production or low-risk environment can often reach supervised testing in two to six weeks when APIs, identities, test fixtures, and owners are available. Incomplete procedures, weak rollback, production-only testing, multi-client boundaries, and complex approvals can extend the schedule.
Official implementation references
- AWS Systems Manager automation approvals
- AWS Systems Manager Automation runbooks
- Azure Automation managed identity
- Azure Automation webhook security
Start with one mature runbook, its rollback, and five failure cases. Datrick can assess control boundaries, identities, approvals, evaluation, execution, verification, and operating ownership before proposing a pilot.