Production incident at 2 a.m.
At 2 a.m. the hard part is finding what changed and deciding whether rollback is safe — fast, without losing the evidence trail.
The work already crosses these systems:
From scattered checks to one governed run.
Today
An on-call SRE gets paged, opens ServiceNow, reviews CI/CD deploy history, checks GitHub commits, then files a Jira follow-up and updates the Confluence postmortem alone.
With NopsAI
An alert-triggered pipeline collects incident, change, deploy and commit context, runs investigation goals bounded by the team’s rollback guardrail, and executes a pre-approved rollback behind the production approval gate the pipeline declares.
A controlled incident workflow instead of a 2 a.m. scavenger hunt.
The run, step by step.
Deterministic work first, reasoning inside the tools and written policies the pipeline bound to it, and a named human at every gate its author placed.
Trigger
The alerting system calls an authenticated external trigger bound to the production scope.
Collect context
Recent deploys, open changes and commit diffs are gathered in parallel, alongside runbook/platform/payments-oncall, which the step binds as required.
Verify state
Deterministic health checks confirm which instances are affected and whether the last deploy is the suspect.
Reason
A Production SRE agent role correlates the signals and proposes rollback or a targeted mitigation. guardrail/platform/rollback-safety bounded the decision: it forbids rolling back across a schema migration, so the model proposes a forward fix instead and says why.
Approve
The pipeline puts a gate before anything touches production, so a named approver from the platform/prod team decides.
Execute and record
The pre-approved rollback runs on an eligible runner; the postmortem starts from the run record, not memory.
What the run leaves behind.
The useful part is not only the automation. It is repeatability with proof.
Trigger and subject
What started the run and which effective identity it ran as.
Authorization snapshot
Which resources were checked, and which decision each check returned.
Knowledge in scope
The guardrails, policies and runbooks that bounded each decision, stored as text with the run, plus the governance level that applied.
Tool and AI activity
Every tool call, the profile that allowed it, and the model usage it consumed.
Task history and timings
Every task, its state transitions, its duration and the logs it produced, with known secret values masked.
Approvals and outputs
Who approved, when, and the deliverables the run produced.
← Sprint planning for the next two weeks
Planning starts with manual research instead of decisions. Blockers surface late and half-ready stories enter the sprint.
QA is blocked on unclear requirements →
QA loses time reconstructing intent. Testing can start from the wrong assumption and clarification cycles cost a day or more.
Map this workflow against your controls.
Bring the trigger, the tools it touches, the approvers, the runtime boundary and the evidence you need to keep.