Triage and evidence
gathering.

To practice, use this illustrative dashboard snapshot: 68 of 1,000 checkout requests failed in the five minutes after the alert (6.8%). In the comparable five-minute window before it, 2 of 1,000 failed (0.2%). The deployment completed between those windows. These figures are authored teaching data; they are not observations from a live service.

Ask which users, routes, regions, and request types are affected. Is latency rising along with failures? Are attempts counted once, or can client retries inflate the denominator? Does the failure prevent purchase, or is a noncritical status panel slow? Impact determines whether to mitigate immediately and what comparison is meaningful.

Align the alert, first failing request, deployment start and completion, dependency alarms, traffic shifts, and any operator changes. Normalize timestamps and time zones. Preserve request IDs, release identifiers, and the original metric window so later comparisons refer to the same events.

Timelines expose gaps: an error could precede the rollout; a dependency slowdown may begin first; or the failure may appear only as traffic shifts to new instances. A timeline supports a causal story, but adjacency alone does not establish one.

Do not collect evidence without a question. Write down a few explanations that account for the same report, then state what each predicts. Choose checks that could make one explanation less likely as well as more likely.

A version comparison by route may tell you whether the new build is implicated; dependency traces may show a shared downstream failure. Check the denominator and traffic mix before interpreting either. A chart aggregated across versions can hide a clear split, while a few handpicked traces can exaggerate one.

Good troubleshooting names the question, the expected observations, the owner, and the next decision. Prefer read-only queries and narrowly scoped comparisons first. Preserve raw events before dashboards roll up or logs expire. Change one variable at a time when the service is stable enough; during a severe incident, choose the fastest safe mitigation and document the tradeoff.

Empirical thinking does not mean waiting for perfect proof while users are harmed. It means choosing an action for an explicit reason, considering its downside, and checking whether the outcome matches the prediction. Pausing rollout can stop exposure to a suspected version. Rolling back may restore service, but can also remove a needed fix or leave incompatible data changes. A feature flag, traffic shift, or dependency failover may be safer in a particular system.

Before acting, say what impact warrants the change, how to reverse it, which signal will show improvement, and who is watching. Record the pre-change state and decision time. Afterward, verify user-facing outcomes; “the dashboard turned green” is not enough if the affected requests still fail.

Keep a short incident record with impact and window, verified observations, current hypotheses, actions and their results, remaining risks, evidence links, and a named next owner. Separate mitigation from the later root-cause analysis. Once service is stable, test the causal explanation against the timeline and look for evidence that could disconfirm it.

For the example: the failure rate increased in the post-deployment window; the deployment is a candidate cause; no version-sliced evidence has yet established that link. The next step is to compare outcomes by version, route, and region while checking dependency signals. If impact worsens before that check completes, act on the agreed mitigation threshold and continue gathering evidence.

References: Google SRE, Effective Troubleshooting and Managing Incidents; Google’s Incident Management Guide. Accessed 2026-10-01.