Incident Response in Platform Engineering: CNPA practice questions

Incident response is everything a team does from the moment something breaks for users until service is restored and the lessons are captured. Platform engineering touches it in two ways. The platform's own capabilities, a shared pipeline, a cluster, an identity service, can fail and affect many teams at once. And the platform gives every other team the tools to detect, diagnose and recover from incidents in their own services.

Detection starts with good alerting. The Prometheus documentation's advice is compact: keep alerting simple, alert on symptoms that are associated with end-user pain rather than every possible cause, have good consoles that help pinpoint the faulty component, and avoid pages where there is nothing to do. For online-serving systems it suggests alerting on high latency and error rates as high up the stack as possible, and paging on user-visible errors. A platform makes this easier by shipping standard dashboards and alert rules with its golden paths.

Diagnosis depends on observability data the platform already collects: metrics to see what changed and when, logs and traces to find where. Kubernetes events, which kubectl describe pod lists at the end of its output, are persisted in etcd and give high-level information about what is happening in the cluster. The Kubernetes debugging guide also walks through reading container logs and attaching an ephemeral debug container to a running Pod.

Recovery should be routine rather than heroic. Many incidents follow a change, so a fast, practiced rollback path is one of the most valuable mitigations. A Deployment keeps a rollout history, and kubectl rollout undo returns it to the previous revision or, with --to-revision, to a specific one; in a GitOps setup the same effect comes from reverting the commit and letting the agent reconcile. DORA measures this ability as failed deployment recovery time, the time it takes to recover from a deployment that fails and requires immediate intervention, next to change fail rate and deployment rework rate, the share of deployments that are unplanned and caused by an incident in production.

Learning closes the loop. DORA's generative culture capability, based on Westrum's typology, contrasts organizations where failure leads to scapegoating with those where failure leads to inquiry, and recommends blameless postmortems. The platform team should feed the resulting actions into its roadmap like any other user requirement.

Misconceptions include treating every alert as a page, searching for a single root cause (DORA's security guidance points out that in complex production systems a defect usually comes from a series of interacting factors), and judging the response by how rarely incidents happen rather than by how quickly users get back to normal.

At associate level, expect questions on symptom-based alerting, choosing the quickest safe mitigation, the meaning of DORA's recovery and failure metrics, and the purpose of blameless reviews.