Skip to content
CNPEObservability and OperationsSingle nodeMedium

Incident response: roll back a bad release and fix an OOM-killed workload

Scenario cnpe-obs-004 · about 14 minutes · 10 points · 7 grader checks

Two workloads in namespace checkout are degraded: diagnose a failed rollout and an OOM-killed container from events and status, remediate both in place and record the findings in an incident ConfigMap.

  • incident-response
  • rollout
  • rollback
  • oomkilled
  • troubleshooting
  • observability

What you will practise

  • Read rollout history, events and container status to find the cause of an incident
  • Roll back a Deployment to its last working revision without recreating it
  • Fix a memory limit that makes a container OOM-kill

Start this lab

With the CLI installed and signed in, this command provisions a single node cluster on your Mac, applies the scenario setup and prints the task below.

goldenkube start cnpe-obs-004

Task

The task text exactly as the CLI shows it. Solve it on the cluster, then run goldenkube submit.

Namespace "checkout" is degraded after the last change: - the Deployment "payments" no longer completes its rollout; - the Deployment "cache" keeps restarting. 1. Return "payments" to its last working revision. Do not delete or recreate the Deployment; both replicas must run the working image. 2. Make "cache" run stably: its container needs a memory limit of exactly 128Mi (leave its command unchanged). 3. Record the incident in namespace "checkout" as the ConfigMap "incident-0001" with the keys: failed_revision the rollout revision number of "payments" that failed cause_payments one word: image, probe, quota or config cause_cache one word: oom, crash, probe or image

Requirements and details

Certification
CNPE · Certified Cloud Native Platform Engineer
Domain
Observability and Operations
Difficulty
Medium
Cluster topology
Single node
Cluster features
None beyond the base cluster
Recommended time
14 minutes
Points
10
Grader checks
7
Kubernetes version
1.35
Minimum CLI version
0.1.0
Scenario version
1
Published

Grading runs locally: 7 state checks read your cluster when you submit; the scenario passes when every check passes. Hints, the reference solution and the check definitions are available only inside the CLI, not on this page.

Install the CLICreate a free account

More Observability and Operations labs

CNPEObservability and OperationsMedium

Monitoring and alerting: ServiceMonitor, alert and recording rule

Get the shop API into the platform Prometheus: scrape it with a ServiceMonitor, alert on its 5xx ratio and precompute its request rate with a recording rule.

Single node · ~18 min · 10 points

  • prometheus
  • servicemonitor
  • prometheusrule
  • alerting
  • recording-rules
  • observability

goldenkube start cnpe-obs-001

CNPEObservability and OperationsMedium

Tracing: add an OTLP traces pipeline to the OpenTelemetry Collector

The platform OpenTelemetry Collector only handles metrics. Give it an OTLP traces pipeline with the right processors, expose it, and prove it with generated spans.

Single node · ~18 min · 9 points

  • opentelemetry
  • otel-collector
  • tracing
  • otlp
  • telemetrygen
  • observability

goldenkube start cnpe-obs-002

CNPEObservability and OperationsHard

Logging: collect one namespace's pod logs with an OpenTelemetry DaemonSet

Ship only the shop namespace's container logs: run the OpenTelemetry Collector as a DaemonSet with the filelog receiver on the node's pod log files and print the records with the debug exporter.

Single node · ~18 min · 9 points

  • opentelemetry
  • logging
  • filelog
  • daemonset
  • hostpath
  • observability

goldenkube start cnpe-obs-003

Independent community project, not affiliated with or endorsed by CNCF or The Linux Foundation. Lab results are practice feedback, not an exam score.