Observability: KCSA practice questions

Observability is how operators see what a cluster and its workloads are doing, and in security work it is how unusual behavior gets noticed. Three signal types matter here: logs, metrics and events, each with its own path through the cluster.

Container logs come from what applications write to standard output and standard error. The kubelet and container runtime capture these streams on the node and handle rotation, and kubectl logs reads them; -c selects one container in a multi-container Pod and --previous shows output from the previous instance of a crashed container. Node-local logs disappear with the node, so cluster-level logging ships them elsewhere. The recommended pattern is a node-level logging agent run as a DaemonSet, which reads every container's log files on its node and pushes them to a backend. A sidecar container with its own logging agent is an alternative, but its logs are not visible through kubectl logs.

Resource metrics follow a separate pipeline. The metrics-server collects CPU and memory usage from the kubelets on each node and exposes them through the Metrics API, which the HorizontalPodAutoscaler and VerticalPodAutoscaler consume. kubectl top pod and kubectl top node read the same API, which is why metrics-server, a cluster add-on, must be installed for them to work. The HorizontalPodAutoscaler uses these figures to change the replica count of a Deployment or StatefulSet, for example to keep average CPU utilization near a target.

For richer, longer-lived monitoring, Prometheus is the common choice. It is an open-source monitoring and alerting toolkit that stores metrics as time series identified by a metric name and key-value labels, queries them with PromQL, and collects them by pulling over HTTP from targets found through service discovery or static configuration. Short-lived jobs can push through an intermediary gateway. Grafana is the usual visualization layer and connects to Prometheus as a data source.

Day-to-day inspection uses kubectl. kubectl get nodes lists the registered nodes, kubectl describe shows an object's configuration, status and recent events, kubectl get pod with -o yaml prints the full stored definition, and kubectl events lists cluster events for a namespace, all namespaces or a single resource.

Security monitoring builds on the same signals: API server audit logs record who did what through the API, while container logs and metrics show what workloads did. Logs kept only on a node are lost with that node, which is why cluster-level collection matters for investigations.