Local Kubernetes Dev — Part 12: Debugging and observability

Sooner or later, myapp will break. A Pod won't come up, an image won't pull, the app can't reach PostgreSQL — that's a normal part of life in Kubernetes. The good news: you already have everything you need to quickly figure out what exactly went wrong. In this chapter we'll cover the basic debugging toolkit — from reading logs to a handy terminal UI — and sketch out where to grow once the basic commands aren't enough.

The key takeaway of this chapter

Debugging in Kubernetes almost always follows the same route. First you look at the events (what happened at the cluster level), then describe (the details of a specific Pod), then the logs (what the application itself had to say). Memorize this chain — events → describe → logs — it will get you out of most situations.

Logs: kubectl logs, -f, --previous, and the Tilt dashboard

Logs are the first place you look when the application has started but isn't behaving as expected. The basic command:

1kubectl logs <pod> -n myapp

To watch logs in real time (like tail -f), add -f (follow) — the stream keeps going until you interrupt it:

1kubectl logs -f <pod> -n myapp

The -p flag (also --previous) deserves a special mention. It shows the logs of the previous, already-crashed container instance. This is the key to debugging CrashLoopBackOff: the current container hasn't started yet (or just crashed again), so a plain kubectl logs will show nothing or the start of a new run. The real cause of the crash lives in the logs of the previous instance:

1kubectl logs -p <pod> -n myapp

Important: -f and -p don't logically combine — you can't "stream the logs of something that's already dead."

A few flags that save time:

1# a specific container in a multi-container Pod
2kubectl logs <pod> -c <container> -n myapp
3
4# the last 50 lines with timestamps
5kubectl logs <pod> --tail=50 --timestamps -n myapp
6
7# only the last 10 minutes
8kubectl logs <pod> --since=10m -n myapp
9
10# logs from all Pods with the label app=myapp, all containers
11kubectl logs -l app=myapp --all-containers -n myapp

--since accepts values like 5s, 2m, 3h. By default --tail shows all lines (or the last 10 if you use the -l selector). For the full list of flags, see the official reference for kubectl logs.

A tip about multi-container Pods: if a Pod has several containers and you didn't specify -c, kubectl will pick the default container — and it can easily turn out to be not the one you want. When the logs look "wrong," the first thing to check is whether you're reading the right container.

Logs in Tilt

If you're already working through Tilt (see the chapter on Tilt), then in most cases you won't have to reach for kubectl logs by hand. Tilt collects the logs of all resources into its web UI at http://localhost:10350. There are two views there:

Resource Overview — a list of all resources with their statuses (build status and runtime status separately), Pod ID, and endpoints.
Resource Detail — focused on a single resource and its logs.

Tilt pulls in not just the application's stdout, but also Docker build errors and Kubernetes events — all in one place, with filters by source and search by substring/regex. This really speeds up the loop when you're iteratively editing myapp and want to see the reaction right away. You can change the UI port if needed with the --host/--port flags. For details, see the Tilt UI tutorial.

kubectl describe pod — reading the Events

If the Pod didn't start at all (which means there are no application logs yet), move on to describe:

1kubectl describe pod <pod> -n myapp

This command gives you the full picture of the Pod: its configuration, current status, conditions, and — most valuable for debugging — the Events section at the bottom of the output. These events aren't in kubectl get; they're available only here (and through kubectl get events, see below).

Read the Events from the bottom up and look for lines with type Warning and reasons like Failed, BackOff, FailedScheduling. Typical lifecycle events:

Scheduled — the Pod has been assigned to a node;
Pulling / Pulled — the image is being pulled / has been pulled;
Failed — something didn't work out (for example, the image didn't pull);
BackOff — Kubernetes is waiting before the next restart attempt.

The crucial nuance: describe tells you what happened at the Kubernetes level (for example, "the container is crashing, so BackOff"), but why the application itself is crashing — only the logs (kubectl logs -p) will tell you that. So there's exactly one connection: you see BackOff in the Events → you go to logs --previous for the real error. For more on debugging running Pods, see the official Kubernetes documentation.

kubectl get events, Pod statuses, and what they mean

To see events not for a single Pod but for the whole namespace or cluster, use kubectl get events. One important point: by default, events are not sorted chronologically, so almost always add --sort-by:

1kubectl get events --sort-by='.lastTimestamp' -n myapp
2
3# all namespaces, in watch mode
4kubectl get events -A --watch

Events are structured API objects with a type (Normal or Warning), a reason, and a message. Useful commands and flags are described in the cluster debugging guide.

An important caveat: events have a limited lifespan (a TTL on the order of an hour). If the incident happened yesterday, the events are already gone — you'll have to rely on logs and the state of the resources.

Pod phases versus what you see in STATUS

This is where newcomers often get confused, so let's go through it carefully. A Pod has five official phases, described in the Pod Lifecycle:

Pending — the Pod has been accepted by the cluster, but the containers haven't started yet. This includes both waiting to be scheduled onto a node and pulling the image.
Running — the Pod is bound to a node, and at least one container is running.
Succeeded — all containers have terminated successfully and won't be restarted.
Failed — all containers have terminated, and at least one of them with an error.
Unknown — the Pod's state could not be obtained.

And now for the surprise: the familiar CrashLoopBackOff and ImagePullBackOff that you see in the STATUS column of kubectl get pods are NOT Pod phases. They're display fields that kubectl assembles from the container states for convenience. The phase, meanwhile, stays, for example, Pending (the Pod never made it to Running). Don't confuse the two — it'll save you some grief when reading the documentation.

What the most common statuses mean:

CrashLoopBackOff — the container starts, crashes, Kubernetes waits and tries again, around and around. The pause between attempts grows (exponential backoff). The primary documentation guarantees only a minimum of 100ms and a cap of 5 minutes; treat the popular sequence like "10s, 20s, 40s…" as an illustration from blog posts, not an exact promise. Restart behavior depends on the Pod's restartPolicy (Always by default, or OnFailure/Never). Diagnose with kubectl logs -p + describe.
ImagePullBackOff — the image can't be pulled. For myapp this is typical if you made a typo in the tag or didn't push the image to the local registry k3d-registry.localhost:5000 (see the chapter on containerization). Look at the Events in describe — the specific cause will be there (no such tag, registry unreachable, etc.).
Pending + FailedScheduling — the scheduler couldn't find a suitable node. Causes: not enough CPU/memory, nodeSelector/affinity set, taints on the nodes, hitting a quota. In a local k3d cluster, this is most often a shortage of resources.

To quickly see the statuses and which node a Pod lives on:

1kubectl get pods -n myapp
2kubectl get pod <pod> -o wide -n myapp

port-forward and exec — checking from the inside

Sometimes a Pod is Running, but it's not clear whether the application is responding and whether it can see the database. Two commands help here.

kubectl port-forward creates a temporary TCP tunnel from a local port of yours to a Pod or Service — the traffic goes through the Kubernetes API server, without any LoadBalancer or Ingress. Handy for poking at myapp directly:

1# through the Service
2kubectl port-forward svc/myapp 8080:80 -n myapp
3
4# or straight into the Pod
5kubectl port-forward pod/<pod> 8080:8080 -n myapp

Pay attention to the ports: when forwarding to the Service, the target port is the port of the Service itself (80), which is then proxied on to the container's targetPort: 8080 (see the chapter on manifests and the chapter on networking). When forwarding straight into the Pod, you hit the application's port 8080 directly — the same port we forwarded through Tilt in the Tilt chapter. The local port on the left (8080) is yours in both cases and can be any free port.

After this, curl http://localhost:8080/healthz will go straight into the Pod. For more on the tunnel model, see the port-forward guide. One quirk: the tunnel breaks if the Pod restarts or gets recreated — that's normal, just run the command again.

kubectl exec runs a command inside a container. The most common use is to open a shell and look around:

1kubectl exec -it <pod> -n myapp -- sh
2
3# a specific container + view the environment variables
4kubectl exec -it <pod> -c <container> -n myapp -- env

From the inside, it's convenient to check the env variables (are DB_HOST/DB_PASSWORD etc. correct?), the presence of files, and network connectivity to PostgreSQL. As with logs, in a multi-container Pod specify -c, otherwise you'll end up in the wrong container.

When the container has no shell

If you built myapp on a distroless image (a minimal image with no shell or utilities — good practice for production), then kubectl exec ... -- sh simply won't work: there's no shell there. For this case there's kubectl debug — it attaches a temporary (ephemeral) container with the tools you need to the Pod, without restarting the Pod or changing its spec (stable as of Kubernetes 1.25):

1kubectl debug <pod> -it --image=busybox --target=<container> -n myapp

--target connects the temporary container to the process namespace of the target one, so you can see its processes and network. All such actions are recorded in the API audit logs. For details and options (--share-processes, --copy-to), see the same Pod debugging documentation and the walkthrough on debugging distroless containers.

k9s: navigating the cluster in a single window

kubectl commands are fine, but switching between get pods, logs, describe, and port-forward dozens of times an hour gets tiring. k9s is a terminal UI that continuously watches the cluster and shows resources in real time, right in a single window. Starting it is trivial:

1k9s -n myapp

From there you navigate Pods, Deployments, and Services with the arrow keys, and you perform routine operations with hotkeys: view logs, describe a resource, open a shell, port-forward, restart, scale. The shortcuts adapt to the context — over a Pod one set is available, over a Deployment another. There's powerful filtering, navigation between related resources, and support for CRDs, RBAC, and theming (skins). For a beginner this is probably the fastest way to "get a feel" for the cluster, and it works perfectly with a local k3d. The project lives at github.com/derailed/k9s.

Under the hood, k9s does exactly the same thing you do by hand through kubectl — just without typing the commands. So you still need to understand the basic commands from this chapter: k9s speeds up the work but doesn't replace understanding what's going on.

Basic observability: where to grow

Everything discussed above is debugging: you react to a specific problem here and now. As myapp matures (especially after moving to production), you'll want to see the state of the service continuously, not just when something has already broken. That's observability, and it rests on three pillars:

Metrics — numeric indicators over time: CPU, memory, network, request count, latency. They answer the question "is there a problem" (for example, latency is creeping up). The industry standard for collection is Prometheus, and for visualization, Grafana. The very first step in a local cluster is to install metrics-server so that kubectl top pods works.
Logs — what we've been covering throughout this chapter. They give you the details of a specific error.
Traces — the path of a single request through several services. They answer the question "why" and where exactly the time was lost. The standard here is OpenTelemetry: the application produces spans, they go to the OTel Collector, and from there to a backend like Jaeger or Zipkin for viewing.

A short mnemonic: metrics — what, logs — details, traces — why. For an overview of the topic, take a look at the materials on observability in Kubernetes from 2026.

It's important not to overdo it. For local development of myapp, a full Prometheus + Grafana + OpenTelemetry stack is usually overkill — at this stage, logs, describe, events, and k9s are enough. Treat this section as a direction of growth: when the service heads to production and you have several interacting components, you'll know which way to dig. We'll touch on some of these topics again when we make the local setup more production-like.