The first 15 minutes of Kubernetes debugging (a checklist that works)

Why a checklist beats improvisation

When something is failing in a Kubernetes cluster, improvisation feels natural. Everyone wants to move fast. Someone checks logs, someone restarts a pod, someone looks at dashboards, and someone else asks whether there was a deployment.

That energy is useful, but without a checklist it can quickly turn into noise.

The first 15 minutes of Kubernetes debugging are not about solving every possible root cause. They are about narrowing the problem quickly and safely. You want to confirm that you are looking at the right cluster, identify the affected namespace or workload, find the first concrete error signal, and capture enough evidence before changing anything.

A checklist works because incidents are stressful. Under pressure, even experienced engineers can skip basic checks. The wrong context, wrong namespace, or wrong pod can waste precious time. A simple repeatable flow prevents that.

The goal of the first 15 minutes

The goal is not to become a Kubernetes expert in the middle of an incident.

The goal is to answer a few practical questions:

• Am I connected to the correct cluster?
• Which namespace or workload is affected?
• Are pods running, restarting, pending, or failing readiness?
• Is the failure in the application, configuration, dependency, or infrastructure?
• What is the first hard error signal?
• What evidence should be captured before making changes?

If you can answer those questions, the incident becomes much easier to reason about.

Step 1: Confirm the Kubernetes context

Always start by confirming context.

kubectl config current-context
kubectl cluster-info

This sounds basic, but it prevents one of the most common incident mistakes: debugging the wrong cluster.

In organizations with multiple environments — development, staging, pre-production, production, customer-specific clusters, or regional clusters — context confusion is easy. A command that looks harmless in one environment can be dangerous in another.

Before running deeper commands, confirm:

• cluster name
• environment
• region, if relevant
• access role
• whether the issue is actually reported in this cluster

Do not assume your terminal is already pointing to the right place.

Step 2: Scope to the namespace

Once the context is correct, scope the investigation.

kubectl get ns
kubectl get all -n <namespace>

Namespaces help reduce the search space. Instead of looking at the entire cluster, you can focus on the workloads that belong to the failing application or platform area.

Start with a broad namespace view:

kubectl get pods -n <namespace>
kubectl get deploy -n <namespace>
kubectl get svc -n <namespace>
kubectl get ingress -n <namespace>

You are looking for obvious signals:

• pods not ready
• pods restarting
• deployments with unavailable replicas
• services with no endpoints
• ingress changes
• jobs or cronjobs failing
• recently created or terminated pods

The first useful clue is often visible before you read a single application log.

Step 3: Find unhealthy pods fast

Pods are usually the fastest path to a concrete signal.

kubectl get pods -n <namespace>
kubectl get pods -n <namespace> -o wide

The -o wide output gives extra context such as node placement and pod IPs. This helps identify whether failures are isolated to one node, one deployment, or one group of pods.

Look at pod status carefully:

• Running does not always mean healthy.
• CrashLoopBackOff points to repeated process failure.
• ImagePullBackOff or ErrImagePull points to image or registry issues.
• Pending often points to scheduling, resource, or node constraints.
• Completed may be normal for jobs.
• ContainerCreating can indicate volume, image, or network setup delay.

Then describe the suspicious pod:

kubectl describe pod <pod> -n <namespace>

The describe output is useful because it includes events, container states, restart counts, readiness/liveness probe results, resource requests, mounted volumes, and scheduling information.

Do not skip the events section. It often tells you whether Kubernetes itself is failing to start the workload.

Step 4: Read logs like a timeline

After identifying the pod, read logs.

kubectl logs <pod> -n <namespace>

If the pod restarted, also check the previous container logs:

kubectl logs <pod> -n <namespace> --previous

This is critical for crash loops. The current container may not show the original crash, while --previous can reveal the actual exception or startup failure.

Read logs as a timeline, not as isolated lines.

Look for:

• startup sequence
• configuration loading
• dependency connection attempts
• authentication or authorization failures
• migration or initialization steps
• first error timestamp
• repeated retry loops
• request or correlation IDs
• shutdown signals
• out-of-memory or fatal errors

The first error is often more useful than the last error. Later errors may be consequences of the original failure.

Step 5: Check rollout and version state

Many incidents follow a deployment.

Check deployment status:

kubectl rollout status deployment/<deployment> -n <namespace>
kubectl describe deployment <deployment> -n <namespace>

Then inspect replica sets:

kubectl get rs -n <namespace>

You want to know:

• Was there a recent rollout?
• Are new pods failing while old pods were healthy?
• Did the image tag change?
• Did environment variables change?
• Are replicas stuck during rollout?
• Are readiness probes preventing traffic?
• Is the deployment partially rolled out?

If the issue started right after a deployment, version evidence matters. Capture the image tag, deployment age, and rollout state before making changes.

Step 6: Validate services and endpoints

A pod can be healthy while the service path is broken.

Check the service:

kubectl get svc -n <namespace>
kubectl describe svc <service> -n <namespace>

Then check endpoints:

kubectl get endpoints <service> -n <namespace>

If a service has no endpoints, traffic has nowhere to go. This often points to selector mismatch, readiness failures, or pods not matching the service labels.

Service debugging should answer:

• Does the service exist?
• Does it select the expected pods?
• Are endpoints populated?
• Are ports correct?
• Are pods ready and eligible for traffic?

A missing endpoint can explain many “service unavailable” symptoms quickly.

Step 7: Inspect readiness and liveness probes

Readiness and liveness probes are common sources of confusion.

A liveness probe decides when Kubernetes should restart a container. A readiness probe decides whether the pod should receive traffic.

A pod can be running but not ready. In that state, the application process exists, but the service should not send traffic to it.

Check probe failures in:

kubectl describe pod <pod> -n <namespace>

Look for events such as:

• readiness probe failed
• liveness probe failed
• connection refused
• HTTP status failures
• timeout
• probe endpoint missing
• startup taking longer than probe thresholds

Probe failures may indicate a real application issue, but they can also indicate misconfigured health checks. The important question is whether the probe reflects actual traffic readiness.

Step 8: Validate configuration and secrets safely

Configuration issues frequently appear after deployments.

Inspect environment variables without exposing secrets:

kubectl describe pod <pod> -n <namespace>
kubectl get deployment <deployment> -n <namespace> -o yaml

Be careful with secrets. Do not paste secret values into chats, tickets, or logs. The goal is to verify that references exist and are mounted correctly, not to expose credentials.

Check config map names, secret references, mounted volume paths, environment variable names, image tags, commands, args, resource limits, and service accounts. A missing or renamed environment variable can cause an application to fail before it even starts accepting traffic.

Step 9: Validate dependencies with quick probes

Once the workload looks suspicious, check whether it can reach its dependencies.

From inside a container or debug pod, validate connectivity to:

• internal services
• databases
• cache systems
• message brokers
• configuration endpoints
• authentication providers
• external APIs, if allowed

Depending on what is available in the container, use tools such as curl, nc, wget, or application-specific health endpoints.

kubectl exec -it <pod> -n <namespace> -- sh

Then run safe read-only checks from inside the container.

Dependency checks help distinguish application failure from environment failure. If the app cannot reach its database, restarting the pod may not solve anything.

Step 10: Check resource pressure

Kubernetes failures often come from resource constraints.

kubectl top pod -n <namespace>
kubectl top node

Then inspect requests and limits:

kubectl describe pod <pod> -n <namespace>

Look for CPU throttling symptoms, memory limit crashes, OOMKilled state, pending pods due to insufficient resources, node pressure, eviction events, high restart counts, or pods concentrated on one node.

If a pod was killed because of memory, the logs may not show a graceful application error. The pod state and events may be the clearest evidence.

Step 11: Capture one hard artifact before changing anything

Before restarting, scaling, rolling back, or editing configuration, capture at least one hard artifact.

Good artifacts include:

• exact error message
• pod status and restart count
• deployment image tag
• event output
• relevant log excerpt with timestamp
• correlation or trace ID
• failing request path
• service endpoint state
• readiness/liveness failure
• resource pressure signal

This matters because mitigation can erase evidence. A restart may clear container logs. A rollback may remove the failing pod. Scaling may change scheduling. Once the evidence is gone, root cause analysis becomes weaker.

A 15-minute command flow

Here is a practical first-pass flow:

kubectl config current-context
kubectl cluster-info

kubectl get ns
kubectl get all -n <namespace>

kubectl get pods -n <namespace> -o wide
kubectl describe pod <pod> -n <namespace>

kubectl logs <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous

kubectl rollout status deployment/<deployment> -n <namespace>
kubectl describe deployment <deployment> -n <namespace>

kubectl get svc -n <namespace>
kubectl get endpoints <service> -n <namespace>

kubectl top pod -n <namespace>
kubectl top node

This flow does not solve every incident. It gives you a structured map of the current state.

After this, the next step becomes more obvious: application debugging, configuration review, dependency investigation, rollback, scaling, or infrastructure escalation.

What not to do in the first 15 minutes

Avoid these common mistakes:

• restarting pods before checking logs
• debugging the wrong cluster or namespace
• assuming Running means healthy
• ignoring --previous logs after restarts
• skipping events in describe
• changing multiple things at once
• treating service errors as application errors without checking endpoints
• exposing secrets while copying configuration
• relying only on dashboards without checking cluster state
• forgetting to capture timestamps

The first 15 minutes should reduce uncertainty, not create more of it.

How to think about common symptoms

Pod is CrashLooping

Check previous logs, startup configuration, dependency connection, migrations, and memory kills. A crash loop usually has a clear cause if you inspect the previous container state.

Pod is Pending

Check node capacity, resource requests, taints, tolerations, affinity rules, PVC binding, and scheduling events. Pending usually means Kubernetes cannot place the pod.

Pod is Running but not Ready

Check readiness probe failures, service dependencies, startup time, and health endpoint behavior. The process may be alive but not eligible for traffic.

Service returns 503

Check endpoints. If endpoints are empty, the service has no ready pods. If endpoints exist, check ingress, network policy, application logs, and upstream dependencies.

Only one replica fails

Check node placement, pod-specific events, configuration mounts, and whether the replica is handling a different traffic pattern. One bad pod can create intermittent failures.

Failures started after deployment

Check image tag, environment variables, config maps, secrets, rollout status, and replica set history. Compare old and new pod behavior.

When to escalate

Escalation is not failure. It is part of incident response.

Escalate when:

• the issue affects production users
• the blast radius is growing
• data integrity may be at risk
• cluster-level resources are unhealthy
• multiple services are failing
• security or credentials may be involved
• you need access you do not have
• mitigation requires a rollback or scaling decision

When escalating, include the evidence you captured: environment, namespace, affected workload, first error timestamp, pod status, recent deployment status, key log line, endpoint/probe/resource findings, and suspected dependency or change.

Good escalation is evidence transfer.

Closing

Most Kubernetes incidents do not need a complex strategy in the first few minutes. They need fast narrowing.

A checklist turns that narrowing into muscle memory.

Confirm the context. Scope the namespace. Find unhealthy pods. Read logs as a timeline. Check rollout state. Validate services and endpoints. Inspect probes and configuration. Test dependencies carefully. Capture one hard artifact before changing anything.

That rhythm prevents avoidable mistakes and helps the team move from “something is broken” to “this workload is failing for this reason.”

Kubernetes debugging is not about memorizing every command. It is about asking the right questions in the right order while preserving enough evidence to act safely.

The first 15 minutes matter because they set the direction for the rest of the incident. A calm checklist keeps that direction clear.