Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Kubernetes CrashLoopBackOff: find out why the container keeps exiting

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

A pod that shows CrashLoopBackOff has a container that starts, exits and is restarted by the kubelet, with a longer wait after each failure. The status is not the cause. The kubelet records the wait as the event Back-off restarting failed container. Read the previous container's log with kubectl logs <pod-name> --previous, then its last state with kubectl describe pod <pod-name>, and fix what those two show.

What the symptom means

The Kubernetes pod lifecycle page says that when a pod is failing to start repeatedly, CrashLoopBackOff may appear in the Status field of some kubectl commands. It adds that Status is a kubectl display field and must not be confused with the pod's phase. The same page defines the Running phase as covering a pod where at least one container "is in the process of starting or restarting", so a crash-looping pod can stay in that phase.

The container restart is decided by restartPolicy. The kubelet restarts the container in the same pod on the same node, and the page gives the delay as "an exponential backoff delay (10s, 20s, 40s, …), that is capped at 300 seconds (5 minutes)". Doubling from 10 seconds gives 10, 20, 40, 80, 160, then 320 seconds, which the cap holds at 300. The first five waits add up to 310 seconds (10 + 20 + 40 + 80 + 160). The same page says that after a container has run for 10 minutes without problems, the kubelet resets the timer.

The event text comes from the kubelet's source code (release tag v1.34.0), which builds it from the format Back-off restarting failed container %s in pod %s. The first %s is the container's name. The second is produced by the source's format.Pod helper, which writes the pod's name, namespace and UID as name_namespace(uid), so a real event reads like ... in pod api-7c9d5b8f6-x2kq4_shop(<uid>) rather than the bare pod name.

Two settings change those numbers, and both are version dependent. The KubeletCrashLoopBackOffMax feature is beta since Kubernetes v1.35 and enabled by default; it lets a node's kubelet configuration set maxContainerRestartPeriod between "1s" and "300s". The ReduceDefaultCrashLoopBackOffDecay feature is alpha since v1.33 and disabled by default; with it, delays start at 1 second and are capped at 60 seconds.

Deployments use restartPolicy: Always, so a container that exits cleanly with code 0 is also restarted. The lifecycle page shows Jobs using OnFailure or Never for one-time work.

Likely causes

Cause What you see How to confirm
The application exits with an error Log lines from the previous container end with an error or a stack trace kubectl logs <pod-name> --previous
Configuration is wrong or missing (environment variables, mounted files, Secrets) Previous log names a variable, file or credential; events may show a mount problem Compare the pod's env and volume mounts with what the application reads, and read the events in kubectl describe pod
The memory limit is too low Last State is Terminated with Reason OOMKilled (the Kubernetes example shows Exit Code 137); the log stops without an error kubectl describe pod shows the Last State block and the container's memory limit
A liveness or startup probe fails Events say the container "failed liveness probe, will be restarted" kubectl describe pod events, and the probe settings in the pod spec
The process finishes normally but the pod is a Deployment Exit Code 0, log shows the work completed, restarts keep counting Last State shows Exit Code 0 under restartPolicy: Always
A dependency is not reachable at start-up Previous log shows connection errors to a database or service Previous log, then test the same address from another pod

Prove the cause before changing anything

All of these are read-only.

1. See the status and the restart count.

kubectl get pods -n shop

The RESTARTS column rises each time the kubelet starts the container again. READY of 0/1 with STATUS CrashLoopBackOff is the symptom.

2. Read the log of the instance that crashed, not the one that is waiting.

kubectl logs <pod-name> -n shop -c <container-name> --previous

The kubectl reference describes --previous as printing "the logs for the previous instance of the container in a pod if it exists".

3. Read the last state and the events.

kubectl describe pod <pod-name> -n shop

Find the container's Last State. A good result for this check is a state, reason and exit code that agree with the log. Terminated with Reason OOMKilled points to the memory limit. The Kubernetes resource page says the OOMKilled reason "shows that the container tried to use more memory than its limit". Exit Code 0 points to a finished task. In the events, a Killing line about a failed probe points to the probe.

4. Read the termination message, if the application writes one.

kubectl get pod <pod-name> -n shop -o go-template="{{range .status.containerStatuses}}{{.lastState.terminated.message}}{{end}}"

The page on determining the reason for pod failure gives this form. The message comes from /dev/termination-log by default and the kubelet truncates messages longer than 4096 bytes. An empty result only means the application wrote none.

5. For a multi-container pod, repeat steps 2 and 3 per container. Use -c with the container name that kubectl describe pod shows with the restarts.

Fix

Fix the cause the checks showed. Do not raise limits or add delays until the previous log and last state agree on one cause.

The application exits with an error, or configuration is missing

These faults are corrected in the workload's own configuration: the setting that the previous log names lives in the manifest, ConfigMap or Secret that your deployment process manages. The Deployment page describes kubectl rollout undo for rolling back a Deployment "when the Deployment is not stable, such as crash looping".

A dependency was not ready

The pod's log names the address. Fix that service first. The kubelet keeps retrying, so the pod recovers by itself at the next restart once the dependency answers.

The memory limit is too low

First save the current Deployment, which holds the present limits:

kubectl get deployment api -n shop -o yaml > /root/api-deployment-2026-11-22.yaml

Risk: the memory-limit change below edits the pod template of the api Deployment, which starts a rollout and replaces its pods.

Run the first command to check the change on the server without applying it, then the second to apply it. The reference shows kubectl set resources with --limits, and lists server as a --dry-run value:

kubectl set resources deployment api -n shop -c=api --limits=memory=512Mi --dry-run=server
kubectl set resources deployment api -n shop -c=api --limits=memory=512Mi

To undo it, set back the limit saved in /root/api-deployment-2026-11-22.yaml (here 256Mi), then read the rollout:

kubectl set resources deployment api -n shop -c=api --limits=memory=256Mi
kubectl rollout status deployment/api -n shop

The Kubernetes resource page says the next step may be to check the application code for a memory leak, and that if the application behaves as you expect, you should consider a higher memory limit.

A probe kills a slow starter

The probes page describes a startup probe for slow-starting containers, sized so that failureThreshold multiplied by periodSeconds covers the worst-case start time. Its example uses 30 and 10, which is 300 seconds, and once the startup probe succeeds the liveness probe takes over. Adding such a probe, or raising initialDelaySeconds, is an edit to the Deployment's pod template made by whoever owns the manifest.

A one-time task runs as a Deployment

A task that should end belongs in a Job, which the lifecycle page shows using OnFailure or Never. Moving a task from a Deployment to a Job is a design change made by the workload's owner, not a repair step.

Example (illustrative): in namespace shop, pod api-7c9d5b8f6-x2kq4 on k8s-node1 shows 0/1 CrashLoopBackOff with 6 restarts. kubectl logs --previous ends in the middle of a request with no error. kubectl describe pod shows Last State Terminated, Reason OOMKilled, Exit Code 137, against a memory limit of 256Mi. The engineer saves the Deployment, sets the limit to 512Mi (2 × 256Mi) after a dry run, and watches the rollout complete. RESTARTS on the new pod stays at 0.

Confirm the fix

  1. kubectl get pods -n shop shows the pod Running with READY 1/1.
  2. RESTARTS stops rising. After 10 minutes of running without problems the kubelet resets the back-off timer, so a repeat fault would again restart quickly.
  3. kubectl describe pod shows no new Back-off restarting failed container event.
  4. For a memory change, kubectl logs runs past the point where the previous container stopped.

What to watch afterwards

  • A RESTARTS count that keeps rising shows the container is still failing. Compare the time between restarts with the 10, 20, 40 second delays.
  • Probe thresholds and start-up time after each application release.
  • Memory limits whenever the application's data set or traffic grows.
  • Pods that exit with code 0 under a Deployment.

Risks and limits

  • These commands have not been run or lab-tested.
  • The event text comes from Kubernetes source code at release tag v1.34.0 and can differ in other releases.
  • kubectl logs --previous shows only the previous instance of the container, so keep logs elsewhere if you need a longer history.
  • The Kubernetes pages give no table of exit codes. Exit code 137 appears there only in the OOMKilled example. For a stop timeout or a manual kill, see "Docker container Exited (137): OOMKilled, a stop timeout, or a kill".
  • Restart delays differ if the two feature settings above are in use on your nodes.
  • A rollout started by kubectl set resources replaces running pods. Do it in a window your users accept.

Sources