1. Install crictl on a worker node

Instructions provided

2. Configure crictl to connect to the container runtime

First, check the container runtime socket (containerd in your environment):

sudo crictl config --list

Then, verify the configuration:

sudo crictl info

3. List and inspect containers using crictl

• Listing all Pods, this is similar to kubectl get pods but at container runtime level

sudo crictl pods

• Listing all containers

sudo crictl ps

• Listing all containers including stopped ones

sudo crictl ps -a

• Getting detailed information about containers

sudo crictl ps -v

• listing images present on the current Node

sudo crictl images

• Checking container resources usage

sudo crictl stats --all

4. Debug a failing Pod using crictl commands

First, create the following Pod from your control plane Node:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: debug-test
  namespace: default
spec:
  containers:
  - name: failing-container
    image: busybox:1.37
    command: ["/bin/sh"]
    args: ["-c", "echo 'Starting...' && sleep 10 && exit 1"]
  - name: working-container
    image: busybox:1.37
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo 'Working...'; sleep 30; done"]
  restartPolicy: Always
EOF

Next, check where the Pod is running:

kubectl get pod debug-test -o wide

Next, SSH onto the corresponding worker Node.

Then, use crictl to get the Pod’s details and to troubleshoot the error you’ll find.

• verify the Pod is on the Node

sudo crictl pods | grep debug-test

• Get the Pod’s identifier

POD_ID=$(sudo crictl pods --name debug-test -q)

• Get detailed pod information

sudo crictl inspectp $POD_ID

• List containers in the pod

sudo crictl ps --pod $POD_ID

• Get logs from the failing container

FAILING_CONTAINER_ID=$(sudo crictl ps --pod $POD_ID --name failing-container -aq)
sudo crictl logs $FAILING_CONTAINER_ID

You should only see:

Starting...

• Get logs with timestamps

sudo crictl logs -t $FAILING_CONTAINER_ID

• Follow logs in real-time

sudo crictl logs -f $FAILING_CONTAINER_ID

• Inspect the failing container

sudo crictl inspect $FAILING_CONTAINER_ID

• Execute commands in the working container

WORKING_CONTAINER_ID=$(sudo crictl ps --pod $POD_ID --name working-container -q)
sudo crictl exec -it $WORKING_CONTAINER_ID /bin/sh

• Check container statistics

sudo crictl stats $WORKING_CONTAINER_ID

5. Compare crictl output with kubectl output

• From control plane (kubectl):

kubectl get pods debug-test -o wide
kubectl describe pod debug-test
kubectl logs debug-test -c failing-container
kubectl exec -it debug-test -c working-container -- /bin/sh

• From worker node (crictl):

sudo crictl pods --name debug-test
sudo crictl inspectp $POD_ID
sudo crictl logs $FAILING_CONTAINER_ID
sudo crictl exec -it $WORKING_CONTAINER_ID /bin/sh

The key differences between these tools:

• crictl works directly with the container runtime
• crictl shows container runtime IDs vs Kubernetes Pod names
• crictl can access containers even if the kubelet is not responsive
• crictl provides runtime-specific information (cgroup, namespaces, etc.)

6. Clean up test resources

Run the Pod deletion command from the control plane Node:

kubectl delete pod debug-test

From the worker Node, verify the containers and the Pod are gone:

sudo crictl ps --name debug-test
sudo crictl pods --name debug-test

You’ll find below examples of crictl commands for various scenario:

When kubectl is not working, but containers are running:

sudo crictl ps
sudo crictl logs <container-id>

When you need to inspect container mounts and volumes:

sudo crictl inspect <container-id> | jq '.status.mounts'

When debugging network issues at container level:

sudo crictl inspectp <pod-id> | jq '.status.network'

When checking container restart history:

sudo crictl ps -a | grep <container-name>

When you need to stop/start containers manually:

sudo crictl stop <container-id>
sudo crictl start <container-id>

When debugging image pull issues:

sudo crictl images
sudo crictl pull <image-name>
sudo crictl rmi <image-id>