Debugging with kubectl

Make sure you have access to the namespace for your app, and that you have the required tools installed. See the how-to article on installing kubectl.

Make sure you know your namespace name, it's described in our Kubernetes reference.

Finding your pods

Taking a look at your pods can be a quick way to figure out what's wrong with your app.

To get the pods in your application:

$ kubectl get -n MY_NAMESPACE pods

# Output: (truncated)
# NAME           READY   STATUS
# my-app-ntfq5   1/1     Running

To get information about that pod:

$ kubectl describe -n MY_NAMESPACE pods/my-app-ntfq5 | less

# Output:
# /* Fills the screen */

View logs with kubectl access

An exciting thing in you can do with kubectl is to view logs.

Logs from containers can be found by somehow selecting containers. To find containers, we must first determine which namespace to use. Let's say we're interested in an app named olas-create-react-app, which is crashing in test:

$ kubectl get namespaces | grep olas-create-react-app

# Output:
# apps-olas-create-react-app-prod          Active   30d
# apps-olas-create-react-app-snap1         Active   1d
# apps-olas-create-react-app-test          Active   31d

We now know that our namespace is "apps-olas-create-react-app-test".

$ man kubectl-logs

Listing logs per container

Find all containers in the namespace:

$ kubectl get pods --namespace apps-olas-create-react-app-test

# Output:
# NAME                                    READY   STATUS             RESTARTS        AGE
# olas-create-react-app-5c4b8d64-ftm5k     1/1     Running            0               2d
# olas-create-react-app-687b84cf85-mkl46   0/1     CrashLoopBackOff   9 (3m22s ago)   24m

We're interested in olas-create-react-app-687b84cf85-mkl46:

$ kubectl logs pods/olas-create-react-app-687b84cf85-mkl46 \
    --namespace apps-olas-create-react-app-test

# Output:
# panic: http: invalid pattern
#
# goroutine 1 [running]:
# net/http.(*ServeMux).Handle(0xc00007a200, {0x0, 0x0}, {0x6f2060?, 0x6b1f28})
#         /usr/local/go/src/net/http/server.go:2510 +0x25f
# net/http.(*ServeMux).HandleFunc(...)
#         /usr/local/go/src/net/http/server.go:2553
# /* snip */

Fetching logs for a deployment

Find the deployment name, and then find the logs for that deployment

# Setting the namespace for all requests:
$ kubectl config set-context --current \
    --namespace=apps-olas-create-react-app-test

# What's the deployment name?
$ kubectl get deployments -o name

# Output:
# deployments.apps/apps-olas-create-react-app-test

# Get the logs using black magic:
# - Get the "selector", which describes how a deployment identifies resources that belong to it
# - Turn the selector from JSON to a key=value,key=value list
# - Get pods selected by this selector
# - Get logs for those pods

$ kubectl get deployments.apps/olas-create-react-app --output json \
    | jq '[ .spec.selector.matchLabels | to_entries | .[] | "\(.key)==\(.value)" ] | join(",") | @sh' -r \
    | xargs -n1 -I "{}" kubectl get pods --selector="{}" --output name \
    | xargs -n1 kubectl logs --prefix --tail=100

Streaming logs

If you want streaming logs, you can install stern.

Stern allows you to tail multiple pods on Kubernetes and multiple containers within the pod.

Stern matches pods and log lines using regular expressions.

# Set the default namespace, so we don't need to pass --namespace all the time:
$ kubectl config set-context --current \
    --namespace=apps-olas-create-react-app-test


# stern pod-query [flags]

# All pods, all containers
$ stern '.*'

# All pods, but not one particular container
$ stern '.*' --exclude-container '^cloudsql-proxy$'

# Ignore health checks
$ stern '.*' --exclude '/health'

# Only "waiting" (failing?) "backend" pods (backend is part of the appling name)
$ stern 'backend' --container-state "waiting"