Skip to content

Kubernetes Cheat Sheet

Find api resources for namespace

kubectl api-resources --verbs=list --namespaced -o name \
  | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace>

Flux

kubectl logs -n flux-system deploy/helm-controller

title: Kubernetes Cheat Sheet permalink: /cheatsheets/kube-commands/


Retries Exceeded

I see this when things take a while because something else is broken (like a csi) but there is no reason to fix the thing that is stuck. First force helmrelease to get fixed:

flux reconcile helmrelease -n home-automation home-assistant --force

Then get the kustomization in a good state again (if the helmrelease was OK):

flux reconcile kustomization -n flux-system zigbee2mqtt

Here are some commands for managing Kubernetes.

Clean Mess

Check what you are deleting:

kubectl get pod --field-selector=status.phase==Succeeded -A
kubectl get pod --field-selector=status.phase==Failed -A

If nothing to debug go ahead and kill em:

kubectl delete pod --field-selector=status.phase==Succeeded -A
kubectl delete pod --field-selector=status.phase==Failed -A

Rebooting sometimes creates a mess and this will clean it but I thing they go away after a bit.

DNS

DNS Test:

kubectl -n ai run dns-test --rm -it --image=busybox --restart=Never -- nslookup volsync-hayesops.s3.amazonaws.com

kubectx & kubens

These are helper scripts that come from here.

Show available contexts (clusters)

kubectx

Switch cluster

kubectx <cluster name>

Show available namespaces

kubens

Switch namespaces

kubens <namespace>

kubectl

This is the beast that drives operations around your k8s cluster. Most of the heavy lifting seems to be done in the manifest yaml or helm charts but debugging and/or setting up requires heavy use of kubectl.

kubectl apply

kubectl apply -f <filename.yaml>

kubectl get

To see what's running, installed, or anything else we use kubectl get <thing>

get nodes

kubectl get nodes

get namespaces

kubectl get namespaces

get pods

kubectl get pods --all-namespaces
kubectl -n <namespace> kubget pods 

get services

This is useful for figuring port mappings.

kubectl get svc --all-namespaces

get volumes

kubectl get pv,pvc -o wide

get volumesnapshots

kubectl get volumesnapshot

get volumesnapshotclass

This will show you what type of snapshots you can take.

kubectl get volumesnapshotclass

get ceph cluster

Since I used rook this tells me how the ceph cluster exposed to k8s is doing.

kubectl -n rook-ceph  get CephCluster

kubectl delete

delete manifest

kubectl delete -f <manifest.yaml>

delete all your volumes (don't do this!)

kubectl delete pvc --all

delete pv

kubectl patch pvc -n namespace PVCNAME -p '{"metadata": {"finalizers": null}}'
kubectl patch pv PVNAME -p '{"metadata": {"finalizers": null}}'

k delete pvc PVCNAME --grace-period=0 --force
k delete pv PVNAME --grace-period=0 --force

kubectl describe

Describe things in the cluster like pods and pvc's. This will give you some basic info to troubleshoot with before going to the logs (or if the logs are gone because the thing is crash looping.

kubectl describe <type> <name>

kubectl logs

For an app:

kubectl logs -n velero -l app.kubernetes.io/name=velero

For a pod:

kubectl logs <podname>

kubectl exec

Like bashing into a docker container but for a pod. Note if you are not kubens'd into the namespace you will need -n <namespace>

kubectl exec -it -n <namespace> <pod_name> -- env

Bash in:

k exec -it -n <namespace>  <pod_name> -- sh
k exec -it  -n <namespace>  <pod_name> -- /bin/bash

Restart

https://kubernetes.io/docs/reference/kubectl/generated/kubectl_rollout/kubectl_rollout_restart/

k rollout restart deployment/prowlarr -n media-management

See what capabilities you have

This will show resources. Here we are checking to see if the cluster can make volumesnapshots or at least is eligible to be setup for volumesnapshots.

kubectl api-resources | grep volumesnapshots

Velero

Create Backup

velero backup create hellothere --include-namespaces=default --wait
 velero backup create --from-schedule=velero-daily-backups
 ```

#### TODO FROM DEBUGGING NEEDS ORGANIZING

```bash
kubectl get events -A
ceph -s
ceph osd lspools
kubectl logs -f velero-764d58dfd9-k47sh

Mass Delete

Namespace w/ Spec Finalizer

This was super hard to get rid of vs. a regular namespace!

kubectl get ns rook-ceph -o json | jq '.spec.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/rook-ceph/finalize" -f -
kubectl get namespace rook-ceph -o json > tmp.json
  • Delete kubernetes finalizer in tmp.json (leave empty array "finalizers": [])
  • Run kubectl proxy in another terminal for auth purposes and run following curl request to returned port
kubectl proxy &
curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json 127.0.0.1:8001/api/v1/namespaces/rook-ceph/finalize

Jobs

kubectl get pod  -n velero --field-selector=status.phase==Succeeded

kubectl delete pod -n velero  --field-selector=status.phase==Succeeded

kubectl delete pod -n velero  --field-selector=status.phase==Failed
kubectl get jobs  -n velero --field-selector status.successful=1

kubectl delete jobs -n velero --field-selector status.successful=1

Pass `--all-namespaces for a good time!

velero delete backup move-data-test -n velero
velero delete backup velero-daily-backups-20240827000047 -n velero
velero delete backup velero-daily-backups-20240826000046 -n velero

TANK CHECK:

velero describe backup move-data-test-smb --details
kubectl -n velero rollout restart daemonset node-agent
kubectl -n velero rollout restart deployment velero

Stuck stuff

kubectl get volumeattachment

See if it's attached, remove finalizes:

kubectl patch pvc {PVC_NAME} -p '{"metadata":{"finalizers":null}}'

Or if it's stuck in released:

kubectl patch pv pv-smb-tank-k8s-media-management -p '{"spec":{"claimRef": null}}'