kubectl node-shell

(formerly known as kubectl-enter)

Start a root shell in the node's host OS running. Uses an alpine pod with nsenter for Linux nodes and a HostProcess pod with PowerShell for Windows nodes.

Installation

using krew:

Plugin can be installed from the official krew repository:

kubectl krew install node-shell

Or from our own krew repository:

kubectl krew index add kvaps https://github.com/kvaps/krew-index
kubectl krew install kvaps/node-shell

or using curl:

curl -LO https://github.com/kvaps/kubectl-node-shell/raw/master/kubectl-node_shell
chmod +x ./kubectl-node_shell
sudo mv ./kubectl-node_shell /usr/local/bin/kubectl-node_shell

Usage

# Get standard bash shell
kubectl node-shell <node># Use custom image for pod
kubectl node-shell <node> --image <image># Use X-mode (mount /host, and do not enter host namespace)
kubectl node-shell -x <node># Skip specific namespace types to enter, choose any of ipc, mount, pid, net, uts
kubectl node-shell <node> --no-ipc
# Execute custom command
kubectl node-shell <node> -- echo 123
# Use stdin
cat /etc/passwd | kubectl node-shell <node> -- sh -c 'cat > /tmp/passwd'# Run oneliner script
kubectl node-shell <node> -- sh -c 'cat /tmp/passwd; rm -f /tmp/passwd'

X-mode

X-mode can be useful for debugging minimal systems that do not have a built-in shell (eg. Talos).
Here's an example of how you can debug the network for a rootless kube-apiserver container without a filesystem:

kubectl node-shell -x <node># Download crictl
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.28.0/crictl-v1.28.0-linux-amd64.tar.gz -O- | \
tar -xzf- -C /usr/local/bin/
# Setup CRI endpointexport CONTAINER_RUNTIME_ENDPOINT=unix:///host/run/containerd/containerd.sock
# Find your container
crictl ps | grep kube-apiserver
#3ff4626a9f10e e7972205b6614 6 hours ago Running kube-apiserver 0 215107b47bd7e kube-apiserver-talos-rzq-nkg# Find pid of the container
crictl inspect 3ff4626a9f10e | grep pid
# "pid": 2152,# "pid": 1# "type": "pid"# "getpid",# "getppid",# "pidfd_open",# "pidfd_send_signal",# "waitpid",# Go to network namespace of the pid, but keep mount namespace of the debug container
nsenter -t 2152 -n

You need to be able to start privileged containers for that.

Mounting External CSI Volumes

You can mount volumes from your CSI storage layer using the -m flag. This allows you to move data to/from node devices seamlessly. The PVC will be mounted at /opt-pvc. This is useful for failover in minimal systems that do not have a built in shell (eg. Talos). Here is an example of how you can retrieve zfs/lvm data from a volume on a failed CSI node and put it back in your distributed storage layer:

k node-shell -n <namespace> -x <node_with_data> -m <pvc_name># install rsync
apk add rsync
# Add lvm/zfs libs# ZFS
mount -o bind /host/dev /dev
mount -o bind /host/usr/local /usr/local
touch /lib/libuuid.so.1
mount -o bind /host/lib/libuuid.so.1 /lib/libuuid.so.1
touch /lib/libuuid.so.1.3.0
mount -o bind /host/lib/libuuid.so.1.3.0 /lib/libuuid.so.1.3.0
touch /lib/libblkid.so.1
mount -o bind /host/lib/libblkid.so.1 /lib/libblkid.so.1
touch /lib/libblkid.so.1.1.0
mount -o bind /host/lib/libblkid.so.1.1.0 /lib/libblkid.so.1.1.0
#LVM
touch /usr/lib/libaio.so.1
mount -o bind /host/usr/lib/libaio.so.1.0.2 /usr/lib/libaio.so.1
touch /usr/lib/libudev.so.1
mount -o bind /host/usr/lib/libudev.so.1 /usr/lib/libudev.so.1
export PATH=$PATH:/host/sbin
mkdir /lib/modules
mount -o bind /host/lib/modules /lib/modules
# look for data to recover
zfs list
NAME USED AVAIL REFER MOUNTPOINT
hdd-1 15.9T 7.52T 96K /hdd-1
hdd-1/SOME-OLD-PVC-FROM-PREVIOUS-NODE-INSTALL 361G 7.52T 361G - -
# mount the failed volume
zfs set mountpoint=/mnt hdd-1/SOME-OLD-PVC-FROM-PREVIOUS-NODE-INSTALL
zfs mount /hdd-1/SOME-OLD-PVC-FROM-PREVIOUS-NODE-INSTALL
# recover the data : copy it to the mounted CSI volume
rsync -avh --info=progress2 /mnt/ /opt-pvc/

the above exemple assumes pvc_name already exists in namespace. You need to be able to start privileged containers.