This content originally appeared on DEV Community and was authored by Mateus Caruccio
NOTE: this is an updated copy of my post in medium, where I'm not writing anymore.
Have you lost ssh access to one of your Kubernetes nodes? Why do you even need ssh access to nodes in the first place? Well, maybe something is stuck, or you need to see a config with your own eyes… I don’t know and I don’t care, they are your servers, not mine…
I’m assuming you have admin level into kubernetes API.
Talk is cheap, show me the code®:
$ NODE_NAME=master-0
$ kubectl create -n kube-system -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: root-shell-$NODE_NAME
namespace: kube-system
spec:
nodeName: $NODE_NAME
containers:
- command:
- /bin/cat
image: alpine:3
name: root-shell
securityContext:
privileged: true
tty: true
stdin: true
volumeMounts:
- mountPath: /host
name: hostroot
hostNetwork: true
hostPID: true
hostIPC: true
tolerations:
- effect: NoSchedule
operator: Exists
- effect: NoExecute
operator: Exists
volumes:
- hostPath:
path: /
name: hostroot
EOF
This pod will create a privileged POD into the node master-0 (change it to your node name) running /bin/cat forever. Now you simply exec into it and change the host’s root to pod’s root:
$ kubectl -n kube-system exec -it root-shell-$NODE_NAME chroot /host /bin/bash
[root@master-0 /]# id
uid=0(root) gid=0(root) groups=0(root)
Profit!
PS: Here is a DaemonSet for the lazy like me.
$ kubectl create serviceaccount -n kube-system root-shell
### For OKD/Openshift clusters only:
$ oc adm add-scc-to-user privileged -n kube-system -z root-shell
$ kubectl create -n kube-system -f - <<EOF
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: root-shell
namespace: kube-system
spec:
revisionHistoryLimit: 0
selector:
matchLabels:
app: root-shell
template:
metadata:
labels:
app: root-shell
spec:
terminationGracePeriodSeconds: 0
containers:
- command:
- /bin/cat
image: alpine:3
name: root-shell
tty: true
stdin: true
volumeMounts:
- mountPath: /host
name: hostroot
securityContext:
privileged: true
hostNetwork: true
hostPID: true
hostIPC: true
serviceAccountName: root-shell
hostNetwork: true
tolerations:
- effect: NoSchedule
operator: Exists
- effect: NoExecute
operator: Exists
volumes:
- hostPath:
path: /
name: hostroot
updateStrategy:
rollingUpdate:
maxUnavailable: 100%
type: RollingUpdate
EOF
AWS Bottlerock
Since now bottlerocket is a really nice OS alternative in EKS clusters, lets use control/admin containers to gain root-level access to the host.
As you known, Bottlerocket is a Read-only, container oriented operating system. This gives us a lot of benefits, but you can't simply kubect exec + chroot into it as we did in the stone age. Turns out there is two containers in the system: control and admin. We whant admin, since control is for adminstrative tasks (like upgrades and reboots).
First, we need a plain container to have access to host's apiclient binary:
$ kubectl create -n kube-system -f - <<EOF
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: apiclient
namespace: kube-system
spec:
revisionHistoryLimit: 0
selector:
matchLabels:
app: apiclient
template:
metadata:
labels:
app: apiclient
spec:
containers:
- command:
- sleep
- infinity
image: fedora
imagePullPolicy: Always
name: regain-access
securityContext:
seLinuxOptions:
level: s0
role: system_r
type: control_t
user: system_u
volumeMounts:
- mountPath: /usr/bin/apiclient
name: apiclient
readOnly: true
- mountPath: /run/api.sock
name: apiserver-socket
restartPolicy: Always
terminationGracePeriodSeconds: 0
volumes:
- hostPath:
path: /usr/bin/apiclient
type: File
name: apiclient
- hostPath:
path: /run/api.sock
type: Socket
name: apiserver-socket
updateStrategy:
rollingUpdate:
maxUnavailable: 100%
type: RollingUpdate
EOF
With that in place, let's use apiclient exec subcommand to activate and enter admin container:
$ kubectl exec -i -t -n kube-system apiclient-xtwxh -- apiclient exec -t control enter-admin-container
Confirming admin container is enabled...
Waiting for admin container to start...
Entering admin container
Welcome to Bottlerocket's admin container!
╱╲
╱┄┄╲ This container provides access to the Bottlerocket host
│▗▖│ filesystems (see /.bottlerocket/rootfs) and contains common
╱│ │╲ tools for inspection and troubleshooting. It is based on
│╰╮╭╯│ Amazon Linux 2, and most things are in the same places you
╹╹ would find them on an AL2 host.
To permit more intrusive troubleshooting, including actions that mutate the
running state of the Bottlerocket host, we provide a tool called "sheltie"
(`sudo sheltie`). When run, this tool drops you into a root shell in the
Bottlerocket host's root filesystem.
[root@admin]#
At the time of this writing, all you have to do now is execute sudo sheltie and voilà!
[root@admin]# sudo sheltie
bash-5.1# ps 1
PID TTY STAT TIME COMMAND
1 ? Ss 14:25 /sbin/init systemd.log_target=journal-or-kmsg systemd.log_color=0 systemd.show_status=true
bash-5.1#
This content originally appeared on DEV Community and was authored by Mateus Caruccio
Mateus Caruccio | Sciencx (2024-06-23T17:08:52+00:00) Kubernetes hack. Retrieved from https://www.scien.cx/2024/06/23/kubernetes-hack/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.