Diagnosing K8s Installations

Well, this was much harder than I thought. I tried to update Minikube and my local Kubernetes installation.

First, I installed everything I thought I needed:

sudo apt-get install -y kubelet kubeadm kubectl kubernetes-cni

and then download and install Minikube per the instructions. Unfortunately, when I did, I'd see it complain that possibly "the kubelet is not running".

Running:

systemctl status kubelet

showed that kubelet was exiting with error code 1.

Running:

journalctl -fu kubelet

showed it puking stack traces but this line is the salient one:

... failed to run Kubelet: running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false...

But where do I put it? As it happened, I had to add --fail-swap-on=false to the ExecStart line in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (I found this by grepping ExecStart in /etc/). You then run:

sudo systemctl daemon-reload

to have changes recognised. Then, it's a matter of configuring Kubernetes system wide:

sudo kubeadm init --ignore-preflight-errors=Swap

(I needed to ignore the fact that my system has Swap space as I'm only working on proof of concepts, not running a K8s cluster in production. Using or not using swap space is nuanced - see here for more information. Basically, what is best depends on the situation. TL;DR; it can speed things up but also delay the inevitable death of a pathological system). 

You can run kubeadm reset if you foul things up.

Also, I had to rm -rf $HOME/.kube and $HOME/.minikube since kubectl config view showed me a profile that was literally years out of data. The .kube config can be regenerated with:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

(lines that kubeadm kindly told me to run). Now, running kubectl get all gives me a sensible output.

The story doesn't quite end there. After a few hibernations of my laptop, minikube was failing to start again with (apparently) certificate errors. This issue helped me: I executed minikube delete && minikube start and all was well with the World once more.