K3s Certificate Expiration

Nov 20, 2023

And not automatically rotated.


Kubectl is not able to connect to the k3s server after months:

kubectl get node
error: You must be logged in to the server (Unauthorized)

Telegram bot cannot connect to api.telegram.org:

telegram.error.NetworkError: urllib3 HTTPError HTTPSConnectionPool(host='api.telegram.org', port=443): Max retries exceeded with url

Verify the problem

sudo kubectl get node
# execute on the k3s server node
error: You must be logged in to the server (Unauthorized)

openssl s_client -connect localhost:6443 -showcerts < /dev/null 2>&1 | openssl x509 -noout -enddate
# check the expiration date of the certificate

Certificate has expired

Force k3s to rotate the certificate:

sudo rm /var/lib/rancher/k3s/server/tls/dynamic-cert.json
sudo kubectl --insecure-skip-tls-verify=true delete secret -n kube-system k3s-serving
sudo systemctl restart k3s

Fix k3s-agent error

Error message: Error syncing pod, skipping

sudo vim /etc/rancher/k3s/k3s.yaml
# update the certificate data according to the server configuration

sudo systemctl restart k3s-agent
sudo systemctl status k3s-agent

Restart kilo if needed; this could also fix the issue that the pod cannot resolve DNS requests.

K3S upgrade (Optional)

/usr/local/bin/k3s -v
k3s version v1.25.3+k3s1 (f2585c16)
go version go1.19.2

sudo wget https://github.com/k3s-io/k3s/releases/download/v1.28.3%2Bk3s2/k3s-arm64 -O /usr/local/bin/k3s
sudo wget https://github.com/k3s-io/k3s/releases/download/v1.28.3%2Bk3s2/k3s -O /usr/local/bin/k3s

/usr/local/bin/k3s -v
k3s version v1.28.3+k3s2 (bbafb86e)
go version go1.20.10

sudo systemctl restart k3s
sudo systemctl restart k3s-agent