K3s Certificate Expiration
And not automatically rotated.
Issue
Kubectl is not able to connect to the k3s server after months:
kubectl get node
error: You must be logged in to the server (Unauthorized)
Telegram bot cannot connect to api.telegram.org:
telegram.error.NetworkError: urllib3 HTTPError HTTPSConnectionPool(host='api.telegram.org', port=443): Max retries exceeded with url
Verify the problem
sudo kubectl get node
# execute on the k3s server node
error: You must be logged in to the server (Unauthorized)
openssl s_client -connect localhost:6443 -showcerts < /dev/null 2>&1 | openssl x509 -noout -enddate
# check the expiration date of the certificate
Certificate has expired
Force k3s to rotate the certificate:
sudo rm /var/lib/rancher/k3s/server/tls/dynamic-cert.json
sudo kubectl --insecure-skip-tls-verify=true delete secret -n kube-system k3s-serving
sudo systemctl restart k3s
Fix k3s-agent error
Error message: Error syncing pod, skipping
sudo vim /etc/rancher/k3s/k3s.yaml
# update the certificate data according to the server configuration
sudo systemctl restart k3s-agent
sudo systemctl status k3s-agent
Restart kilo if needed; this could also fix the issue that the pod cannot resolve DNS requests.
K3S upgrade (Optional)
/usr/local/bin/k3s -v
k3s version v1.25.3+k3s1 (f2585c16)
go version go1.19.2
sudo wget https://github.com/k3s-io/k3s/releases/download/v1.28.3%2Bk3s2/k3s-arm64 -O /usr/local/bin/k3s
sudo wget https://github.com/k3s-io/k3s/releases/download/v1.28.3%2Bk3s2/k3s -O /usr/local/bin/k3s
/usr/local/bin/k3s -v
k3s version v1.28.3+k3s2 (bbafb86e)
go version go1.20.10
sudo systemctl restart k3s
sudo systemctl restart k3s-agent