K3s failed to destroy or setup network for sandbox

Sep 5, 2024

Issue: failed to destroy network for sandbox

Aug 30 16:33:45 k8s k3s[3407]: E0830 16:33:45.180187    3407 kuberuntime_manager.go:1075] "killPodWithSyncResult failed" err="failed to \"KillPodSandbox\" for \"fadedcde-ce8d-42cc-8467-a8d5bb826fce\" with KillP
Aug 30 16:33:45 k8s k3s[3407]: E0830 16:33:45.180300    3407 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"fadedcde-ce8d-42cc-8467-a8d5bb826fce\" with KillPodSandbox
Aug 30 16:33:45 k8s k3s[3407]: E0830 16:33:45.182928    3407 remote_runtime.go:222] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"cb4
Aug 30 16:33:45 k8s k3s[3407]: E0830 16:33:45.183053    3407 kuberuntime_manager.go:1375] "Failed to stop sandbox" podSandboxID={"Type":"containerd","ID":"cb4e620aa2a90ea1a358bc214a5c87015a398159ac5d934dd24aef5
Aug 30 16:33:45 k8s k3s[3407]: E0830 16:33:45.183182    3407 kuberuntime_manager.go:1075] "killPodWithSyncResult failed" err="failed to \"KillPodSandbox\" for \"7c96104e-de03-4c26-bf41-28e9cae1f015\" with KillP

Solution

k3s-killall.sh
rm -rf /var/lib/cni
sudo systemctl start k3s

Issue: failed to setup network for sandbox

Warning  FailedCreatePodSandBox  15s               kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "da475fac888cf16bc86a42c2b9bee6db460624a6ba674f3cf0357b622af96a71": plugin type="flannel" failed (add): incompatible CNI versions; config is "1.0.0", plugin supports ["0.1.0" "0.2.0" "0.3.0" "0.3.1" "0.4.0"]

Solution

cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml

# Check if the soft link of "current" folder matches to the below "bin_dir" folder
[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/var/lib/rancher/k3s/data/ea4b37218d11c27f3eb1c216194bb1a839b6e5c28be33cc92b92bd008bac1ad8/bin"

# If it doesn't match
sudo rm /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl

Issue: failed to get sandbox image

Warning  FailedCreatePodSandBox  2m4s   kubelet            Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image "rancher/mirrored-pause:3.6": failed to pull image "rancher/mirrored-pause:3.6": failed to pull and unpack image "docker.io/rancher/mirrored-pause:3.6": failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/62/6270bb605e12e581514ada5fd5b3216f727db55dc87d5889c790e4c760683fee/data?verify=1725011232-tD3vGnDILkddFPTG79vdgX7yYCU%!D(MISSING)": dial tcp 103.73.161.52:443

Solution

sudo vim /etc/containerd/config.toml

[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
    endpoint = ["https://atomhub.openatom.cn"]

Knowledge

Ubuntu 18.04.5 LTS DNS Settings

sudo vim /etc/netplan/01-netcfg.yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    ens18:
      dhcp4: yes
      nameservers:
        addresses:
          - 114.114.114.114
        search: []

[back]