k0s reset fails on leftover container mounts under the run dir, and the failed attempt makes further resets impossible

[vikramhh]

Before creating an issue, make sure you've checked the following

• You are running the latest released version of k0s
• Make sure you've searched for existing issues, both open and closed
• Make sure you've searched for PRs too, a fix might've been merged already
• You're looking at docs for the released version

Platform

Ubuntu 22.04.5 LTS, kernel 6.8.0-1061-aws, x86_64
AWS EC2 t3.medium (2 vCPU, 3.7 GiB), single node

Version

v1.36.3+k0s.0

Also reproduced identically on v1.35.4+k0s.0. pkg/cleanup is byte-identical between the two — git diff v1.35.4+k0s.0 v1.36.3+k0s.0 -- pkg/cleanup/ is empty.

Sysinfo

k0s sysinfo

Total memory: 3.7 GiB (pass)
File system of /var/lib: ext4 (pass)
Disk space available for /var/lib/k0s: 26.8 GiB (pass)
Relative disk space available for /var/lib/k0s: 92% (pass)
Name resolution: localhost: [127.0.0.1] (pass)
Containerd config snippets in /etc/k0s/containerd.d/: no config snippets found (pass)
Operating system: Linux (pass)
  Linux kernel release: 6.8.0-1061-aws (pass)
  Max. file descriptors per process: current: 1048575 / max: 1048576 (pass)
  AppArmor: active (pass)
  Executable in PATH: modprobe: /usr/sbin/modprobe (pass)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: module (pass)
      CONFIG_NF_NAT: module (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: module (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: module (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connection tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
      CONFIG_NF_DEFRAG_IPV4: module (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connection tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: module (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
      CONFIG_NF_DEFRAG_IPV6: module (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass)
      CONFIG_LLC: module (pass)
      CONFIG_STP: module (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: built-in (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

For context, the three directories the reset cleanup deals with:

• dataDir — the k0s data directory, /var/lib/k0s by default (constant.DataDirDefault), overridable with --data-dir. Holds the bundled binaries in bin/ (runc, containerd, containerd-shim-runc-v2, etcd, …), the containerd content store and overlayfs snapshots, etcd data and PKI.
• kubeletRootDir — <dataDir>/kubelet by default, overridable with --kubelet-root-dir. (The reproduction below passes /var/lib/kubelet explicitly, which is not the default.)
• runDir — runtime state: supervised PID files and sockets, containerd.sock, containerd-cri.toml, and containerd's task state under io.containerd.runtime.v2.task/<ns>/<id>/, which includes each running container's mounted rootfs overlay. This is /run/k0s whenever k0s runs as root and <dataDir>/run otherwise; there is no flag to change it.

Two separate problems, the second more serious than the first.

1. The reset cleanup never unmounts under runDir, but then deletes it.

In pkg/cleanup/directories.go the unmount loop is guarded by:

if isUnderPath(v.Path, d.kubeletRootDir) || isUnderPath(v.Path, d.dataDir) {

runDir is not included. Container task rootfs overlays live under it — /run/k0s/containerd/io.containerd.runtime.v2.task/<ns>/<id>/rootfs — so any mount that survives the containers cleanup step is still mounted when the same function reaches:

if err := os.RemoveAll(d.runDir); err != nil {
    return fmt.Errorf("failed to delete %s: %w", d.runDir, err)
}

UnmountLazy (MNT_DETACH) already exists in the package but is only reachable from the dataDir/kubeletRootDir loop, and the errorIsUnlinkat mountpoint tolerance is likewise applied only to dataDir. Mount-awareness is present in one half of the function and absent in the other.

2. A failed reset cannot be retried.

The same step deletes dataDir before runDir. Once the runDir delete fails, /var/lib/k0s/bin is already gone, so the next k0s reset cannot extract runc/containerd-shim, cannot start containerd, skips container cleanup entirely, and fails on the same mount in ~140 ms. The first failure destroys what any retry needs; the node then requires a manual unmount or a reboot.

This matters for any automated stop → reset → restore flow, which cannot recover by retrying.

Steps to reproduce

k0s install controller --single && k0s start        # wait for the node to be Ready
kubectl run holder --image=busybox:1.36 -- sleep 100000

# Any container the CRI-based cleanup cannot see will do. Simplest:
k0s ctr -n demo images pull docker.io/library/busybox:1.36
k0s ctr -n demo run -d docker.io/library/busybox:1.36 straggler sleep 100000

k0s stop
k0s reset --data-dir /var/lib/k0s --kubelet-root-dir /var/lib/kubelet   # fails on /run/k0s
k0s reset --data-dir /var/lib/k0s --kubelet-root-dir /var/lib/kubelet   # ~140ms, containerd cannot start

A control run without the extra container succeeds and leaves zero task mounts, confirming the containers step is what normally clears runDir.

Expected behavior

k0s reset releases mounts under runDir before deleting it, as it already does for dataDir and kubeletRootDir. A failed reset leaves the node in a state where reset can be retried.

Actual behavior

Error: errors occurred during clean-up: failed to delete /run/k0s: unlinkat
/run/k0s/containerd/io.containerd.runtime.v2.task/<ns>/<id>: directory not empty

and on every subsequent attempt:

level=warning msg="Failed to initialize containerd, skipping container cleanup"
  error="open /var/lib/k0s/bin/.runc.<random>.tmp: no such file or directory"

Screenshots and logs

Three sequential resets after a single surviving mount

v1.36.3+k0s.0:

Attempt	Duration	Outcome
1	1835 ms	failed to delete /run/k0s: unlinkat …: directory not empty; dataDir and dataDir/bin now gone
2	143 ms	Failed to initialize containerd, skipping container cleanup; same delete failure
3	136 ms	same

v1.35.4+k0s.0: 2292 ms / 130 ms / 131 ms, identical messages.

Single-straggler case, both versions, 2 runs each

v1.36.3+k0s.0  reset_rc=1  ENOTEMPTY  mounts_after_stop=9   mounts_after_reset=1
v1.36.3+k0s.0  reset_rc=1  ENOTEMPTY  mounts_after_stop=8   mounts_after_reset=1
v1.35.4+k0s.0  reset_rc=1  ENOTEMPTY  mounts_after_stop=11  mounts_after_reset=1
v1.35.4+k0s.0  reset_rc=1  ENOTEMPTY  mounts_after_stop=11  mounts_after_reset=1

In every run the containers step cleaned every CRI-visible mount and left exactly the one straggler; the delete then failed while unlinking it:

Error: errors occurred during clean-up: failed to delete /run/k0s: unlinkat
/run/k0s/containerd/io.containerd.runtime.v2.task/demo/straggler: directory not empty

Controls

With no surviving mount, reset exits 0 and leaves zero task mounts (3/3 runs on v1.36.3+k0s.0, 2/2 on v1.35.4+k0s.0). With the containers step disabled — --cri-socket pointing at a non-existent socket, so k0s does not start its own containerd — reset fails on /run/k0s in 3/3 runs at each version.

Whether the delete surfaces ENOTEMPTY or EBUSY varies between identical runs (readdir order); it is the same defect either way.

Additional context

Suggested fixes:

1. Include runDir in the unmount loop, or extend the existing errorIsUnlinkat tolerance to the runDir delete.
2. Make reset idempotent under failure — delete runDir before dataDir, or otherwise preserve the binaries a later attempt needs.

FYI on overlap with #6472. PR #6472 ("Don't unmount kubelet-root-dir on k0s reset") modifies the same (d *directories) Run() function: it adds kubeletRootDirMounted/kubeletRootDirSeparate, changes the same procMounts loop (also refactoring it to slices.Backward), and extends the errorIsUnlinkat tolerance to kubeletRootDir. That is the same pattern suggested above for runDir, so the two changes look complementary — but they will conflict textually, and merging them will likely need coordinating.

Related: #5657 — k0s stop leaving containers running is what creates the precondition. That is expected containerd behaviour (shims are deliberately decoupled) and is tracked there, not part of this report. #2852 — same family: reset not fully cleaning up.