k0s reset can hang indefinitely in the containers cleanup step: StopPodSandbox is called with context.TODO()

[vikramhh]

Before creating an issue, make sure you've checked the following

• You are running the latest released version of k0s
• Make sure you've searched for existing issues, both open and closed
• Make sure you've searched for PRs too, a fix might've been merged already
• You're looking at docs for the released version

Platform

Ubuntu 22.04.5 LTS, kernel 6.8.0-1061-aws, x86_64
AWS EC2 t3.medium (2 vCPU, 3.7 GiB), single node

Version

v1.36.3+k0s.0

pkg/cleanup/containers.go and pkg/container/runtime/cri.go are identical between v1.36.3+k0s.0 and v1.35.4+k0s.0, so both are affected.

Sysinfo

k0s sysinfo

Total memory: 3.7 GiB (pass)
File system of /var/lib: ext4 (pass)
Disk space available for /var/lib/k0s: 26.8 GiB (pass)
Relative disk space available for /var/lib/k0s: 92% (pass)
Name resolution: localhost: [127.0.0.1] (pass)
Containerd config snippets in /etc/k0s/containerd.d/: no config snippets found (pass)
Operating system: Linux (pass)
  Linux kernel release: 6.8.0-1061-aws (pass)
  Max. file descriptors per process: current: 1048575 / max: 1048576 (pass)
  AppArmor: active (pass)
  Executable in PATH: modprobe: /usr/sbin/modprobe (pass)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: module (pass)
      CONFIG_NF_NAT: module (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: module (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: module (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connection tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
      CONFIG_NF_DEFRAG_IPV4: module (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connection tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: module (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
      CONFIG_NF_DEFRAG_IPV6: module (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass)
      CONFIG_LLC: module (pass)
      CONFIG_STP: module (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: built-in (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

k0s reset can block forever in its containers cleanup step. If the container runtime does not answer a StopPodSandbox call, reset waits indefinitely, because the call is made with a context that has neither a deadline nor cancellation.

In pkg/cleanup/containers.go, stopAllContainers():

func (c *containers) stopAllContainers() error {
	var errs []error

	var pods []string
	ctx := context.TODO()
	err := retry.Do(func() error {
		logrus.Debugf("trying to list all pods")
		var err error
		pods, err = c.containerRuntime.ListContainers(ctx)
		...
	}, retry.Context(ctx), retry.LastErrorOnly(true))
	...
	for _, pod := range pods {
		logrus.Debugf("stopping container: %v", pod)
		err := c.containerRuntime.StopContainer(ctx, pod)

Note the asymmetry: ListContainers is wrapped in retry.Do, but the StopContainer loop is not bounded in any way. That ctx reaches pkg/container/runtime/cri.go, where it is handed straight to the gRPC call:

func (cri *CRIRuntime) StopContainer(ctx context.Context, id string) error {
	...
	request := &pb.StopPodSandboxRequest{PodSandboxId: id}
	logrus.Debugf("StopPodSandboxRequest: %v", request)
	r, err := client.StopPodSandbox(ctx, request)

With context.TODO() there is nothing to bound the wait, so a runtime that is slow or wedged stops reset permanently rather than producing an error.

I reproduced it by making one container genuinely unstoppable: pinning its rootfs mount so it cannot be released. StopPodSandbox then never returns.

Steps to reproduce

k0s install controller --single && k0s start        # wait for the node to be Ready
kubectl run holder --image=busybox:1.36 -- sleep 100000
k0s stop

# pin one container's rootfs so it cannot be unmounted: a process whose cwd is the mountpoint
RF=$(mount | awk '/io.containerd.runtime.v2.task/ {print $3}' | head -1)
setsid bash -c "cd '$RF' && exec tail -f /dev/null" &
sleep 3
umount "$RF"    # confirms the pin: "target is busy"

k0s reset --data-dir /var/lib/k0s --kubelet-root-dir /var/lib/kubelet   # never returns

An open file descriptor inside the mountpoint works as well as a cwd.

Expected behavior

The CRI calls in the cleanup path are bounded, so an unresponsive container runtime makes k0s reset fail with a clear error instead of hanging. Reset should terminate one way or the other.

Actual behavior

Reset never returns. Observed still running at 30 s, 60 s and 90 s in the instrumented run, and for 25 minutes in an earlier unattended one.

Process state while hung:

State:   S (sleeping)
Threads: 9

per-thread wchan:
  futex_wait_queue   (x7)
  do_wait
  ep_poll

The container's rootfs was still mounted throughout (1 task mount present).

Screenshots and logs

SIGQUIT with GOTRACEBACK=all. Goroutine 1 is the reset itself, parked waiting for gRPC response headers:

goroutine 1 gp=0x38dfa6d261e0 m=nil [select]:
runtime.gopark(...)
	runtime/proc.go:462
runtime.selectgo(...)
	runtime/select.go:351
google.golang.org/grpc/internal/transport.(*ClientStream).waitOnHeader(0x38dfa7b688c0)
	google.golang.org/grpc@v1.82.1/internal/transport/client_stream.go:130
google.golang.org/grpc/internal/transport.(*ClientStream).RecvCompress(...)
	google.golang.org/grpc@v1.82.1/internal/transport/client_stream.go:145
google.golang.org/grpc.(*csAttempt).recvMsg(...)
	google.golang.org/grpc@v1.82.1/stream.go:1156
google.golang.org/grpc.(*clientStream).RecvMsg.func1(...)
	google.golang.org/grpc@v1.82.1/stream.go:1021
...
github.com/k0sproject/k0s/pkg/container/runtime.(*CRIRuntime).StopContainer(...)
	github.com/k0sproject/k0s/pkg/container/runtime/cri.go:87
github.com/k0sproject/k0s/pkg/cleanup.(*containers).stopAllContainers(...)
	github.com/k0sproject/k0s/pkg/cleanup/containers.go:86
github.com/k0sproject/k0s/pkg/cleanup.(*containers).Run(...)
	github.com/k0sproject/k0s/pkg/cleanup/containers.go:54
github.com/k0sproject/k0s/pkg/cleanup.(*Config).Cleanup(...)
	github.com/k0sproject/k0s/pkg/cleanup/cleanup.go:33

So the wait is inside the gRPC client with no deadline available to break it.

On signals, one observation I can offer but not fully characterise: in an earlier unattended run a SIGTERM sent at the 4-minute mark did not take effect, the process was still running 21 minutes later, and a second SIGTERM then terminated it promptly. I have only that single observation and cannot say whether the first signal was lost, deferred, or handled by a path that itself blocked. SIGQUIT always produced the dump and exited.

Additional context

Relationship to earlier reports. k0s reset hanging in this step has been reported before, and both reports were closed by the stale bot rather than fixed:

• k0s reset hangs #4211 "k0s reset hangs" (closed, Stale) — four reporters, last debug output before the hang is * containers steps / started containerd successfully / trying to list all pods, i.e. the same function. Two reporters were stuck permanently. Its author asked for exactly what is missing here: "At least it should timeout with an error."
• k0s reset hangs when containerd is broken #4783 "k0s reset hangs when containerd is broken" (closed, Stale).

In #4211 a maintainer explained the general shape — reset must stop running containers before it can clean the data directory, because their mounts block deletion. What those threads never established is why the wait can be unbounded, which is what the stack above pins down.

Relationship to #4434. PR #4434 ("Make k0s reset fail if it can't reach containerd") addressed the neighbouring case: it bounded the gRPC dial retries and wrapped ListPods in retries so an absent socket no longer hangs forever. That fix stopped short of StopContainer, which still receives context.TODO() and is not retried or bounded. This report is that remaining half.

Note the same context.TODO() is also used for managedContainerd.Init/Start in containers.Run, so the exposure is not limited to StopContainer.

Suggested fix. Give the cleanup path's CRI calls a bounded context — a per-call timeout in stopAllContainers(), or plumb a cancellable context down from k0s reset so the command can be interrupted. Either turns an unbounded hang into a reportable failure.

Related: #8048 — a different defect in the same k0s reset cleanup, where a leftover mount under the run dir makes the run-dir delete fail. The reproduction here uses the same pinned-mount trick to make a container unstoppable, but the failure is in the containers step and is independent of that one. #5657 — k0s stop leaving containers running, which is what leaves a container around for the cleanup step to trip over.