Upgrading from v1.35.4 to v1.36.2 broke kube-controller-manager

[rnalrd]

Before creating an issue, make sure you've checked the following:

• You are running the latest released version of k0s
• Make sure you've searched for existing issues, both open and closed
• Make sure you've searched for PRs too, a fix might've been merged already
• You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

Linux 6.18.37-0-lts #1-Alpine SMP PREEMPT_DYNAMIC 2026-06-29 20:26:07 x86_64 GNU/Linux
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.23.5
PRETTY_NAME="Alpine Linux v3.23"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"

Version

v1.35.4+k0s.0

Sysinfo

`k0s sysinfo`

Total memory: 62.9 GiB (pass)
File system of /var/lib/k0s: xfs (pass)
Disk space available for /var/lib/k0s: 821.2 GiB (pass)
Relative disk space available for /var/lib/k0s: 98% (pass)
Name resolution: localhost: [::1 127.0.0.1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 6.18.35-0-lts (pass)
  Max. file descriptors per process: current: 4095 / max: 4096 (warning: < 65536)
  AppArmor: unavailable (pass)
  Executable in PATH: modprobe: /sbin/modprobe (pass)
  Executable in PATH: mount: /bin/mount (pass)
  Executable in PATH: umount: /bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: module (pass)
      CONFIG_NF_NAT: module (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: module (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: module (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connection tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: unknown (warning)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: unknown (warning)
        CONFIG_IP_NF_MANGLE: Packet mangling: unknown (warning)
      CONFIG_NF_DEFRAG_IPV4: module (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connection tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: unknown (warning)
        CONFIG_IP6_NF_MANGLE: Packet mangling: unknown (warning)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: unknown (warning)
      CONFIG_NF_DEFRAG_IPV6: module (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass)
      CONFIG_LLC: module (pass)
      CONFIG_STP: module (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: module (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

The upgrade process performed via k0sctl apply -c k0sctl.yaml fails after attempting to upgrade the first worker node. Controller logs report:

time="2026-07-01 19:59:27" level=info msg="E0701 19:59:27.027799   27169 controllermanager.go:278] \"Error building controllers\" err=\"failed to mark cidr[fd00:172:16::1000/117] at idx [0] as occupied for node: k8sw-1: cidr fd00:172:16::1000/117 is out the range of cluster cidr 172.16.0.0/16\"" component=kube-controller-manager stream=stderr
time="2026-07-01 19:59:27" level=info msg="E0701 19:59:27.028034   27169 controllermanager.go:413] \"leaderelection lost/stopped\"" component=kube-controller-manager stream=stderr
time="2026-07-01 19:59:27" level=error msg="Process terminated unexpectedly: exit status 1" component=kube-controller-manager

Steps to reproduce

1. Prepare a k0sctl.yaml like:

---
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: dev-txa-1
spec:
  hosts:
    - role: controller
      openSSH:
        address: 10.196.36.205
        user: root
    - role: worker
      openSSH:
        address: 10.196.36.227
        user: root
    - role: worker
      openSSH:
        address: 10.196.36.229
        user: root
    - role: worker
      openSSH:
        address: 10.196.36.230
        user: root
  k0s:
    version: v1.35.4+k0s.0
    dynamicConfig: true
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: ClusterConfig
      metadata:
        name: dev-txa-1
      spec:
        api:
          address: 10.196.36.205
          extraArgs:
            oidc-issuer-url: https://sts.windows.net/%TENANT_UUID%/
            oidc-client-id: %CLIENT_UUID%
            oidc-username-claim: upn
            oidc-groups-claim: groups
            oidc-username-prefix: "-"
          sans:
            - <REDACTED>:36::205
            - dev-txa-1.mydomain.com
        network:
          dualStack:
            enabled: true
            IPv6podCIDR: fd00:172:16::/108
            IPv6serviceCIDR: fd00:172:31::/108
          kubeProxy:
            mode: nftables
          podCIDR: 172.16.0.0/16
          provider: kuberouter
          serviceCIDR: 172.31.0.0/16
        storage:
          type: etcd
        telemetry:
          enabled: false

2. Run k0sctl appy -c k0sctl.yaml
3. Login into the controller node. Observe that the "--cluster-cidr" option values are fd00:172:16::/108,172.16.0.0/16 (IPv6/IPv4)

ps ax | grep controller-manager

5. Change k0s.version to v1.36.2+k0s.0 in k0sctl.yaml
6. Run k0sctl appy -c k0sctl.yaml
7. Wait for the upgrade to fail at the first worker node
8. Login into controller node and check for logs entries containing the string failed to mark cidr
9. Observe that the "--cluster-cidr" option values are now inverted: 172.16.0.0/16,fd00:172:16::/108 (IPv4/IPv6)

Expected behavior

To be able to complete the upgrade successfully

Actual behavior

kube-controller-manager fails to start because it is validating an IPv6 node pods' CIDR against an IPv4 CIDR.

Screenshots and logs

Error building controllers" err="failed to mark cidr[fd00:172:16::1000/117] at idx [0] as occupied for node: k8sw-1: cidr fd00:172:16::1000/117 is out the range of cluster cidr 172.16.0.0/16"

Additional context

I've tried to revert the cluster-cidr values to the original installation order using the option described here but the order did not change.

I also tried to use option primaryAddressFamily=IPv6 but it clashes with spec.api.address which is an IPv4 address.

cc @ncopa

A couple of issues that might be related:

#5116

#7186

[twz123]

Right, you're apparently suffering from a bug fix in #7684. Before that, k0s used the primary address family detection only for the Service CIDRs, and never for the Pod CIDRs. The latter were IPv6 first unconditionally. Now they're in lock step. This is definitely the right thing to do, but it breaks existing dual-stack clusters using IPv4 as primary family that have been bootstrapped with k0s < 1.36, when k0s used IPv4 for the Service CIDR, but IPv6 for the Pod CIDR. That problem got completely unnoticed during review. However, you might just specify the cluster CIDR explicitly in your config to get the pre 1.36 behavior:

  spec:
    controllerManager:
      extraArgs:
        cluster-cidr: "fd00:172:16::/108,172.16.0.0/16"

[rnalrd]

It doesn't look like that setting controllerManager.extraArgs is having any effect:

~# grep -A5 -B5 controllerManager /etc/k0s/k0s.yaml 
    - 10.116.36.244
    - a:b:101:36::252
    - a:b:101:36::245
    - a:b:101:36::244
    - qa-wkl-1.mydomain.com
  controllerManager:
    extraArgs:
      cluster-cidr: "fd00:172:16::/108,172.16.0.0/16"
  network:
    dualStack:
      IPv6podCIDR: fd00:172:16::/108
~# rc-service k0scontroller stop
 * Stopping k0s ...                                                                                                                                                                                                                                      [ ok ]
~# ps ax | grep k0s
 2904 root      0:00 grep k0s
~# rc-service k0scontroller start
 * Starting k0s ...                                                                                                                                                                                                                                      [ ok ]
~# ps ax | grep kube-controller-manager
 3229 kube-api  0:00 /var/lib/k0s/bin/kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/k0s/pki/ccm.conf --authorization-kubeconfig=/var/lib/k0s/pki/ccm.conf --bind-address=127.0.0.1 --client-ca-file=/var/lib/k0s/pki/ca.crt --cluster-cidr=172.16.0.0/16,fd00:172:16::/108 --cluster-name=k0s --cluster-signing-cert-file=/var/lib/k0s/pki/ca.crt --cluster-signing-key-file=/var/lib/k0s/pki/ca.key --controllers=*,bootstrapsigner,tokencleaner --feature-gates= --kubeconfig=/var/lib/k0s/pki/ccm.conf --leader-elect=true --node-cidr-mask-size-ipv4=24 --node-cidr-mask-size-ipv6=117 --profiling=false --requestheader-client-ca-file=/var/lib/k0s/pki/front-proxy-ca.crt --root-ca-file=/var/lib/k0s/pki/ca.crt --service-account-private-key-file=/var/lib/k0s/pki/sa.key --service-cluster-ip-range=172.31.0.0/16,fd00:172:31::/108 --terminated-pod-gc-threshold=12500 --use-service-account-credentials=true --v=1
 3246 root      0:00 grep kube-controller-manager

[twz123]

@rnalrd According to the k0s config you've shared, you're using dynamic config. That means you have to edit the k0s config stored in the cluster, not the one on disk. You can use something like kubectl -n kube-system edit clusterconfig k0s, or, if you prefer a one-shot yolo command: kubectl -n kube-system patch clusterconfig k0s --type merge -p '{"spec":{"controllerManager":{"extraArgs":{"cluster-cidr":"fd00:172:16::/108,172.16.0.0/16"}}}}'.

[flyzstu]

One possible compatibility approach would be to derive the Pod CIDR family order from the existing Nodes before starting kube-controller-manager, instead of unconditionally changing the order during an upgrade.

The important distinction is that legacy dual-stack clusters may legitimately have different persisted orders for the two CIDR types because of the old k0s behavior:

• Service CIDRs: IPv4, IPv6
• Pod CIDRs / existing Node.spec.podCIDRs: IPv6, IPv4

Therefore, changing the cluster-wide primary address family is probably not safe: it would also change the Service CIDR order. Reverting #7684 globally is also risky because clusters created by 1.36 may already have IPv4-first Pod CIDRs.

A conservative resolution could be:

1. For single-stack clusters, keep the current behavior.
2. If controllerManager.extraArgs.cluster-cidr is explicitly set, keep treating it as the highest-priority override.
3. Otherwise, for a dual-stack cluster, inspect existing Nodes with two allocated spec.podCIDRs values.
4. If all existing Nodes consistently use IPv6-first Pod CIDRs, start kube-controller-manager with the configured IPv6 Pod CIDR first. If they consistently use IPv4-first, use IPv4 first.
5. If there are no allocated Nodes (a new cluster), fall back to the current BuildPodCIDR(primaryAddressFamily) behavior.
6. If Nodes contain inconsistent family ordering, fail with an actionable error instead of letting kube-controller-manager fail later with an out the range of cluster cidr message.

This would preserve the persisted NodeIPAM ordering during upgrades without mutating Node objects or changing Service CIDR semantics. It should work for k0sctl, Autopilot, and manual binary upgrades because the compatibility logic would live in k0s itself.

Suggested regression coverage:

• pre-1.36 legacy Nodes with IPv6-first Pod CIDRs remain operational after upgrade;
• newly bootstrapped 1.36+ clusters still use the configured primary family first;
• existing IPv4-first Nodes remain IPv4-first;
• explicit cluster-cidr overrides always win;
• mixed Node family ordering produces a clear diagnostic;
• ideally, an integration test covering a 1.35 dual-stack cluster upgraded to the patched release.

BuildPodCIDR is also used by kube-proxy, so it would be worth checking whether kube-proxy must use the detected legacy Pod CIDR order as well. I would avoid coupling this to Service CIDR ordering, since the legacy cluster can have opposite primary families for Pod and Service CIDRs.

Would this runtime compatibility detection be acceptable, or would you prefer a preflight validation plus documented/manual override instead?

[rnalrd]

@rnalrd According to the k0s config you've shared, you're using dynamic config. That means you have to edit the k0s config stored in the cluster, not the one on disk. You can use something like kubectl -n kube-system edit clusterconfig k0s, or, if you prefer a one-shot yolo command: kubectl -n kube-system patch clusterconfig k0s --type merge -p '{"spec":{"controllerManager":{"extraArgs":{"cluster-cidr":"fd00:172:16::/108,172.16.0.0/16"}}}}'.

Thanks, you're right. I confirm it works.

[rnalrd]

Would this runtime compatibility detection be acceptable, or would you prefer a preflight validation plus documented/manual override instead?

This compatibility detection looks solid to me. Thanks!