Host Entries got removed from CoreDNS

[NileshChandekar]

Before creating an issue, make sure you've checked the following:

• You are running the latest released version of k0s
• Make sure you've searched for existing issues, both open and closed
• Make sure you've searched for PRs too, a fix might've been merged already
• You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

[root@CTSRVITP0066 ~]# uname -srvmo; cat /etc/os-release || lsb_release -a
Linux 5.15.0-302.167.6.el9uek.x86_64 #2 SMP Mon Nov 4 23:41:59 PST 2024 x86_64 GNU/Linux
NAME="Oracle Linux Server"
VERSION="9.5"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="9.5"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Oracle Linux Server 9.5"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:9:5:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://github.com/oracle/oracle-linux"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 9"
ORACLE_BUGZILLA_PRODUCT_VERSION=9.5
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=9.5
[root@CTSRVITP0066 ~]#

Version

[root@CTSRVITP0066 ~]# k0s version v1.33.4+k0s.0 [root@CTSRVITP0066 ~]#

Sysinfo

`k0s sysinfo`

[root@CTSRVITP0066 ~]# k0s sysinfo
Total memory: 629.6 GiB (pass)
File system of /var/lib/k0s: xfs (pass)
Disk space available for /var/lib/k0s: 506.7 GiB (pass)
Relative disk space available for /var/lib/k0s: 56% (pass)
Name resolution: localhost: [::1 127.0.0.1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 5.15.0-302.167.6.el9uek.x86_64 (pass)
  Max. file descriptors per process: current: 65536 / max: 65536 (pass)
  AppArmor: unavailable (pass)
  Executable in PATH: modprobe: /usr/sbin/modprobe (pass)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: built-in (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: built-in (pass)
      CONFIG_NF_NAT: module (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: module (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: module (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connection tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
      CONFIG_NF_DEFRAG_IPV4: module (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connection tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: module (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
      CONFIG_NF_DEFRAG_IPV6: module (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: built-in (pass)
      CONFIG_LLC: built-in (pass)
      CONFIG_STP: built-in (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: module (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)
[root@CTSRVITP0066 ~]# 

What happened?

• Trying to add host entries in coreDNS

[root@CTSRVITP0066 ~]# cd /var/lib/k0s/manifests/coredns/
[root@CTSRVITP0066 coredns]# 

[root@CTSRVITP0066 coredns]# cp  coredns.yaml  /root/coredns.yaml-$(date +%F)

[root@CTSRVITP0066 coredns]# vi coredns.yaml 

 

49 ---
50 apiVersion: v1
51 kind: ConfigMap
52 metadata:
53   name: coredns
54   namespace: kube-system
55 data:
56   Corefile: |
57     .:53 {
58         errors
59         health
60         ready
61         kubernetes cluster.local in-addr.arpa ip6.arpa {
62           pods insecure
63           ttl 30
64           fallthrough in-addr.arpa ip6.arpa
65         }
66         hosts {
67           10.115.176.106 gitea.ct.lan ct.repo.local
68           10.115.72.81 taxii-server
69           10.115.176.109 hdfs-ext-svc-lb
70           10.115.176.109 csqas10g3hadoop1.cpqas10g3.ct.lan
71           10.115.176.110 csqas10G3hadoop2.cpqas10g3.ct.lan
72           10.115.176.233 csqas10g3hadoop3.cpqas10g3.ct.lan
73           172.50.33.253 probe1-ext-svc
74           10.115.72.242 omx1-ext-svc
75           192.168.0.25 omx2-ext-svc
76           10.115.72.243 i2400-ext-svc
77           10.115.176.182 object-storage-ext-svc
78           10.116.164.20 omx-simulator-ext-svc
79           10.116.164.40 speech-ext-svc
80           10.115.104.184 sm4t-ext-svc
81           10.115.74.190 phonexia-ext-svc
82           10.115.74.173 osint1-ext-svc
83           10.115.74.170 osint2-ext-svc
84           192.168.0.10 llm-ext-svc
85           172.50.33.253 probeitp0019
86           10.115.176.195 gitea.ct.lan ct.repo.local
87           10.115.176.203 gitea.ct.lan ct.repo.local
88           fallthrough
89         }
90         prometheus :9153
91         forward . /etc/resolv.conf
92         cache 30
93         loop
94         reload
95         loadbalance
96     }
97 ---

• save and exit

[ansible@CSQASINGULARCONTROLLER02 ~]$ kubectl rollout restart deployment coredns -n kube-system

[ansible@CSQASINGULARCONTROLLER02 ~]$ kubectl get cm coredns -n kube-system -oyaml
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          ttl 30
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts {
          10.115.176.106 gitea.ct.lan ct.repo.local
          10.115.72.81 taxii-server
          10.115.176.109 hdfs-ext-svc-lb
          10.115.176.109 csqas10g3hadoop1.cpqas10g3.ct.lan
          10.115.176.110 csqas10G3hadoop2.cpqas10g3.ct.lan
          10.115.176.233 csqas10g3hadoop3.cpqas10g3.ct.lan
          172.50.33.253 probe1-ext-svc
          10.115.72.242 omx1-ext-svc
          192.168.0.25 omx2-ext-svc
          10.115.72.243 i2400-ext-svc
          10.115.176.182 object-storage-ext-svc
          10.116.164.20 omx-simulator-ext-svc
          10.116.164.40 speech-ext-svc
          10.115.104.184 sm4t-ext-svc
          10.115.74.190 phonexia-ext-svc
          10.115.74.173 osint1-ext-svc
          10.115.74.170 osint2-ext-svc
          192.168.0.10 llm-ext-svc
          172.50.33.253 probeitp0019
          10.115.176.195 gitea.ct.lan ct.repo.local
          10.115.176.203 gitea.ct.lan ct.repo.local
          fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  annotations:
    k0s.k0sproject.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"Corefile":".:53 {\n    errors\n    health\n    ready\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n      pods insecure\n      ttl 30\n      fallthrough in-addr.arpa ip6.arpa\n    }\n    hosts {\n      10.115.176.106 gitea.ct.lan ct.repo.local\n      10.115.72.81 taxii-server\n      10.115.176.109 hdfs-ext-svc-lb\n      10.115.176.109 csqas10g3hadoop1.cpqas10g3.ct.lan\n      10.115.176.110 csqas10G3hadoop2.cpqas10g3.ct.lan\n      10.115.176.233 csqas10g3hadoop3.cpqas10g3.ct.lan\n      172.50.33.253 probe1-ext-svc\n      10.115.72.242 omx1-ext-svc\n      192.168.0.25 omx2-ext-svc\n      10.115.72.243 i2400-ext-svc\n      10.115.176.182 object-storage-ext-svc\n      10.116.164.20 omx-simulator-ext-svc\n      10.116.164.40 speech-ext-svc\n      10.115.104.184 sm4t-ext-svc\n      10.115.74.190 phonexia-ext-svc\n      10.115.74.173 osint1-ext-svc\n      10.115.74.170 osint2-ext-svc\n      192.168.0.10 llm-ext-svc\n      172.50.33.253 probeitp0019\n      10.115.176.195 gitea.ct.lan ct.repo.local\n      10.115.176.203 gitea.ct.lan ct.repo.local\n      fallthrough\n    }\n    prometheus :9153\n    forward . /etc/resolv.conf\n    cache 30\n    loop\n    reload\n    loadbalance\n}\n"},"kind":"ConfigMap","metadata":{"name":"coredns","namespace":"kube-system"}}
    k0s.k0sproject.io/stack-checksum: 1f56b59483f401051eb312caf41714fe
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"Corefile":".:53 {\n    errors\n    health\n    ready\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n      pods insecure\n      ttl 30\n      fallthrough in-addr.arpa ip6.arpa\n    }\n    hosts {\n      10.115.176.106 gitea.ct.lan ct.repo.local\n      10.115.72.81 taxii-server\n      10.115.176.109 hdfs-ext-svc-lb\n      10.115.176.109 csqas10g3hadoop1.cpqas10g3.ct.lan\n      10.115.176.110 csqas10G3hadoop2.cpqas10g3.ct.lan\n      10.115.176.233 csqas10g3hadoop3.cpqas10g3.ct.lan\n      172.50.33.253 probe1-ext-svc\n      10.115.72.244 omx1-ext-svc\n      192.168.0.242 omx2-ext-svc\n      10.115.72.243 i2400-ext-svc\n      10.115.176.131 object-storage-ext-svc\n      10.116.164.20 omx-simulator-ext-svc\n      10.116.164.40 speech-ext-svc\n      10.115.104.184 sm4t-ext-svc\n      10.115.74.190 phonexia-ext-svc\n      10.115.74.173 osint1-ext-svc\n      10.115.74.170 osint2-ext-svc\n      192.168.0.10 llm-ext-svc\n      172.50.33.253 probeitp0019\n      10.115.176.195 gitea.ct.lan ct.repo.local\n      10.115.176.203 gitea.ct.lan ct.repo.local\n      10.115.176.109 csqas10G3hadoop1 hdfs-ext-svc-lb\n      fallthrough\n    }\n    prometheus :9153\n    forward . /etc/resolv.conf\n    cache 30\n    loop\n    reload\n    loadbalance\n}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"k0s.k0sproject.io/stack":"coredns"},"name":"coredns","namespace":"kube-system"}}
  creationTimestamp: "2025-12-27T08:50:14Z"
  labels:
    k0s.k0sproject.io/stack: coredns
  name: coredns
  namespace: kube-system
  resourceVersion: "68286752"
  uid: fe6469ea-cd0b-4806-9664-458f21cf21cc
[ansible@CSQASINGULARCONTROLLER02 ~]$ 

[ansible@CSQASINGULARCONTROLLER02 ~]$ kubectl get pod -A | grep -i dns
kube-system      coredns-55d9465cff-kmqtr                                                 1/1     Running            0                 15m
kube-system      coredns-55d9465cff-p79dg                                                 1/1     Running            0                 15m
[ansible@CSQASINGULARCONTROLLER02 ~]$ 

 

Steps to reproduce

1. rebooted the node
2. after the reboot lost coreDNS configmap config

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          ttl 30
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
---

3. It should not get removed
4. we have tried edit the cm of coredns as well, after reboot same observed.

Expected behavior

• host entries should not get removed.

Actual behavior

• Should be present after reboot as well

49 ---
50 apiVersion: v1
51 kind: ConfigMap
52 metadata:
53   name: coredns
54   namespace: kube-system
55 data:
56   Corefile: |
57     .:53 {
58         errors
59         health
60         ready
61         kubernetes cluster.local in-addr.arpa ip6.arpa {
62           pods insecure
63           ttl 30
64           fallthrough in-addr.arpa ip6.arpa
65         }
66         hosts {
67           10.115.176.106 gitea.ct.lan ct.repo.local
68           10.115.72.81 taxii-server
69           10.115.176.109 hdfs-ext-svc-lb
70           10.115.176.109 csqas10g3hadoop1.cpqas10g3.ct.lan
71           10.115.176.110 csqas10G3hadoop2.cpqas10g3.ct.lan
72           10.115.176.233 csqas10g3hadoop3.cpqas10g3.ct.lan
73           172.50.33.253 probe1-ext-svc
74           10.115.72.242 omx1-ext-svc
75           192.168.0.25 omx2-ext-svc
76           10.115.72.243 i2400-ext-svc
77           10.115.176.182 object-storage-ext-svc
78           10.116.164.20 omx-simulator-ext-svc
79           10.116.164.40 speech-ext-svc
80           10.115.104.184 sm4t-ext-svc
81           10.115.74.190 phonexia-ext-svc
82           10.115.74.173 osint1-ext-svc
83           10.115.74.170 osint2-ext-svc
84           192.168.0.10 llm-ext-svc
85           172.50.33.253 probeitp0019
86           10.115.176.195 gitea.ct.lan ct.repo.local
87           10.115.176.203 gitea.ct.lan ct.repo.local
88           fallthrough
89         }
90         prometheus :9153
91         forward . /etc/resolv.conf
92         cache 30
93         loop
94         reload
95         loadbalance
96     }
97 ---

Screenshots and logs

[root@CTSRVITP0066 ~]# journalctl --no-pager --since "2 hours ago" | grep -w CoreDNS | head -10 
Jul 11 18:10:00 CTSRVITP0066 k0s[5512]: time="2026-07-11 18:10:00" level=info msg="initializing CoreDNS"
Jul 11 18:10:10 CTSRVITP0066 k0s[5512]: time="2026-07-11 18:10:10" level=info msg="starting CoreDNS"
Jul 11 18:10:10 CTSRVITP0066 k0s[5512]: time="2026-07-11 18:10:10" level=debug msg="checkpoint recorded" checkpoint=running-CoreDNS component=performance-timer duration=15.753336ms target=component-start
Jul 11 18:10:10 CTSRVITP0066 k0s[5512]: time="2026-07-11 18:10:10" level=debug msg="checkpoint recorded" checkpoint=running-CoreDNS-done component=performance-timer duration=15.779755ms target=component-start
Jul 11 18:10:10 CTSRVITP0066 k0s[5512]: time="2026-07-11 18:10:10" level=info msg="starting to reconcile *controller.CoreDNS"
Jul 11 18:10:10 CTSRVITP0066 k0s[5512]: time="2026-07-11 18:10:10" level=debug msg="reconcile method called for: CoreDNS"
Jul 11 18:10:20 CTSRVITP0066 k0s[5512]: time="2026-07-11 18:10:20" level=debug msg="reconcile method called for: CoreDNS"
Jul 11 18:10:30 CTSRVITP0066 k0s[5512]: time="2026-07-11 18:10:30" level=debug msg="reconcile method called for: CoreDNS"
Jul 11 18:10:40 CTSRVITP0066 k0s[5512]: time="2026-07-11 18:10:40" level=debug msg="reconcile method called for: CoreDNS"
Jul 11 18:10:50 CTSRVITP0066 k0s[5512]: time="2026-07-11 18:10:50" level=debug msg="reconcile method called for: CoreDNS"
[root@CTSRVITP0066 ~]# journalctl --no-pager --since "2 hours ago" | grep -w CoreDNS | tail  -10 
Jul 11 19:18:40 CTSRVITP0066 k0s[5512]: time="2026-07-11 19:18:40" level=debug msg="reconcile method called for: CoreDNS"
Jul 11 19:18:50 CTSRVITP0066 k0s[5512]: time="2026-07-11 19:18:50" level=debug msg="reconcile method called for: CoreDNS"
Jul 11 19:19:00 CTSRVITP0066 k0s[5512]: time="2026-07-11 19:19:00" level=debug msg="reconcile method called for: CoreDNS"
Jul 11 19:19:10 CTSRVITP0066 k0s[5512]: time="2026-07-11 19:19:10" level=debug msg="reconcile method called for: CoreDNS"
Jul 11 19:19:20 CTSRVITP0066 k0s[5512]: time="2026-07-11 19:19:20" level=debug msg="reconcile method called for: CoreDNS"
Jul 11 19:19:30 CTSRVITP0066 k0s[5512]: time="2026-07-11 19:19:30" level=debug msg="reconcile method called for: CoreDNS"
Jul 11 19:19:40 CTSRVITP0066 k0s[5512]: time="2026-07-11 19:19:40" level=debug msg="reconcile method called for: CoreDNS"
Jul 11 19:19:50 CTSRVITP0066 k0s[5512]: time="2026-07-11 19:19:50" level=debug msg="reconcile method called for: CoreDNS"
Jul 11 19:20:00 CTSRVITP0066 k0s[5512]: time="2026-07-11 19:20:00" level=debug msg="reconcile method called for: CoreDNS"
Jul 11 19:20:10 CTSRVITP0066 k0s[5512]: time="2026-07-11 19:20:10" level=debug msg="reconcile method called for: CoreDNS"
[root@CTSRVITP0066 ~]# 

Additional context

No response

[NileshChandekar]

https://docs.k0sproject.io/v1.33.4+k0s.0/manifests/

[NileshChandekar]

This is my understanding as well:-

every 10s

Jul 11 19:20:00 CTSRVITP0066 k0s[5512]: time="2026-07-11 19:20:00" level=debug msg="reconcile method called for: CoreDNS"
Jul 11 19:20:10 CTSRVITP0066 k0s[5512]: time="2026-07-11 19:20:10" level=debug msg="reconcile method called for: CoreDNS"

k0s's CoreDNS component controller regenerates the Corefile from its own internal Go template and writes/applies it via a kubectl apply-style operation (that's why the ConfigMap has a kubectl.kubernetes.io/last-applied-configurationand a k0s-specific k0s.k0sproject.io/stack-checksum/last-applied-configuration annotation).

That internal template has no concept of your custom hosts {} block. So whenever the controller reconciles — on boot, or on its normal loop — it recomputes the "desired" Corefile from the template and re-applies it, silently dropping anything you added by hand.

[NileshChandekar]

On running production system:-

Plan A - Not sure whether this will work

++++++++++
Action Plan:
++++++++++

- ssh:
      address: 10.115.176.104
      user: ansible
      port: 22
    role: controller+worker
    noTaints: true
    installFlags:
    - --debug
    - --kubelet-extra-args="--max-pods=500"
    - --disable-components=coredns
    uploadBinary: true
    k0sBinaryPath: /opt/k0s-releases/k0s-v1.33.4+k0s.0-amd64

k0sctl apply -c cpqas10g3.yaml

k0sctl will detect the changed installFlags, re-run the k0s install step on 10.115.176.104 with the new flag, and restart the k0scontroller service on that node. Since it's your only controller you'll get a short API/DNS blip during the restart — pick a low-traffic window.

Safer cutover order for your single controller+worker setup:

1. Grab the exact backups (Step 1 from before) — CM, Deployment, Service — while CoreDNS is still healthy.
2. Build your replacement manifest now, with your hosts block folded into the Corefile, using the same names (coredns Deployment, kube-dns Service, same namespace/labels) so kube-proxy/kubelet DNS config (--cluster-dns) keeps pointing at the same ClusterIP without needing kubelet changes.
3. Save that replacement YAML locally (not yet applied) and have the kubectl apply -f command ready to run immediately.
4. In one maintenance window: update cpqas10g3.yaml with --disable-components=coredns, run k0sctl apply -c cpqas10g3.yaml — this will restart the controller and prune the old CoreDNS objects — then immediately kubectl apply -f your-coredns-replacement.yaml to bring DNS back up under your own management.
5. Verify pods come up and resolution works before considering it done.

Plan B:

Put all the host entries in /etc/hosts on all the nodes via ansible this is airgapped environemnt, no external DNS , no forwarder.

[NileshChandekar]

#4021

[NileshChandekar]

I dont think so on running cluster we can run this or running this is very risky

- ssh:
      address: 10.115.176.104
      user: ansible
      port: 22
    role: controller+worker
    noTaints: true
    installFlags:
    - --debug
    - --kubelet-extra-args="--max-pods=500"
    - --disable-components=coredns
    uploadBinary: true
    k0sBinaryPath: /opt/k0s-releases/k0s-v1.33.4+k0s.0-amd64

[NileshChandekar]

Runbook: Disable k0s-managed CoreDNS and take over with custom-coredns

Cluster: 3-node HA control-plane, k0s v1.33.4+k0s Nodes (all run control-plane + --enable-worker=true):

Role	Hostname	IP
Master	CTRLNODE01	10.0.0.11
Master	CTRLNODE02	10.0.0.12
Master	CTRLNODE03	10.0.0.13

Purpose: Stop k0s from reverting custom hosts {} entries in CoreDNS by disabling the k0s-managed coredns component on all three controllers and running an independently managed CoreDNS.

Audience: L1 — follow every step in order. Do not skip verification steps. Do not skip ahead to the next node until the current node's health check passes.

Window required: Yes. This is a 3-node HA cluster — each node's control-plane restart is individually brief, but the full cutover across all three plus final DNS cutover should be treated as one maintenance window from start to finish.

Why this must be done on all three nodes, not just one: every controller runs its own reconciler for the coredns manifest. If you disable it on only one node, the other two will keep re-applying k0s's default CoreDNS config and silently undo your custom hosts entries the next time either of them handles the reconcile — the exact problem this runbook exists to fix, just intermittent and much harder to diagnose. All three must be disabled before you apply the custom manifests.

---

0. Before you start — hard stop conditions

Do not proceed if any of these are true. Escalate instead:

• hostname does not return one of CTRLNODE01, CTRLNODE02, or CTRLNODE03. Confirm which of the three you're on before running any per-node step.
• You are not inside an approved maintenance window.
• kubectl get nodes does not show all three nodes as Ready right now.
• kubectl -n kube-system get pods -l k8s-app=kube-dns does not show 2 CoreDNS pods Running.
• Any step below produces an error you don't understand. Stop and escalate — do not guess.
• You are about to restart a second controller before the previous one has come back fully Ready. Never have more than one controller down or restarting at a time — this is an etcd quorum cluster; taking two of three down loses quorum and the whole API goes down.

---

1. Pre-checks (read-only, safe to run any time)

kubectl get nodes -o wide
kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
Baseline — confirm DNS works before you touch anything
kubectl run dns-baseline --image=busybox:1.35 --restart=Never --rm -it -- 

nslookup gitea.example.internal

Expected: all 3 nodes Ready, 2 CoreDNS pods Running, nslookup resolves gitea.example.internal. If this fails before you've changed anything, stop and escalate.

---

2. Backups — mandatory, do not skip

Run once, from any node with kubectl access (these are cluster-wide objects, not per-node):

mkdir -p ~/coredns-cutover-backup && cd ~/coredns-cutover-backup
kubectl -n kube-system get cm coredns -o yaml                        > coredns-cm-backup.yaml

kubectl -n kube-system get deploy coredns -o yaml                    > coredns-deploy-backup.yaml

kubectl -n kube-system get svc kube-dns -o yaml                       > coredns-svc-backup.yaml

kubectl -n kube-system get pdb coredns -o yaml                       > coredns-pdb-backup.yaml

kubectl -n kube-system get sa coredns -o yaml                        > coredns-sa-backup.yaml            2>/dev/null

kubectl -n kube-system get clusterrole system-coredns -o yaml        > coredns-clusterrole-backup.yaml       2>/dev/null

kubectl -n kube-system get clusterrolebinding system-coredns -o yaml > coredns-clusterrolebinding-backup.yaml 2>/dev/null
ls -la ~/coredns-cutover-backup

Confirm all files are non-empty before continuing.

Confirm the cluster IP — known fixed value for k0s CoreDNS is 10.96.0.10:

grep -A2 "clusterIP" ~/coredns-cutover-backup/coredns-svc-backup.yaml

If this does not show 10.96.0.10, stop and escalate.

Confirm install flags are identical on all three nodes — run on each node separately, they must all match:

ps aux | grep '[k]0s controller'

Confirmed for this cluster (all 3 nodes, verified identical):

/usr/local/bin/k0s controller --config=/etc/k0s/k0s.yaml --data-dir=/var/lib/k0s \
  --debug=true --enable-worker=true --kubelet-extra-args=--max-pods=500 --no-taints=true

Re-run this on each node right before that node's Step 7a — if any node's output no longer matches the line above, stop and escalate before proceeding on that node.

---

3. Prepare the custom hosts ConfigMap (new file)

Carries forward every entry currently in the live hosts {} block, unchanged. Save as coredns-custom-hosts-cm.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom-hosts
  namespace: kube-system
data:
  hosts.db: |
    10.0.1.10 gitea.example.internal repo.example.internal
    10.0.1.11 taxii-server
    10.0.1.12 hdfs-ext-svc-lb
    10.0.1.12 hadoop-node1.cluster1.example.internal
    10.0.1.13 hadoop-node2.cluster1.example.internal
    10.0.1.14 hadoop-node3.cluster1.example.internal
    10.0.1.15 probe1-ext-svc
    10.0.1.16 omx1-ext-svc
    10.0.1.17 omx2-ext-svc
    10.0.1.18 i2400-ext-svc
    10.0.1.19 object-storage-ext-svc
    10.0.1.20 omx-simulator-ext-svc
    10.0.1.21 speech-ext-svc
    10.0.1.22 sm4t-ext-svc
    10.0.1.23 phonexia-ext-svc
    10.0.1.24 osint1-ext-svc
    10.0.1.25 osint2-ext-svc
    10.0.1.26 llm-ext-svc
    10.0.1.15 probe-node-alt
    10.0.1.27 gitea.example.internal repo.example.internal
    10.0.1.28 gitea.example.internal repo.example.internal
    10.0.1.12 hadoop-node1 hdfs-ext-svc-lb
    10.0.1.29 omx1-ext-svc
    10.0.1.30 omx2-ext-svc
    10.0.1.31 object-storage-ext-svc

⚠ Known issue, not fixed automatically by this runbook: several names resolve to more than one IP — gitea.example.internal/repo.example.internal (.106 / .195 / .203), omx1-ext-svc (.244 / .242), omx2-ext-svc (10.0.1.17 / .25), and object-storage-ext-svc (.131 / .182). The hosts plugin returns all matching IPs for a duplicated name, which can cause clients to pick the wrong / stale one at random. Carried forward as-is deliberately — Nilesh must confirm the correct current IP for each of these four names and prune the stale entries in a follow-up change.

---

4. Prepare the patched CoreDNS ConfigMap

Save as coredns-cm-patched.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          ttl 30
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts /etc/coredns/custom/hosts.db {
          reload 30s
          fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }

---

5. Prepare the patched Deployment (verified against k0s's own source manifest)

Save as coredns-deploy-patched.yaml. This is k0s's actual rendered CoreDNS manifest with exactly two additions marked below — everything else (image, replicas, tolerations, affinity, probes, security context) is unchanged:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/name: "CoreDNS"
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '9153'
    spec:
      priorityClassName: system-cluster-critical
      serviceAccountName: coredns
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      nodeSelector:
        kubernetes.io/os: linux
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - topologyKey: "kubernetes.io/hostname"
            labelSelector:
              matchExpressions:
              - key: k8s-app
                operator: In
                values: ['kube-dns']
      containers:
      - name: coredns
        image: quay.io/k0sproject/coredns:1.12.2
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            cpu: 100m
            memory: 70Mi
        args: [ "-conf", "/etc/coredns/Corefile" ]
        volumeMounts:
        - name: config-volume
          mountPath: /etc/coredns
          readOnly: true
        - name: custom-hosts-volume          # <-- added
          mountPath: /etc/coredns/custom     # <-- added
          readOnly: true                     # <-- added
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          periodSeconds: 10
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /ready
            port: 8181
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 2
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 3
      dnsPolicy: Default
      volumes:
        - name: config-volume
          configMap:
            name: coredns
            items:
            - key: Corefile
              path: Corefile
        - name: custom-hosts-volume          # <-- added
          configMap:                         # <-- added
            name: coredns-custom-hosts        # <-- added
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/name: "CoreDNS"
spec:
  minAvailable: 50%
  selector:
    matchLabels:
      k8s-app: kube-dns

Before applying this, confirm it still matches this cluster's actual rendered manifest. If /root/coredns.yaml or an equivalent exists on one of these three nodes, diff it against the above. If it differs (different k0s version rendering different probe ports, tolerations, etc.), update this file to match reality — do not apply blindly.

---

6. Pin the Service (edit, don't rewrite from scratch)

Copy coredns-svc-backup.yaml to coredns-svc-patched.yaml. No changes needed — confirm clusterIP: 10.96.0.10 is present before continuing.

---

7. Cutover — Phase A: disable coredns on all three controllers, one at a time

Do this node by node. Do not start the next node until the current node passes its health check.

7A.1 — Node CTRLNODE01 (10.0.0.11)

# On CTRLNODE01 — confirm you're on the right node first
hostname   # must print CTRLNODE01
Re-confirm flags haven't drifted since Step 2
ps aux | grep '[k]0s controller'
k0s install controller --force 

--config=/etc/k0s/k0s.yaml 

--data-dir=/var/lib/k0s 

--debug=true 

--enable-worker=true 

--kubelet-extra-args=--max-pods=500 

--no-taints=true 

--disable-components=coredns
systemctl list-units | grep k0s     # confirm exact unit name first

sudo systemctl restart k0scontroller

Health check before continuing — all must pass:

kubectl get nodes                                    # CTRLNODE01 back to Ready
kubectl -n kube-system get pods -l k8s-app=kube-dns   # still 2 pods Running
kubectl get --raw='/readyz?verbose' | grep -i etcd    # etcd healthy, quorum intact

If kubectl get nodes doesn't respond within 2 minutes, or any check fails, stop and escalate — do not proceed to node 2.

7A.2 — Node CTRLNODE02 (10.0.0.12)

Repeat the exact same sub-steps as 7A.1, replacing the hostname check with CTRLNODE02. Run the same health check before continuing to node 3.

7A.3 — Node CTRLNODE03 (10.0.0.13)

Repeat the exact same sub-steps as 7A.1, replacing the hostname check with CTRLNODE03. Run the same health check before continuing to Phase B.

Do not proceed to Phase B until all three nodes show --disable-components=coredns in their running process (re-check with ps aux | grep '[k]0s controller' on each) and all three nodes are Ready.

---

8. Cutover — Phase B: apply the custom CoreDNS stack (once, from any node)

Only after all three controllers are confirmed disabled:

cd ~/coredns-cutover-backup
Only if these existed in Step 2 (skip any that errored as "not found")
kubectl apply -f coredns-sa-backup.yaml

kubectl apply -f coredns-clusterrole-backup.yaml

kubectl apply -f coredns-clusterrolebinding-backup.yaml
kubectl apply -f coredns-custom-hosts-cm.yaml   # Step 3

kubectl apply -f coredns-cm-patched.yaml        # Step 4

kubectl apply -f coredns-deploy-patched.yaml    # Step 5 (Deployment + PDB)

kubectl apply -f coredns-svc-patched.yaml       # Step 6

---

9. Verification — do not close the window until every check passes

kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50
kubectl run dns-test --image=busybox:1.35 --restart=Never --rm -it -- 

nslookup kubernetes.default.svc.cluster.local
kubectl run dns-test2 --image=busybox:1.35 --restart=Never --rm -it -- 

nslookup gitea.example.internal
kubectl run dns-test3 --image=busybox:1.35 --restart=Never --rm -it -- 

nslookup hadoop-node1.cluster1.example.internal
Confirm k0s is no longer reconciling: edit the hosts ConfigMap, wait, confirm it sticks
kubectl -n kube-system edit cm coredns-custom-hosts
add a harmless test line, save, wait 60s, then:
kubectl -n kube-system get cm coredns-custom-hosts -o yaml   # confirm your edit is still there

All must pass. If any pod is not Running/Ready, or any nslookup fails, go to Rollback immediately — do not leave the cluster half-migrated.

---

10. Rollback (if verification fails)

Re-enable coredns on all three nodes (reverse order doesn't matter, but still one at a time with health checks between):

# On each node in turn:
k0s install controller --force \
  --config=/etc/k0s/k0s.yaml \
  --data-dir=/var/lib/k0s \
  --debug=true \
  --enable-worker=true \
  --kubelet-extra-args=--max-pods=500 \
  --no-taints=true
sudo systemctl restart k0scontroller

kubectl get nodes   # wait for that node to be Ready before moving to the next

Once all three are back to baseline flags, restore the original objects (once, from any node):

cd ~/coredns-cutover-backup
kubectl apply -f coredns-cm-backup.yaml
kubectl apply -f coredns-deploy-backup.yaml
kubectl apply -f coredns-svc-backup.yaml
kubectl apply -f coredns-pdb-backup.yaml
kubectl -n kube-system delete cm coredns-custom-hosts

Re-run the Step 1 baseline check to confirm you're back to the starting state, then escalate to Nilesh with logs from Step 9.

---

11. Sign-off

• All Step 9 verification checks passed
• Confirmed all 3 nodes (CTRLNODE01/002/003) show --disable-components=coredns
• Duplicate-IP entries (Step 3 warning) flagged to Nilesh for cleanup
• Backups from Step 2 retained (do not delete ~/coredns-cutover-backup)
• Maintenance window closed / stakeholders notified

Executed by: _______________ Date/time: _______________ Result: ☐ Success ☐ Rolled back

[NileshChandekar]

Hi @jnummelin @twz123 , hope you're doing well. Could you please help review/check my message above when you get a chance? Would really appreciate your input on this. Thanks so much for your time!

[NileshChandekar]

Hi Any update?

[jnummelin]

the general plan looks good, of course I'm not able to test it with your environment etc. :D

We've recently, in 1.36, introduced also a way to patch any of the k0s managed Kubernetes resources: https://docs.k0sproject.io/stable/configuration/#component-patches

Your use case looks like you'd want to "edit" the CoreDNS configmap, so you could use the patches feature to basically provide your own coredns configmap. Somethin like:

spec
  network
    coreDNS
      patches:
        - target:
            kind: ConfigMap
            name: coredns
            namespace: kube-system
            patch:
              type: StrategicMergePatch
              content: |
                data:
                  Corefile: |
                     .:53 {
                     .....

[NileshChandekar]

Thanks, @jnummelin