etcd restore does not support restoring onto a different node (changed IP)

[vikramhh]

Summary

k0s backup / k0s restore does not work when a backup taken on one controller is restored onto a different node with a different IP address. This is exactly the disaster-recovery (DR) scenario — rebuild a cluster from a backup onto fresh hardware/VMs — and it currently fails on the etcd member identity.

The restore always reuses the backed-up k0s config. If that config pinned the etcd peer address (spec.storage.etcd.peerAddress), the restored single-member etcd comes up still advertising the old node's peer URL in its membership store, and k0s then fails to start.

Environment

• k0s: (controller+etcd, single-node restore target)
• OS where reproduced: RHEL 8.10 and 9.6 (also affects others; SELinux enabled raises the hit rate because the slower startup lets the failure window widen)
• Scenario: backup taken on node A (NODE_A_IP), restored onto node B (NODE_B_IP)

Steps to reproduce

1. Create a cluster on node(s) with IP NODE_A_IP; take a backup (k0s backup).
2. On a fresh node with a different IP NODE_B_IP, run k0s restore <backup>.tgz and start the controller.
3. Controller fails to start cleanly; service crash-loops (and on RHEL/SELinux can fail to restart at all).

Expected behavior

Restoring a backup onto a node with a different IP should succeed — the restored single-member etcd should adopt the new node's identity/peer URL so the controller starts.

Actual behavior

k0s fails to start with:

failed to start cluster components: failed to create EtcdMember object for this controller: peer not found: https://<NEW_IP>:2380

The whole component manager (including etcd) is then shut down; systemd restarts the unit, producing a crash loop.

Root cause

1. k0s restore restores etcd from the snapshot using the backed-up config's peer address (pkg/backup/etcd_unix.go, uses nodeSpec.Storage.Etcd.PeerAddress for PeerURLs / InitialCluster):

   k0s/pkg/backup/etcd_unix.go

   Lines 72 to 93 in 6075e93

   	m := utilsnapshot.NewV3(lg)
   	name, err := os.Hostname()
   	if err != nil {
   	return err
   	}
   	u := &url.URL{
   	Scheme: "https",
   	Host: net.JoinHostPort(e.peerAddress, "2380"),
   	}
   	peerURL := u.String()
   	restoreConfig := utilsnapshot.RestoreConfig{
   	SnapshotPath: snapshotPath,
   	OutputDataDir: e.etcdDataDir,
   	PeerURLs: []string{peerURL},
   	Name: name,
   	InitialCluster: fmt.Sprintf("%s=%s", name, peerURL),
   	}
   	err = m.Restore(restoreConfig)
   	if err != nil {
   	return err
   	}

2. etcd itself starts fine and elects itself leader, but its membership store still holds the old peer URL (https://<OLD_IP>:2380). etcd's publish step only updates the advertised client URLs, not the peer URL in the membership.
3. On startup, EtcdMemberReconciler.Start() → createMemberObject → GetPeerIDByAddress(ctx, "https://<NEW_IP>:2380"). MemberList returns only the old peer URL, so there's no match → peer not found: https://<NEW_IP>:2380.
4. The retry in Start() only retries on apierrors.IsInternalError (Kubernetes API 500s). This is a custom etcd "peer not found" error, so it is never retried — all attempts fail immediately, Start() returns an error, and the component manager cancels startup, stopping etcd along with everything else.

As confirmed in discussion with maintainers: backup/restore was never designed for restoring onto another node; the config (and thus the pinned peer address) is carried over verbatim from the backup.

Why this matters

Restoring a backup onto a different node is the core of any disaster-recovery workflow — the original host is, by definition, often gone. For enterprise customers this is a hard requirement and a deal-blocker, not an edge case. DR restore always targets a single-node cluster initially, even when the original cluster had multiple members.

[twz123]

My current idea how to tackle this:

• Reconcile etcd endpoint's peer URL for single-member clusters #7875

[github-actions]

The issue is marked as stale since no activity has been recorded in 30 days

[cnd4]

/remove-lifecycle stale

Edit: oops, didn't work 😔