[release-4.20] OCPBUGS-77313: Wait for revision stability before removing etcd members by hasbro17 · Pull Request #1571 · openshift/cluster-etcd-operator

hasbro17 · 2026-03-13T17:33:12Z

Manual cherry-pick of #1555

/cherrypick release-4.19 release-4.18

Replaces automated cherry-pick in #1559

Needs to wait on DeleteAll test from openshift/origin#30882
/hold

Previously, the ClusterMemberRemovalController would remove etcd members during revision rollouts, causing cluster degradation when simultaneously deleting multiple control plane machines with the OnDelete strategy. During a revision rollout, etcd members can temporarily appear unhealthy while their pods are reinstalled to the latest revision. This is different from members being indefinitely unhealthy on a stable revision. Additionally, the EtcdEndpointsController pauses during revision rollouts, so when a replacement machine is added and triggers a rollout, the etcd-endpoints configmap won't update. This causes API servers on the old revision to use removed member endpoints, leading to API unavailability. This change adds a revision stability check before allowing member removal, ensuring we only remove members when revisions are stable and unhealthy members are truly unhealthy. This explicitly codifies the 4.17 behavior where the operator waited for all revisions to complete before removing members and lifecycle hooks. Additionally, the ClusterMemberRemovalController now verifies that the live etcd membership matches the configmap before proceeding with member removal, preventing potential issues during rapid member deletion (cherry picked from commit 0168733)

openshift-ci-robot · 2026-03-13T17:33:19Z

@hasbro17: This pull request references Jira Issue OCPBUGS-74151, which is invalid:

expected the bug to target either version "4.20." or "openshift-4.20.", but it targets "4.22.0" instead
expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is Verified instead
release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
expected Jira Issue OCPBUGS-74151 to depend on a bug targeting a version in 4.21.0, 4.21.z and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Manual cherry-pick of #1555

/cherrypick release-4.19 release-4.18

Replaces automated cherry-pick in #1559

Needs to wait on DeleteAll test from openshift/origin#30882
/hold

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-03-13T17:33:27Z

Important

Review skipped

Auto reviews are limited based on label configuration.

🚫 Review skipped — only excluded labels are configured. (1)

do-not-merge/work-in-progress

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 785fc5a7-43a6-49dc-b0d9-0a935fd3c58c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

📝 Coding Plan

Generate coding plan for human review comments

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

openshift-ci-robot · 2026-03-13T17:34:26Z

@hasbro17: This pull request references Jira Issue OCPBUGS-77313, which is valid.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.20.z) matches configured target version for branch (4.20.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
release note type set to "Release Note Not Required"
dependent bug Jira Issue OCPBUGS-77097 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-77097 targets the "4.21.z" version, which is one of the valid target versions: 4.21.0, 4.21.z
bug has dependents

Requesting review from QA contact:
/cc @geliu2016

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Manual cherry-pick of #1555

/cherrypick release-4.19 release-4.18

Replaces automated cherry-pick in #1559

Needs to wait on DeleteAll test from openshift/origin#30882
/hold

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2026-03-13T20:56:45Z

@hasbro17: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

dusk125 · 2026-03-16T14:08:34Z

/lgtm
/label backport-risk-assessed

openshift-ci · 2026-03-16T14:14:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dusk125, hasbro17

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [dusk125,hasbro17]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

hasbro17 · 2026-04-22T17:36:03Z

/test e2e-aws-ovn-etcd-scaling

hasbro17 · 2026-04-22T17:43:27Z

/payload-aggregate e2e-aws-ovn-etcd-scaling 5

openshift-ci · 2026-04-22T17:43:31Z

@hasbro17: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

e2e-aws-ovn-etcd-scaling

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/c56265b0-3e72-11f1-8209-442772e910fc-0

hasbro17 · 2026-04-28T16:58:27Z

/verified by ci

openshift-ci-robot · 2026-04-28T16:58:39Z

@hasbro17: This PR has been marked as verified by ci.

Details

In response to this:

/verified by ci

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

hasbro17 · 2026-04-28T17:00:56Z

/unhold

openshift-ci-robot · 2026-04-28T17:04:34Z

@hasbro17: Jira Issue Verification Checks: Jira Issue OCPBUGS-77313
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-77313 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

Manual cherry-pick of #1555

/cherrypick release-4.19 release-4.18

Replaces automated cherry-pick in #1559

Needs to wait on DeleteAll test from openshift/origin#30882
/hold

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-cherrypick-robot · 2026-04-28T17:05:21Z

@hasbro17: new pull request created: #1603

Details

In response to this:

Manual cherry-pick of #1555

/cherrypick release-4.19 release-4.18

Replaces automated cherry-pick in #1559

Needs to wait on DeleteAll test from openshift/origin#30882
/hold

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-merge-robot · 2026-04-29T15:17:32Z

Fix included in release 4.20.0-0.nightly-2026-04-29-032740

hasbro17 · 2026-05-07T17:22:13Z

/cherrypick release-4.19

openshift-cherrypick-robot · 2026-05-07T17:23:02Z

@hasbro17: new pull request could not be created: failed to create pull request against openshift/cluster-etcd-operator#release-4.19 from head openshift-cherrypick-robot:cherry-pick-1571-to-release-4.19: status code 422 not one of [201], body: {"message":"Validation Failed","errors":[{"resource":"PullRequest","code":"custom","message":"A pull request already exists for openshift-cherrypick-robot:cherry-pick-1571-to-release-4.19."}],"documentation_url":"https://docs.github.com/rest/pulls/pulls#create-a-pull-request","status":"422"}

Details

In response to this:

/cherrypick release-4.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

hasbro17 · 2026-05-07T17:25:02Z

/cherrypick release-4.19

openshift-cherrypick-robot · 2026-05-07T17:25:50Z

@hasbro17: new pull request created: #1613

Details

In response to this:

/cherrypick release-4.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

hasbro17 requested a review from dusk125 March 13, 2026 17:33

openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 13, 2026

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Mar 13, 2026

openshift-ci Bot requested a review from ironcladlou March 13, 2026 17:33

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 13, 2026

hasbro17 changed the title ~~OCPBUGS-74151: Wait for revision stability before removing etcd members~~ [release-4.20] OCPBUGS-77313: Wait for revision stability before removing etcd members Mar 13, 2026

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Mar 13, 2026

openshift-ci Bot requested a review from geliu2016 March 13, 2026 17:34

hasbro17 mentioned this pull request Mar 13, 2026

[release-4.20] OCPBUGS-77313: Wait for revision stability before removing etcd members #1559

Closed

openshift-ci Bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Mar 16, 2026

openshift-ci Bot assigned dusk125 Mar 16, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Mar 16, 2026

openshift-ci-robot mentioned this pull request Apr 22, 2026

[release-4.20]: OCPBUGS-77313: Add test for CPMS OnDelete strategy with full master replacement openshift/origin#30882

Merged

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Apr 28, 2026

openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 28, 2026

openshift-merge-bot Bot merged commit 9fb46bc into openshift:release-4.20 Apr 28, 2026
14 of 15 checks passed

openshift-cherrypick-robot mentioned this pull request Apr 28, 2026

[release-4.19] OCPBUGS-77313: Wait for revision stability before removing etcd members #1603

Closed

openshift-cherrypick-robot mentioned this pull request May 7, 2026

[release-4.19] OCPBUGS-85275: Wait for revision stability before removing etcd members #1613

Merged

Conversation

hasbro17 commented Mar 13, 2026

Uh oh!

openshift-ci-robot commented Mar 13, 2026

Uh oh!

coderabbitai Bot commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

openshift-ci-robot commented Mar 13, 2026

Uh oh!

openshift-ci Bot commented Mar 13, 2026

Uh oh!

dusk125 commented Mar 16, 2026

Uh oh!

openshift-ci Bot commented Mar 16, 2026

Uh oh!

hasbro17 commented Apr 22, 2026

Uh oh!

hasbro17 commented Apr 22, 2026

Uh oh!

openshift-ci Bot commented Apr 22, 2026

Uh oh!

hasbro17 commented Apr 28, 2026

Uh oh!

openshift-ci-robot commented Apr 28, 2026

Uh oh!

hasbro17 commented Apr 28, 2026

Uh oh!

Uh oh!

openshift-ci-robot commented Apr 28, 2026

Uh oh!

openshift-cherrypick-robot commented Apr 28, 2026

Uh oh!

openshift-merge-robot commented Apr 29, 2026

Uh oh!

hasbro17 commented May 7, 2026

Uh oh!

openshift-cherrypick-robot commented May 7, 2026

Uh oh!

hasbro17 commented May 7, 2026

Uh oh!

openshift-cherrypick-robot commented May 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

coderabbitai Bot commented Mar 13, 2026 •

edited

Loading