Skip to content

Defer annotation update until after watchdog is started - #334

Open
kwohlfahrt wants to merge 4 commits into
medik8s:mainfrom
kwohlfahrt:watchdog-timeout
Open

Defer annotation update until after watchdog is started#334
kwohlfahrt wants to merge 4 commits into
medik8s:mainfrom
kwohlfahrt:watchdog-timeout

Conversation

@kwohlfahrt

@kwohlfahrt kwohlfahrt commented Aug 4, 2026

Copy link
Copy Markdown

Why we need this PR

Currently, the watchdog timeout is read upon creating the watchdog, before starting it. This always returns a value of 0, because the timeout is only available after the watchdog's start.

Changes made

  • Add a Started() method to the watchdog interface, that notifies when the watchdog is started
  • Add a new runnable that waits for the watchdog to start, and then calls the annotation updater
  • Fix a bug where watchdog initialization errors were not surfaced when software fallback is disabled

Which issue(s) this PR fixes

Fixes #332

Test plan

Added new unit test. I'm open to suggestions for a more thorough e2e test, but am not familiar enough with the test setup to come up with a clean idea.

kwohlfahrt and others added 2 commits August 4, 2026 12:55
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@openshift-ci
openshift-ci Bot requested review from jmontleon and mshitrit August 4, 2026 12:13
@openshift-ci

openshift-ci Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kwohlfahrt
Once this PR has been reviewed and has the lgtm label, please assign mpryc for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci

openshift-ci Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Hi @kwohlfahrt. Thanks for your PR.

I'm waiting for a medik8s member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Summary by CodeRabbit

  • Bug Fixes
    • Improved startup status reporting for the agent’s reboot watchdog.
    • Reboot capability is now updated only after watchdog startup completes.
    • Reboot capability is correctly cleared when no watchdog is available.
    • Watchdog start failures and software-reboot fallback are handled more reliably, with consistent timeout and malfunction behavior.
  • Tests
    • Added coverage for watchdog start signalling and start-failure scenarios (including software reboot enabled).

Walkthrough

The watchdog now signals startup completion through Started(). The agent waits for this signal before updating the node annotation with watchdog state and timeout. Tests cover successful startup and software-reboot fallback.

Changes

Watchdog startup and annotation

Layer / File(s) Summary
Watchdog startup signal
internal/watchdog/interface.go, internal/watchdog/synchronized.go, internal/watchdog/watchdog_test.go
The Watchdog interface exposes Started(). The synchronized watchdog closes the channel after hardware startup or software fallback. Tests verify startup signalling, timeout, and malfunction state.
Startup-aware annotation update
cmd/main.go
Watchdog-enabled agents wait for startup before updating the node annotation. Agents without a watchdog set non-reboot-capable state with a zero timeout.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Agent
  participant synchronizedWatchdog
  participant NodeAnnotation
  Agent->>synchronizedWatchdog: wait for Started()
  synchronizedWatchdog-->>Agent: startup channel closes
  Agent->>NodeAnnotation: update armed state and timeout
Loading

Suggested reviewers: jmontleon, mshitrit

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The changes address issue #332 by updating the annotation after startup when the configured watchdog timeout is available.
Out of Scope Changes check ✅ Passed The changes remain focused on watchdog startup signaling, annotation accuracy, startup errors, and related tests.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Title check ✅ Passed The title clearly states that the annotation update is deferred until after the watchdog starts.
Description check ✅ Passed The description explains the watchdog timeout bug, the Started() change, the annotation update, error handling, and testing.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cmd/main.go`:
- Around line 358-364: Update the runnable setup around wd.Started() to call
UpdateNodeAnnotations with the unavailable state (false and timeout 0) before
registering the runnable with mgr. Preserve the existing context cancellation
behavior, then call UpdateNodeAnnotations again after wd.Started() closes using
wd.Status() == watchdog.Armed and wd.GetTimeout() so the reachable state is
published.

In `@internal/watchdog/watchdog_test.go`:
- Around line 90-94: Move the context.WithCancel(context.Background()) setup out
of the BeforeEach callback into a package-level test helper, then invoke that
helper from BeforeEach while preserving the existing ctx, cancel, and
DeferCleanup behavior. Update the test setup around BeforeEach without changing
the test lifecycle semantics.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3bab509f-bd01-4260-a4f9-aeb4c5618c05

📥 Commits

Reviewing files that changed from the base of the PR and between 08359bb and f6eacc7.

📒 Files selected for processing (4)
  • cmd/main.go
  • internal/watchdog/interface.go
  • internal/watchdog/synchronized.go
  • internal/watchdog/watchdog_test.go

Comment thread cmd/main.go
Comment thread internal/watchdog/watchdog_test.go Outdated
kwohlfahrt and others added 2 commits August 5, 2026 14:46
Report the correct error if software reboot is disabled. This tears down
the manager and agent if the watchdog cannot be started.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Node annotation self-node-remediation.medik8s.io/watchdog-timeout always 0

1 participant