fix: stop memory leak from orphaned CR reflector goroutines on repeated CRD discovery

[bhope]

Elevated and unbounded memory growth introduced in v2.18.0 when custom resource state config is in use.

Root Causes

1. AppendToMap overwrites stop channels and appends duplicate kinds on every call (internal/discovery/types.go). Since PollForCacheUpdates calls it for every known GVK each cycle, old stop channels were silently replaced, orphaning any reflector goroutine blocking on them.
2. CR reflectors ignore context cancellation (internal/store/builder.go). Unlike standard reflectors started with reflector.Run(b.ctx.Done()), custom resource reflectors were started with only their GVK-specific stop channel - no context cancellation path at all.

Fix

• AppendToMap: skip the append if the kind already exists; skip make(chan struct{}) if a channel already exists for the GVK.
• startReflector: wrap the GVK stop channel with a bridge goroutine that also selects on b.ctx.Done(), so CR reflectors stop on both CRD deletion and context cancellation.

Also, added tests to cover idempotency and cleanup in the discovery package - verifying no duplicate kinds or channel replacement on repeated AppendToMap calls, and that RemoveFromMap closes channels so reflectors stop cleanly.

Test Results:

TestMemoryLeakSimulation - 5 GVKs × 500 poll cycles

	Buggy (pre-fix)	Fixed (post-fix)
Kind entries in map	2500	5
Stop channels live	5	5
Heap growth (KB)	+88	-8

TestGoroutineLeakSimulation - 5 GVKs × 20 store rebuilds

	Buggy (pre-fix)	Fixed (post-fix)
Goroutines leaked	100	0

Fixes #2867

[Copilot]

Pull request overview

Fixes elevated/unbounded memory growth and goroutine leaks when custom resource state config is enabled by making CRD discovery idempotent and ensuring custom-resource reflectors stop on both CRD removal and context cancellation.

Changes:

• Make CRDiscoverer.AppendToMap idempotent (no duplicate kinds; don’t replace existing stop channels).
• Ensure custom-resource reflectors stop when either the GVK stop channel fires or the builder context is cancelled.
• Add/extend tests covering idempotency, channel cleanup, and leak simulations.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
internal/store/builder.go	Updates CR reflector stop behavior to also honor builder context cancellation.
internal/store/builder_test.go	Adds unit tests around the combined stop channel behavior for CR reflectors.
internal/discovery/types.go	Prevents duplicate kind entries and stop-channel replacement in repeated discovery updates.
internal/discovery/types_test.go	Adds deterministic unit tests for Append/Remove idempotency and channel closure.
internal/discovery/memleak_test.go	Adds simulation-style tests intended to demonstrate pre/post fix memory & goroutine behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[bhope]

Hi @mrueg addressed the copilot suggestions and CI is now green. Ready for a review when you get a chance.

[jullianow]

Any idea when this will be released?

[bhope]

@jullianow This will be included in the upcoming release, we are working towards it. Please stay tuned. Thanks.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mrueg]

Hi @mrueg addressed the copilot suggestions and CI is now green. Ready for a review when you get a chance.

Thanks for looking into those comments.
Unfortunately I won't have access to a way to test it until mid May due to private travel.

@rexagod can you take a look?

[rexagod]

The tests introduced here copy the behavior as it is now, and test it. However, since the behavior is not encapsulated (as a testable function), this needs to be updated every time there's a change. I'll suggest doing so and calling that here, or dropping these otherwise.

[bhope]

Thanks, makes sense. Extracted the bridge goroutine into newCRReflectorStopCh() in builder.go. Both tests now call that directly instead of copying the logic inside of the tests.

[rexagod]

No need to reproduce the current faulty behavior. It'd be fine to just see if the patched behavior works as expected. Tests implore maintenance too, so we'd benefit from testing traits in a way that balances manageable maintenance (PTAL at my other comment regarding reuse) with reasonable coverage, and dropping any extraneous ones that are arguably redundant in the long-term.

[bhope]

Thanks. Dropped:

1. appendToMapBuggy - as you called out, no need to test the existence of bug
2. TestGoroutineLeakSimulation - it was duplicating the bridge goroutine logic from the store package and is already covered by the builder_test.go tests

Also, replaced TestMemoryLeakSimulation with TestAppendToMapStability which only asserts fixed behavior.

[rexagod]

/triage accepted

[alexandernorth]

Hi all, I took a look at this fix and I believe that the issue with the hanging channels would also be fixed by my PR #2872. However, I see that my PR is missing the parent context cancellation logic from builder.go.
Would you also be open to the approach I use regarding storing the stopChans, but then adding the logic from this PR's builder.go to ensure that parent context cancellation also propagates correctly?
Let me know your thoughts!

[bhope]

Hi @alexandernorth - Thanks for the heads up. Since we already have done a few rounds of review on this one and left with a few test comments, I'd suggest we get this one merged first. Post that, I am happy to review your PR and get you the help needed to move forward on that one. Hope that works!

[bhope]

Hi @rexagod - addressed both your inline comments. PTAL when you get a chance. Thanks!

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[rexagod]

/approve
/lgtm
/hold

For other maintainers to review, feel free to unhold.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bhope, rexagod

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [rexagod]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[bhope]

/unhold

[jullianow]

@bhope When is this expected to be released?

[bhope]

@jullianow very soon, we are actively preparing the release. Stay tuned.

[jullianow]

thanks you @bhope