feat: make manifest hydration queue concurrency configurable (#27926)

[GuruduGanesh]

What / why

Closes #27926.

The manifest hydration queue used by the Source Hydrator is currently drained by a single goroutine, so source repos are hydrated one at a time. In hub-spoke topologies where a single Argo CD instance hydrates many repos, this serial processing becomes the bottleneck for how quickly changes become deployable.

This PR makes the hydration queue concurrency tunable, mirroring the existing --status-processors / --operation-processors pattern:

• Adds --hydration-processors (env ARGOCD_APPLICATION_CONTROLLER_HYDRATION_PROCESSORS), default 5, minimum 1.
• The hydrationQueue is drained by N workers; appHydrateQueue stays single-worker. The queue is keyed by {SourceRepoURL, SourceTargetRevision, DestinationBranch} and is rate-limiting, so the same key is never processed by two workers at once — additional workers only parallelize across distinct keys.
• Threaded through Run(...), the application-controller manifests, argocd-cmd-params-cm, and the HA/operator docs.

Additional correctness fix (required to enable concurrency safely): running multiple workers makes a pre-existing race likely — a hydration item can be processed before all apps for a key are marked Hydrating (nil CurrentOperation), which previously panicked and could overwrite status. To handle this safely the hydrator now always commits the complete app set for a key in a single pass (the commit server records a git note per dry SHA and short-circuits later commits, so a partial commit would leave some apps marked hydrated without their manifests), and only the per-app status writes are guarded (apps not yet Hydrating are skipped and finalize on a later refresh/resync pass). The worker count is clamped to a minimum of 1 so a 0/negative flag value cannot silently stall hydration.

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Title of the PR
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.
• Optional. My organization is added to USERS.md.
• Optional. For bug fixes, I've indicated what older releases this fix should be cherry-picked into (this may or may not happen depending on risk/complexity).

[bunnyshell]

❌ Preview Environment deleted from Bunnyshell

Available commands (reply to this comment):

• 🚀 /bns:deploy to deploy the environment

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.70%. Comparing base (6bb2027) to head (447b20a).

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #27948      +/-   ##
==========================================
+ Coverage   64.67%   64.70%   +0.02%     
==========================================
  Files         423      423              
  Lines       58428    58437       +9     
==========================================
+ Hits        37790    37811      +21     
+ Misses      17116    17111       -5     
+ Partials     3522     3515       -7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[crenshaw-dev]

@GuruduGanesh picking this up this week. :-)

[codecov]

Bundle Report

Bundle size has no change ✅

[GuruduGanesh]

Thanks @crenshaw-dev, addressed.

I moved the per-app Hydrating status update into ProcessHydrationQueueItem, so the hydration queue worker now gathers the full app set for the hydration key, marks the group Hydrating, then validates/hydrates/finalizes status from there. The app hydrate queue is now only the per-app detector/enqueuer.

This is safe under parallel hydration workers because the workqueue dedups by key: ProcessHydrationQueueItem always holds exclusive ownership of the whole app group when it runs.

I also fixed a pre-existing data race that the latest CI run surfaced. That failure was in TestFinalizeAppDeletion/DeleteWithInvalidClusterName: the test was mutating the fake client ReactionChain while an informer-driven Patch could read it through Fake.Invokes. I reproduced it on plain upstream/master with -race to confirm it was not introduced by this PR. I wrapped that test setup mutation with the fake client lock, then fixed the staticcheck lint finding by using the embedded lock directly.

Local validation:

• go test -race ./controller -run "TestFinalizeAppDeletion/DeleteWithInvalidClusterName" -count=50
• go test -race ./controller -run "TestFinalizeAppDeletion" -count=20
• go test -race ./controller -count=1
• go test -race ./controller/hydrator/... -count=3

[crenshaw-dev]

Thank you, @GuruduGanesh!