Skip to content

perf(pm): add resolver pm wiring#3038

Draft
elrrrrrrr wants to merge 1 commit into
perf/pm-split-resolver-http-client-poolfrom
perf/pm-split-resolver-pm-wiring
Draft

perf(pm): add resolver pm wiring#3038
elrrrrrrr wants to merge 1 commit into
perf/pm-split-resolver-http-client-poolfrom
perf/pm-split-resolver-pm-wiring

Conversation

@elrrrrrrr
Copy link
Copy Markdown
Contributor

Summary

Resolver stack split from #3028 / #2948, stacked after #3036.

Adds the PM-side wiring needed by the resolver source stack:

  • uses a resolver-specific manifest concurrency default for non-semver/npmjs mode;
  • keeps the user-visible install/download concurrency default unchanged;
  • updates CLI help to document the resolver default;
  • moves manifest-store persistence onto the writer queue shape used by the source PR;
  • removes the obsolete compact sync JSON helper after manifest-store no longer uses it;
  • removes preload progress event handling from the PM logger and adjusts the pipeline receiver test;
  • keeps project-cache/retry docs aligned with resolver-owned cache semantics.

No resolver scheduling logic lands in this PR; it only prepares PM-side integration for the upcoming mainloop PRs.

Size

Validation

  • cargo fmt
  • cargo check -p utoo-pm
  • cargo test -p utoo-pm util::user_config::tests::test_resolver_manifest_concurrency_raises_npmjs_default
  • cargo test -p utoo-pm service::pipeline::receiver::tests::test_pipeline_receiver_filters_events
  • cargo clippy --all-targets -- -D warnings --no-deps

pack-napi still warns locally because next.js is a symlink in this worktree; clippy exits successfully.

@elrrrrrrr elrrrrrrr added A-Pkg Manager Area: Package Manager benchmark Run pm-bench on PR labels May 21, 2026
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adjusts manifest fetch concurrency, increasing the default to 256 for non-semver resolvers while maintaining 64 for others. It also refactors manifest persistence by removing the bounded write queue and switching from streaming JSON serialization to in-memory buffering. Feedback highlights concerns regarding the removal of backpressure in the manifest writer, which could lead to unbounded memory growth, and the increased memory overhead caused by allocating full vectors for JSON serialization instead of streaming to disk.

impl ManifestWriter {
fn spawn() -> Self {
let (tx, rx) = mpsc::sync_channel(MANIFEST_WRITE_QUEUE_CAPACITY);
let (tx, rx) = mpsc::channel();
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The switch from a bounded sync_channel to an unbounded channel removes the backpressure mechanism. The previous implementation used a capacity of 1024 and dropped writes when full, which prevented unbounded memory growth if disk I/O stalled. Since manifest persistence is described as 'opportunistic' (line 150), dropping writes is a safer strategy than risking an OOM during large-scale dependency resolution.

Comment on lines +152 to +158
let bytes = match serde_json::to_vec(value) {
Ok(b) => b,
Err(e) => {
tracing::debug!("Failed to serialize {path:?}: {e}");
return;
}
};
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using serde_json::to_vec allocates a full Vec<u8> for every manifest before writing. The previous implementation used serde_json::to_writer with a BufWriter, which streamed the JSON directly to the file. For large manifests, this new approach increases memory pressure and allocation overhead, which seems contrary to the 'perf' goal of this PR.

@elrrrrrrr elrrrrrrr force-pushed the perf/pm-split-resolver-pm-wiring branch from 70806c8 to d1db0d2 Compare May 21, 2026 22:32
@elrrrrrrr elrrrrrrr force-pushed the perf/pm-split-resolver-http-client-pool branch from 5b73204 to e68b1b8 Compare May 21, 2026 23:09
@elrrrrrrr elrrrrrrr force-pushed the perf/pm-split-resolver-pm-wiring branch from d1db0d2 to 94579a5 Compare May 21, 2026 23:09
@elrrrrrrr elrrrrrrr force-pushed the perf/pm-split-resolver-http-client-pool branch from e68b1b8 to 5b0b1ce Compare May 21, 2026 23:39
@elrrrrrrr elrrrrrrr force-pushed the perf/pm-split-resolver-pm-wiring branch from 94579a5 to 56101d8 Compare May 21, 2026 23:39
@github-actions
Copy link
Copy Markdown

📊 pm-bench-phases · b91b56d · linux (ubuntu-latest)

Workflow run — ant-design

PMs: utoo (this branch) · utoo-npm (latest published) · bun (latest)

npmjs.org

p0_full_cold

PM wall ±σ user sys RSS pgMinor
bun 9.22s 0.26s 10.43s 10.19s 712M 327.8K
utoo-next 7.95s 0.17s 10.50s 12.08s 944M 118.4K
utoo-npm 8.14s 0.02s 10.85s 12.19s 1010M 115.3K
utoo 8.52s 0.18s 11.03s 12.22s 980M 124.1K
PM vCtx iCtx netRX netTX cache node_mod lock
bun 14.2K 18.7K 1.19G 6M 1.88G 1.76G 1M
utoo-next 104.4K 75.4K 1.16G 4M 1.71G 1.70G 2M
utoo-npm 114.8K 79.5K 1.16G 4M 1.71G 1.70G 2M
utoo 113.5K 61.3K 1.16G 5M 1.71G 1.70G 2M

p1_resolve

PM wall ±σ user sys RSS pgMinor
bun 1.96s 0.10s 4.06s 0.98s 514M 157.3K
utoo-next 2.84s 0.13s 5.27s 1.63s 619M 84.4K
utoo-npm 3.01s 0.09s 5.54s 1.92s 609M 77.8K
utoo 3.17s 0.07s 5.86s 1.85s 624M 90.5K
PM vCtx iCtx netRX netTX cache node_mod lock
bun 7.8K 4.4K 203M 3M 107M - 1M
utoo-next 45.4K 66.4K 199M 2M 7M 3M 2M
utoo-npm 68.9K 87.4K 199M 2M 7M 3M 2M
utoo 51.1K 71.0K 201M 3M 7M 3M 2M

p3_cold_install

PM wall ±σ user sys RSS pgMinor
bun 6.60s 0.11s 6.22s 9.82s 638M 205.8K
utoo-next 8.03s 2.20s 5.03s 11.16s 535M 61.2K
utoo-npm 5.99s 0.09s 5.02s 10.86s 473M 59.4K
utoo 5.96s 0.59s 5.01s 10.61s 484M 64.9K
PM vCtx iCtx netRX netTX cache node_mod lock
bun 5.0K 6.9K 1018M 3M 1.76G 1.76G 1M
utoo-next 120.0K 50.1K 989M 3M 1.70G 1.70G 2M
utoo-npm 94.8K 46.2K 989M 2M 1.70G 1.70G 2M
utoo 88.4K 50.2K 989M 2M 1.70G 1.70G 2M

p4_warm_link

PM wall ±σ user sys RSS pgMinor
bun 3.29s 0.05s 0.20s 2.36s 133M 32.7K
utoo-next 2.33s 0.05s 0.51s 3.80s 79M 18.6K
utoo-npm 2.38s 0.09s 0.49s 3.80s 79M 18.2K
utoo 2.30s 0.03s 0.49s 3.79s 78M 18.6K
PM vCtx iCtx netRX netTX cache node_mod lock
bun 422 25 5M 16K 1.91G 1.75G 1M
utoo-next 44.7K 18.9K 3K 25K 1.70G 1.70G 2M
utoo-npm 42.1K 18.2K 2K 9K 1.70G 1.70G 2M
utoo 42.2K 20.0K 1K 5K 1.71G 1.70G 2M

npmmirror.com: no output captured.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-Pkg Manager Area: Package Manager benchmark Run pm-bench on PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant