perf(pm): add resolver pm wiring#3038
Conversation
There was a problem hiding this comment.
Code Review
This pull request adjusts manifest fetch concurrency, increasing the default to 256 for non-semver resolvers while maintaining 64 for others. It also refactors manifest persistence by removing the bounded write queue and switching from streaming JSON serialization to in-memory buffering. Feedback highlights concerns regarding the removal of backpressure in the manifest writer, which could lead to unbounded memory growth, and the increased memory overhead caused by allocating full vectors for JSON serialization instead of streaming to disk.
| impl ManifestWriter { | ||
| fn spawn() -> Self { | ||
| let (tx, rx) = mpsc::sync_channel(MANIFEST_WRITE_QUEUE_CAPACITY); | ||
| let (tx, rx) = mpsc::channel(); |
There was a problem hiding this comment.
The switch from a bounded sync_channel to an unbounded channel removes the backpressure mechanism. The previous implementation used a capacity of 1024 and dropped writes when full, which prevented unbounded memory growth if disk I/O stalled. Since manifest persistence is described as 'opportunistic' (line 150), dropping writes is a safer strategy than risking an OOM during large-scale dependency resolution.
| let bytes = match serde_json::to_vec(value) { | ||
| Ok(b) => b, | ||
| Err(e) => { | ||
| tracing::debug!("Failed to serialize {path:?}: {e}"); | ||
| return; | ||
| } | ||
| }; |
There was a problem hiding this comment.
Using serde_json::to_vec allocates a full Vec<u8> for every manifest before writing. The previous implementation used serde_json::to_writer with a BufWriter, which streamed the JSON directly to the file. For large manifests, this new approach increases memory pressure and allocation overhead, which seems contrary to the 'perf' goal of this PR.
70806c8 to
d1db0d2
Compare
5b73204 to
e68b1b8
Compare
d1db0d2 to
94579a5
Compare
e68b1b8 to
5b0b1ce
Compare
94579a5 to
56101d8
Compare
📊 pm-bench-phases ·
|
| PM | wall | ±σ | user | sys | RSS | pgMinor |
|---|---|---|---|---|---|---|
| bun | 9.22s | 0.26s | 10.43s | 10.19s | 712M | 327.8K |
| utoo-next | 7.95s | 0.17s | 10.50s | 12.08s | 944M | 118.4K |
| utoo-npm | 8.14s | 0.02s | 10.85s | 12.19s | 1010M | 115.3K |
| utoo | 8.52s | 0.18s | 11.03s | 12.22s | 980M | 124.1K |
| PM | vCtx | iCtx | netRX | netTX | cache | node_mod | lock |
|---|---|---|---|---|---|---|---|
| bun | 14.2K | 18.7K | 1.19G | 6M | 1.88G | 1.76G | 1M |
| utoo-next | 104.4K | 75.4K | 1.16G | 4M | 1.71G | 1.70G | 2M |
| utoo-npm | 114.8K | 79.5K | 1.16G | 4M | 1.71G | 1.70G | 2M |
| utoo | 113.5K | 61.3K | 1.16G | 5M | 1.71G | 1.70G | 2M |
p1_resolve
| PM | wall | ±σ | user | sys | RSS | pgMinor |
|---|---|---|---|---|---|---|
| bun | 1.96s | 0.10s | 4.06s | 0.98s | 514M | 157.3K |
| utoo-next | 2.84s | 0.13s | 5.27s | 1.63s | 619M | 84.4K |
| utoo-npm | 3.01s | 0.09s | 5.54s | 1.92s | 609M | 77.8K |
| utoo | 3.17s | 0.07s | 5.86s | 1.85s | 624M | 90.5K |
| PM | vCtx | iCtx | netRX | netTX | cache | node_mod | lock |
|---|---|---|---|---|---|---|---|
| bun | 7.8K | 4.4K | 203M | 3M | 107M | - | 1M |
| utoo-next | 45.4K | 66.4K | 199M | 2M | 7M | 3M | 2M |
| utoo-npm | 68.9K | 87.4K | 199M | 2M | 7M | 3M | 2M |
| utoo | 51.1K | 71.0K | 201M | 3M | 7M | 3M | 2M |
p3_cold_install
| PM | wall | ±σ | user | sys | RSS | pgMinor |
|---|---|---|---|---|---|---|
| bun | 6.60s | 0.11s | 6.22s | 9.82s | 638M | 205.8K |
| utoo-next | 8.03s | 2.20s | 5.03s | 11.16s | 535M | 61.2K |
| utoo-npm | 5.99s | 0.09s | 5.02s | 10.86s | 473M | 59.4K |
| utoo | 5.96s | 0.59s | 5.01s | 10.61s | 484M | 64.9K |
| PM | vCtx | iCtx | netRX | netTX | cache | node_mod | lock |
|---|---|---|---|---|---|---|---|
| bun | 5.0K | 6.9K | 1018M | 3M | 1.76G | 1.76G | 1M |
| utoo-next | 120.0K | 50.1K | 989M | 3M | 1.70G | 1.70G | 2M |
| utoo-npm | 94.8K | 46.2K | 989M | 2M | 1.70G | 1.70G | 2M |
| utoo | 88.4K | 50.2K | 989M | 2M | 1.70G | 1.70G | 2M |
p4_warm_link
| PM | wall | ±σ | user | sys | RSS | pgMinor |
|---|---|---|---|---|---|---|
| bun | 3.29s | 0.05s | 0.20s | 2.36s | 133M | 32.7K |
| utoo-next | 2.33s | 0.05s | 0.51s | 3.80s | 79M | 18.6K |
| utoo-npm | 2.38s | 0.09s | 0.49s | 3.80s | 79M | 18.2K |
| utoo | 2.30s | 0.03s | 0.49s | 3.79s | 78M | 18.6K |
| PM | vCtx | iCtx | netRX | netTX | cache | node_mod | lock |
|---|---|---|---|---|---|---|---|
| bun | 422 | 25 | 5M | 16K | 1.91G | 1.75G | 1M |
| utoo-next | 44.7K | 18.9K | 3K | 25K | 1.70G | 1.70G | 2M |
| utoo-npm | 42.1K | 18.2K | 2K | 9K | 1.70G | 1.70G | 2M |
| utoo | 42.2K | 20.0K | 1K | 5K | 1.71G | 1.70G | 2M |
npmmirror.com: no output captured.
Summary
Resolver stack split from #3028 / #2948, stacked after #3036.
Adds the PM-side wiring needed by the resolver source stack:
No resolver scheduling logic lands in this PR; it only prepares PM-side integration for the upcoming mainloop PRs.
Size
9 files changed, 115 insertions(+), 113 deletions(-)Validation
cargo fmtcargo check -p utoo-pmcargo test -p utoo-pm util::user_config::tests::test_resolver_manifest_concurrency_raises_npmjs_defaultcargo test -p utoo-pm service::pipeline::receiver::tests::test_pipeline_receiver_filters_eventscargo clippy --all-targets -- -D warnings --no-depspack-napistill warns locally becausenext.jsis a symlink in this worktree; clippy exits successfully.