Commit b9915ff
authored
reduce bbolt overhead in containerd metadata stores (#13117)
* engine: reduce bbolt freelist memory usage
Marcos has seen severe memory usage during some dagger generate commands that do not reproduce on my machine: 30GB+ RSS versus around 800MB. The heap profile points at bbolt's internal freelist mergeSpans path, with more than 100GB of allocations over one run.
bbolt PR #1179 improves hashmap freelist span merging: etcd-io/bbolt#1179
Keep using that main-branch bbolt commit and switch Dagger's bbolt opens to FreelistMapType so we actually use the hashmap path. This covers the Dagger-managed snapshotter metastore and containerd metadata DB. It also sets bolt.DefaultOptions.FreelistType before snapshotter construction because some containerd snapshotters open bbolt internally with nil options.
Upgrade containerd to v2.2.3 while here. The previous v2.0.8 release does not compile against opencontainers/runtime-spec v1.3.0: it assigns an int64 to LinuxPids.Limit, which is now *int64. v2.2.x supports the new runtime-spec API.
containerd v2.2.x requires go.etcd.io/bbolt v1.4.3, which is a higher semver than the main-branch pseudo-version containing PR #1179. The go.mod replace keeps the selected bbolt code on that main commit even though MVS selects v1.4.3.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
* engine: disable bbolt statistics
bbolt's main branch now has Options.NoStatistics, added by etcd-io/bbolt#977, to skip DB-level statistics collection. We do not consume these stats for the engine metadata databases, and bbolt notes this can improve performance under high-concurrency read-only transactions.
Set NoStatistics on the explicit engine bbolt opens and in the process defaults used by containerd snapshotter paths that open bbolt internally with nil options.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
* chore: update x/sys module versions
Keep the repository module files in sync with the dependency solver now selecting golang.org/x/sys v0.44.0. This removes the generated check diff across module fixtures, SDK runtimes, and toolchains.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
* chore: update dagger module x/sys version
Update the hidden .dagger module files to match the golang.org/x/sys v0.44.0 selection used by the rest of the repository modules.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
---------
Signed-off-by: Erik Sipsma <erik@sipsma.dev>1 parent 5f9ec2e commit b9915ff
2 files changed
Lines changed: 3 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
51 | | - | |
| 51 | + | |
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
79 | 79 | | |
80 | 80 | | |
81 | 81 | | |
82 | | - | |
83 | | - | |
| 82 | + | |
| 83 | + | |
84 | 84 | | |
85 | 85 | | |
86 | 86 | | |
| |||
0 commit comments