Skip to content

Commit b9915ff

Browse files
authored
reduce bbolt overhead in containerd metadata stores (#13117)
* engine: reduce bbolt freelist memory usage Marcos has seen severe memory usage during some dagger generate commands that do not reproduce on my machine: 30GB+ RSS versus around 800MB. The heap profile points at bbolt's internal freelist mergeSpans path, with more than 100GB of allocations over one run. bbolt PR #1179 improves hashmap freelist span merging: etcd-io/bbolt#1179 Keep using that main-branch bbolt commit and switch Dagger's bbolt opens to FreelistMapType so we actually use the hashmap path. This covers the Dagger-managed snapshotter metastore and containerd metadata DB. It also sets bolt.DefaultOptions.FreelistType before snapshotter construction because some containerd snapshotters open bbolt internally with nil options. Upgrade containerd to v2.2.3 while here. The previous v2.0.8 release does not compile against opencontainers/runtime-spec v1.3.0: it assigns an int64 to LinuxPids.Limit, which is now *int64. v2.2.x supports the new runtime-spec API. containerd v2.2.x requires go.etcd.io/bbolt v1.4.3, which is a higher semver than the main-branch pseudo-version containing PR #1179. The go.mod replace keeps the selected bbolt code on that main commit even though MVS selects v1.4.3. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * engine: disable bbolt statistics bbolt's main branch now has Options.NoStatistics, added by etcd-io/bbolt#977, to skip DB-level statistics collection. We do not consume these stats for the engine metadata databases, and bbolt notes this can improve performance under high-concurrency read-only transactions. Set NoStatistics on the explicit engine bbolt opens and in the process defaults used by containerd snapshotter paths that open bbolt internally with nil options. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * chore: update x/sys module versions Keep the repository module files in sync with the dependency solver now selecting golang.org/x/sys v0.44.0. This removes the generated check diff across module fixtures, SDK runtimes, and toolchains. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * chore: update dagger module x/sys version Update the hidden .dagger module files to match the golang.org/x/sys v0.44.0 selection used by the rest of the repository modules. Signed-off-by: Erik Sipsma <erik@sipsma.dev> --------- Signed-off-by: Erik Sipsma <erik@sipsma.dev>
1 parent 5f9ec2e commit b9915ff

2 files changed

Lines changed: 3 additions & 3 deletions

File tree

runtime/go.mod

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ require (
4848
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
4949
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.41.0 // indirect
5050
golang.org/x/net v0.52.0 // indirect
51-
golang.org/x/sys v0.43.0 // indirect
51+
golang.org/x/sys v0.44.0 // indirect
5252
golang.org/x/text v0.35.0 // indirect
5353
google.golang.org/genproto/googleapis/api v0.0.0-20260226221140-a57be14db171 // indirect
5454
google.golang.org/genproto/googleapis/rpc v0.0.0-20260226221140-a57be14db171 // indirect

runtime/go.sum

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -79,8 +79,8 @@ golang.org/x/net v0.52.0 h1:He/TN1l0e4mmR3QqHMT2Xab3Aj3L9qjbhRm78/6jrW0=
7979
golang.org/x/net v0.52.0/go.mod h1:R1MAz7uMZxVMualyPXb+VaqGSa3LIaUqk0eEt3w36Sw=
8080
golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
8181
golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
82-
golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI=
83-
golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
82+
golang.org/x/sys v0.44.0 h1:ildZl3J4uzeKP07r2F++Op7E9B29JRUy+a27EibtBTQ=
83+
golang.org/x/sys v0.44.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
8484
golang.org/x/text v0.35.0 h1:JOVx6vVDFokkpaq1AEptVzLTpDe9KGpj5tR4/X+ybL8=
8585
golang.org/x/text v0.35.0/go.mod h1:khi/HExzZJ2pGnjenulevKNX1W67CUy0AsXcNubPGCA=
8686
gonum.org/v1/gonum v0.16.0 h1:5+ul4Swaf3ESvrOnidPp4GZbzf0mxVQpDCYUQE7OJfk=

0 commit comments

Comments
 (0)