Skip to content

[storage] Fix update-before-await pattern #4148

Description

@danlaine

There are a number of places where we clear a struct's accumulated mutations before attempting to make them durable with (e.g. blob.write_sync().await).

This is fragile -- if the callback is dropped, the durable write may never complete. This makes the state inconsistent -- the struct's accumulated mutations are gone, but they have not been durably persisted, and they can't be recovered.

Location Incorrect Pattern
runtime/src/utils/buffer/write.rs Write::write_at overlap flush Drains the buffered tip with buffer.take(), then awaits the blob write that persists those drained bytes.
runtime/src/utils/buffer/write.rs Write::resize grow flush buffer.resize(len) can drain buffered bytes, then the method awaits the blob write for those bytes.
runtime/src/utils/buffer/write.rs Write::sync Drains the buffered tip with buffer.take(), then awaits the write/sync path for those drained bytes.
runtime/src/utils/buffer/paged/writer.rs Writer::flush_internal Removes full pages from the tip and publishes page/cache state, then awaits physical page writes.
runtime/src/utils/buffer/paged/writer.rs Writer::append_owned large direct path Caches/publishes full pages and replaces the tip with the suffix, then awaits the direct blob write.
storage/src/journal/segmented/manager.rs Manager::clear Empties self.blobs with take(&mut self.blobs), then awaits removal of each underlying blob.
storage/src/metadata/storage.rs Metadata::sync overwrite path Clears modified-key tracking and mutates in-memory target data, then awaits the writes and sync.
storage/src/metadata/storage.rs Metadata::sync cursor/version update Advances cursor and next_version, then awaits persistence of the selected target blob.

Update: I’ve spent some time fixing individual instances of this problem, and others remain unresolved. The recurring difficulty is that a dropped future can leave an operation partly complete, forcing later code to infer which I/O landed from bounds, counters, offsets, and other bookkeeping.

We should consider a more structural design: represent interrupted mutations explicitly in the owning data structure, make their I/O steps safe to repeat, and publish their final state in one synchronous step. Where necessary, recovery should use a single durable commit point or intent. The goal is for incomplete work to be a valid, understood state rather than an inconsistent state that each implementation must repair differently. This should make cancellation safety easier to reason about, review, and test systematically.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

Status
Staged

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions