There are a number of places where we clear a struct's accumulated mutations before attempting to make them durable with (e.g. blob.write_sync().await).
This is fragile -- if the callback is dropped, the durable write may never complete. This makes the state inconsistent -- the struct's accumulated mutations are gone, but they have not been durably persisted, and they can't be recovered.
| Location |
Incorrect Pattern |
runtime/src/utils/buffer/write.rs Write::write_at overlap flush |
Drains the buffered tip with buffer.take(), then awaits the blob write that persists those drained bytes. |
runtime/src/utils/buffer/write.rs Write::resize grow flush |
buffer.resize(len) can drain buffered bytes, then the method awaits the blob write for those bytes. |
runtime/src/utils/buffer/write.rs Write::sync |
Drains the buffered tip with buffer.take(), then awaits the write/sync path for those drained bytes. |
runtime/src/utils/buffer/paged/writer.rs Writer::flush_internal |
Removes full pages from the tip and publishes page/cache state, then awaits physical page writes. |
runtime/src/utils/buffer/paged/writer.rs Writer::append_owned large direct path |
Caches/publishes full pages and replaces the tip with the suffix, then awaits the direct blob write. |
storage/src/journal/segmented/manager.rs Manager::clear |
Empties self.blobs with take(&mut self.blobs), then awaits removal of each underlying blob. |
storage/src/metadata/storage.rs Metadata::sync overwrite path |
Clears modified-key tracking and mutates in-memory target data, then awaits the writes and sync. |
storage/src/metadata/storage.rs Metadata::sync cursor/version update |
Advances cursor and next_version, then awaits persistence of the selected target blob. |
Update: I’ve spent some time fixing individual instances of this problem, and others remain unresolved. The recurring difficulty is that a dropped future can leave an operation partly complete, forcing later code to infer which I/O landed from bounds, counters, offsets, and other bookkeeping.
We should consider a more structural design: represent interrupted mutations explicitly in the owning data structure, make their I/O steps safe to repeat, and publish their final state in one synchronous step. Where necessary, recovery should use a single durable commit point or intent. The goal is for incomplete work to be a valid, understood state rather than an inconsistent state that each implementation must repair differently. This should make cancellation safety easier to reason about, review, and test systematically.
There are a number of places where we clear a struct's accumulated mutations before attempting to make them durable with (e.g.
blob.write_sync().await).This is fragile -- if the callback is dropped, the durable write may never complete. This makes the state inconsistent -- the struct's accumulated mutations are gone, but they have not been durably persisted, and they can't be recovered.
runtime/src/utils/buffer/write.rsWrite::write_atoverlap flushbuffer.take(), then awaits the blob write that persists those drained bytes.runtime/src/utils/buffer/write.rsWrite::resizegrow flushbuffer.resize(len)can drain buffered bytes, then the method awaits the blob write for those bytes.runtime/src/utils/buffer/write.rsWrite::syncbuffer.take(), then awaits the write/sync path for those drained bytes.runtime/src/utils/buffer/paged/writer.rsWriter::flush_internalruntime/src/utils/buffer/paged/writer.rsWriter::append_ownedlarge direct pathstorage/src/journal/segmented/manager.rsManager::clearself.blobswithtake(&mut self.blobs), then awaits removal of each underlying blob.storage/src/metadata/storage.rsMetadata::syncoverwrite pathstorage/src/metadata/storage.rsMetadata::synccursor/version updatecursorandnext_version, then awaits persistence of the selected target blob.Update: I’ve spent some time fixing individual instances of this problem, and others remain unresolved. The recurring difficulty is that a dropped future can leave an operation partly complete, forcing later code to infer which I/O landed from bounds, counters, offsets, and other bookkeeping.
We should consider a more structural design: represent interrupted mutations explicitly in the owning data structure, make their I/O steps safe to repeat, and publish their final state in one synchronous step. Where necessary, recovery should use a single durable commit point or intent. The goal is for incomplete work to be a valid, understood state rather than an inconsistent state that each implementation must repair differently. This should make cancellation safety easier to reason about, review, and test systematically.