tbc: store tx byte location in tx index, bump DB to v6#1052
Open
marcopeereboom wants to merge 2 commits into
Open
tbc: store tx byte location in tx index, bump DB to v6#1052marcopeereboom wants to merge 2 commits into
marcopeereboom wants to merge 2 commits into
Conversation
BlockByHash calls btcutil.NewBlockFromBytes on every read, eagerly deserializing the entire block into heap objects. Callers needing one tx pay for all txs. CPU profiles show 50% GC pressure from this. Add lazyBlock type wrapping raw []byte from the block cache with lazy per-tx access — no deserialization until a specific tx is requested. Single-pass boundary scan finds tx offsets without parsing. Per-tx txid computation handles both witness and non-witness serialization. Per-tx output value extraction reads only the outputs section. Add BlockRawByHash to the DB interface — the cache-check + LevelDB read path from BlockByHash without the NewBlockFromBytes call. Existing BlockByHash callers are unchanged — this is an opt-in parallel path for callers that need lightweight access. 100% test coverage on lazyblock.go. Every method cross-checked against btcutil.NewBlockFromBytes output as oracle. Test blocks include: genesis, segwit single/multi input, mixed segwit/non-segwit, 50-tx blocks, empty witness items, large inscription-style witness, FullBlock byte-identical round-trip, and exhaustive error path coverage for all truncation boundaries.
marcopeereboom
force-pushed
the
marco/lazy-block
branch
from
9f1a394 to
99d768f
Compare
marcopeereboom
force-pushed
the
marco/tx-offset
branch
from
461b684 to
4a29640
Compare
This was referenced May 29, 2026
joshuasing
reviewed
May 29, 2026
| return l.MetadataPut(ctx, versionKey, v) | ||
| } | ||
|
|
||
| func (l *ldb) v6(ctx context.Context) error { |
Contributor
There was a problem hiding this comment.
How long does this upgrade take?
Store TxLoc (offset + length within raw block) in the t entry value instead of nil. This allows callers to jump directly to a tx's bytes in the raw block without scanning — O(1) instead of O(txs_in_block). BlockHashByTxId now returns (*chainhash.Hash, wire.TxLoc, error). All callers updated. No separate method needed — callers that only need the hash use bh, _, err := BlockHashByTxId(...). processTxs calls block.TxLoc() and stores the location via NewTxMappingWithLoc. Errors from TxLoc() are logged at Errorf and the indexer falls back to nil values (legacy format). BlockTxUpdate uses stack-allocated reusable buffers instead of slicing loop variables. The previous code sliced the range variable and passed the slice to leveldb.Batch.Put. appendRec copies immediately, but the interaction between range variable reuse, map deletion, and GC is not guaranteed safe. Stack buffers are zero-alloc and independent per iteration. DB version 5 -> 6. Upgrade path wipes the transactions index for rebuild with TxLoc values. The index is fully derived from block data. Ref: #1050
marcopeereboom
force-pushed
the
marco/lazy-block
branch
from
99d768f to
e103e96
Compare
marcopeereboom
force-pushed
the
marco/tx-offset
branch
from
4a29640 to
1a04a83
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Store tx byte location (
TxLoc: offset + length within raw block) in the tx index't'entry value, which was previously nil. This enables O(1) tx lookup by jumping directly to the tx's bytes in the raw block — no scanning, no SHA256 hashing, no full block deserialization.Depends on #1051 (lazy block reader).
Problem
Every tx lookup via
BlockHashByTxIdrequires a subsequent full block deserialization to find the tx. The't'entry value was nil — wasted space that could carry the byte offset. CPU profile shows 60% in SHA256 hashing fromFindTxscanning every tx in the block to find a match.Solution
BlockHashByTxIdsignature changed to return(*chainhash.Hash, wire.TxLoc, error)— all callers updatedprocessTxsnow callsblock.TxLoc()and stores offset+length viaNewTxMappingWithLocBlockTxUpdateuses stack-allocated reusable buffers instead of slicing loop variables (addresses potential data integrity issue documented in tbc: tx index intermittently loses entries during IBD #1050)TxById(RPC) — deserializes only the target txtxOutFromOutPoint(UTXO unwind) — deserializes only the target txhandleBlockHashByTxIdRequest(RPC) — hash only, ignores TxLochemictl— hash only, ignores TxLocblock.TxLoc()logged at Errorf, falls back to nil valuesTesting
TestDbUpgradeV6— seeds v5 DB, runs upgrade, verifies index wiped and version bumpedTestTxLocRoundTrip— stores TxLoc, reads back, verifies offset+length matchTestTxLocOffsetCorrectness— stores raw block, uses offset to extract tx bytes, deserializes, verifies txid and output values matchTestTxLocLegacyNilValue— nil-value entry returns zero TxLoc gracefullyImpact
With TxLoc, each cache miss costs: 1 LevelDB read (tx index) + 1 LevelDB/cache read (raw block) + parse ~200 bytes of one tx. No full block deserialization. No SHA256 scanning. Eliminates the need for parallel lookup strategies.
Related
Files changed
database/tbcd/database.go—BlockHashByTxIdreturns TxLoc,NewTxMappingWithLocdatabase/tbcd/level/level.go— implementation, stack buffers, DB v6database/tbcd/level/level_test.go— 4 new testsdatabase/tbcd/level/upgrade.go—v6()wipes tx indexservice/tbc/txindex.go— stores TxLocservice/tbc/tbc.go—TxByIduses TxLocservice/tbc/utxoindex.go—txOutFromOutPointuses TxLocservice/tbc/rpc.go— caller updatedservice/tbc/cpfp_test.go— stub updatedservice/tbc/tbc_test.go— version expectations updatedcmd/hemictl/hemictl.go— caller updated