[WIP] all: implement chain freezer#884

Open

gzliudan wants to merge 90 commits into

XinFinOrg:dev-upgradefrom

gzliudan:chain_freezer

Commits on Apr 15, 2025

all: chain freezer on top of db rework (19244)
gzliudan
committed
core/rawdb, eth/downloader: align 64bit atomic fields (19591)
gzliudan
committed
core/rawdb: keep genesis in key-value store for full sync too (19628)
gzliudan
committed
core: concurrent database reinit from freezer dump (19604)
gzliudan
committed
core/rawdb: avoid O_APPEND (19676)
gzliudan
committed
core/rawdb: expose various counter metrics for grafana (19692)
gzliudan
committed
core: fix fast head updating logic (19764)
gzliudan
committed
core/rawdb: remove unused function print
gzliudan
committed
core/rawdb: add missing error checks (19871)
gzliudan
committed
core/rawdb: switch some invalid counters to gauges (20047)
gzliudan
committed
core/rawdb: check hash before return data from ancient db (20195)
gzliudan
committed
core/rawdb: fix reinit regression caused by the hash check PR (20403)
gzliudan
committed
core: write chain data in atomic way (20287)
gzliudan
committed
core/rawdb: fix freezer table test error check (20779)

meowsbits
authored and
gzliudan
committed
core/rawdb: fix data race between Retrieve and Close (20919)

MariusVanDerWijden
authored and
gzliudan
committed
core/rawdb: stop freezer process as part of freezer.Close() (21010)

AusIV
authored and
gzliudan
committed
all: tx background indexing/unindexing, improve chain initiation from freezer (20302)
gzliudan
committed
core/rawdb: swap tailId and itemOffset for deleted items in freezer (21220)

zhangsoledad
authored and
gzliudan
committed
core/rawdb: fix high memory usage in freezer (21243)

AusIV
authored and
gzliudan
committed

Commits on Apr 16, 2025

all: historical data garbage collection (19570)
gzliudan
committed
core/rawdb: better log messages for ancient failure (21327)

rjl493456442
authored and
gzliudan
committed
core/rawdb: convert some comments to godoc convention (21384)

meowsbits
authored and
gzliudan
committed
core/rawdb: refactor freezer (21105)
gzliudan
committed
core: define and test chain reparation cornercases (21409)
gzliudan
committed
core/rawdb: only complain loudly if truncating many items (21483)

karalabe
authored and
gzliudan
committed
eth/downloader: fix memory leak (21491)
gzliudan
committed
eth/downloader: only roll back light sync if not fully validating (21537)
gzliudan
committed
core/rawdb: added counters to the db inspect report (21495)
gzliudan
committed
eth/downloader: dynamically move pivot even during chain sync (21529)
gzliudan
committed
eth/downloader: fix data race around the ancientlimit (21681)

rjl493456442
authored and
gzliudan
committed
core: refactor function SetHead (21594)
gzliudan
committed
core: aiming genesis when middle block is missing (22135)
gzliudan
committed
core/rawdb: add function newFreezerTable (21967)
gzliudan
committed
core/rawdb: refactor function InspectDatabase (22014)
gzliudan
committed
all: add read-only option to database (22407)
gzliudan
committed
cmd/XDC, core/rawdb: add db-command to inspect freezer index (22633)
gzliudan
committed
core/rawdb: fix datarace in freezer (22728)

holiman
authored and
gzliudan
committed
core: remove old conversion to shuffle leveldb blocks into ancients (22739)
gzliudan
committed
core/rawdb: wait for background freezing to exit when closing freezer (22878)

holiman
authored and
gzliudan
committed
core/rawdb: db inspect move 'config' and 'shutdown' into 'meta data' (22978)
gzliudan
committed
core/rawdb: implement sequential reads in freezer_table (23117)

holiman
authored and
gzliudan
committed
eth/downloader: fix typo in comment (23413)

chuwt
authored and
gzliudan
committed
core/rawdb: freezer batch write (23462)

authored and
gzliudan
committed
core/rawdb, ethdb: introduce batched/atomic reads from ancients (23566)

holiman
authored and
gzliudan
committed
core/rawdb: better error message in freezer (23901)

holiman
authored and
gzliudan
committed
core, eth/downloader: fix resetting below freezer threshold (23949)
gzliudan
committed
core/rawdb: use AncientRange when initializing leveldb from freezer (23612)

holiman
authored and
gzliudan
committed
core/rawdb: improve delivery speed on header requests (23105)

authored and
gzliudan
committed
core/rawdb: enforce readonly in freezer instantiation (24119)

authored and
gzliudan
committed
cmd/XDC: add db cmd to show metadata (23900)
gzliudan
committed
core/rawdb, cmd, ethdb, eth: implement freezer tail deletion (23954)

rjl493456442
authored and
gzliudan
committed
core/rawdb: fix db commands (24540)

rjl493456442
authored and
gzliudan
committed
core/rawdb: simple legacy receipt converter (24028)

authored and
gzliudan
committed
core/rawdb: use T.TempDir to create temporary test directories (24633)
gzliudan
committed
core/rawdb: fix typo (24731)

s7v7nislands
authored and
gzliudan
committed
core/rawdb: untie freezer and ancient chain data (24684)

rjl493456442
authored and
gzliudan
committed
core/rawdb: fix typo in comment (25191)

noirgif
authored and
gzliudan
committed
cmd, core, ethdb, node: move chain freezer one folder deeper (25487)

rjl493456442
authored and
gzliudan
committed
core/rawdb: fix some typos (25551)
gzliudan
committed
core/rawdb: fix leak of backoff timer (25776)

ucwong
authored and
gzliudan
committed
core/rawdb: provide more info on 'gap in the chain' error (25938)

holiman
authored and
gzliudan
committed
core/rawdb: open meta file in read only mode (26009)

rjl493456442
authored and
gzliudan
committed
core/rawdb: refactor db inspector for extending multiple ancient store (25896)
gzliudan
committed
core/rawdb: improve freezerTable.Sync (26245)

fjl
authored and
gzliudan
committed
core/rawdb: fix freezer validation (26251)

rjl493456442
authored and
gzliudan
committed
core/rawdb: implement resettable freezer (26324)

rjl493456442
authored and
gzliudan
committed
core/rawdb: fsync head data file before closing it (26490)

rjl493456442
authored and
gzliudan
committed
core/rawdb: fix cornercase shutdown behaviour in freezer (26485)

holiman
authored and
gzliudan
committed
core/rawdb: expose chain freezer constructor without internals (26748)

karalabe
authored and
gzliudan
committed
core/rawdb, node: use standalone flock dependency (26633)

holiman
authored and
gzliudan
committed
core/rawdb: use atomic int added in go1.19 (26935)

s7v7nislands
authored and
gzliudan
committed
core/rawdb: update freezertable read meter (26946)

jsvisa
authored and
gzliudan
committed
core/rawdb: replace noarg fmt.Errorf with errors.New (27332)
gzliudan
committed
core/rawdb: support freezer batch read with no size limit (27687)

rjl493456442
authored and
gzliudan
committed
core/rawdb: no need to run truncateFile for readonly mode (28145)

jsvisa
authored and
gzliudan
committed
core/rawdb: use readonly file lock in readonly mode (28180)

authored and
gzliudan
committed
trie/triedb/pathdb, core/rawdb: enhance error message in freezer (28198)

rjl493456442
authored and
gzliudan
committed
core/rawdb: add logging and fix comments around AncientRange function. (28379)

jakubpajek
authored and
gzliudan
committed
core/rawdb: fsync the index file after each freezer write (28483)

rjl493456442
authored and
gzliudan
committed
core/rawdb: implement size reporting for live items in freezer_table (28525)

authored and
gzliudan
committed
core/rawdb: find smallest block stored in key-value store when chain gapped (26719)

authored and
gzliudan
committed
core/rawdb: fix ancient root folder (29697)

rjl493456442
authored and
gzliudan
committed
core/rawdb: fix freezer read-only option (29823)
gzliudan
committed
core/rawdb: more accurate description of freezer in docs (30393)

rjl493456442
authored and
gzliudan
committed
core/rawdb: freezer index repair (29792)

rjl493456442
authored and
gzliudan
committed
ethdb: refactor Database interface (30693)
gzliudan
committed
core/rawdb: fix panic in freezer (30973)

MariusVanDerWijden
authored and
gzliudan
committed
core/rawdb: introduce flush offset in freezer (30392)

authored and
gzliudan
committed
core/rawdb: skip setting flushOffset in read-only mode (31173)

rjl493456442
authored and
gzliudan
committed
core/rawdb: allow for truncation in the freezer (31362)

authored and
gzliudan
committed