feat: add --engine.no-backfill flag for CL-driven nodes (archive/verifier)#840
Open
agentotto[bot] wants to merge 1 commit into
Open
feat: add --engine.no-backfill flag for CL-driven nodes (archive/verifier)#840agentotto[bot] wants to merge 1 commit into
agentotto[bot] wants to merge 1 commit into
Conversation
…fier) Adds a new `--engine.no-backfill` CLI flag that swaps the default `PipelineSync` backfill implementation for a no-op. When set, P2P peer announcements more than 32 blocks ahead of the canonical tip no longer put the engine into pipeline mode, so `engine_forkchoiceUpdated` is never short-circuited to `SYNCING` from a peer-triggered backfill. ## Motivation World Chain (a fork of op-reth) inherits reth's behaviour of triggering a pipeline-based backfill whenever P2P peers announce blocks far ahead of the canonical tip. On archive/verifier nodes where a consensus client (e.g. `kona-node`) is the sync authority and drives the chain via the Engine API, this behaviour deadlocks the CL/EL handshake: - reth transitions the engine into pipeline mode from a peer announcement. - Pipeline mode causes reth to return `SYNCING` to every `engine_forkchoiceUpdated` from the CL. - kona-node sends a single FCU at startup, receives `SYNCING`, and enters `AwaitingELSyncCompletion` — waiting for the EL to finish syncing before sending another FCU. - The EL is meanwhile waiting for the CL to advance its target. Neither side progresses. ## Implementation - `NoopBackfillSync` (`crates/node/src/backfill.rs`) implements `BackfillSync` with no-ops: actions are dropped, `poll` is always `Poll::Pending`, no events are emitted. The engine tree's internal `backfill_sync_state` therefore stays `Idle`. - `BoxedBackfillSync` (same module) is a thin `Box<dyn BackfillSync>` wrapper so the orchestrator has a single concrete type regardless of which implementation is active. - `WorldChainEngineNodeLauncher` (`crates/node/src/launch.rs`) is a fork of `reth_node_builder::EngineNodeLauncher` that accepts a `no_backfill: bool` and constructs the `ChainOrchestrator` with either `PipelineSync` (default) or `NoopBackfillSync`. The fork was necessary because upstream's `build_engine_orchestrator` hardcodes `PipelineSync<N>` in its return type. - The `--engine.no-backfill` (env `ENGINE_NO_BACKFILL`) flag is wired into `WorldChainArgs` and `main.rs` calls `WorldChainEngineNodeLauncher::new(.., no_backfill)`. ## When to use - Set on CL-driven nodes (archive, verifier) — the CL drives the chain and P2P-triggered backfill causes deadlock. - Do NOT set on standalone (non-CL-driven) nodes — without pipeline sync the node cannot catch up over P2P from far behind the tip. Co-authored-by: Otto <otto@toolsforhumanity.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new
--engine.no-backfillCLI flag that swaps the defaultPipelineSyncbackfill implementation for a no-op. When set, P2P peer announcements more than 32 blocks ahead of the canonical tip no longer put the engine into pipeline mode, soengine_forkchoiceUpdatedis never short-circuited toSYNCINGfrom a peer-triggered backfill.Background — the deadlock
World Chain (a fork of op-reth) inherits reth's behaviour of triggering a pipeline-based backfill whenever P2P peers announce blocks far ahead of the canonical tip. On archive/verifier nodes where a consensus client (e.g.
kona-node) is the sync authority and drives the chain via the Engine API, this behaviour deadlocks the CL/EL handshake:SYNCINGto everyengine_forkchoiceUpdatedfrom the CL.kona-nodesends a single FCU at startup, receivesSYNCING, and entersAwaitingELSyncCompletion— waiting for the EL to finish syncing before sending another FCU.Fix
Bypass the P2P-driven pipeline entirely on CL-driven nodes. The CL retains exclusive control over chain progression via the Engine API.
Implementation
NoopBackfillSync(crates/node/src/backfill.rs) — implements theBackfillSynctrait with no-ops: actions are dropped,pollis alwaysPoll::Pending, no events are emitted. The engine tree's internalbackfill_sync_statetherefore staysIdle.BoxedBackfillSync(same module) — a thinBox<dyn BackfillSync>wrapper so the orchestrator has a single concrete type regardless of which implementation is active. Needed becauseChainOrchestrator<T, P>is generic overPand we want to select betweenPipelineSync<N>andNoopBackfillSyncat runtime.WorldChainEngineNodeLauncher(crates/node/src/launch.rs) — a fork ofreth_node_builder::EngineNodeLauncherthat accepts ano_backfill: booland constructs theChainOrchestratorwith eitherPipelineSync(default) orNoopBackfillSync. The fork is necessary because upstream'sbuild_engine_orchestratorhelper hardcodesPipelineSync<N>in its return type. The launcher is otherwise a straight port; behaviour for the default (backfill-enabled) path is unchanged.--engine.no-backfill(envENGINE_NO_BACKFILL) — wired intoWorldChainArgsand passed through to the launcher frommain.rs.When to use
Testing
cargo check -p world-chain -p world-chain-node -p world-chain-cli— clean.backfill.rsverifyingNoopBackfillSync::pollnever yields.Full E2E validation (CL/EL handshake with kona-node) should be done in staging on an archive/verifier node before rolling out.
Notes
EngineNodeLauncher. This is unavoidable given upstream's helper returns a concretely-typedPipelineSync<N>. A follow-up could push theBackfillSyncselection upstream (e.g. behindTreeConfig) and drop the fork..inspect(...)hardforks/pruning log block was dropped from the fork because it references privateLaunchContextWith<Attached<WithConfigs<..>, ..>>types that are not accessible from outsidereth_node_builder. All other diagnostics still fire from the underlying subsystems.Note
High Risk
Changes the consensus engine launch and backfill path (~570-line launcher fork); enabling the flag on standalone nodes removes P2P catch-up, while the default path should match upstream behavior.
Overview
Adds
--engine.no-backfill(envENGINE_NO_BACKFILL) so CL-driven nodes can ignore P2P-triggered pipeline backfill and avoid a kona-node / EL deadlock where FCU staysSYNCINGand neither side advances.When the flag is set, the engine uses
NoopBackfillSyncinstead ofPipelineSync: peer backfill actions are dropped andengine_forkchoiceUpdatedis not short-circuited toSYNCINGfrom P2P.BoxedBackfillSynclets the orchestrator pick the implementation at runtime.Node startup no longer uses the default reth
launch()path; it uses a forkedWorldChainEngineNodeLauncherwired vialaunch_with_fninmain.rs, withno_backfillpassed from CLI config. Default behavior (flag off) still usesPipelineSync.Test helpers and e2e structs initialize
no_backfill: false. A unit test assertsNoopBackfillSyncnever completes a poll.Reviewed by Cursor Bugbot for commit 97c8017. Bugbot is set up for automated code reviews on this repo. Configure here.