Skip to content

feat: add --engine.no-backfill flag for CL-driven nodes (archive/verifier)#840

Open
agentotto[bot] wants to merge 1 commit into
mainfrom
feat/engine-no-backfill
Open

feat: add --engine.no-backfill flag for CL-driven nodes (archive/verifier)#840
agentotto[bot] wants to merge 1 commit into
mainfrom
feat/engine-no-backfill

Conversation

@agentotto

@agentotto agentotto Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a new --engine.no-backfill CLI flag that swaps the default PipelineSync backfill implementation for a no-op. When set, P2P peer announcements more than 32 blocks ahead of the canonical tip no longer put the engine into pipeline mode, so engine_forkchoiceUpdated is never short-circuited to SYNCING from a peer-triggered backfill.

Background — the deadlock

World Chain (a fork of op-reth) inherits reth's behaviour of triggering a pipeline-based backfill whenever P2P peers announce blocks far ahead of the canonical tip. On archive/verifier nodes where a consensus client (e.g. kona-node) is the sync authority and drives the chain via the Engine API, this behaviour deadlocks the CL/EL handshake:

  1. reth transitions the engine into pipeline mode from a peer announcement.
  2. Pipeline mode causes reth to return SYNCING to every engine_forkchoiceUpdated from the CL.
  3. kona-node sends a single FCU at startup, receives SYNCING, and enters AwaitingELSyncCompletion — waiting for the EL to finish syncing before sending another FCU.
  4. The EL, meanwhile, is waiting for the CL to advance its target. Neither side progresses.

Fix

Bypass the P2P-driven pipeline entirely on CL-driven nodes. The CL retains exclusive control over chain progression via the Engine API.

Implementation

  • NoopBackfillSync (crates/node/src/backfill.rs) — implements the BackfillSync trait with no-ops: actions are dropped, poll is always Poll::Pending, no events are emitted. The engine tree's internal backfill_sync_state therefore stays Idle.
  • BoxedBackfillSync (same module) — a thin Box<dyn BackfillSync> wrapper so the orchestrator has a single concrete type regardless of which implementation is active. Needed because ChainOrchestrator<T, P> is generic over P and we want to select between PipelineSync<N> and NoopBackfillSync at runtime.
  • WorldChainEngineNodeLauncher (crates/node/src/launch.rs) — a fork of reth_node_builder::EngineNodeLauncher that accepts a no_backfill: bool and constructs the ChainOrchestrator with either PipelineSync (default) or NoopBackfillSync. The fork is necessary because upstream's build_engine_orchestrator helper hardcodes PipelineSync<N> in its return type. The launcher is otherwise a straight port; behaviour for the default (backfill-enabled) path is unchanged.
  • --engine.no-backfill (env ENGINE_NO_BACKFILL) — wired into WorldChainArgs and passed through to the launcher from main.rs.

When to use

  • Set on CL-driven nodes (archive / verifier / any node where kona-node drives the chain). Without this flag, spontaneous P2P-triggered pipeline sync deadlocks the CL/EL handshake.
  • Do NOT set on standalone (non-CL-driven) nodes. Without pipeline sync, a node cannot catch up from far behind the tip over P2P.

Testing

  • cargo check -p world-chain -p world-chain-node -p world-chain-cli — clean.
  • Unit test in backfill.rs verifying NoopBackfillSync::poll never yields.

Full E2E validation (CL/EL handshake with kona-node) should be done in staging on an archive/verifier node before rolling out.

Notes

  • The launcher fork duplicates ~450 lines of upstream EngineNodeLauncher. This is unavoidable given upstream's helper returns a concretely-typed PipelineSync<N>. A follow-up could push the BackfillSync selection upstream (e.g. behind TreeConfig) and drop the fork.
  • Upstream's .inspect(...) hardforks/pruning log block was dropped from the fork because it references private LaunchContextWith<Attached<WithConfigs<..>, ..>> types that are not accessible from outside reth_node_builder. All other diagnostics still fire from the underlying subsystems.

Note

High Risk
Changes the consensus engine launch and backfill path (~570-line launcher fork); enabling the flag on standalone nodes removes P2P catch-up, while the default path should match upstream behavior.

Overview
Adds --engine.no-backfill (env ENGINE_NO_BACKFILL) so CL-driven nodes can ignore P2P-triggered pipeline backfill and avoid a kona-node / EL deadlock where FCU stays SYNCING and neither side advances.

When the flag is set, the engine uses NoopBackfillSync instead of PipelineSync: peer backfill actions are dropped and engine_forkchoiceUpdated is not short-circuited to SYNCING from P2P. BoxedBackfillSync lets the orchestrator pick the implementation at runtime.

Node startup no longer uses the default reth launch() path; it uses a forked WorldChainEngineNodeLauncher wired via launch_with_fn in main.rs, with no_backfill passed from CLI config. Default behavior (flag off) still uses PipelineSync.

Test helpers and e2e structs initialize no_backfill: false. A unit test asserts NoopBackfillSync never completes a poll.

Reviewed by Cursor Bugbot for commit 97c8017. Bugbot is set up for automated code reviews on this repo. Configure here.

…fier)

Adds a new `--engine.no-backfill` CLI flag that swaps the default
`PipelineSync` backfill implementation for a no-op. When set, P2P peer
announcements more than 32 blocks ahead of the canonical tip no longer
put the engine into pipeline mode, so `engine_forkchoiceUpdated` is
never short-circuited to `SYNCING` from a peer-triggered backfill.

## Motivation

World Chain (a fork of op-reth) inherits reth's behaviour of triggering
a pipeline-based backfill whenever P2P peers announce blocks far ahead
of the canonical tip. On archive/verifier nodes where a consensus
client (e.g. `kona-node`) is the sync authority and drives the chain
via the Engine API, this behaviour deadlocks the CL/EL handshake:

- reth transitions the engine into pipeline mode from a peer
  announcement.
- Pipeline mode causes reth to return `SYNCING` to every
  `engine_forkchoiceUpdated` from the CL.
- kona-node sends a single FCU at startup, receives `SYNCING`, and
  enters `AwaitingELSyncCompletion` — waiting for the EL to finish
  syncing before sending another FCU.
- The EL is meanwhile waiting for the CL to advance its target.
  Neither side progresses.

## Implementation

- `NoopBackfillSync` (`crates/node/src/backfill.rs`) implements
  `BackfillSync` with no-ops: actions are dropped, `poll` is always
  `Poll::Pending`, no events are emitted. The engine tree's internal
  `backfill_sync_state` therefore stays `Idle`.
- `BoxedBackfillSync` (same module) is a thin `Box<dyn BackfillSync>`
  wrapper so the orchestrator has a single concrete type regardless of
  which implementation is active.
- `WorldChainEngineNodeLauncher` (`crates/node/src/launch.rs`) is a
  fork of `reth_node_builder::EngineNodeLauncher` that accepts a
  `no_backfill: bool` and constructs the `ChainOrchestrator` with
  either `PipelineSync` (default) or `NoopBackfillSync`. The
  fork was necessary because upstream's `build_engine_orchestrator`
  hardcodes `PipelineSync<N>` in its return type.
- The `--engine.no-backfill` (env `ENGINE_NO_BACKFILL`) flag is
  wired into `WorldChainArgs` and `main.rs` calls
  `WorldChainEngineNodeLauncher::new(.., no_backfill)`.

## When to use

- Set on CL-driven nodes (archive, verifier) — the CL drives the
  chain and P2P-triggered backfill causes deadlock.
- Do NOT set on standalone (non-CL-driven) nodes — without pipeline
  sync the node cannot catch up over P2P from far behind the tip.

Co-authored-by: Otto <otto@toolsforhumanity.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants