All zeam nodes stall on devnet: snappyz.decodeWithMax rejects every block gossip message as error.Corrupt

[ch4r10t33r]

Symptoms

Fresh-genesis devnet restart on 2026-05-27 produces an immediate zeam-fleet-only consensus split. Every zeam container freezes at head_slot ≈ 52–56 while every non-zeam client advances normally. Wall-clock is at slot 845+ across the fleet.

Client	head_slot	current_slot	Status
zeam_8..15 (all 8)	52–56	844	stuck
ethlambda_0	850	850	live
ethlambda_8	845	849	live
lantern_0	834	850	live
grandine_0	845	851	live
gean_0	845	851	live

The chain network is healthy. Only zeam stops processing blocks.

Root cause (from log)

Starting at zeam's local slot 52, every incoming block-topic gossip message fails the snappyz.decodeWithMax call at pkgs/network/src/ethlibp2p.zig:538 with error.Corrupt:

[error] [network] Error in snappyz decoding the message for
  topic=/leanconsensus/12345678/block/ssz_snappy from peer=unknown_peer:
  error.Corrupt

Same line, same error, every block, until the operator restarts. validateGossipSnappyHeader (the pre-decode varint guard at line 529) passes — the failure is inside the snappyz decoder itself.

Slots 0–51 decode fine. Slot 52 onward fails. Every zeam container in the fleet hits this at almost exactly the same slot, so it correlates with chain content (a particular block shape) rather than zeam-local state.

What's not the cause

• Not a peer issue. ethlambda/lantern/grandine/gean/ream are all gossiping fresh blocks correctly to each other.
• Not the size limit. MAX_GOSSIP_BLOCK_SIZE = 50 MB (ethlibp2p.zig:45) is well above any real block.
• Not PR node, types, metrics, xmss: cut recursive-STARK aggregation tail + per-call instrumentation (#940) #941. None of node, types, metrics, xmss: cut recursive-STARK aggregation tail + per-call instrumentation (#940) #941's changes touch the gossip-ingress path. The aggregator never runs on data it never receives.

Investigation hints

1. Content correlation. Slot 52 corresponds to roughly when the network starts producing blocks with aggregated-attestation payloads (justified=20 by then, aggregator activity in full swing). Empty/near-empty early blocks decode fine; blocks carrying STARK aggregate proofs do not. The Corrupt error may surface only when the compressed payload exceeds a particular size or shape.

2. Two snappy implementations in tree. build.zig.zon pins both zig_snappy 0.0.5 (imported as snappyz) and another older revision used as snappyz0. The gossip path uses snappyz.decodeWithMax. Worth checking whether the 0.0.5 decoder mishandles a payload shape the 0.0.3 decoder (or upstream Go snappy reference) accepts.

3. Failure is silent past the metric. The decoded byte length never reaches lean_gossip_block_size_bytes — rejection happens before observation — so the symptom is not visible on Prometheus dashboards. Suggest adding a zeam_gossip_decode_failure_total{topic, error} counter so operators see the stall via metrics, not only via head_slot falling behind.

4. Failed dumps. The decode-failure path writes the raw bytes to failedbytes_* files via writeFailedBytes when shouldPersistMalformedDump() returns true (ethlibp2p.zig:544). On the current devnet image the dump is gated by an env var; turning it on for a single replay would give us the exact byte pattern that triggers error.Corrupt.

Acceptance criteria

• Identify the specific byte pattern (or content shape) that makes snappyz.decodeWithMax return error.Corrupt on payloads peers produced correctly.
• Fix or replace the decode path so zeam can ingest the same gossip messages every other lean-consensus client accepts.
• Add zeam_gossip_decode_failure_total (or equivalent) so this class of stall is detectable from metrics before head_slot drifts.
• Regression test on a captured failure-bytes corpus.

Severity

Critical for any release deployed to a network where blocks carry aggregated attestations. zeam falls out of consensus within ~50 slots of every fresh-genesis devnet and cannot participate further until the operator intervenes. Found while validating #940; not caused by #940.

[ch4r10t33r]

Investigation update — 2026-05-28 devnet runs

PR #945 (observability + bridge message_id_fn fix) is deployed. The original symptom no longer presents the same way — zeam now progresses normally for a few minutes then stalls — and the new instrumentation lets us localize what's actually going on. Closing some hypotheses, opening a new and clearer root cause.

Updated headline

The bytes on the wire ARE broken snappy (both snappyz.decodeWithMax and snap::raw::Decoder reject them; python-snappy too). zeam handles the decode failures correctly (log + counter + return). The stall is NOT caused by the decode-failure handling.

The stall is caused by CPU contention on zeam's single-threaded libxev event loop: rust-libp2p gossipsub delivery, chain block import, signature verification, and the leanMultisig STARK aggregator all run on the same thread. When chain CPU work blocks libxev for 0.5+ seconds at the slot cadence, gossipsub peer scoring on remote peers progressively demotes zeam, peers prune zeam from their meshes, and gossip delivery silently dries up.

What was ruled out

Hypothesis	Status	Evidence
FFI handoff slice/pointer bug	Ruled out	Rust-side first32= (added in PR #945) matches Zig-side first32= byte-for-byte.
Zig snappyz.decode bug on these bytes	Ruled out	Rust snap::raw::Decoder rejects the exact same byte sequences at the exact same offsets. Python-snappy also rejects.
Zig snappyz.encode bug on high-entropy inputs	Ruled out	Three new round-trip tests on PR #945 cover 280 KB PRNG, sizes around the 64 KB chunk boundary, and a captured 65,866-byte real-block prefix — all round-trip cleanly.
tagCopy4 byte emission by a spec-compliant encoder	Ruled out	snappyz.emitCopy (snappy.zig:367–397) and Rust snap::raw::Block::emit_copy (compress.rs:323) both only ever emit tagCopy1 / tagCopy2; the wire bytes contain tagCopy4 tags with offsets in the tens of millions. These bytes were not produced by any standard snappy encoder.
Single client family is the publisher	Ruled out	Failure forwarder distribution shifts run-to-run (run 1: gean dominated 66 %; run 2: ethlambda dominates 55 %). No single family is consistently the source.

What we now know

1. zeam itself forwards the bad bytes too

Run-2 propagation-peer breakdown (44 failures):

Client family	Failed forwards
ethlambda (2, 0, 8, 6, 5)	24
gean (1, 5, 3, 7)	8
qlean (0)	7
zeam (9, 10, 14)	4
lantern (3)	1

zeam_9, zeam_10, zeam_14 are forwarding broken bytes to zeam_8. That is expected gossipsub behaviour — mesh forwarding does not re-decode; once a peer accepts bytes into the mesh, every mesh peer gets a verbatim copy. The byte corruption is upstream of any single client; clients merely propagate.

2. The bytes ARE structurally raw SSZ with a varint prepended

Decoding the captured wire-failure dumps via Python tag-walk gets ~65 KB to ~131 KB of cleanly-decoded output (which is valid SSZ block content — first bytes match the standard BeamBlock layout) before hitting a tagCopy4 with a tens-of-millions offset and rejecting. We have multiple distinct wire dumps showing the same pattern at different offsets.

Suggests the bytes look "snappy-shaped enough to start decoding" but contain garbage past a chunk boundary. The original publisher is producing output a standard snappy encoder cannot have produced.

3. We now have real SSZ block content captured

PR #945's decoded_block_*.bin dump path (default-on for diagnostics, throwaway) now captures successful uncompressed decodes ≥ 100 KB. Five dumps so far this run, sizes 286 KB to 914 KB. The 286 KB dump is staged at pkgs/network/src/testdata_issue_942_real_block.bin so any future test can use real SSZ block content for round-trip experiments.

Python-snappy round-trips this real 286 KB block (encode → decode == original) so the content itself is legal SSZ that any spec-compliant snappy encoder handles correctly.

4. The mesh-silence stall pattern

Current devnet run shows the same eventual stall, but later in the process than before the message_id_fn fix:

Time	head	current	mesh_peers	decode_fails	What's happening
07:36	5	14	23	28	Bootstrapping, catching up
07:38:59	53	53	23	44	Caught up via gossip + RPC
07:39:01	53	53	23	44	Last gossip ingress logged
07:42:27	53	95	17	44	Status RPCs timing out, mesh shrinking
07:41:06	53	85	17	44	Full stall — 32 slots behind

Two telltale signs the cause is libxev contention rather than gossipsub-side decode handling:

• decode_failures stops incrementing at the same time as gossip delivery — peers stopped sending zeam ANY gossip, not just broken bytes.
• zeam_xev_clock_until_done_slow_ge_500ms_total grows at a constant 40 per 30 s (~1.3/s) both during the healthy catch-up phase AND the stalled phase. The slow ticks are a constant background cost from chain work on libxev, independent of decode-failure rate.

Confirmed root cause

zeam's libxev event loop is single-threaded and shared between:

• rust-libp2p Tokio runtime delivering gossipsub events via FFI
• chain.onBlock (state transition, signature verify) on inbound blocks
• The leanMultisig STARK aggregator prove (per node, aggregation: flat 8-raw XMSS prove tail (~4s p95) vs lantern/qlean on 8-validator subnet #940 / node, types, metrics, xmss: cut recursive-STARK aggregation tail + per-call instrumentation (#940) #941, still ~1.6 s mean even with deferred aggregation; cli, node, types: cap aggregator-worker children per call to bound STARK latency (#940) #943's children cap would drop to ~0.4 s)
• All other libxev-driven work

When STARK prove or block STF runs on this thread, rust-libp2p can't deliver gossipsub events to zeam during that window. From the remote peer's view, zeam stops sending IHAVE/IWANT responses for 0.5–2 s. Gossipsub's peer-score mechanism on every other client reduces zeam's score for each such delay; after enough degradation, mesh peers PRUNE zeam. zeam's local mesh_peers metric still reads the high number (gossipsub's local view), but no peer is actually sending mesh-forwarded messages anymore.

Other Rust / Go clients (ethlambda, grandine, gean) don't have this contention because their gossip-processing runs on a dedicated Tokio runtime / goroutine pool, separate from chain CPU. They handle the SAME broken bytes on the wire (we confirmed via rust_snap=err on identical fingerprints) and stay live.

Two separate problems

1. Wire-side broken-snappy bytes: real, but a Lean-network-wide phenomenon, not zeam-specific. zeam handles them correctly. Filing a separate question to client teams to identify which encoder is producing the tagCopy4-bearing bytes is the right next step here — out of scope for the zeam-side fix.
2. zeam stalls in the face of mesh-silence pressure: zeam-specific, fixable on the zeam side without changing the wire protocol or encoder.

Path forward (zeam-side)

In order of effort:

• Land cli, node, types: cap aggregator-worker children per call to bound STARK latency (#940) #943 (children cap) — already merged-pending. Drops mean aggregator prove time from 1.64 s to ~0.4 s. Smallest change, reduces slow-tick density ~4×. May be enough on its own to keep zeam in the mesh long-term.
• Move STARK FFI off libxev — wrap xmss_aggregate calls in a ThreadPool job, resume via xev.Async. The pkgs/thread-pool worker is the natural home. This is the correct fix and prevents future contention.
• Move signature verify and block STF off libxev as well — same pattern. Rule: libxev only ever touches FFI for sub-millisecond work.
• Tune gossipsub peer scoring to be more lenient on slow-but-alive peers (workaround; not preferred over actually fixing the contention).

PR #945 closes the observability gap on this issue. Once it merges I'll convert the throwaway diagnostic dump (decoded_block_*.bin + Rust-side [gossip ingress] full-line logging) into a clean-up follow-up PR.

Acceptance-criteria update on this issue

• Identify the specific byte pattern that makes snappyz.decodeWithMax return error.Corrupt — captured byte previews and dump files cover this.
• Add a counter so this class of stall is visible from metrics before head_slot drifts (PR network, node, metrics, libp2p-glue: gossip decode + per-aggregate + Rust-side bridge observability (#942) #945 ships zeam_gossip_decode_failures_total{topic_kind, reason}).
• Regression-test scaffolding on a captured corpus (PR network, node, metrics, libp2p-glue: gossip decode + per-aggregate + Rust-side bridge observability (#942) #945 ships three round-trip tests + a fixture file).
• Reframe: the underlying user-visible problem isn't decode failure, it's libxev contention. The follow-up issue is the thread-affinity refactor (move FFI off libxev). I'll open a separate issue for that and link both ways.
• Separately: identify the network-side producer of tagCopy4-bearing snappy bytes (cross-client coordination, not zeam-side).

CC @ch4r10t33r for visibility — particularly the libxev-contention reframe and the proposal to move STARK / sig-verify FFI off libxev.

[ch4r10t33r]

Root cause located — mplex muxer in zeam's libp2p transport

Updated diagnosis from today's devnet (2026-05-28).

Confirmed: zeam-specific reception bug, not encoder bug

Cross-checked across the fleet:

• ream_8 (Rust libp2p): 0 gossip snappy decode failures
• grandine_4 (Rust libp2p): 0 snappy decode failures; 1 gossip warning that is "invalid aggregated XMSS signature" — meaning grandine decompressed the snappy fine and only failed on downstream signature check
• zeam_8 / 9 / 10 / 11 / 12 / 13: 38 block + 28 aggregation snappy decode failures each, identical counts across all zeam nodes, all stuck at head_slot=51-61

Other clients successfully decode the exact same gossip messages zeam fails on. This is not a publisher / encoder bug — it is a zeam-specific reception bug.

Failure pattern: gossip messages > ~100 KB only

Message type	Received	Failed	Rate
Regular attestations (/attestation_X/, ~2.6 KB)	840	0	0%
Aggregations (/aggregation/, 175–217 KB)	402	28	7%
Blocks (/block/, 286 KB – 4 MB)	72	27	38%
req/resp (blocks_by_range, blocks_by_root)	many	0 snappy errors	n/a

Small messages never fail. Failure rate scales with message size — perfect fit for a muxer fragmentation / reassembly bug.

Failure happens before the Zig FFI handoff

The Rust-side [gossip ingress] log in rust/libp2p-glue/src/lib.rs (added in PR #945) shows rust_snap=err(Offset { offset: 4244805804, dst_pos: 98638 }) — i.e., the Rust snap::raw::Decoder also rejects the bytes, before any FFI handoff to Zig's snappyz. The bytes are already corrupt in message.data as delivered by libp2p-rust's gossipsub behaviour.

Python's reference snappy (Google's C library) also rejects the same captured payload offline. Three independent decoders agree — the bytes ARE malformed snappy when zeam receives them.

Non-determinism rules out a deterministic encoder bug

Same logical message (same data_len, same first 32 bytes) arrives at zeam_9 four times with different mid-stream decode failure positions:

data_len=286682 first32=ed c5 11 24 08 00 00 00 f0 00 00 00 01 00 09 01 ...
  rust_snap=err(Offset { offset: 2898906373, dst_pos: 131519 })
  rust_snap=err(Offset { offset: 3827619729, dst_pos: 78567 })
  rust_snap=err(Offset { offset: 3827619729, dst_pos: 98698 })
  rust_snap=err(Offset { offset: 43031490,   dst_pos: 57268 })

The publisher published one block (slot=1, proposer=ethlambda_8 from slot % validator_count). Four receipts at zeam have identical first-32-byte prefixes but bytes that diverge at different positions mid-stream. That isn't an encoder producing different output — it's the same byte stream getting corrupted in zeam's receive path, with the corruption happening at different positions on each retry.

Root cause: deprecated mplex muxer

rust/libp2p-glue/Cargo.toml declares libp2p-mplex = "0.43", and build_transport configures both yamux and mplex in SelectUpgrade:

// rust/libp2p-glue/src/lib.rs:2345-2359
let mut mplex_config = libp2p_mplex::Config::new();
mplex_config.set_max_buffer_size(256);
mplex_config.set_max_buffer_behaviour(libp2p_mplex::MaxBufferBehaviour::Block);
let yamux_config = yamux::Config::default();
let tcp = libp2p::tcp::tokio::Transport::new(...)
    .upgrade(core::upgrade::Version::V1)
    .authenticate(generate_noise_config(&local_private_key))
    .multiplex(core::upgrade::SelectUpgrade::new(
        yamux_config,
        mplex_config,
    ))
    .timeout(Duration::from_secs(10));

mplex was deprecated in rust-libp2p in 2023 and is known to have correctness problems with large messages. ream and grandine likely use yamux-only transports, which is why they don't see this.

The size correlation (only messages > 100 KB fail) is the fingerprint of muxer-level frame corruption: mplex fragments the substream into protocol frames and reassembles on the receiver side; the bug appears only when fragmentation actually has to do work (i.e., on large messages).

Auxiliary findings

• InvalidPostState slot=44 errors on zeam_8 / 9 are a downstream symptom — they fire when zeam syncs blocks via blocks_by_range whose pre-state would have been built from aggregations zeam never received via gossip.
• The deferred-hashTreeRoot work in PR node, metrics: defer gossip-block hashTreeRoot to chain-worker + libxev callback histogram (#942) #948 is unrelated to this — it just made the symptoms more visible because mesh-demotion under the old (decompress-in-message_id_fn) code path was masking how many gossip messages zeam was missing.

Fix

Drop libp2p-mplex from the transport. Use yamux as the only TCP-side muxer (QUIC keeps native streams). One-line code change in build_transport; follow-up PR incoming on top of #948.

What to verify post-fix

1. zeam_gossip_decode_failures_total{topic_kind="block",reason="snappy_decode"} stays at 0 on the next devnet
2. zeam head_slot tracks wall-clock instead of stalling at slot 51–61
3. InvalidPostState errors disappear (they were downstream of the missed-block problem)