Repository navigation
feat: add Beacon Chain support - #661
Open
MegaRedHand wants to merge 54 commits into
Open
MegaRedHand wants to merge 54 commits into
MegaRedHand wants to merge 54 commits into
Conversation
* build(deps): track libssz from git, and add ethereum-types and sha2 The Beacon Chain containers landing in the next commit hand out `&mut` into an `SszList` (the validator registry, through `altair_validator_lists_mut`), which published libssz-types 0.3.0 cannot do: it exposes only `Deref` and `Index`. `DerefMut` and `IndexMut` landed after that release, in lambdaclass/libssz#33. A git dependency rather than a `[patch.crates-io]` override: nothing outside this workspace depends on libssz, so there are no two copies to unify, which is the job a patch table would be doing. Pointing the four dependencies at the repository says the same thing with less machinery, and avoids committing a `[patch]` table that `shadow/cargo-patch.toml` would then collide with, since that fragment is appended verbatim to this manifest and declares one itself. All four move together because libssz-types depends on libssz and libssz-merkle by path inside that repository, so a mixed set would put two incompatible copies of libssz in the graph. `Cargo.lock` pins the exact commit, so tracking a branch does not make the build drift on its own; it moves when someone runs `cargo update`. Return these to a crates.io version once a release carrying #33 exists. `ethereum-types` and `sha2` join the workspace dependencies for the same incoming containers. `sha2`'s `asm` feature selects the CPU's SHA-256 instructions, which merkleizing a beacon state feels directly, since a state's `hash_tree_root` is almost entirely SHA-256 compressions. * feat(types): add the Beacon Chain containers under a beacon namespace The Beacon Chain is a different protocol from the Lean consensus this repository implements, but the two need to meet at one state type: a single BlockChainServer should be able to dispatch on the state variant rather than existing twice. That means the containers cannot live in a crate that also carries the state transition, because everything reaching for a container would then also pull in blst and c-kzg. They live here instead. Seven forks, phase0 through fulu. Containers that change between forks are defined once per fork as plain structs deriving their SSZ encoding and merkleization, wrapped in an enum. Deriving is the reason for that shape: the per-fork field lists are not a growing tail. Two of phase0's fields are *replaced* in altair, one field changes type in five separate forks, and the state's merkle tree gains a level at electra, so a single container with fork-conditional serialization would have to reproduce all of that by hand. The preset is a compile-time choice, `preset-minimal` against an implicit mainnet default, because SSZ list and vector bounds are const-generic arguments and cannot be selected at run time. Fork *scheduling* is runtime configuration in `config` instead, since fork epochs move per case in the upstream test suites. `constants` holds what the spec fixes outright, `primitives` the scalar aliases and fixed-length byte strings, `error` what the containers return, and `fork` the `ForkName` ordering that later gating reads. This is a port of code verified against the consensus-specs v1.6.1 fixture suites, where it passes 5705 mainnet and 40009 minimal cases across both presets. Those suites exercise the state transition that consumes these containers and are not part of this crate, so what runs here is the containers' own round-trip and shape tests. * feat(types): give ForkName and BeaconState a Lean variant This is the point of moving the containers here. One BlockChainServer should dispatch on the state variant rather than existing once per chain, and that needs a single state type spanning both. `BeaconState::Lean` holds lean's `State`, which gains `PartialEq` because `BeaconState` derives it. `ForkName::Lean` is declared last, so the derived `Ord` puts it after every beacon fork. That is what `fork >= ForkName::X` gating reads, so such a gate reads as true for a lean state rather than accidentally selecting phase0 rules. It is deliberately absent from `ForkName::ALL`. `ALL` is what `parse`, `previous`, `next`, and the upstream fixture harness search, so its absence is what makes `parse("lean")` return `None`, keeps `Fulu.next()` at `None` so a fork upgrade cannot walk off the end into lean, and stops a fixture directory from ever resolving to a lean case. Lean is not a point on the Beacon Chain's fork timeline and must not be reachable by traversing it. Only `fork_name()` and `from_ssz()` are real arms. Every other exhaustive match on `BeaconState` answers `unreachable!()` naming itself, and the accessors on `ForkName` do the same, except those already returning `Result`, which answer `UnsupportedForFork`. That asymmetry is the design, not an oversight. The guarantee that a lean state never reaches a beacon accessor is the single `match` at the top of each handler, not the type system, so the failure mode when that guarantee is broken should be a named panic pointing at the function that was reached, rather than a silent wrong answer computed from a state of the wrong shape. * feat(types): add what the storage and networking layers need Four additions, each pulled here by a crate that must not depend on a Beacon Chain state transition. `fork_digest` computes `compute_fork_data_root` from the fork schedule. Gossip topic names and the eth2 ENR entry both need it, and both belong to the networking layer, which cannot reach for a crate carrying blst and c-kzg just to name a topic. The fulu branch is EIP-7892's: a blob-parameter-only fork perturbs the digest, so mainnet's is 8c9f62fe rather than fulu's bare 82fae541. `fork_choice` holds `LatestMessage` and `PowBlock`. Both are data a fork choice store holds rather than behavior it runs, and the storage layer cannot depend on the consensus layer because the dependency runs the other way. `ForkName` gains a stable one-byte storage selector. A stored state has to record which fork's shape it has, and the derived discriminant is not a stable encoding: it shifts whenever a variant is added, which silently reinterprets every state already on disk. An explicit byte survives that. Lean's `primitives` gains conversions between the beacon `Root` and lean's `H256`. Same 32 bytes in two types, and the boundary is crossed by every store lookup on the beacon path, so the conversion belongs here rather than open-coded at each call site. * docs(types): resolve the beacon namespace's doc links Ten rustdoc warnings, inherited from the source these containers were ported from rather than introduced here, but there is no reason to carry them onto main. Four kinds: - `shared_state_accessors` is a `macro_rules!` with no `#[macro_export]`, so rustdoc has no item to point at and the link never resolved. It becomes a plain code span, which is what it always meant. - `SYNC_COMMITTEE_SUBNET_COUNT` is a private const. altair.rs already refers to it as a plain code span thirteen lines further down, so this follows the convention the file had already settled on. - capella's three links spelled out a target identical to what the label already resolved to. - preset.rs's two sat in an outer doc comment on `pub mod retuned`, which resolves in the parent scope rather than inside the module, so they pointed at nothing. Qualifying them with `retuned::` makes them mean what they say. `cargo doc -p ethlambda-types --no-deps` is now warning-free. * docs: describe the beacon namespace in ethlambda-types Records what a reader needs before touching these types: that the Beacon Chain is a separate protocol sharing the crate only so one state type spans both, that ForkName::Lean sits outside ForkName::ALL on purpose, why the beacon accessors panic rather than error on the Lean variant, and that the preset is a compile-time feature every lean crate inherits. Also records where the verification lives, since it is not in this repo: the state transition that exercises these containers against the consensus-specs fixtures is on feat/beacon-chain-stf. * build(deps): pin the libssz crates by rev, not branch `branch = "main"` leaves a routine `cargo update` free to move the SSZ encoder and merkleizer that decides every `hash_tree_root`. The requirement is a commit carrying lambdaclass/libssz#33, not whatever `main` holds today, so pin the rev `Cargo.lock` already resolved to and make moving it an explicit edit. Also correct what sha2's `asm` feature actually buys: on x86/x86_64 the SHA-NI dispatch happens through `cpufeatures` with or without it, and `asm` only swaps that dispatch's fallback. aarch64 is where the feature is the whole hardware path. * refactor(types): dispatch the beacon fork ladders from one place Every beacon accessor spelled out its own seven-arm match plus a lean arm, so a new fork meant editing twenty ladders, and the boundary panic's wording had already drifted between them. Route the arms through `dispatch_state!`, `dispatch_state_from!` and `dispatch_block!`, and write the panic once as `lean_unreachable!`, in the two forms the two boundaries are crossed by. Also, from the same reviewing pass: - Make the fork matches in `preset::retuned` exhaustive, so a fork added later is a build error instead of falling silently into bellatrix's value. - Keep `get_blob_parameters` on `Config`, where `max_blobs_per_block` and `compute_fork_digest` both reach it; the digest hashes the pair's epoch as well as its limit, so the search cannot be written once per caller. - Convert both presets' `TERMINAL_TOTAL_DIFFICULTY` at compile time, with a test pinning the limbs against the decimal strings the config files carry. - Move the `Root` <-> `H256` conversion to the beacon side of the namespace: beacon is the half that knows about lean, so lean's `primitives` names nothing beacon. * ci: check the minimal preset The beacon containers' SSZ bounds are const-generic arguments, so the minimal preset is a whole second compilation of ethlambda-types that no other step reaches: everything else builds the default mainnet preset only. The fixture suites that select the feature live on another branch, so without this step it would rot between merges. * refactor(types): turn lean_unreachable! into two functions The macro's two arms were already two unrelated diagnostics: one for a beacon accessor handed lean's state, one for a beacon-only function handed ForkName::Lean. Nothing about either needed macro expansion, so they become lean_state_unreachable and lean_fork_unreachable. Both return `!` so they still type-check in every arm position the macro occupied: value arms, the Ok/Err arms of dispatch_state_from!, and the unit arm of Config::with_fork_epoch. #[track_caller] is what a plain function would otherwise have cost us, since without it every panic would report beacon/mod.rs instead of the arm that was actually reached. * refactor(types): replace byte_vector! with plain newtypes Four expansions did not earn a macro, and the $(#[$doc:meta])* passthrough it needed to carry each type's documentation was the least readable part of the module. Written out rather than made newtypes over a generic ByteVector<N>, which would have deduplicated Default and AsRef but turned every construction site into BlsPubkey(ByteVector([9; 48])) and every field read into .0.0. Type aliases are not an option at all: BLS_PUBKEY_SIZE and KZG_POINT_SIZE are both 48, so aliasing would make BlsPubkey and KzgCommitment the same type and give up exactly the confusion the newtypes exist to catch. Default stays hand-written because the standard library implements it for arrays only up to length 32, so a derive has nothing to call for [u8; 48] or [u8; 96]. The Debug bodies share a debug_byte_vector helper so the four cannot drift into printing the same kind of value four ways. * test(types): drop a_state_equals_its_own_clone The assertion was that a derived PartialEq agrees with a derived Clone, which holds by construction and would not fail short of a compiler bug. It was the only test in partial_eq_tests, so the module goes with it. * ci: scope the minimal-preset check to --lib, so the lint job can pass The step runs in the `lint` job, which downloads no fixtures. `cargo test -p ethlambda-types --features preset-minimal` builds every test target of the crate, including `ssz_spectests`, whose datatest-stable harness walks `leanSpec/fixtures/consensus/ssz` and panics with "error reading directory" when that tree is absent. Only the `test` job fetches it, so the step exits 101 on every run, on pull requests and on main alike, without ever reaching the thing it exists to check. `--lib` runs the 102 unit tests instead, which read nothing from disk. Those are what the preset actually changes: it selects the const-generic bounds the beacon containers are declared with, so the whole point is that the crate compiles under it and its own assertions still hold. The fixture suite is preset-independent and already runs, at the default preset, in the `test` job. * feat(types): let BeaconState encode and merkleize its lean variant `BeaconState::from_ssz(ForkName::Lean, ..)` builds `BeaconState::Lean`, while `to_ssz` and `hash_tree_root` dispatched through `dispatch_state!` and so panicked on it. Decoding a state and re-encoding it therefore panicked: the enum accepted a value on the way in that it refused on the way out. Nothing reachable today hits it, since no production caller constructs the variant and `ForkName::parse` cannot return `Lean`. What makes it worth closing anyway is who the first caller will be. The variant exists so one `BlockChainServer` can dispatch on a single state type; that server will decode a state whose fork came from configuration, where `Lean` is a legal value, and re-encoding it to store it is the next thing it does. The other `Lean` arms panic because there is genuinely no beacon answer to give: asking a lean state for its `validators` or its `slot` has no meaning, and a named panic beats a wrong number. These two are not like that. Every variant is an SSZ container, so encoding and merkleizing mean the same thing whichever chain the state belongs to, and the answer was already sitting in the variant. `dispatch_state_including_lean!` is the arm list for exactly those cases, kept separate from `dispatch_state!` so that a beacon accessor cannot reach it by accident: it takes no `$function`, because it has no arm that panics. One subtlety worth naming, since it is what makes a single macro body typecheck for all eight arms. This crate has two `HashTreeRoot` traits, both blanket implemented over `libssz_merkle::HashTreeRoot`, differing only in the wrapper type around the digest. `containers/mod.rs` imports the beacon one alone, so lean's state answers it too, with the same bytes lean's own trait would produce. The new test asserts that equality rather than merely asserting the call does not panic, because a block's `state_root` is checked against this digest. It names lean's trait through its full path: importing it would make the call ambiguous. `SignedBeaconBlock` is left as it is. Its `from_ssz` returns `Error::UnsupportedForFork` for `Lean` and it has no `Lean` variant to encode, so it has no asymmetry to fix. * refactor(types): declare the H-types instead of depending on ethereum-types The beacon namespace pulled `H160`, `H256` and `U256` from `ethereum-types`, which meant the crate carried two 32-byte hashes: that one and lean's own `primitives::H256`. Every beacon block root reaching a store lookup had to be converted between them, and the two `HashTreeRoot` convenience traits over the two hashes were otherwise identical. `beacon::primitives::Root` is now `primitives::H256` itself and `beacon::primitives::HashTreeRoot` re-exports lean's trait, so both conversions and the duplicate trait are gone. What an external crate was actually supplying beyond the hash is two newtypes, `H160` and `U256`, which this module now declares. `U256` holds the 32 little-endian bytes SSZ encodes it as, so encoding and merkleizing are the inner array's. That puts the stored byte order opposite the numeric one, so `Ord` is written out: derived, it would compare the least significant byte first and invert the `terminal_total_difficulty` comparisons that are the only place the specification orders a `uint256`. Mainnet's value fits 128 bits and is now spelled as its decimal digits rather than as limbs; both values stay pinned against the configuration files' decimal strings. `H160` writes out `libssz_merkle::HashTreeRoot` because the derive drops `is_basic_type`, and at 20 bytes wide that answer decides whether a collection of addresses packs or pads. No container holds one today, so both answers agree everywhere it is currently asked; writing it out is what keeps the first container that does hold one from silently getting the other tree. `ethereum-types` stays in `Cargo.lock` as an indirect dependency of `ethrex-common`, which `ethlambda-p2p` needs for discv5. * refactor(types): print an H256 as hex under Debug, not as 32 numbers The derived `Debug` printed the inner array, so a hash came out as `H256([171, 171, ...])`: unreadable alone, and unreadable in bulk inside the `Debug` of a container holding thousands of roots. `ethereum-types` printed hex, so with the beacon `Root` now being this type, keeping the derive would have made every spec-suite assertion failure that prints a state harder to read than before. Same output as `Display`. Nothing asserted on the old form. `ShortRoot` is still the way to truncate at a call site that wants it short.
feat(state-transition): Beacon Chain state transition, phase0 through fulu
feat(beacon): follow mainnet gossip behind an `ethlambda beacon` sub-command
ci: split the lint and test jobs, and drop the disk-cleanup action
* docs: design the unified lean/beacon storage layer
The beacon fork choice holds its whole world in memory, including three
unbounded maps of ~350 MB beacon states, so a mainnet follower can
neither survive a restart nor bound its memory. This designs the move
onto the DB-backed store the lean chain already runs on: one set of
methods over one block enum, one runtime config persisted for both
chains, and a byte-domain state delta that makes recovering an arbitrary
beacon state bounded.
The diff layer is the one deliberate exception to sharing. Lean's
field-shaped StateDiff depends on validators never changing, which a
beacon registry breaks every epoch, and it is the better algorithm for
lean, so beacon gets xdelta3 byte deltas beside it rather than instead
of it.
Working artifacts, not reference docs: both are removed once the work
lands, the way the beacon STF design document was.
* build: check debug assertions under the test profile
Tests run under release-fast, which inherits release and therefore had
debug assertions off, so a debug_assert! in any crate ran nowhere. A dev
build is not a fallback: signature verification and aggregation
stack-overflow without release-grade opt-level.
overflow-checks is pinned rather than inherited so arithmetic behaviour
stays identical to release.
Verified both directions: debug_assertions appears in `cargo rustc
--profile release-fast -- --print cfg` and not under release.
* test(types): pin the genesis state root
`State.config` is hashed into every lean state root and is the field
leanSpec declares as `config: GenesisConfig`. Naming it after the spec was
this commit's original job, and main has since done exactly that under the
name `StateConfig`, keeping `ChainConfig` for the node's runtime
configuration: a different type for a different job.
What is left to add is the guard the rename came with. This pins the
genesis state root, so a field reorder, an added field, or a type change
disguised as a rename fails here rather than in cross-client interop.
* feat(types): let the runtime config persist, and give lean one
Config becomes the store-level configuration for both chains, so it needs
an encoding it did not have and a genesis_time it did not carry.
min_genesis_time could not serve as the clock: it is the earliest the
deposit-driven rules permit, 23 seconds before mainnet actually started.
Config::lean fills every beacon field with a placeholder chosen to fail
closed, FAR_FUTURE_EPOCH for every fork epoch, so a beacon-shaped gate
reading a lean config never sees an activated fork where epoch 0 would
have activated all seven.
blob_schedule becomes a bounded SSZ list because SSZ has no unbounded
one; the bound is a storage limit, which is what the database format
version exists to let us change.
* feat(storage): tag a data directory with its chain and format version
A directory is one chain for its whole life and one on-disk format, and
neither is recoverable when wrong: there is no migration, because the
States value layout is about to change and every state already written
would decode as the wrong shape.
Both checks come after the "has this ever held a chain" reads, so a fresh
directory is still Ok(None) rather than a version mismatch.
Also folds four Store struct literals into one constructor, which is how
the next few fields avoid drifting between the bootstrap paths.
* feat(storage): hold the runtime config, not the in-state one
Metadata["config"] now holds the unified runtime Config for both chains
rather than the two-field container that lives inside a lean State. The
in-state one is still what a state root commits to; this is what the node
runs on, and a beacon directory needs the fork schedule in it.
Returned behind an Arc because every beacon fork-choice call takes
&mut Store alongside the config, so a caller has to hold it across a
mutable borrow of the store it came from.
The config row's layout changes without a DB_VERSION bump on purpose:
the version was introduced one commit ago on this same branch and has
never shipped, so there is no released format to be compatible with.
* feat(types): give the block enum a lean variant
BeaconState already carries Lean; blocks now do too, so the storage layer
can take one block type and split inside its methods instead of growing a
method per chain.
The accessors are macro-generated, so the arm lands in the macros: the
existing message:/outer: split turns out to be exactly the lean boundary,
since lean's Block declares all four message fields under the same names
and has no BLS signature to answer the one outer accessor with.
The enum is not renamed. It names 156 sites across 19 files, eight of
them the concrete per-fork structs that share the identifier, and
BeaconState already carries Lean under a beacon-shaped name.
Adding the variant also required PartialEq on SignedBlock, Block,
BlockBody and AggregatedAttestation, since SignedBeaconBlock derives it
and now has to compare a lean payload too; and a lean_block_unreachable
boundary in ethlambda-state-transition's beacon module, mirroring the one
ethlambda-types already has, since the crate's own beacon process_block
dispatcher matched the enum exhaustively.
* docs: withdraw the block-enum rename, and record how the Lean arm lands
Two things the design got wrong about blocks, found while sizing the work.
The rename to SignedBlock is withdrawn. SignedBeaconBlock names 156 sites
across 19 files and eight of those are the concrete per-fork structs that
share the identifier, so a rename is a large edit with a real corruption
mode and no functional gain. BeaconState already carries Lean under a
beacon-shaped name, which is the precedent that makes the rename
unnecessary rather than merely risky.
The plan also assumed the block accessors were hand-written match arms.
They are macro-generated, and there is a fifth accessor the plan never
accounted for: signature, returning a BlsSignature that a lean block has
no equivalent of. The state side already solved this with a pair of
dispatch macros, one panicking on Lean and one running every arm, so
blocks mirror it. The happy part is that the existing message:/outer:
split in the accessor macro turns out to be exactly the lean boundary,
since lean's Block declares all four message fields under the same names
and cannot answer the one outer accessor.
Also corrects the claim that current_slot() reads genesis_time. It
divides the store clock by INTERVALS_PER_SLOT and never reads the config,
so it is lean-shaped as written and the beacon path needs its own slot
derivation from the Unix clock.
* feat(storage): tag States values with the fork that shapes them
One table now has to hold a lean State and seven beacon shapes, and SSZ
carries no type tag, so the reader cannot recover the shape from the
bytes. A leading ForkName::selector byte is what makes one table serve
both without the caller having to already know which chain it opened.
Safe to change the value layout only because the format version and
chain tag landed first: an older directory is rejected outright rather
than misdecoded, and there is deliberately no migration.
The selector is not a variant index, so it always goes back through
from_selector: Lean takes 255 so beacon forks after fulu can keep taking
the next free value.
* feat(storage): the beacon fork-choice table and its scratch
Splits the beacon fork-choice store's fields by whether they are worth
persisting. Only the unrealized justification is: get_voting_source reads
it for every block from a prior epoch, and recomputing an entry means
replaying epoch processing on a copy of that block's post-state, so it
gets the ninth table.
The rest are per-slot or per-epoch scratch and go in memory. Proposer
boost resets every slot, equivocators come back from replaying attester
slashings, latest messages from one epoch of attestations, and PoW blocks
stand in for an execution-client call a restarted node makes again.
The latest-message read takes a closure because the map is behind a
mutex, and it filters equivocators there because get_weight has to
exclude such a vote entirely rather than let it count for either side of
the fork it created.
* feat(storage): one block insert for both chains
insert_signed_block takes the fork-generic enum and splits inside it,
with both arms spelled out rather than delegating to a helper per chain:
the two write different tables, and that difference is what a reader
needs to see. Because a beacon Root is already H256 there is one key type
and one value codec, so nothing here needs a bridging generic.
BlockRoots and BlockProof stay unwritten on the beacon arm: a slot holds
several blocks once a fork exists, so only the parent links say which is
canonical, and a range request walks those from the fork-choice head.
The lean arm keeps its post-commit attestation-vote recording, which the
beacon arm has nothing to do: a beacon block carries no lean
attestations, so the match hands that decision back out rather than
running it unconditionally.
* feat(storage): the beacon checkpoints, head, and an empty beacon directory
Both bootstrap paths write every metadata key they will later read, so an
absent key is a bug reaching the wrong chain's accessor rather than a
condition a caller could recover from. The reader now panics naming the
key and the chain, which is what says which mistake was made, and the
helpers stop returning a Result they could never populate.
Beacon's accessors return values directly: their callers' error type
lives in ethlambda-types and cannot carry a storage error, so a Result
here would buy a map_err at every call site and nothing else.
The fork-choice head is the one row deliberately left unseeded, since
absent means no head has been computed yet, which is a real state.
Metadata["time"] is shared between the chains with different units, which
is safe exactly because a directory holds one chain for its whole life.
from_db_state's existence gate previously relied on KEY_LATEST_FINALIZED,
a lean-only key init_beacon never writes; a fully-formed beacon directory
would otherwise read as empty and let the lean bootstrap write into it.
The gate now consults the chain tag first.
* feat(storage): read a block back as the fork it was written as
insert_signed_block already took the fork-generic enum, but the read side
still returned the lean type, so a beacon block could be written and
never read back. The fork choice migration needs to read blocks, so the
asymmetry had to go.
Dispatches on the cached chain tag rather than trial-decoding: a data
directory holds one chain for its whole life, so the tag is authoritative
and guessing would be wrong as well as slower.
The lean arm keeps the two synthesis behaviours its callers depend on,
the empty proof for the slot-0 anchor and the empty body for a header
whose body_root says there is none.
* feat(storage): persist and read a beacon post-state
The fork choice migration has post-states to keep, so insert_state and
get_state take the fork-generic state type and split inside, the way the
block methods already do.
The beacon arm writes a full snapshot per block on purpose. That is
wasteful at mainnet scale and is a placeholder: the delta layer and the
bounded cache land in later tasks, and at the fixture presets a
standalone snapshot per block is entirely adequate. Beacon deliberately
does not borrow lean's StateDiff, which omits validators on the
assumption they never change; a beacon registry changes every epoch.
The lean arm is unchanged: same snapshot interval, same diff chain, same
reconstruction, same cache.
* feat(beacon): put the fork choice on the DB-backed store
The spec-shaped in-memory Store held three unbounded maps of whole
beacon states, so a mainnet follower could neither survive a restart nor
bound its memory. Its fields move to the storage layer the lean chain
already runs on: blocks to the block tables, checkpoints and the clock to
Metadata, unrealized justifications to their own table, and the per-slot
scratch to an in-memory struct that is cheap to rebuild.
The children scan takes a block index built once per tree walk rather
than a lookup per hop, since get_weight calls get_ancestor once per
active validator.
checkpoint_state is derived on every call for now; a later task restores
a bounded cache. That also removes the file's only outright whole-state
clone, which existed because two maps each needed their own copy of the
anchor.
The fixture suite passes with an unchanged case count at both presets.
* feat(beacon): record the head fork choice computed, and narrow the rest
set_beacon_head existed with tests but had no caller, so a restarted
node had to recompute a whole weighted tree walk before it could answer
what its head was. get_head now writes the head it just computed, every
time rather than only on change: a value written once and left alone is
a second source of truth a bug can let drift.
That also earns get_head's &mut Store. The six read-only helpers around
it go back to &Store: checkpoint_state was given &mut on the theory that
a future cache would need it, but the cache accessors are &self by
design and use interior mutability precisely so a read-only helper can
record a derived value. The module's invariant is restored, so which
functions can mutate consensus state is visible again at a glance.
Fixture suite unchanged at both presets.
* feat(storage): byte-domain state deltas for the beacon chain
VCDIFF rather than xor-plus-compress: an SSZ list that grows shifts every
offset after it, which an xor delta cannot survive and a COPY/ADD stream
handles natively. That is what lets the three big arrays stay untouched
instead of needing length-aware handling of their own.
Beacon needs its own algorithm because lean's StateDiff omits validators
on the assumption they never change, which a beacon registry breaks every
epoch. Lean keeps its own, which is smaller and cheaper there and keeps a
C dependency off the path every devnet runs.
Sized through the *_with_output_len variants rather than the crate's
convenience wrappers, which ask for (input + src) * 2: at beacon scale
that is a ~1.4 GB transient per call and the u32 cast wraps above a
~1.07 GB source. The frame carries the target length so a decode can size
its output exactly.
default-features = false because the crate's stream feature does not
compile at this rev; the one-shot API used here is outside that module.
* docs: replace the delta layer's guessed costs with measured ones
A standalone probe of the codec, before wiring it in, answered two of
this design's open questions and corrected a third.
The dependency does not build as specified: its default `stream` feature
pulls an async module that fails to compile at the pinned rev, so
`default-features = false` is mandatory rather than tidy. The one-shot
in-memory API this design uses is outside that module.
The source-window risk is deleted. xdelta3's 64 MiB window belongs to its
streaming API; this crate wraps the one-shot memory API, which takes the
whole source as one buffer. A one-byte change 79 MB into an 80 MB base
still yields a tiny delta.
And the cost estimates were pessimistic by roughly eight times. Measured
on a 64 MiB buffer with three quarters of its bytes rewritten to
index-derived values: delta 1 MiB, encode 229 ms, decode 6 ms. Scaled to
a mainnet state that puts encode near 1.3 s, inside a slot, and a
31-delta fold near one second rather than eight. The epoch-sized
snapshot interval survives.
The benchmark task still earns its place: the extrapolation is linear and
xdelta3's hash table need not be, and the probe's change pattern is more
structured than a real reward distribution, so it flatters the delta size
more than the timings.
* feat(storage): wire the byte-domain delta codec into beacon state storage
A mainnet BeaconState is on the order of 350 MB, so a snapshot per block
(the placeholder insert_state/get_state shipped with) is not viable past
the minimal preset the fixture suites run against. insert_state now
snapshots only at fork-scoped intervals (one epoch for beacon, unchanged
for lean) and diffs everything in between with the byte-domain VCDIFF
codec added in the previous commit; get_state folds the resulting delta
chain in the byte domain and SSZ-decodes exactly once, at the end, since
folding bytes is memcpy-speed while a mainnet-scale SSZ decode is not.
ForkName::snapshot_interval centralizes the interval so lean's storage
layer and the new beacon path share one source of truth instead of each
picking its own constant.
* refactor(storage): one bounded state cache for both chains
The beacon fork choice used to hold whole states in maps with no cap,
which is not a tuning question at a few hundred megabytes each. It now
shares the LRU the lean chain already had, so one capacity bounds
everything.
Behind an Arc because a hit must not copy: returning an owned state would
hand back much of what the cache saves, and the fixture presets are too
small for a test to notice.
Block and checkpoint states share one bound through a key enum rather
than getting a map each, since two separate caps overshoot the total. A
checkpoint is keyed by epoch as well as root, because a checkpoint's root
is the last block at or before its boundary slot and the same root can
serve different epochs.
The accessors take &self so the read-only fork-choice helpers can record
what they derive on a miss; that is what lets checkpoint_state stay
&Store instead of widening its callers.
* test(storage): measure beacon delta cost on a real mainnet-scale state
The design's encode-cost risk was marked "largely retired by measurement"
on a synthetic 64 MiB probe whose 75%-rewritten, single-pass change
pattern is far more structured than a real epoch transition, and whose
timings were extrapolated linearly past a hash-table codec that need not
scale that way. Both assumptions mattered: measuring a real 2,000,000-
validator electra state (280.7 MB) shows encode cost tracks how
fragmented a change is, not just buffer size.
One ordinary slot: 924 B delta, 375 ms encode, 26 ms decode, comfortably
inside a 4 s slot. Across an epoch boundary: 26.0 MB delta, 3.04 s
encode, 111 ms decode, roughly 2.3x the earlier linear guess and three
quarters of a slot on its own. The epoch-sized snapshot interval still
survives, but only because `is_anchor` already routes the expensive
shape around xdelta3 entirely (always a full snapshot at that boundary,
never a delta): its safety is contingent on staying epoch-aligned, not
on the codec being fast for any shape.
Recorded in the design doc alongside the probe for contrast.
* docs: price the delta measurements against a beacon slot, not a lean one
The measurement section compared both encode times to a 4 second slot,
which is the lean chain's slot time. Mainnet's seconds_per_slot is 12, so
the ordinary-slot delta is about 3% of a slot rather than "most of it
free", and the epoch-boundary shape is about a quarter of a slot rather
than three quarters.
The conclusion is unchanged, and so is the reason: the epoch-transition
shape never reaches the codec, because is_anchor writes that state as a
raw snapshot. But a quarter of a slot and three quarters read very
differently to whoever has to decide whether the contingency is
acceptable, so the arithmetic is worth getting right.
* chore: remove implementation plan
* refactor(storage): the unrealized justification is scratch, not a table
`get_voting_source` reads a block's unrealized justification for every
block from a prior epoch, and recomputing a missing entry means replaying
epoch processing on a copy of that block's post-state. That is what earned
it a column family of its own, but it does not make the map chain history:
no beacon path resumes from a data directory today, and the one that
eventually will re-imports the unfinalized window from its anchor, refilling
the map as it goes.
So move it into `BeaconScratch` beside the other beacon fork-choice maps and
drop `Table::BeaconForkChoice`, putting the table count back at eight. The
accessors keep their signatures, so fork choice is untouched; a read is now a
mutex rather than a backend read plus an SSZ decode, and a write no longer
commits a batch per block.
The struct's bound claim changes with it. `block_timeliness` and the
justifications are both one entry per block imported and neither is pruned,
so "bounded by the unfinalized window" was already loose; say what is
actually true of each map instead.
A data directory written by an earlier commit of this branch will no longer
open, since RocksDB requires every on-disk column family to be listed.
Nothing outside this branch ever had the table, so no lean directory is
affected.
* refactor(types): peel a lean value off the shared enum in one place
`Store::get_state` and `get_signed_block` hand back the fork-generic
`BeaconState`/`SignedBeaconBlock` for both chains, so every lean-only
caller narrowed the result again by hand: thirteen copies of
`let BeaconState::Lean(x) = y else { unreachable!(..) }` across five
crates, each carrying its own wording for the same impossibility, and one
that logged and skipped rather than panicking at all.
Give both enums an `expect_lean` accessor and put its panic behind
`beacon_value_unreachable`, beside the `lean_state_unreachable` pair that
already answers the opposite direction. The invariant that a data
directory holds one chain for its whole life is now asserted in one place
instead of thirteen, and a new lean caller cannot word it differently or
weaken it into a silent skip.
* refactor(storage): one head and checkpoint writer for both chains
The beacon arm kept its own checkpoint rows because the two chains'
`Checkpoint` types carry different fields, lean's `{root, slot}` against
beacon's `{epoch, root}`. That reason does not survive contact with the
fact that an epoch names its own start slot: storing it that way round
trips exactly through a division, so one slot-denominated row serves
either chain and `Store::update_checkpoints` becomes the single writer of
the realized pair. `KEY_BEACON_JUSTIFIED` and `KEY_BEACON_FINALIZED` go
away, and with them the four setters that fed them.
`KEY_BEACON_HEAD` goes too. The head is `KEY_HEAD` on both chains now, so
`beacon_head` derives its slot from the head's own header row rather than
keeping a second `slot || root` value that could drift from the first.
Recording a head through the shared writer also keeps the canonical
`BlockRoots` index in step on a beacon directory, which it never was:
that index holds one block per slot because it holds only the branch fork
choice picked, so forks are no more an obstacle here than on lean.
What the unification actually needed was a chain-agnostic reader for a
stored block's slot and parent root. `block_root_index_changes` decoded a
lean `BlockHeader`, so it could not have run on a beacon directory at
all; `block_entry` now dispatches on the chain tag and both walks share
it. The beacon fork choice reads through it too, in place of the seven
sites that fully decoded a `SignedBeaconBlock`, execution payload
included, to reach two integers.
Pruning stays lean-only, deliberately. The gossip-signature and payload
buffers hold lean attestations a beacon directory never has, and
`prune_live_chain` would be worse than a no-op: `filter_block_tree` asks
`get_checkpoint_block` for the ancestor at the finalized epoch's start
slot, and `get_ancestor` keeps walking while a parent's slot exceeds the
one asked for, so an empty start slot lands below the horizon. A row
missing there is a hard `SpecAssert`, not a degraded read, so beacon
needs its own horizon before it prunes anything.
Also here, from the same read-through: `block_index` was a byte-for-byte
copy of `get_live_chain` and now delegates; `get_weight` takes the block
index instead of rescanning the whole table once per candidate child at
every level of `get_head`'s walk; `set_metadata_batch` puts related
metadata rows under one commit; and the index diff short-circuits when
the head has not moved, which is what a checkpoint-only advance looks
like on the beacon arm.
* refactor(storage): keep a beacon block in one row, not two
Lean splits a block across `BlockHeaders` and `BlockBodies` for two
reasons: a header-only query need not pay for the body, and a block whose
body is empty can leave the body row out entirely. Neither reason holds
for a beacon block. It has no empty-body case, and nothing reads a beacon
header without wanting the block behind it, so the split bought a second
write per import and a second way for the two rows to disagree.
`BlockHeaders` now holds the whole `SignedBeaconBlock`, fork-selector
tagged the way a `States` value is, and the beacon arm writes no
`BlockBodies` row at all. `BeaconBlockEntry`, the two-field struct that
stood in for a header, goes away with it, and the tag-then-decode that
`get_signed_block` spelled out inline is now `encode_beacon_block_value`
and its inverse, beside the pair that already does this for states.
The cost is that `block_entry` decodes a whole block, execution payload
included, to answer a slot and a parent root. Its own documentation now
says so and points a caller walking a chain of them at
`Store::block_index`, which reads the same links out of `LiveChain`.
Also corrects the comment on why `BlockRoots` is not written during
import. It claimed a slot can hold several blocks once a fork exists, but
that index only ever holds the branch fork choice picked, so it holds one
block per slot on either chain; what makes it wrong to write here is that
import order does not determine which branch is canonical.
`update_checkpoints` maintains it as the head moves.
* refactor(beacon): read a block's slot from the caller that already has it
Three of the seven places the beacon fork choice reached into the store
for a block's slot were asking for a value the caller already held. Now
that a beacon `BlockHeaders` row holds the whole block, each of those was
a full SSZ decode, execution payload included, to recover one integer.
`get_voting_source` was the worst of them: `filter_block_tree` calls it
once per leaf on every `get_head`, and one frame up it already has that
block's `(slot, parent_root)` out of the block index it is walking. It
now takes the index, like `get_ancestor`, `get_checkpoint_block` and
`get_weight` do, which is also the closer reading of the specification,
whose `store.blocks[block_root].slot` this index stands in for.
`compute_pulled_up_tip` is called only by `on_block`, which has the block
itself in hand, so the slot is passed in. `validate_on_attestation` built
the index a few lines below its own point lookup; building it first lets
the known-block check and the LMD walk share one scan.
The four that remain, in `get_proposer_head` and
`should_override_forkchoice_update`, are proposal-time and have no index
in scope, where a whole-table scan to serve two lookups would be the
worse trade.
* chore(storage): drop a comment for an import that is gone
`BeaconBlockEntry` was the only thing in `store.rs` deriving SSZ, and it
took the `libssz_derive` import with it when 5a8f8be1 folded a beacon
block into one row. The comment above that import stayed.
It explained a real hazard: `libssz` exports traits named `SszDecode`
and `SszEncode`, `libssz_derive` exports derive macros of the same two
names, so a file wanting both has to bind one pair under another name.
Nothing here wants both any more, and a comment describing a collision
that no longer exists reads as a warning against a change that is
already made.
* refactor(storage): drop the beacon import watermark
`beacon_highest_imported_slot` is the highest slot any beacon block was
imported at: monotone, advanced only forward, and deliberately not the
fork-choice head, so that a reorg lowering the head does not make
forward sync re-fetch slots it already has.
Nothing on this branch reads it. Its three callers live on
feat/mainnet-network, which forked from #591 and keeps its own copy in
`beacon_store.rs`, the file this branch folded into `store.rs`: the
range-sync origin, the `Status` head fallback before the first head
computation, and the `lean_sync_local_head_slot` metric. So what stood
here was the storage half of a feature whose other half is on a sibling
line, kept alive by tests written for it and by nothing else.
Carrying it costs a metadata key seeded at every beacon bootstrap, a
point read on every `insert_signed_block` including the lean arm that
has no use for it, and a conditional write. Whichever line lands second
has to reconcile the two copies anyway; there is no version of that
merge made harder by this side being empty.
Also drops half of the `batch.commit()` rationale, which justified the
single commit partly by the watermark never getting ahead of the block
it counts. The other half, that a half-written block is visible to the
`LiveChain` scan without being decodable, is the whole reason now.
* feat(storage): one store clock in seconds, plus a preset tag on the directory
`Metadata["time"]` carried two different units under one key and one accessor:
intervals-since-genesis on a lean directory, Unix seconds on a beacon one.
`Store::time`'s own documentation asserted the lean meaning unconditionally,
and `Store::current_slot` divided by `INTERVALS_PER_SLOT`, so it answered a
number that meant nothing on a beacon store. Nothing read it there yet, which
is the only reason that was not already a bug.
`time` is now a Unix second on both chains, which is what the beacon
specification's `Store.time` is, and `current_slot` derives the slot from it
for either chain. Lean keeps its interval clock under a second row,
`intervals_since_genesis`, because it cannot be derived from the first: with
`INTERVALS_PER_SLOT` intervals to a slot most interval boundaries fall
strictly between two whole seconds, and `on_tick` steps that grid one interval
at a time, running a duty per step. Both advance from the same tick, the
shared one forward-only so that `tick_to_slot` handing back an earlier
timestamp cannot rewind it.
Every lean site that meant "which interval are we in" now says so: the
attestation and block future-slot guards, the tick idempotency guard, the
arrival-metrics bound, the early-aggregation gate, the fork-choice fixtures'
`checks.time` and the Hive driver snapshot. The two block/attestation guards
in particular are documented as mirrors of each other, so both read the clock
`on_tick` rewinds to replay a slot rather than one of each.
`get_slots_since_genesis` now goes through `Store::current_slot` instead of
keeping a second copy of the arithmetic. It had been reading `genesis_time`
off the store and `seconds_per_slot` off its `config` parameter, and those are
not the same value: `init_beacon` overwrites `genesis_time` from the anchor
state, so a caller holding a config with a placeholder there got a split
clock.
Separately, a data directory now records which SSZ preset the build that wrote
it used, beside the chain tag and for the same reason: a raw byte, readable
before anything else in the directory is decoded, by a build that would decode
the rest into the wrong shape. `db_version` cannot stand in for it, because
both presets write the same layout at the same version while bounding every
SSZ container differently; a `preset-minimal` build opening a mainnet
directory passed both existing gates and then misread it. `from_db_state`
refuses a mismatch, and refuses a directory with no preset recorded, since the
whole point of the row is that nothing else in the directory says which shapes
it holds.
* refactor(storage): one clock row in milliseconds, every other grid derived
The previous commit gave the two chains one `Metadata["time"]` in seconds and
handed lean a second row for its intervals, because four of every five
interval boundaries fall between two whole seconds and a seconds row cannot
name them. Two rows is the wrong answer to that: they are the same clock, and
anything that must be kept in step can fall out of step.
The row is milliseconds instead, which is fine enough for the finest grid
either chain schedules on, and the intervals row is gone. `intervals_since_
genesis` now derives from the row rather than holding its own copy of it, as
`current_slot` already did, and both go through one `ms_since_genesis` that
owns the genesis subtraction and its saturation. Nothing stores a derived
reading, so no two readings can disagree.
`on_tick` walks that one row forward an interval at a time, which is what it
was already doing to the intervals row, and leaves it on the boundary it
processed rather than on the tick's own timestamp: the leftover milliseconds
carry no duty that has run. The forward-only guard the previous commit needed
for the second row goes away with it, since the only backwards write left is
the deliberate rewind that replays a slot.
The beacon specification denominates its `Store.time` in seconds, so
`on_tick_per_slot` and `get_forkchoice_store` multiply on the way in and the
fixture harness divides on the way out. That is three call sites against a
second stored unit, and it is why the accessor is `time_ms` and not `time`:
the two quantities are one unmarked factor of a thousand apart, which is the
confusion this whole change exists to remove.
`is_proposing_on_time` and `on_block`'s timeliness check get more precise for
free. Both were computing a whole-second difference and multiplying it back up
to compare against a basis-point deadline within the slot; both now read
`ms_since_genesis` directly. The fixtures drive those through whole seconds,
so the suites see no change.
* feat(types): read a beacon state's slot without decoding it * feat(types): answer the genesis identity reads for lean states too Both chains need to answer "is this state ours?" with the pair (genesis_time, genesis_validators_root). Beacon stores the root as a field; lean's validator registry never mutates after genesis, so the root it carries at any slot equals the root it had at genesis, which lets one comparison recognize either chain's own state. The writes stay beacon-only: genesis construction sets them, and lean has no such field to hand out a &mut to. * feat(types): commit a lean genesis config's registry to a root * refactor(types): identify a genesis by its registry root, on both chains verify_state_genesis compared a lean State's validator registry against the genesis config field by field: count, sequential indices, and both pubkeys per validator. Beacon cannot use that check as-is, since a mainnet state carries around a million validators unrelated to the genesis registry and identifies its genesis by a stored genesis_validators_root instead. Re-type the function to take a BeaconState and compare genesis_validators_root, which the lean variant now derives by merkleizing the registry it carries (valid because that registry never mutates). The root subsumes the four field-by-field checks it replaces: any difference in count, an index, or a pubkey changes the root. This lets both chains, and both entry points (resume from disk, checkpoint sync), share one check. * feat(storage): read the finalized state root on either chain * refactor(storage): load a data directory without judging whose it is Store::from_db_state took a lean GenesisConfig to verify the persisted chain matched it, which was the only thing tying the function to the lean chain. Judging identity needs the configured network, which the storage crate has no business knowing; loading a directory does not. Split them: from_db_state now only loads whatever chain a backend holds and hands back its Store, and the two callers in main (resuming on disk, and the checkpoint-sync path once it lands) check Store::chain() and run the genesis check themselves, with their own wording for the two distinct ways startup can be asked to touch a foreign chain. * fix(beacon): pair the anchor on its header root, not its state root A checkpoint-synced anchor's state sits at its finalized epoch's boundary slot; when that slot was empty, the state has advanced past its own latest_block_header and the block's state_root no longer equals the state's root, so the specification's own assertion cannot hold. Validate against the header root instead, substituting the state's own root for the header's placeholder zero, the same deviation Lighthouse makes in weak_subjectivity_state. * docs(storage): name only the from_db_state caller that exists The doc comment claimed `fetch_initial_state` and `fetch_initial_beacon_state` were both callers discharging the genesis check. Only the first exists; the beacon one arrives with the beacon checkpoint-sync path. Stating it in the present tense made the claim false today and unverifiable by grep. * refactor(cli): fetch the lean anchor state-first, as beacon must * feat(cli): fetch and verify a beacon checkpoint anchor Adds the beacon counterpart to lean's checkpoint sync path, against a standard Beacon API server rather than /lean/v0/...: fetch the finalized state, decode it at the fork its own slot names (SSZ carries no type tag), fetch the block at the state's latest_block_header.slot, and verify the pair on the header root rather than block.state_root == hash_tree_root(state) -- the finalized-state endpoint resolves to the epoch boundary slot, which may be empty, in which case the state has already advanced past its own header. get_forkchoice_store makes the same deviation for the same reason. Not wired into run_node yet, so every new item is unreachable from main and carries #[allow(dead_code)] until the anchor-and-follow task wires it in. * refactor(bin): open the data directory once, for either chain Both chains will keep a RocksDB store, so lift the data-directory resolution, creation and RocksDB open above the network match instead of duplicating it in the lean arm. The checkpoint-URL cleaning moves with it, since fetch_initial_state (still lean-only) reads both bindings. * feat(beacon): anchor the node from a checkpoint, and resume from disk Wire fetch_initial_beacon_state into run_node's mainnet arm, replacing the placeholder empty in-memory store: a beacon follower now resumes a resumable data directory, checkpoint-syncs against a Beacon API when there is none, and aborts rather than parking at slot 0 when neither is available, since mainnet's built-in genesis is not a legitimate anchor for a node that imports nothing. This makes the beacon checkpoint-sync path (fetch_beacon_anchor_with_retry and friends) reachable from main, so the allow(dead_code) attributes that fenced it off come out too. * refactor(beacon): assert the anchored head rather than substituting a slot `beacon_head` is provably `Some` where the resume path reads it: `init_beacon` writes the head in the same batch as the finalized checkpoint, and `finalized_state_root` has already succeeded by then. Substituting slot 0 for an impossible case would read as maximally stale and force a re-sync of a directory that should have resumed. * refactor(cli): keep the beacon decode error, and cover the anchor's checks BeaconDecode(&'static str) discarded the diagnostic on every beacon container decode failure. Split it into a slot-read failure and a fork-resolved container failure for the state, plus one for the block that keeps the gossip decoder's own DecodeError, so an operator gets the byte lengths and resolved fork a real failure needs. verify_beacon_checkpoint_state's four other branches, and the anchor's header-root pairing and fork checks, had no test driving them to fail. Extracted the pairing/fork checks out of fetch_beacon_anchor into a pure verify_beacon_anchor_pairing so they're reachable without an HTTP server, and added a test per branch, building anchor pairs off the built-in mainnet genesis fixture. * docs: describe beacon checkpoint sync and the shared genesis check * refactor(storage): follow the base's shared checkpoint keys This branch was written against an earlier feat/unified-storage, which was then rewritten. The base now seeds a beacon directory's `head`, `latest_justified` and `latest_finalized` from its anchor, the same rows `init_store` seeds on a lean directory, so `update_checkpoints` is one writer for both chains. It also derives `beacon_head` from `head` plus the head block's own entry rather than storing a `slot || root` row that could drift, and it drops the import watermark. That supersedes this branch's own version of the same idea, so that commit is gone and what remains follows the base: - `finalized_state_root` loses its chain match. Both chains keep the finalized checkpoint in `latest_finalized`, and the epoch-to-slot conversion the beacon accessors apply touches only the slot, never the root, so one read answers for either chain. - The `init_beacon` call sites take the base's `anchor_block_root` and a slot-denominated `Checkpoint`, via `beacon_checkpoint_as_stored`. - `data_storage.md` documents the shared rows and the two keys that really are beacon's alone, and says why no head row is stored.
* feat(p2p): serve beacon blocks by range and by root
The beacon follower registered no block protocol, so a peer asking for one
got a stream-negotiation refusal. Register `beacon_blocks_by_range/2` and
`beacon_blocks_by_root/2`, answer both off the store, and add the request
side for the anchor-to-head fetch to call.
Both chains answer a block request as one chunk per block, so the loop, its
per-chunk metrics and its skip-on-error-code rule move out of
`lean::encoding` into `req_resp::encoding` and each chain now supplies only
what differs: its `ChunkLimits` (whether a chunk carries a fork digest, and
how many chunks an answer may run to), and how a chunk body becomes a block.
`write_success_chunk` grows a `context` parameter for the same reason. That
loop was also unbounded on both chains, reading until the peer closed, so it
now stops at the protocol's own request ceiling.
Two details of these two protocols are not guessable from the response they
produce, and both are wire-visible:
- The range request is still three fields. `step` is deprecated and must be
1, but altair says the v2 request is unchanged from phase0's and lighthouse
pins the protocol's request length to `min == max == 24 bytes`, so a
two-field body is refused before it is ever decoded. A body naming any
other step is refused as malformed: the spec says a requester MUST set it
to 1, and honouring the phase0 leniency would mean carrying a dead field
through every layer above the codec.
- The root request is a bare SSZ list, not a container holding one. The spec
says this body is an SSZ *field* where the range body is an SSZ
*container*, and a container prefixes its one variable-length field with a
four-byte offset.
Neither difference reaches above the codec. What a peer is asking for is the
same question on either chain, so `req_resp::messages` owns one
`BlocksByRangeRequest` and one `BlocksByRoot` variant for both wires and each
chain's encoder converts to its own container: lean's has no `step` and fills
it with 1 on the way in, beacon's carries all three fields. The shared struct
is deliberately not SSZ-derived, so it cannot be written to either wire by
accident. The root list needs no conversion at all, since `Root` *is* `H256`
and both lists are `SszList<H256, 1024>`.
`step` is carried as far as the handler rather than checked at decode, so a
peer that sets it wrong gets `INVALID_REQUEST` and learns which rule it broke;
refusing in the codec would close the stream with nothing on it.
The answer is shared the same way. `ResponsePayload::Blocks` is one variant for
all four block protocols: `SignedBeaconBlock` already carries a `Lean` variant,
and both stores hand blocks back in that type, so the serving path no longer
converts at all. The narrowing to lean's `SignedBlock` happens once, where a
fetched block is handed to the chain actor, and lean's block writer refuses to
put a block of any other fork on its wire.
Neither enum records which chain a message came from. A node speaks one wire
for its whole life, so `P2PServer::wire` already answers that, and the dispatch
reads `Wire::is_beacon`.
Every successful chunk carries a four-byte `ForkDigest` context, and it is
the digest of that block's own epoch rather than the one this node runs on,
so serving history needs `genesis_validators_root`: `BeaconWire` and the
codec both carry it now. The codec is no longer `Default`, because a
contextless one would negotiate these protocols and then fail every chunk;
`build_swarm` decides the context in the same match that decides the
protocol set.
Reading a chunk goes the other way round. The fork comes from the slot inside
the payload, through the decode path gossip already uses, and the context
bytes are then checked against the digest that slot implies. That is a
stronger test than using them as the decoder key: it catches a peer whose
`genesis_validators_root` or fork schedule is not ours, which is what the
digest exists to say and is not otherwise visible until a signature fails.
Serving is live; asking is not driven from anywhere yet. This node has no
beacon `BlockChain` actor to import an answer into, so a fetch loop would
spend a peer's bandwidth on blocks it then drops. The two request functions
are exported for the anchor-to-head fetch to call. A node whose data
directory is not a beacon one refuses both protocols with
`RESOURCE_UNAVAILABLE`, the spec's own code for a peer unable to reply to
block requests.
One gap this exposes rather than introduces: `build_status` still advertises
zeroes for every checkpoint. That was the honest answer for a follower holding
no chain, and is no longer one now that the anchor gives it a store to serve
from. Peers pick who to ask for blocks by reading `Status`, so that
understatement, not any limit in the serving path, is what will keep these two
protocols quiet. Deriving `Status` from the store wants the same work that
gives this node an importer.
* feat(beacon): advertise the store this node already serves from
`beacon_blocks_by_{range,root}/2` answer from the checkpoint-anchored
store, but `build_status` reported zeroes for every checkpoint. That was
the honest answer for a follower holding no chain and stopped being one
once there was a store behind those handlers. Peers pick who to ask for
blocks by reading `Status`, so the understatement, not any limit in the
serving path, is what kept that path quiet.
Derive it: head from `Store::beacon_head`, the finalized checkpoint from
`Store::beacon_finalized_checkpoint`, and `earliest_available_slot` from
`Store::latest_finalized`'s own slot rather than
`finalized_epoch * SLOTS_PER_EPOCH`. This node anchors at a checkpoint
and keeps only the unfinalized window above it, so the epoch's first
slot is not necessarily one it holds, while the anchor checkpoint is
exactly the oldest block the directory can still produce. Naming that
slot tells a peer not to ask for anything older; naming zero would claim
genesis is in reach.
`Store::beacon_head` answers `None` in one window, between `init_beacon`
seeding `KEY_HEAD` with the anchor root and fork choice inserting the
block that root names. Every field but the fork digest falls back to
zero there, which stays honest: lighthouse's relevance check exempts a
zero `finalized_root` from its finalized-root comparison, reading it as
"this peer is syncing" rather than as a conflicting chain, so the
connection survives instead of earning an `IrrelevantPeer` disconnect.
`handle_status_request` stops going through `beacon_wire_or_refuse`,
which returns a `&BeaconWire` borrowed from the whole `&mut P2PServer`
and so would make the `&server.store` this now needs a conflicting
borrow. Reading `server.wire.beacon()` directly borrows that one field.
* refactor(p2p): pick the by-root protocol from the wire
`fetch_block_from_peer` asked for a missing block through
`Request::BlocksByRoot` on `BLOCKS_BY_ROOT_PROTOCOL_V1` unconditionally.
`Handler<FetchBlock>` calls it without knowing which chain is running,
so on a beacon node that puts a lean-framed request on beacon streams.
Peer selection stays where it is, since which peers already failed to
answer for a root has nothing to do with which wire is speaking. Only
the send branches, and the beacon arm goes through
`request_beacon_block_by_root`, which was already written and exported
with no in-tree caller. Its `outbound_requests` bookkeeping is its own,
so the lean arm keeps recording that entry itself and the
`pending_root_requests` entry that dedupes a repeated fetch is shared
below both arms.
Nothing exercises the beacon arm on this branch: reaching it needs a
chain actor asking for a missing parent, and this one spawns none. It is
here so that the request cannot go out mis-framed once one does.
* feat(beacon): sync the block range from the anchor to a peer's head
`Status` was recorded and nothing was driven off it, so this node asked
no peer for the range between its anchor and their head. The two beacon
request senders existed and were exported with no in-tree caller.
A peer's Status now starts or extends a range session, reusing lean's
`RangeSyncState` wholesale: same merge-or-create, same one-batch-in-
flight rule, same per-peer failure elimination. What differs is only
where the batch is sent and what "behind" is measured against.
Sent through `request_beacon_blocks_by_range`, so the beacon protocol id
applies and `MAX_REQUEST_BLOCKS_DENEB` is the real per-request ceiling
rather than the larger `MAX_REQUEST_BLOCKS` that `next_batch` plans
against. That sender records its own `outbound_requests` entry with
whatever it actually sent, so `complete_batch` still advances by the true
request span when a batch was clamped.
Measured against a new `beacon_fetched_through` watermark, not the
store's head. Nothing here imports, so the store's head never moves and
a sync driven off it would re-request the same range forever; the
watermark is seeded from the anchor at spawn and advances on arrival.
The field's doc comment carries the rest of the reasoning, which holds
once an importer does exist: delivery is a message and import is work,
so a head-driven sync trails a delivered batch by the whole actor
mailbox.
`handle_beacon_blocks_response` grows the session bookkeeping to match:
an empty answer fails the peer out of the session (or, for a by-root
request, goes through `handle_fetch_failure` like lean's), a successful
one completes the batch and asks for the next. An exhausted session
clears back to `None`, which is what keeps the resync path usable, since
both status handlers only ever merge into an existing session and never
replace one that is stuck. That predicate is now `range_session_exhausted`,
shared with the lean handler rather than spelled out twice.
Blocks are still counted and dropped, exactly as gossiped beacon blocks
are: there is no actor to import them on this branch. The range and root
checks run anyway, since they are what makes an answer trustworthy.
* docs(beacon): describe the wire that now asks, and advertises honestly
Both notes said the request side had no in-tree caller and that
`build_status` reported zeroes. Neither is true any more: a peer's Status
starts a range session, `fetch_block_from_peer` picks its protocol from
the wire, and Status is derived from the anchored store.
What replaces them keeps the limitation that is still real and was
previously implied by "no caller": the range path runs while nothing
imports, so its blocks are checked and dropped. Worth saying outright,
since it is bandwidth spent on discarded data rather than a path that
merely sits idle.
* refactor(p2p): answer a by-root response with one handler for both wires
The two by-root paths had drifted, and only the beacon one had drifted
into bugs. An empty answer re-inserted its `outbound_requests` entry for
a failure event that never comes again, so the entry stayed forever. An
answer carrying blocks under other roots fell through with no
`handle_fetch_failure` at all, leaving the root in `pending_root_requests`
where it deduplicates every later fetch of that block for the life of the
process.
Both follow from treating an empty answer as a case of its own. It is not
one: a request carries exactly one root, so `find` over an empty response
is already the no-match case, and collapsing the two leaves a single
handler that serves either wire. The import at the end is where the chains
genuinely part, and that split is already made for us, since `blockchain`
is `None` on a beacon node.
The dispatch now reads the request kind first and the wire second, which
is what gives the root arm one callee. `handle_beacon_blocks_response`
loses its root half and becomes `handle_beacon_blocks_by_range_response`,
which is also what lean's side has always looked like.
* refactor(p2p): read a canonical slot range through one function
`canonical_beacon_blocks_by_range` was written as the counterpart of
`canonical_blocks_by_range`, but there is nothing left for a counterpart
to do: both return `SignedBeaconBlock` and both call the same store
method, which dispatches on the chain itself. The only difference was a
`count == 0` early return that `checked_sub(1)` already covers, since a
count of zero has no last offset to add.
The surviving name is the bare one, which is how this module already
spells a handler that serves both wires.
* docs(p2p): give the `Request` enum back its doc comment
`BlocksByRangeRequest` was inserted between the enum's doc block and the
enum, with no blank line, so rustdoc attached the whole "Every request
either chain can send" block to the struct and `Request` documented
nothing at all. The block moves back down to the item it describes; the
struct keeps the paragraph that was always its own.
* docs(p2p): correct the rustdoc this branch left behind
`cargo doc --document-private-items` reported two links that no longer
resolve, and four passages had gone stale under the same renames:
- `Codec::for_wire` became `Codec::lean()` / `Codec::beacon()`.
- `crate::lean::messages::BlocksByRangeRequest` became
`LeanBlocksByRangeRequest`, and the shared request it was standing in
for is `req_resp`'s own struct, which belongs to neither wire. Said in
three places, all three now say it.
- The `pub use` of the two senders still claimed nothing in the crate
calls them. `fetch_block_from_peer` and `request_next_beacon_range_batch`
both do; what is still true is that the answer stops here, with no
importer to take it.
- `decode_single_chunk` still claimed every registered beacon protocol is
single-chunk, which the two block protocols are not. Its link to the
multi-chunk shape did not resolve either, which predates this branch.
`DISCOVERY_DIAL_INTERVAL` in `discovery/dial.rs` is left alone: it is the
one unresolved link older than this branch and outside its diff.
* fix(p2p): log a planned beacon batch as planned, not as sent
`request_next_beacon_range_batch` traced the batch size `next_batch`
planned, which is bounded by `MAX_REQUEST_BLOCKS`, under the name `count`,
while the request actually sent is clamped to `MAX_REQUEST_BLOCKS_DENEB`.
Reading the log, a clamped batch looked like a request for several times
what went out.
The field is now `planned` and the message says the batch is being
planned. What went on the wire keeps being traced by
`request_beacon_blocks_by_range`, which is where the clamp lives and so
the only place that can report it without a second copy of the ceiling.
* test(p2p): drop the range-session test that tested itself
The second half of `a_range_session_exhausted_by_completing_its_batch_
clears_to_none` built its own `Option`, re-implemented the handlers'
clear-to-`None` inline, and asserted that its own two lines had done what
they say. No production code ran in it, so it could not fail for any
reason worth knowing about.
Its one real assertion, that a session whose whole range arrives in one
batch is exhausted, is the `current_range.is_empty()` branch of
`range_session_exhausted`, which no other test covered. It moves to the
test that already covers the other branch, whose name now says which
branches it checks rather than only the case it opens with.
* chore: simplify doc
* feat(blockchain): drive the beacon chain from BlockChainServer The actor was the last lean-only layer in a stack that has already unified types, storage, the state transition, fork choice, p2p dispatch and startup: ChainSetup.chain was None on mainnet, so run_node spawned nothing and gossip blocks were decoded only to be dropped. Nothing is wired up here. Feeding the actor beacon gossip and spawning it from run_node are later slices, so this changes no binary behaviour and the beacon arm is exercised by tests rather than by a running node. Blocks now cross the actor boundary as SignedBeaconBlock. Its Lean variant already exists and its message accessors already answer for it, so widening P2PToBlockChain::new_block and Store::insert_pending_block makes the whole pending-parent cascade chain-generic instead of duplicated, and removes two expect_lean assertions that held only because of a chain tag elsewhere. ChainDuties moves the seven lean-only fields behind a Lean variant, so they are unreachable on the beacon arm by construction rather than sitting inert there. The store, the pending maps, the sync-status tracker and the event bus stay shared. An early beacon block is deferred rather than dropped. Beacon's on_block requires a block's slot to be in the past, and the specification's own instruction above that assertion is that such a block's consideration "must be delayed until they are in the past". One dropped early block froze the live mainnet follower permanently on 2026-09-03: it arrived 273ms before its slot, and every later block was then an orphan of a root the node would never obtain. The window is wider than any clock-disparity tolerance, since the beacon store clock advances only in the tick handler and a multi-second import blocks that same thread, so the rule is really "before the first tick at or after the slot boundary". Deferral is bounded by BEACON_DEFERRAL_HORIZON_SLOTS: it persists the block, so deferring on "is in the future" alone would let one peer spend this node's disk on slots years away. The lean arm keeps its existing whole-slot margin and is not touched. It cannot be shared either way: beacon carries a specification assertion that the fork-choice fixture suite tests. Beacon ticks once per slot boundary. on_tick_per_slot resets proposer boost at a slot boundary and pulls up unrealized checkpoints at an epoch boundary, and does nothing between them; Config's attestation_due_bps and its siblings are validator duty deadlines that a follower never reads. Chain events needed fixing to get here. diff_and_emit and checkpoint_state_root both read through Store::get_block_header, which decodes a lean BlockHeader and so panics on a beacon directory, where that table holds the whole signed block. Both wanted the same two fields, so Store::block_slot_and_state_root answers for either chain in one read, keeping a slot and a state root from ever being paired across two blocks. Known gaps, all p2p-side and all deliberate: there is no beacon by-root fetch, so request_missing_block is a logged no-op on beacon and a block with a missing parent pends indefinitely. This arm can follow a contiguous run and nothing else until that lands. * feat(beacon): follow mainnet, from the anchor to the tip The wire and the chain actor existed and did not touch each other. The mainnet arm spawned no actor, gossip blocks were decoded and dropped, fetched blocks were validated and dropped, both request senders were exported and called from nowhere, and Status advertised zeroes so peers had no reason to ask us for anything. Connect them: - `run_node` spawns a chain actor on both arms. `ChainSetup.chain` becomes a `ChainActor` enum rather than a lean-shaped Option, so the InitP2P/InitBlockChain wiring and `wait_for_shutdown` now cover the beacon follower too. - Beacon gossip blocks go to `new_block` with `BlockSource::Gossip`. Aggregates still do not: that topic carries ~1000 per slot at ~30ms each in `on_attestation`, more work per slot than a slot lasts. - Fetched blocks go to `new_block` with `BlockSource::Sync`, keeping every per-block range and root check. - A peer's Status starts range sync, keyed off a new `beacon_fetched_through` watermark rather than the store's head. The head lags a delivered batch by the whole actor mailbox, which is what once cost 11,213 blocks off the wire to import 100. - `fetch_block_from_peer` picks its protocol from the wire, so `BlockChainToP2P::fetch_block` stays one chain-agnostic method and the actor's `request_missing_block` no longer no-ops on beacon. - `build_status` reads the store: head, finalized checkpoint, and an `earliest_available_slot` of the anchor rather than 0, since this node cannot serve genesis. Two of the three defects that froze the 2026-09-03 follower are fixed here. A by-root fetch now arms its own deadline, `RootFetchTimeout`, because `pending_root_requests` otherwise clears only on a libp2p outcome and a request drawing none locks that root for the life of the process. A finished range session now clears back to None instead of disabling the resync path forever. Verified against mainnet, and the wire half works end to end: anchored at slot 15171200 (fulu, 2,361,891 validators), Status drove range sync, 70 blocks came back on beacon_blocks_by_range_v2 and 2 on beacon_blocks_by_root_v2, gossip blocks reached the actor and pended on their missing parent, which then asked for it. It does not catch up yet, for a reason outside this change: one import at that validator count ran over ten minutes at 100% CPU, and a stack sample puts it in `process_attestation` calling `get_total_active_balance` and `get_active_validator_indices`, both of which walk the whole registry per attestation. Because the actor is single-threaded, that also stops its ticks, so `store.time()` freezes and head recomputation never runs. Making those accessors cheap is its own change. * fix(beacon): feed a block's own attestations to fork choice The follower imported blocks and computed nothing from them. Measured on ethlambda-4 against mainnet: a contiguous chain from slot 15171489 to 15171523 landed off the anchor at 15171488, while head, justified and finalized all stayed pinned at the anchor and only the wall-clock slot advanced. `fork_choice::on_block` does not iterate a block's attestations, since the specification leaves that to the caller, and the actor did not either. So the latest-messages map was permanently empty, every block weighed nothing, and `get_head` never descended past the justified root. Gossip aggregates are deliberately not forwarded, that topic being ~1000 aggregates per slot at ~30ms each, so block bodies are the only vote source there is meant to be. This makes that true. `on_block_attestation` takes the carrying block's post-state as the committee source rather than each attestation's target checkpoint state. The two name the same committees, and asking the checkpoint instead is the expensive way to reach a verdict this node already has: `process_attestation` ran the same `is_valid_indexed_attestation` on the way to producing the block, and once a target's post-state falls out of the recency cache, rebuilding it replays every block since the last pinned boundary. On the earlier follower that was one import at 78.9s against a 3.6s steady state. `on_attestation` is untouched, so the spec-conformance fixtures keep exercising the verifying path. A body item that cannot be evaluated is not an import failure. The block is in the store whatever the loop returns, and returning `Err` would make the caller stop the pending-block cascade, leaving held descendants stuck behind a block that did import. A checkpoint-synced follower reaches this legitimately: an attestation may name a target up to `SLOTS_PER_EPOCH` slots back, and for the first epochs after the anchor that target is below it and was never fetched, so `validate_on_attestation` rejects it. Each call therefore logs at trace and continues. Ported from `feat/mainnet-network`, without that branch's `fork_choice::block_state` helper or its block-root-returning `on_block`: the actor already holds the root, and the post-state was written under it by the import a moment earlier, so `Store::get_state` is a direct hit. * refactor(beacon): hold an early block in a timer, not in a map A beacon block that arrives before its own slot starts is held rather than dropped, but the hold cost the follower a map and the DB a write: the root went into `BeaconFollower::deferred_blocks`, the block into the pending-block table, and every tick ran a range query to find what had become replayable. The actor can wait on its own. `process_or_pend_block` now hands an early block back to `run_import_cascade`, which re-delivers it to this same actor with `send_after`, timed to the start of the block's own slot. The block rides in the message, so nothing is written for it and nothing is left to reconcile if the process stops before its slot arrives. `BEACON_DEFERRAL_HORIZON_SLOTS` still bounds what one peer can make this node hold, in memory now rather than on disk. The re-delivery handler runs the slot's tick before importing. Its timer is set off the wall clock, but every slot comparison the import makes reads the store clock, which only the tick advances, and that tick is armed for the same instant. `on_beacon_tick` is idempotent through its own store-clock guard, so running it here settles the ordering rather than betting on which message the mailbox delivers first. `BeaconFollower` held nothing else, so `ChainDuties::Beacon` becomes a unit variant and the beacon tick loses its replay pass. The unit tests that asserted on the map's contents go with it. * refactor(p2p): drop the by-root fetch deadline Every by-root fetch armed a `RootFetchTimeout` self-message as a backstop for a request that drew neither a response nor an `OutboundFailure`, and so left its `pending_root_requests` entry stranded for the life of the process. Nothing is left for it to catch. The response paths that ended an attempt without retiring the root were fixed in #608, and a request that draws no reply at all is already bounded by the request-response behaviour's own 10s `request_timeout`, which arrives as `OutboundFailure::Timeout` and goes through `handle_fetch_failure` like any other failure. What remains is a second mechanism over the same entry, and a `ctx` parameter that `fetch_block_from_peer` carried for nothing else. * refactor(beacon): re-deliver a held block as a NewBlock, not a message of its own Holding an early block until its slot needs no protocol message of its own. The actor already has one for "here is a block": `NewBlock` carries a `BlockSource`, and what a deferral changes is the source, not the question being asked. So `BlockSource` grows a `Deferred` variant, the one the p2p layer never sends, and `defer_early_block` arms its timer with `NewBlock` instead. `Handler<NewBlock>` turns its `if source == Gossip` into a match, so every source now has to be answered for: gossip keeps the `ChainEvent::BlockGossip` emission and the arrival histogram, sync keeps skipping both, and deferred skips them because the block's first arrival is what they already described. The tick that has to run before a deferred import, and the reason it does, move into that arm with it. * refactor(beacon): decide an early block at arrival, against the wall clock The hold for a block whose slot has not started was decided a whole slot wide and against the store clock, inside the import cascade. Both halves were larger than they needed to be. The window is now `MAXIMUM_GOSSIP_CLOCK_DISPARITY`, 500ms, the phase0 p2p-interface constant that already says how far apart two honest clocks may read. A block early by more than that is not early, it is wrong, and a whole slot of headroom was only ever a bound on how much memory one peer could make this node hold. The decision moves to `Handler<NewBlock>`, where it is one comparison of the block's own slot-start timestamp against the wall clock. Arrival is the only place a hold is still possible anyway: by the time the cascade has the block, its caller is a loop with nowhere to put it back, which is what the `Option<SignedBeaconBlock>` return and the `ctx` threaded through `on_block` and `run_import_cascade` were paying for. All three go back to their previous shapes. Reading the wall clock costs one thing the store clock gave for free. The old comparison also covered a tick that had not run yet, since an unticked slot simply deferred the block again; a wall-clock comparison would instead hand it to `on_block` and have the spec assertion reject it. So the admit path catches the store clock up first, and only when it still reads a slot below the block's, which keeps sync backfill out of it entirely. * refactor(blockchain): one tick pipeline for both chains `on_lean_tick` and `on_beacon_tick` were two functions because their duties differ, but a tick is the same four steps on either chain: decode the clock, advance the chain, run this chain's duties, refresh the gauges. Only the middle two actually differ, and both were carrying their own copy of the other two. `on_tick` is now that sequence, and each step answers for the chain it runs on. `begin_tick` decodes the wall clock onto the chain's grid and drops a tick the store clock has already covered, returning a `Tick` that every later step reads instead of deriving a slot of its own. `advance_chain` moves the store clock: `store::on_tick` on lean, which carries its own fork-choice work, and `fork_choice::on_tick` plus `get_head` on beacon, where nothing else records a head. `run_duties` is lean's interval grid, and returns immediately on a chain whose tick carries no interval. `observe_tick` refreshes the head, justified and finalized gauges, and lean's safe target. Only the chain advance stays inside the event-diff window. A lean duty can import a block, and `process_block` diffs and emits around its own import, so a window stretched over the duty phase would emit that block's head and finality moves a second time. That constraint is why beacon's `get_head` belongs to the advance rather than to the duties. Three orderings change, all of them for the better: lean's sync status and current-slot metric now read post-promote state rather than the state before its own store tick, beacon's sync status reads the head `get_head` just computed, and lean refreshes the justified and finalized gauges every tick as beacon already did, rather than only on import. The interval match, the aggregator snapshot and the XMSS key advance move verbatim. * refactor: simplify new block handler guard * refactor(blockchain): keep the tick the function it already was The previous commit unified the two ticks by inventing a shape for them: a `Tick` value and four phases to pass it through. One tick pipeline was the right call, but the shape was not the one this actor already had, and the interval grid is easier to follow written out in the order it happens than split across `begin_tick`, `advance_chain`, `run_duties` and `observe_tick`. So `on_tick` goes back to the function main has, and beacon is woven into it in place. Six edits carry it: the idempotency guard expresses this tick's position in whichever unit the store clock keeps (intervals on lean, Unix seconds on beacon); the empty-validator fail-fast asks first whether this is a lean store, since `head_state` panics on the other one; the sync-status and head-metric reads go through a `head_slot` accessor that decodes the right table; the aggregator flag goes through an `is_aggregator` accessor that answers false for a follower; the one `store::on_tick` line becomes a match, lean's call on one arm and beacon's `fork_choice::on_tick` plus `get_head` on the other; and the duty match is wrapped in a lean guard, arms untouched. The justified and finalized gauges stay in the tail for both chains. Beacon needs them there: its clock advance can pull up unrealized checkpoints at an epoch boundary, moving both without importing anything. Two properties of the code already here are why the rest needed no edit at all. `get_our_proposer` answers `None` on a follower, so the `is_proposer` line is correct unchanged, and a beacon tick fires on slot boundaries, so `interval` resolves to `BlockPublication`, whose arm is empty. The guard on the match is for the tick that lands late. * refactor(blockchain): give the interval duties their own function Weaving beacon into `on_tick` left the interval match nested inside a lean guard, indented a level away from where it reads best and sitting in the middle of a function that is otherwise a sequence of one-line steps. It is the one part of the tick that is a chain's duties rather than every chain's bookkeeping, so it moves out. `run_interval_duties` takes the interval, the slot and the aggregator flag the tick already resolved, and asks the lean question itself, so the call site stays the single line the match displaced. Its arms are unchanged and back at their original indentation. The safe-target metric stays in `on_tick`. It reports what `store::on_tick` did at interval 3 rather than anything a duty did, so it keeps its place after the call, with the chain guard the surrounding `if` used to give it. One thing improves in passing: the `let ChainDuties::Lean(lean) = .. else { return }` guards inside the arms now return from the duties function rather than from the whole tick, so a miss (unreachable, since the function checked already) can no longer skip the head, justified and finalized gauges or the key advance behind them. * docs(beacon): say that the follower now imports what it fetches The base branch's notes describe a wire that asks and advertises honestly but drops what it fetches, since it has no actor. Both are true there and neither is true here: fetched blocks reach the actor as `BlockSource::Sync`, and the by-root path is what resolves a gossiped block's missing parent rather than an unexercised guard. * refactor: simplify * perf(beacon): skip a block whose post-state the store already holds `fork_choice::on_block` does not short-circuit on a known root. It goes from cloning the parent state straight into `state_transition`, so every re-delivery of a block the store already has pays the whole import a second time. Measured on a mainnet follower 2026-09-08: 116 imports produced 79 distinct blocks, so 37 of them were duplicates, a third of the actor's import budget spent recomputing post-states it already held. The duplicate gaps were indistinguishable from first-pass gaps (6.1 to 7.7 seconds), which is how you tell a real re-import from a cheap no-op. Re-deliveries are ordinary, not pathological: a block reaches the actor from gossip, from a `BlocksByRange` response, and from the backward `BlocksByRoot` walk an orphan starts, and nothing upstream dedupes. Check for the post-state before anything else in `process_or_pend_block` and return, so a re-delivery costs a lookup. Its pending children are still collected: the root did import, whether or not this delivery is what imported it. Beacon-only, because lean's `store::on_block` has its own already-imported early return; the comment in `process_block` that claimed both did is corrected here too, since it is what made this look like a solved problem. * fix(beacon): recompute the head when a block arrives, not only on the tick `lean_head_slot` sat flat at the checkpoint-sync anchor for a follower's entire catch-up, while imports landed every few seconds. That is what "the head is not advancing" looked like on the dashboard, and it was not a reporting artifact: the head really was not moving. `fork_choice::on_block` computes no head, and `Store::beacon_head` only reads back whatever the last `fork_choice::get_head` recorded. So an import adds a block and a post-state and moves no head. The sole caller of `get_head` was the tick, which is one message per slot in the same mailbox as every arriving block; a follower catching up imports back-to-back and never drains that mailbox, so the tick did not run and the recorded head stayed at the anchor. `process_block` refreshed the gauge after every import, but from that same stale recorded value. Re-run fork choice at the end of `on_block`, the arrival boundary the store clock is already brought up to real time at, and republish the head there. Measured at ~96ms per head computation against imports costing seconds, and it is once per arrival rather than once per block in the cascade, matching how that boundary already treats the clock. The tick keeps a call of its own, which is still needed for a slot in which nothing arrived; both now go through one function. Publish `lean_current_slot` from the same place, since it had the same single writer: with both gauges only written by a starved tick, `lean_current_slot - lean_head_slot` was a difference between two stale numbers rather than the head lag every panel and alert reads it as. It comes from the wall clock, since the store clock only moves when the tick or an early arrival moves it; `process_block` already needed that reading for its head-recency gate, so the two now share `wall_clock_slot`. Observed on a mainnet follower on the previous two commits: 29 blocks imported to slot 15178365, head gauge still reading the anchor 15178336, current-slot gauge 6 slots behind wall clock, and zero blocks rejected. * Update crates/net/p2p/src/lib.rs * style: rustfmt
… per seat (#10) * perf(beacon): stop rescanning the validator registry per attester and per seat A mainnet follower on this branch could not import a single block. It sat at 100% CPU for over ten minutes per block and never finished one, which also froze the actor's ticks, so `store.time()` stopped advancing and head recomputation never ran. Four places asked an O(registry size) question inside a loop whose iteration count also grows with the registry or the committee: - `get_flag_index_deltas` called `get_base_reward` per eligible validator, and that helper's second factor is `get_base_reward_per_increment`, an unconditional registry scan. At 2.36M validators that is O(n squared), and it runs three times per epoch boundary. Measured at 2^15 validators: ~11.9s unhoisted against ~384us hoisted. - `process_attestation` did the same per attester. - `process_attestation` also called `get_committee_count_per_slot` and `get_beacon_committee` per named committee, two more scans each, for up to `MAX_COMMITTEES_PER_SLOT` committees per attestation. One `EpochCommittees` now shares the single scan their common `(state, epoch)` pair actually costs. - `process_attestation` recomputed `get_attesting_indices` for an attestation `get_indexed_attestation` had just resolved against the same unmutated state. - `process_sync_aggregate` resolved each of `SYNC_COMMITTEE_SIZE` seats with its own `state.validators().iter().position(...)`, roughly 1.2 billion 48-byte pubkey comparisons per block at mainnet scale. It now builds the seat map first and makes one pass over the registry, keyed pubkey to a list of seats because the committee is drawn with replacement. Every substitution is an equivalence, not a behaviour change, and both hoists keep the helper's own order of operations so the arithmetic stays bit-identical: these feed consensus rewards. The minimal-preset beacon spec suite (40009 cases) passes unchanged, which is what proves it. Measured on the mainnet follower, anchored around slot 15171200 at 2,361,891 validators: | build | per block | |------------------------|------------------| | before | no import in 600s| | the four hoists | ~12.4s | | plus the seat inversion | 8.11s median | 8.11s against a 12s slot is the difference between falling behind and catching up. `EpochCommittees` is ported from `feat/mainnet-network` without that branch's shuffle-permutation cache, which lives in a `shuffling.rs` change not taken here, so it shares the registry scan but still pays one shuffle per committee. Adding the cache is a drop-in change to its internals. * feat(beacon): measure what one fork-choice head computation costs Nothing timed `get_head`, so the only evidence that it was the follower's bottleneck came from attaching `perf` to a live node: 62% of the process's cycles sat in that call, over half of them inside `SipHash` on the block index's own keys, while imports ran at 14s against 12s slots. A histogram makes that visible from the dashboard instead. Buckets run to a whole mainnet slot because that is the question being asked: a head computation costing seconds is one the chain actor cannot afford between blocks. The guard is taken in `recompute_beacon_head`, ahead of the early return on failure, since a computation that fails still spent the time and the failing one was the expensive one. * perf(beacon): weigh the whole tree in one pass, not once per candidate `get_weight` is the specification's per-root definition: for one root it scans the whole validator registry and walks each voter's latest message back to that root's slot. `get_head` called it once per candidate child at every level of its descent, so a node with the justified checkpoint tens of slots behind head re-walked every vote tens of times to pick one head. At mainnet's 2.36M validators that dominated the follower outright. `perf` on the live node put 62% of all cycles in that descent, 54% of the total inside `SipHash` alone, hashing the block index's 32-byte keys once per voter per hop. Merkleizing the state twice per import, the previously suspected cost, was 17%. Imports took ~14s against 12s slots, so the follower lost ground every slot and could never converge. `compute_weights` produces every root's weight in one bottom-up fold, since a vote counts for a root exactly when the voted block descends from it: sum each vote at its own block, then fold each block's total into its parent, highest slot first so a child is complete before its parent reads it. Parent links always point at a strictly earlier slot, which makes that a valid topological order. One index lookup per voter over a map small enough to stay in cache, and no registry scan at all. Two behaviours worth naming. Membership is `is_active_validator` against the justified checkpoint's state rather than `get_active_validator_indices`, which allocates a two-million-entry vector to answer a membership question. And a vote for a block no longer indexed weighs nothing instead of raising: `promote_beacon_anchor` prunes below the oldest kept anchor, and a validator whose freshest vote is down there keeps it until it attests again. Such a vote cannot separate candidates above the justified checkpoint anyway, and the specification's version aborting the whole descent on it is what once pinned a live follower's head for an hour. `get_weight` stays exactly as the specification writes it, and is what the new pass is tested against, root by root, across a fork with the boost applied. The 151 beacon `fork_choice` fixture cases still pass. Ported from `feat/mainnet-network` (c09ad18c), where the same descent was found by the same method. * perf(beacon): hoist the base reward out of altair's and deneb's attestation loops `38e2945d` took `feat/mainnet-network`'s `2f3ef3e1` for electra only. Altair and deneb carry the same loop: `process_attestation` calls `get_base_reward` per attester per flag, and that helper's second factor is `get_base_reward_per_increment`, which reaches `get_total_active_balance`, scans the whole validator registry and allocates the active set on every call, with no cache anywhere beneath it. The read phase does not mutate `state`, so that factor is constant across the loop. It is hoisted once and `get_base_reward` inlined against it in the helper's own order of operations, so the arithmetic stays bit-identical: these feed consensus rewards. Nothing changes on the live mainnet follower, which checkpoint-syncs and so only ever transitions fulu blocks. This is for a node that replays altair or deneb history, and for keeping the three copies of this loop saying the same thing rather than leaving two of them as the version that could not finish a block at mainnet scale. Both presets pass unchanged: 5705 mainnet and 40009 minimal fixture cases. * perf(beacon): record a block's post-state root in the state, don't recompute it An import merkleizes a full state twice. `state_transition` hashes the post-state to check the block's committed `state_root`, and one slot later `process_slot` hashes that same state again to record it as the pre-state root. At mainnet's registry size each is over a second, so a third of an import goes on hashing bytes this node has already hashed and already checked. `process_slot` fills `latest_block_header.state_root` with exactly the value it just computed. So `on_block` fills it a slot early instead, from the block field `state_transition` accepted the block for matching, and the new `BeaconState::compute_state_root` takes it back out rather than merkleizing. The cached root is used only while the state is still inside that header's own slot, which is the shape a stored post-state has and the shape nothing the specification produces ever has, since the field is written on the way out of that slot. Every other state, fixture ones included, merkleizes as before. This is what lean has always done: `store::on_block` writes the root into the same field in the same position, and lean's `process_slots` reads it back the same way, minus the slot equality, which it does not need because it keeps no `state_roots` vector. Doing it in the state rather than beside it means the value survives a restart, needs no cache and no second lookup, and reaches `checkpoint_state` and `should_override_forkchoice_update` without either of them asking for it. `get_forkchoice_store` establishes the same field for the anchor, the one block that never goes through `state_transition`, by clearing it, merkleizing, and writing the result back rather than trusting whatever a checkpoint provider sent. `Store::init_store` normalizes a lean anchor the same way. The state stored under a block root now differs from the state that block commits to, by this field. Nothing reads it back expecting the committed shape; serving states over an API would be the change that makes it matter. Rather than assert the equivalence in a test of its own, the `sanity`, `finality`, `random` and `transition` runners now cache each block's state root for the next one, the way `on_block` does. Every multi-block case in both suites then drives the cached arm, and each one's `post` comparison is the assertion that the state comes out the same.
…ailability gate (#11) * feat(beacon): define the das-core custody settings once The five custody values are identical across configs/mainnet.yaml and configs/minimal.yaml, so they belong as constants in the types crate rather than Config fields: a Config field changes the SSZ encoding persisted under Metadata[KEY_CONFIG], forcing DB_VERSION up and refusing every existing lean and beacon data directory. The p2p crate's own CUSTODY_REQUIREMENT now re-exports the shared constant so the ENR cgc entry, MetaDataV3, subnet subscription, and the future availability check cannot drift apart. * feat(beacon): compute the custody groups a node id is assigned * test(beacon): run the das-core custody fixtures * docs(beacon): correct the sidecar serve window's direction The window is the range a node must serve, so a node may refuse below it, not within it. Also drops the forward-looking hedge from the multiple-of test now that compute_columns_for_custody_group exists. * feat(beacon): verify a sidecar's commitments against its block Adds verify_data_column_sidecar_inclusion_proof, the third and last of the fulu column-sidecar checks: it proves the commitments a sidecar carries are the ones its named block actually committed to, closing the gap where a peer could otherwise pair a valid column with any block header it liked and still pass the existing structural and KZG checks. * refactor(beacon): express the custody asserts the way the crate does * feat(storage): keep custodied data column sidecars The beacon follower needs its data column sidecars before a block that depends on them imports (for the availability check), after a restart (a resync shouldn't re-fetch what was already verified), and when peers ask for them by root or by range. Add a DataColumns table keyed slot-first, like BlockProof, so a by-range scan and a future pruner can both work in slot order and stop early. Nothing writes to it yet; the gossip path does in a later task. * test(beacon): report which custody group differs * feat(p2p): subscribe to the column subnets this node custodies ethlambda beacon advertised a custody group count in its ENR while subscribing to no data column subnet at all. Subscribe to exactly the subnets this node's own discv5 node id selects for custody, computed from the *sampling* size rather than the custody requirement, since a node samples more groups than it strictly custodies and the sampled columns need a source too. * docs(beacon): credit the check that actually catches a shifted field The subtree-index unit test only pins BLOB_KZG_COMMITMENTS_SUBTREE_INDEX against the fixture's stated generalized index; it never touches BeaconBlockBody, so a field shifting blob_kzg_commitments's real position would pass it unchanged. The merkle_proof fixtures' end-to-end check, which decodes a real body and drives it through verify_data_column_sidecar_inclusion_proof, is what actually catches that. Reword both comments to credit it correctly. Also: name KZG_COMMITMENTS_INCLUSION_PROOF_DEPTH instead of spelling out its leaf count in prose, and select merkle_proof's fulu cases by the case's own name field instead of an id() meant for failure messages plus a redundant fork check that would silently drop a future fork's cases from this suite. * docs(beacon): state the custody index precedent at its real granularity CustodyIndex's justification for staying local cited RowIndex's precedent but phrased it as staying beside one container, when RowIndex's own comment is single-struct scoped ("only MatrixEntry needs it"). CustodyIndex is used across several functions of a whole module, so reword the analogy to match: staying beside its only consumers, not beside a single container. * fix(storage): make the earliest custodied slot honest under concurrency put_data_column_sidecar's earliest-slot read and its conditional write are two separate backend calls with no isolation between them. Investigated whether a second writer can exist: the chain actor owns its Store outright and processes one message at a time (GenServer pattern), and every other Store clone (ethlambda-rpc, ethlambda-p2p) only reads it, confirmed by grep for mutating calls in both crates. A single writer is genuinely guaranteed today, so document that assumption precisely at the call site instead of adding a mutex the evidence doesn't call for, and correct the doc comment that used to cite "concurrent readers/writers" as the reason for `&self`, which implied a guarantee the code was never given. Also: name the actual reason data_column_sidecars_in_range scans per slot (StorageReadView has no range-iterator counterpart to delete_range) rather than only the multi-block-per-slot reason, which is separate and stays; add a test that stores several column indices across two sibling blocks at one slot and asserts a range query returns exactly the requested subset, since every existing test wrote and queried only column index 0; and drop the test_beacon_store wrapper in favor of the existing beacon_test_store(Arc::new(InMemoryBackend::new())) call pattern already used twelve other times in this module, rather than leaving two spellings of the same setup. * feat(beacon): verify and keep the column sidecars we custody Wires the fulu data-availability sampling gossip path end to end: the p2p actor decodes a data_column_sidecar_N message, runs the checks that need nothing but the sidecar and the store's own checkpoints (subnet match, structural validity, finality, seen-dedup), and hands what survives to the chain actor. The chain actor runs the checks that need state and fork choice (bounded slot, known and finalized-descendant parent, inclusion proof, KZG batch, proposer signature) and writes accepted sidecars to the DataColumns table. Adds a future-slot bound before the proposer check's process_slots call and a finalized-ancestor check alongside the existing finalized-slot check, both present in fork_choice::on_block's own import path but missing from the task's reference implementation; without the first, a header naming an arbitrarily large slot would drive process_slots one slot at a time on the actor's own thread. release_block_if_columns_complete is a no-op notification point for the data-availability gate a later task adds. * fix(p2p): dedupe column topics through the map and reuse the typed key error Two code-review fixes for 31ff52e2. BeaconTopics::new built `topics` in parallel with `column_topics` instead of deriving it, so a network where NUMBER_OF_CUSTODY_GROUPS and DATA_COLUMN_SIDECAR_SUBNET_COUNT diverge would double-subscribe a shared subnet; column_topics is now built first (as a BTreeMap, for a stable ascending order) and topics derives its column tail from it. node_id_from_secret_key returned a bare secp256k1::Error instead of the crate's existing DiscoveryError::NodeKey, so an invalid --node-key reported a different message depending on which of two code paths parsed it first. Also: routed main.rs's key-to-id derivation through a new beacon::beacon_node_id wrapper so that composition is unit-testable, an intra-doc link to a pub(crate) item, and two stale doc/log wordings. * feat(p2p): serve and request data column sidecars The node now custodies the columns its node id selects, so the two DataColumnSidecars protocols no longer need to lie at stream negotiation: by-root answers whichever named columns this node holds, skipping the rest, and by-range serves the custodied slot window, refusing RESOURCE_UNAVAILABLE below the earliest slot ever custodied. * fix(beacon): close the unbounded seen_data_columns growth and untested check Spec review on the sidecar gossip path (eedae86f) found the p2p future-slot filter was more than a nice-to-have: nothing checked a sidecar's slot magnitude before p2p inserted its (slot, proposer, index) tuple into seen_data_columns, so a single gossip message per fabricated far-future slot grew the set forever, since pruning only drops entries finality has actually passed and finality never reaches a fake slot. The chain actor's own future-slot guard cannot help: the entry is already in the set before that actor ever runs. Adds the p2p-side check, ahead of the dedup insert, using MAXIMUM_GOSSIP_CLOCK_DISPARITY moved to ethlambda_types::beacon::constants (milliseconds, matching every other value in that module, rather than the Duration it was as a blockchain-crate-private constant) since it is now a spec value two crates need. The blockchain crate re-derives its own Duration-typed constant from the shared number instead of defining it. Moves seen_data_columns pruning onto the discovery tick (every 1-5s, independent of peer churn) rather than relying solely on new beacon connections, which a stable, well-scored node might not open for a long time. Extracts the finalized-ancestor check into its own method so it can be unit-tested directly: the existing end-to-end sidecar test could not actually distinguish a correct implementation from one that always accepted, since the checks after it in the pipeline reject the same default-shaped test sidecar on their own. * feat(p2p): fetch the columns a held block is missing Adds the asking side of DataColumnsByRoot: fetch_data_columns on BlockChainToP2P, a Handler<FetchDataColumns> mirroring FetchBlock's peer selection and retry ladder, and a PendingColumnRequest/Columns pending-kind pair for the response routing and COLUMN_LOOKUP_MAX_DURATION bookkeeping. Every fetched sidecar is forwarded through new_data_column_sidecar, the same entry point a gossiped one takes, so the spec's "treat as gossip" requirement holds regardless of how a sidecar arrived. The consumer that decides a block is missing columns and calls this is the next task. * feat(beacon): hold a block until its custody columns are available Fulu blocks imported with fork_choice::DataAvailability::NotRequired unconditionally, so a node accepted a block whether or not the columns it sampled for it were actually present. Gate process_block on this node's custody columns: hold the block (persisted, invisible to fork choice) until every column arrives, ask peers for what's missing, and release it through the same import path a new block takes once the set is complete. Gated by --beacon-da-enforce, on by default. * test(p2p): cover the data-column future-slot check and stop duplicating its clock handle_beacon_data_column had no unit tests, so nothing would catch a regression of the future-slot check back to running after the dedup insert (the exact shape of the bug it fixed). Adds a test pinning the property that actually matters — a rejected far-future sidecar must never enter seen_data_columns — plus one confirming the disparity tolerance is neither zero nor inverted. Verified both by temporarily breaking the check (deleting it, then zeroing the tolerance) and watching the matching test fail before restoring it. Also lifts unix_now_ms into ethlambda_types::time, the same precedent this round already applied to MAXIMUM_GOSSIP_CLOCK_DISPARITY: the p2p and blockchain crates have no dependency on each other but both depend on types, and the function was a verbatim private copy in both. Named it out of constants.rs rather than into it, since that module is documented as holding spec-fixed values, not helpers. Minor: names the discovery-tick constants instead of spelling out their interval in prose, and drops the connection-time seen_data_columns prune, which the tick already makes redundant. * fix(storage): scope a data-column range answer to each slot's canonical root data_column_sidecars_in_range prefix-scanned by slot and filtered only by column index, with no root filter. Gossip import requires a known, finalized-descendant parent rather than a canonical one, so a live fork can leave both siblings' columns stored at one slot; Table::DataColumns is never pruned, so an orphaned sidecar sat there forever and leaked into every future by-range answer covering that slot, violating the spec's requirement that a response be consistent from a single chain. Fixes it by resolving each slot's canonical root through Table::BlockRoots before scanning, the same index get_signed_blocks_by_slot_range and the block-range handler already key off. Confirmed BlockRoots is populated on the beacon path too: BlockChainServer's tick calls fork_choice::get_head (crates/blockchain/src/lib.rs:1737), which writes through Store::update_checkpoints unconditionally on every call, on both chains. By-root needed no change, since the peer names the root directly. Two existing tests assumed the unfiltered behavior (one wrote sibling roots at a slot and asserted both leaked through); updated them and added a test pinning that only the canonical root's columns come back. Verified the new test by reverting to the unfiltered scan and watching it fail before restoring the fix. Also adds a round-trip test for write_data_column_sidecars_response into decode_data_column_sidecars_response (per-item context-digest derivation across a fork boundary, and abort on a genesis_validators_root mismatch), verified failing the same way with each check disabled in turn. Minor: fixes a stale function name in protocols.rs's own doc, and updates has_context's doc for the two column protocols that now set it. * fix(beacon): close the fetch-path verification gap and a dead retry bound A fetched data column sidecar skipped verify_data_column_sidecar: gossip runs it in the p2p actor before forwarding, but a DataColumnsByRoot answer went straight to new_data_column_sidecar, and the chain actor never ran it either. Not an acceptance bypass (c-kzg fails rather than returns false on length-mismatched input, and the inclusion proof pins the commitments), but a cost asymmetry: a peer answering a fetch could force a full KZG batch on garbage gossip rejects for free. Fixes it by running the check in on_gossip_data_column itself, the one entry point both paths already share, so it is a single choke point rather than a second copy on the fetch side. Gossip keeps its own copy where it is, since it still saves a mailbox hop. COLUMN_LOOKUP_MAX_DURATION could never fire: MAX_FETCH_RETRIES exhausts in roughly a hundred seconds (ten attempts, each bounded by the request-response layer's ten-second timeout, spaced by the doubling backoff), and the attempts check ran before the duration check, so the ladder always gave up first. The value came from Lighthouse without checking it against this codebase's timings. No path exists where wall time advances without either an attempt completing or the lookup resolving, so there is no way to make it reachable; deleted it and documented the retry ladder as the actual bound instead. The fetch handler's dedup dropped a second FetchDataColumns for a root already pending, which is only safe if a re-ask's columns are always a subset of what is in flight — an invariant nothing enforces, and one the availability gate (committed after this fetch path) may not honor as columns trickle in. Merges the requested columns into the pending entry instead: a column added this way is not in the request already on the wire, but rides the next retry of this lookup or a fresh fetch once this one resolves. Adds a test for the structural check now running as a shared choke point, verified failing with the check disabled. The dedup-merge fix has no dedicated test, matching how FetchBlock's parallel dedup logic is untested today: both live in an actor Handler that takes a live Context, which this crate has no harness for (see 64a6bce9). * fix(beacon): stop a held block's fan-out from livelocking the import cascade process_block returned a bare Ok(()) for a held block, so process_or_pend_block treated the hold exactly like a genuine import and called collect_pending_children on it. A child block naming the still-held block as parent re-enqueued it via the missing-parent ancestor walk, which re-held it and re-collected the same child, forever, on run_import_cascade's synchronous loop with no yield point. Give process_block an ImportOutcome so only a real import unblocks pending children; a hold now waits for release_block_if_columns_complete instead. Also: sweep blocks_awaiting_columns when finality advances, since nothing else evicted a withheld hold and a gossiped block's claims are not signature-checked before the gate can hold it; make the deneb/electra no-blob-pipeline warning observable per occurrence (fork, root, counter) instead of a single shared one-shot log; guard the vacuous-accept case of an empty custody set; and fix stale ChainActor docs left over from the previous change. * docs(beacon): describe the DAS wire honestly and watch the disk it grows Ten tasks landed custody-column subscription, storage and serving without updating beacon_wire.md or data_storage.md, which still described a node that subscribed to no column subnet and served nothing back. Bring both in line with the code, add the ninth table's metrics entries, and close the two operational gaps the docs surfaced along the way: the never-pruned DataColumns table had no dedicated growth signal an operator could watch before disk fills it, and a beacon follower without --node-key silently custodies a different column set on every restart with no warning next to the custody log that would make it obvious. The growth signal turns out to already exist: lean_table_bytes{table= "data_columns"} covers it, since DataColumns has been in ALL_TABLES since 865ec49a. No new metric needed, just the documentation saying so. * docs(beacon): name constants instead of quoting their values in the DAS wire section Follow-up to 364174b: name NUMBER_OF_COLUMNS, MAX_FETCH_RETRIES and INITIAL_BACKOFF_MS instead of restating their current values, matching house style for prose, and drop a confusing, redundant claim about sidecar re-announcement now that the new subsection already covers cross-seeding directly. * fix(beacon): stop the DAS startup warning and docs from denying what is now served Three strings still described this node as storing/serving no custody columns and as advertising a count wider than what it serves. Both are now false: columns are stored and served, and the advertised cgc is a floor this node's actual custody always meets or exceeds. Rewrite the wire_params startup warning, the ENR cgc doc, and build_metadata's doc to say what remains true (no attestation/sync-committee subnet subscription, no publishing) instead of contradicting the neighboring custody log line. * test(blockchain): cover the release path draining a parked child, and fix the doc it falsified The livelock fix's regression test proves the cascade terminates but never reaches collect_pending_children on release: its structurally-fake block fails real RANDAO verification, so the release attempt takes the Err arm. Add a test that seeds a released root's post-state directly (the same shape an independent import racing the release would leave) and asserts the parked child drains from both pending maps through process_or_pend_block's real "already in the store" branch. Reaching process_block's own Ok(ImportOutcome::Imported) arm from this path needs either a real BLS-signed fulu block (this crate has no signing helper, only blst-backed verification) or process_block's `_ if !is_new` shortcut, which is unreachable here since process_or_pend_block's guard already intercepts a known root before process_block ever runs. Both are documented in the new test rather than faked. Also update docs/beacon_wire.md's live-network transcript: it quoted the startup warning this branch's earlier commit rewrote, plus a now-stale caveat apologizing for wording that commit already fixed. * fix(p2p): answer DAS by-root through the chain-aware block accessor `handle_data_column_sidecars_by_root_request` recovered a block's slot from its root through `Store::get_block_header`, which decodes `Table::BlockHeaders` as lean's fixed-size `BlockHeader`. A beacon directory keeps the whole signed block in that row, so the decode failed inside SSZ and panicked the actor holding it. The eth-4 mainnet follower died that way: a peer asked for columns by root, the swarm adapter exited, and what was left was a process that still served metrics and imported nothing for fourteen hours. A peer's request must not be able to do that, and this handler only ever runs on a beacon store, so it was never going to be right. `block_entry` reads the same fields and decodes per chain, which is what every other caller spanning both chains already uses. The four lean-only accessors now say so themselves. Reaching one from a beacon path used to surface as an SSZ length mismatch that named neither the accessor nor the caller, which is most of why this took a log dive to find; it now names the accessor and, through `#[track_caller]`, the line that called it. * fix(beacon): stop the availability gate deadlocking on its own held blocks A follower running --beacon-da-enforce could not follow mainnet. One slot whose custody columns did not arrive in time stopped the chain permanently, at the tip as much as during backfill: head frozen, and 161 of 161 subsequent gossip sidecars rejected. The cycle is self-sustaining. A held block never reaches on_block, so it never writes a post-state; on_gossip_data_column resolves a sidecar's parent with get_state and dropped the sidecar outright when that returned None; so from the moment block S is held, every sidecar of every child of S fails that lookup and is discarded. The node cannot collect the next block's columns while it waits for this one's, and a column is gossiped once, in its own slot. The specification does not ask for that drop. Its rule is `[IGNORE] The sidecar's block's parent has been seen (MAY be queued for processing once the parent block is retrieved)`, distinct from `[REJECT] The sidecar's block's parent passes validation`. A held parent is neither: it was seen, and it did not fail. So park those sidecars against the parent root instead and replay them when it gains a post-state, bounded by MAX_SIDECARS_AWAITING_PARENT and swept by finality on the same schedule held blocks are. Parking happens ahead of the inclusion-proof and KZG checks, so a replay pays for them once. The reject reason `unknown_parent` goes with it: no path reaches it now, and while it existed it made the deadlock invisible, reading as a verdict on the sidecar when the fault was upstream of it. `awaiting_parent_full` takes its place in the label set, and lean_sidecars_awaiting_parent reports the depth. Second fix, same area: the gate applied to every fulu block at any age. Below the availability boundary no peer is obliged to answer for a column at all (`MAY respond with error code 3: ResourceUnavailable`), so gating there holds a block against data the network is entitled to have dropped. da_check_required_for_slot bounds it to the spec's own max(current_epoch - MIN_EPOCHS_FOR_DATA_COLUMN_SIDECARS_REQUESTS, FULU_FORK_EPOCH), mirroring lighthouse's da_check_required_for_epoch. That boundary is why a_childs_fan_out_does_not_livelock_a_held_parent now builds its store with fulu at genesis: its blocks sit at single-digit slots, which under mainnet's real schedule are thousands of epochs before fulu and so are not blocks the gate has any business holding. * fix(p2p): ask the peers that custody a column, and fetch columns by range Two halves of the same problem: a follower could not obtain a data column it had missed on gossip, so the availability gate had nothing to pass on. Peer selection ignored custody entirely. `fetch_data_columns_from_peer` chose with `pool.choose(&mut rand::thread_rng())`, and at mainnet's CUSTODY_REQUIREMENT a peer holds 8 of 128 columns, so a request for specific columns nearly always came back empty and was charged as a failed attempt. The information to do better is public and needs no handshake: the spec notes that "due to the deterministic custody functions, a node knows exactly what a peer should be able to respond to". So compute each peer's set from its node id and its advertised custody group count, and send each column to someone who holds it, splitting one lookup across several peers and preferring the one already carrying fewest of its columns. The count comes from `metadata/3`, now requested once per connection behind Status, and is seeded from the ENR `cgc` for a dialed peer. Both sources, with metadata authoritative, the way lighthouse splits them; `cgc` outside CUSTODY_REQUIREMENT..=NUMBER_OF_CUSTODY_GROUPS is discarded rather than clamped, since it describes no custody set the spec defines. A peer that has supplied neither is not assumed to custody anything and not assumed to custody everything: unknown means the by-root path still asks it at random, which is all it could ever do. The node id comes from the PeerId itself. libp2p stores a key of 42 bytes or fewer directly in the multihash, so a secp256k1 key reads back out and goes through the same keccak256(uncompressed) our own id does; the round trip is asserted against node_id_from_secret_key, because a divergence would aim every request at the wrong peers silently. Second half: backfill had no bulk path at all. `DataColumnsByRange` was registered inbound-only ("has no sender yet"), so the only source of a historical column was the by-root lookup a block triggers after it has already arrived and been held, one block at a time. The spec names the protocol for exactly this case: "DataColumnSidecarsByRange is primarily used to sync data columns that may have been missed on gossip and to sync within the MIN_EPOCHS_FOR_DATA_COLUMN_SIDECARS_REQUESTS window". Every beacon range batch now sends one alongside its BeaconBlocksByRange, over the same capped span, so the columns are normally stored before their blocks reach the gate. Nothing waits on it: a short, empty or refused answer costs nothing and is not retried, and the by-root path stays as the backstop. Splitting a lookup across peers needed the retry ladder taught what a round is. Each failure used to be its own attempt and schedule its own fan-out, so one unanswered lookup against eight custodians would have become eight retries, then sixty-four. `PendingColumnRequest::in_flight` counts the requests of a round and only the last to report back decides the attempt. That arithmetic is `retire_column_attempt`, split out from the handler so it can be tested without an actor context. * fix(beacon): keep asking for a held block's columns, and dial the peers that have them With the gate on, the follower reached the tip and then stopped on the first block whose sampling columns no connected peer custodied. Every peer answered `DataColumnsByRoot` with an empty list, the lookup spent its retry ladder in seconds, and the hold was left with nothing that would ever disturb it again: `hold_block_for_columns` asks once, and the only other thing that revisits a hold is a sidecar for that exact block arriving on gossip, which by definition was not coming. Head froze while the chain kept moving and the parked-sidecar queue filled behind it. Every peer connected at the time advertised the minimum custody_group_count, so a dozen of them held a small fraction of the columns between them, and two of the eight this node samples were not among them. Three changes, at the three points where that state can end: The chain actor re-drives every held block once a slot, releasing the ones whose columns have quietly completed and re-asking for what the rest still lack. Peers churn constantly, so the ask that fails now is worth repeating. The p2p dedup entry expires. It exists so a second ask for a root in flight merges rather than opening a parallel ladder, which is only correct while the lookup it defers to is alive; past STALE_COLUMN_LOOKUP the entry is replaced, so a round that never reported back cannot swallow the re-drive above. Discovery ranks candidates by the custody columns they would cover before the attestation subnets they would cover. A column nobody holds cannot be fetched at all and stops the chain; an uncovered subnet only narrows the view. The ordering also puts a supernode ahead of everything else while any column is uncovered, which is the fastest way out of that state. * fix(beacon): fetch a backlog's columns by range, not one block at a time A held block at the tip is answered by gossip within a second: this node subscribes to every subnet it samples, and the columns are usually on their way before the block is. A held block well behind the tip has no such path, and the blocks queued behind it are each reduced to their own by-root lookup. One round trip per block is slower than the chain produces blocks, so a follower that falls behind stays behind: measured on mainnet, 1 slot imported per 15s against a 12s slot, with the gap growing. A range ask is also far less sensitive to how the peer set happens to be made up. A peer answers only for the columns it custodies, and most of mainnet custodies the minimum four of a hundred and twenty-eight, so a by-root ask for one column is a lottery over the connected peers; one range answered by one supernode covers every column of every slot in the span. So the per-slot re-drive now also asks for the whole span the node is behind by, whenever a held block is more than BACKLOG_PREFETCH_SLOTS behind the current slot. Nothing waits on that answer and nothing retries it: it either shortens the work the by-root lookups have left or it does not, and the next tick asks again while the backlog lasts. * fix(beacon): stop paying full verification for a sidecar this node already has The backlog prefetch re-asks an overlapping span of slots every tick while the node is behind, and a peer answering the same span twice re-delivers every sidecar in it. During a drain most arrivals are therefore ones already in the store, and each was paying the whole of on_gossip_data_column: a full post-state read for the proposer check, the inclusion proof, the KZG batch and a signature verification. All of that runs on the single-threaded chain actor, which is the same thread the imports the drain is waiting on run on. Measured on eth-4 while 170 slots behind: 684 sidecars a minute arriving against 5 blocks a minute importing, the actor pinned at 91% of a core, and the head losing a slot every 24 seconds against a 12-second slot. A presence check on (slot, block root, index) is one store read and sits ahead of everything expensive. It also covers the gossip case it was always true of: a duplicate on a subnet this node subscribes to. * fix(p2p): let a range prefetch guess at peers that have said nothing The follower stalls for minutes at a time on a fresh checkpoint backfill: a block is held, its columns have no known custodian among the connected peers, and neither fetch path can reach them. The by-root path at least asks someone; the range path asked no one, so the columns only arrived when the peer set happened to churn into a custodian. Observed on eth-4: head stuck at 15194370 and then 15194372 for minutes each, with the by-root ladder giving up on a fresh set of peers every slot. The gap is what "uncovered" means. A column is uncovered when no peer is *known* to custody it, and custody is known only once a peer's metadata/3 answer or its ENR cgc has been read. A peer that has supplied neither is not a peer that lacks the column, and there are always a few: freshly connected, or answering metadata/3 on a version this node's codec rejects. So the uncovered columns now go to a couple of those, and only those. A peer that has said what it keeps has already answered the question, and asking it for what it does not keep is the waste the original comment was right about. Two peers rather than all of them, because a range answer is megabytes when it does land. * fix(beacon): park the sidecars nearest the head, not the ones that arrived first A follower behind the tip receives gossip for the tip continuously, and every one of those sidecars names a parent it does not have yet, so the parked queue fills with blocks it will not reach for minutes. Arrival order then decides what is lost, which means it refuses the sidecars for the block it is about to import while holding the ones it cannot use. Measured on the eth-4 follower about a hundred slots behind a fresh checkpoint anchor: the queue sat pinned at its cap and dropped 2,561 sidecars in ten minutes, while the chain ground through by-root lookups for exactly the slots whose columns gossip had already handed over and this function had thrown away. That is the difference between bursts at twice chain rate and stalls of a minute or two, which is what the head was doing. A full queue now gives up its furthest-ahead entry to admit a nearer one, and refuses an arrival further ahead than everything it holds. One sidecar is lost either way, so the counter moves either way; what changes is which one. * fix(p2p): reserve a third of the connection ceiling for peers we dial The mainnet follower sat at 178 inbound peers and zero outbound ones for two days. Every block waited a median 217s on custody columns no connected peer held, against 6.6s to actually import it, so its head advanced at roughly 13 slots an hour while the chain produced 300. Two things combined to produce that. The inbound allowance was 90% of the ceiling, and `dial_tick` gated on the *total* peer count, so once inbound demand filled the table the dial loop stopped entirely and the 20 reserved outbound slots were never used. A reservation the dial loop stops trying to fill reserves nothing. Outbound peers are the only ones this node chooses, so they are also the only lever it has on its own custody-column coverage: inbound peers arrive at random with respect to what they custody. Inbound is now capped at 70% of the ceiling, and the dial budget takes the larger of the total and outbound shortfalls so inbound saturation cannot suppress dialing. Lean is unaffected: it runs unlimited connections and has nothing to reserve against. Expressing any of that needs connections tagged, so `connected_peers` carries a `ConnectionDirection` per peer rather than being a bare set. That also fixes an accounting drift: a peer holding both an inbound and an outbound connection was attributed on disconnect to whichever socket closed last, so the per-direction connect and disconnect counters could diverge from each other. Three gauges, all set from a full re-count rather than incremented and decremented per event, because a gauge meant to reveal leaks must not be able to leak itself: lean_peers_by_direction peers by which side opened the connection lean_swarm_established_connections libp2p's own counters, which the limits are enforced against, so a connection charged to the cap that no live peer is using becomes visible rather than assumed absent lean_custody_column_peers peers known to custody each column this node samples, where a zero is a stall waiting to happen and is invisible in a total peer count The allowances are derived from the percentage and guarded by three compile-time asserts, including one pinning the direction integer division rounds, so a ceiling that is not a multiple of 100 can never round part of the reservation back to inbound. * perf(p2p): pace dialing on ethrex's curve instead of a flat tick The dial loop ran on a flat 5s tick and dialed a batch of 8, which is 1.6 dials a second. Measured on the eth-4 follower: ~3,360 dials were possible in 35 minutes and 21 happened, while five of its eight sampled columns sat at zero known custodians and blocks waited minutes on columns nobody connected held. The rate is now the tick, not the batch. `dial_interval` is ethrex's `lookup_interval_function` — the easeInOutCubic curve it paces both discv5 lookups and RLPx dialing with — so all three layers ramp the same way: progress 0.00 0.25 0.50 0.75 1.00 interval 20ms 56ms 310ms 564ms 600ms rate 50/s 17.8/s 3.2/s 1.8/s 1.7/s The ceiling is ethrex's `LOOKUP_INTERVAL_MS` exactly, because that end is reached while the table is nearly full and peers still churn: a slower ceiling replaces losses at the rate they happen rather than ahead of it. The floor is 50/s against ethrex's 100ms, since 96% of outbound dials to mainnet never establish and landing the other 4% is a numbers game. Dialing stops outright at target, where `dial_budget` returns 0, so the rate reaches zero rather than merely flattening. `progress` is the *minimum* of the total and outbound-reservation ratios, not `connected / target`. With 140 inbound peers and no outbound ones the total ratio alone reads 0.7 and would pace the loop down to half a second between dials, which is the state that stalled this follower for two days. Taking the minimum holds it at the floor until the reservation fills, mirroring `dial_budget` taking the maximum of the same two shortfalls. Two things follow from a tick that can fire 50 times a second. `MAX_PENDING_OUTBOUND_CONNECTIONS` bounds dials in flight. The connection ceiling and its two halves count only *established* connections, and a dial that never establishes is invisible to them. At 1.6 dials a second that gap was harmless; at 50 it is the dial rate times however long the slowest peer takes not to answer. Four seconds of dialing at full rate, far above what a healthy node has outstanding and still a hard ceiling on file descriptors. `prune_seen_data_columns` gets its own clock. It rode the discovery tick back when that was a steady 1-5s heartbeat, and it costs a store read plus a scan of the whole seen-column set. Inheriting the new cadence would have run it hundreds of times a second on the p2p actor's single thread, to re-derive a boundary that moves once an epoch. * feat(beacon): make the data-availability gate unconditional `--beacon-da-enforce` existed as an off switch because the gate could not follow mainnet: a block whose custody columns never arrived stalled the head behind it until finality cleared it, and that had to be escapable without a rebuild. The column-fetch work on this branch removed that failure mode, so the switch now guards nothing and the specification's own behaviour is the only one worth having. Dropping `da_enforced` from `BlockChainServer` leaves lean's import path untouched. The field is read in one place, the `beacon_block` arm of `process_block`, and lean passed `false` to say "no column evidence to gate on" rather than to disable anything, so the gate is now governed by `da_check_required_for_slot` alone. * refactor(storage): record the anchor slot instead of the earliest custodied one `earliest_column_slot` answered "what is the lowest slot we can serve columns from" by tracking, on the gossip write path, the lowest slot a sidecar had ever been written at. That put a read-decide-write span in `put_data_column_sidecar` whose correctness rested entirely on there being exactly one writer, and it cost a backend round trip on every `data_column_sidecars_by_range/1` request to read a value that cannot change while the process runs. The question it was answering has a fixed answer: where this directory's chain begins. That is the anchor block's slot, so record that instead, once, at bootstrap. `Store` caches it in a field beside `config` and `chain`, for the same reason those are cached. It has to be persisted rather than derived, because `from_db_state` is a constructor too and has nothing to recover it from: `latest_finalized` is seeded to the anchor but climbs away from it at the first finalization. Hence `KEY_ANCHOR_SLOT`, written by both bootstrap paths and read back on resume, and hence the `DB_VERSION` bump: a version 1 directory does not carry the key, and is refused at the version check rather than opened with a defaulted 0 that would have it advertise data it does not hold. Both bootstrap paths take the value from the anchor block's own slot rather than from the anchor checkpoint. On beacon that distinction is load-bearing: `beacon_checkpoint_as_stored` converts an epoch to that epoch's start slot, so a mid-epoch anchor would otherwise record a floor below anything the directory holds. This also fixes `build_status`, which reported `latest_finalized().slot` as `earliest_available_slot` and so understated the servable range by a margin that grew with every finalization. The by-range floor and the `Status` advertisement are now the same value, which is what keeps a refusal from contradicting an advertisement. Callers keep the full range semantics with one behavior change: a request in the window between the anchor and the first column actually custodied is answered with an empty list rather than `RESOURCE_UNAVAILABLE`. A node that range-synced has already backfilled columns alongside blocks across that window, so the two floors converge in practice. * refactor(p2p): ask for a block and its columns in one request The chain actor had three ways to ask the p2p layer for something: `fetch_block` for a root, `fetch_data_columns` for a root's columns, and `fetch_data_columns_by_range` for a span of them. That left the caller choosing which protocol to reach for, which is the p2p layer's decision, and the range one duplicated work p2p already does: every `BeaconBlocksByRange` batch has pulled its own span's columns alongside the blocks since 4854755f, so the chain's per-tick backlog prefetch was a second, worse-paced copy of a request already on the wire. One `fetch_block(FetchRequest)` replaces all three. `FetchRequest` names the root, whether the block itself is missing, and which columns are, so a held block's per-slot re-drive asks for columns without also putting a redundant `BlocksByRoot` on the wire. `needs_block` is explicit rather than inferred from an empty column list: the two cases are disjoint only by coincidence of today's callers, and a p2p layer reading its own store to find out would pay a DB read to re-derive what the caller knew. `new_data_column_sidecar` becomes `new_data_column_sidecars` and takes a batch. Every producer but gossip has one to hand -- a by-root answer carries every column of a block, a by-range answer a whole span -- and sending them one at a time put a mailbox hop per sidecar between the answer and the actor draining the backlog that was waiting on it. Parked sidecars move out of memory. `sidecars_awaiting_parent` held whole `DataColumnSidecar`s, cells and all, for a queue whose size a peer gets to choose; it now holds only the key each one was written under, and the bytes go to a new `Table::PendingDataColumns`. A table of its own, not `DataColumns`: a parked sidecar has passed none of the checks that matter, and `data_column_indices_for` is what the availability gate believes, so an unverified row there would let a peer release a held block with a column it invented. A row moves between the two only by passing every check on replay. `MAX_SIDECARS_AWAITING_PARENT` goes with the move. It existed to bound this actor's memory, which the move already does, and a follower behind the tip hit it constantly: gossip for the tip arrives continuously, so the queue filled with sidecars for blocks minutes away and then shed the ones for the block about to import. What is left bounding the queue is the finality sweep, which bounds how long a row lives but not how fast rows arrive -- `on_gossip_data_column` does not require `parent_root` to name a known block, and the p2p seen-set dedups on header fields a fabricated header chooses freely -- so `lean_sidecars_awaiting_parent` is now the only warning that a peer is parking rows it never means to resolve. Documented on the field, the metric, and in data_storage.md. Two things the move to disk needs. Parking dedups, because the fetch paths have no seen-set and a re-delivery would otherwise strand a key whose row the first replay took. And the actor clears the whole table at startup, since the map it indexes from does not survive a restart and every row written before one is unreachable by construction. * refactor(p2p): derive the connection limits from the peer target A review of the five commits since ee123ae7 turned up one mechanism whose depth was wrong and a handful of duplications. The reservation the dial loop chases was a fixed 60 slots, a share of a connection ceiling that had nothing to do with what the operator asked for. It read as "30% of target" only because the default target and the ceiling were both 200. Set `--discovery.target-peers 50` and the loop kept dialing to 60 outbound peers, past the 50 asked for; set it to 0, documented as "never dial", and the outbound shortfall was still 60 and the loop dialed against an explicit opt-out. `max_connections`, `max_inbound_connections` and `max_outbound_connections` are now functions of `target_peers`, and `SwarmConfig` carries it so `build_swarm` derives the limits libp2p enforces from the same number the dial loop reads. What this node refuses and what it goes looking for are two readings of one number, so a reservation the swarm does not keep can no longer be one the loop chases. A target of 0 now holds no peers at all, which `cli.rs` and `docs/discovery.md` both say instead of "discover and serve, never dial". The share is taken in `u64`: the product overflows `u32` above a ceiling of about 61 million, and a saturated one would silently stop being a percentage. Two of the three compile-time asserts pinned facts that were true by construction of the consts they guarded; with no const ceiling left they are a test over targets that actually exercise the rounding. The dial loop was also paced on a shortfall a network may be unable to close: lean's default target is 200 against a devnet of 32, which held the tick at its 20ms floor for the life of the process, re-drawing a candidate pool of peers it was already connected to, hundreds of times a second on the p2p actor's thread. `dial_tick` now reports whether it opened a dial, and one that did not waits the full ceiling. `lean_custody_column_peers` was refreshed from the two `connected_peers` mutation sites, but its other input `peer_custody` is written by `record_peer_custody`, which for the `metadata/3` path runs after the peer connected. On a stable peer set every column therefore read the zero it had before the peer said anything, which is the exact reading the gauge was added to mean "a stall waiting to happen". It republishes from that writer now, and counts through `columns_custodied_by` so the gauge cannot claim custodians a lookup would not find. The rest is duplication the same five commits introduced. `dial_interval` transcribed ethrex's easeInOutCubic curve, which is `pub`, so it calls it and `DIAL_INTERVAL_AT_TARGET` reads `LOOKUP_INTERVAL_MS` rather than repeating 600. `dial_progress` and `dial_budget` each derived the outbound count and the reservation; they share an accessor. `dial_budget` was called twice per tick. `put_data_column_sidecar` and `put_pending_data_column_sidecar` became byte-identical when the anchor slot moved out of the write path. `ParkedColumn::key` and `forget_parked_columns` were two shims between one caller and one store method. The parked queue deduped by scanning a `Vec` that the same commit uncapped, and is a `HashSet` now. A replayed sidecar was decoded and then re-encoded to bytes it already had. Three doc links pointed at symbols those commits deleted, and the three peer gauges they added were undocumented. * fix(p2p): dial a peer's quic address before its tcp one Both addresses went into one dial and libp2p's default concurrency factor started both handshakes, so the transport a peer ended up on was decided by whichever won the race. That read as a neutral choice between two equal wires and is not one. Mainnet beacon peers answer `na` to a yamux-only proposal, so every TCP win negotiates mplex, and a 20s profile of the eth-4 follower put `libp2p_mplex::io::NotifierWrite::register` at 24.4% of process cycles, the single largest symbol, ahead of the entire beacon state transition. QUIC multiplexes natively and reaches none of that code. TCP was winning 62% of outbound establishments over six hours (3,817 against 2,319) while inbound peers, who choose for themselves, picked QUIC 71% of the time. So the race was steering us onto the expensive transport precisely where we had a say. `dial_concurrency_factor` is pinned to one, which makes libp2p walk the address list in order instead of racing it, and `dial_addrs` already puts `quic` first. The cost is the one the race avoided, now bounded rather than removed: a peer advertising a `quic` port nothing answers waits out libp2p-quic's handshake timeout before its `tcp` address is tried. A quarter of dial targets advertise no `quic` at all and are unaffected, as is lean, whose records carry `quic` alone. That ordering is now load-bearing where it used to be incidental, so it is asserted in a test and both doc comments that told the reader to read nothing into it are rewritten. * fix(blockchain): run the chain actor off the shared tokio runtime This actor's handlers are long stretches of synchronous CPU with no await in them: a state transition, a merkleization, a fork-choice pass. As a task on the shared runtime that starves every other task the runtime is serving, and the P2P actor is the one that suffers, because it owns both this node's sockets and its dial loop. `spawned_concurrency` documents `Backend::Thread` as the backend for CPU-bound work and even ships a `WarnOnBlocking` detector for this exact shape, but that detector is `#[cfg(debug_assertions)]`, so a release build says nothing and every actor here took the default. Measured on the mainnet follower, one node, one build, one fresh-anchor backfill, with this actor pinned at ~100% of a core either way: Backend::Async Backend::Thread dial tick samples 12,494 19,251 ticks over 10s late 5 0 ticks over 0.5s late 16 0 worst sample over 10s 25-100ms mean lateness 23-37ms 1.06ms longest gap in gossip 214.92s 5.28s The whole multi-second tail is gone while the actor works exactly as hard, so this is the cause rather than a correlation: the only thing that changed is which runtime it runs on. The symptom points away from the cause, which is what made it expensive to find. Everything on the P2P actor measures fast, 64,000 handler samples with none over 10ms, which reads as a healthy actor. It was not blocked. It was not being polled. Both chains, from the one `start_actor` that `spawn` and `spawn_beacon` share. Lean's state transition is far cheaper than the beacon one, but leanVM aggregation on an aggregator node is not, and it runs on this same actor with the same P2P actor beside it. The cost either way is a second tokio runtime, which `spawned_rt::threads::block_on` builds on the new thread; on the follower that took the process from 2 non-tokio threads to 19. Verified on the beacon path against a live mainnet node. The lean path is compiled and unit-tested but has not been run on a devnet. * feat(beacon): make the custody group count a flag The follower advertised `CUSTODY_REQUIREMENT` and custodied the columns that implies, with no way to say otherwise. That is the least useful a node is allowed to be, and on a network where a `cgc=4` peer holds 4 of 128 columns it is also what decides whether anyone's lookup finds a custodian: the chance no connected peer holds a given column is `(1 - 4/128)^N`, still better than even at twenty such peers. Running a supernode, or anything above the floor, was a recompile. `--custody-group-count` now sets it, and one value feeds all three places the old constant was read: the custody set via `sampling_size`, the ENR's `cgc`, and the startup line that says what is being advertised. It lives in a new `MainnetOptions` rather than in `CommonOptions`, mirroring `LeanOptions`. Lean has no data-availability domain and omits `cgc` from its ENR entirely, so a flag in the common struct would be one the lean path must carry and ignore. That makes `Network::Mainnet` a tuple variant, which is what its doc comment had been promising would happen once this chain had a flag of its own. Range is `CUSTODY_REQUIREMENT..=NUMBER_OF_CUSTODY_GROUPS`, enforced by clap at parse time rather than clamped: a node that quietly serves a different set than the operator asked for is the failure hardest to notice from outside. Two things worth knowing, both now in `docs/cli.md`. The flag is not the number of columns custodied: `sampling_size` is the larger of it and `SAMPLES_PER_SLOT`, so the default still custodies 8 and the two only converge once it is raised past 8. And changing it changes which columns this node custodies, because the custody set is a function of the node id and the count together, so sidecars already on disk belong to the old set. Nothing is corrupted by that; the node advertises a set it has not finished filling until it backfills the difference. * docs: the mainnet network variant is no longer a unit `--custody-group-count` gave `beacon` its first flag that is not a common one, so `Network::Mainnet` carries a `MainnetOptions` now. The startup section still described it as a unit variant and gave "every flag `beacon` takes is a common one" as the reason. * refactor(metrics): drop the blocks-admitted-without-blob-check counter `lean_blocks_admitted_without_blob_check_total` counted, by fork, what `warn_no_blob_pipeline` already logs on every occurrence with both the fork and the block root: a deneb or electra block carrying commitments that this node admits unchecked, having no blob-and-proof pipeline to source evidence from. Two reports of one event, the narrower of which could not say which block. The warning stays and is unchanged. Going with the counter are its zero-seeding roster `BLOB_PIPELINE_MISSING_FORKS`, the `init` loop that made both forks reachable series, the `inc_` wrapper, and the metric's row in metrics.md and its mention in beacon_wire.md's DAS list. * perf(p2p): poll the peer table off the actor's thread `get_contact_to_initiate` is an actor round trip, and `draw_candidates` awaited up to `DISCOVERY_CANDIDATE_BATCH` of them inline in `dial_tick`, which holds `&mut P2PServer` for the length of a tick. Every refill therefore parked the whole p2p actor -- gossip forwarding, req/resp, swarm events, every other tick -- behind the peer table. Pacing the dial loop on a curve made that worse rather than better: the refill is due whenever the candidate queue runs out, and the loop now reaches its 20ms floor whenever this node is short of peers. The round trips move to a tokio task. It draws contacts, runs the same `LeanFilter::dial_target` admission, and sends already-judged `DiscoveredPeer`s down an mpsc channel; `DiscoveryState` holds the receiver in place of the `PeerTable` and the filter, and the refill is a `try_recv` drain that awaits nothing. Ranking, the dial path and the pacing are untouched, and an error from the table is still read as "nothing right now" and retried, as `draw_candidates` did. The channel is bounded and the task reserves its slot *before* asking, which is the part that matters. ethrex marks every contact tried on the way out, so a contact drawn with nowhere to put it is not offered again and would be lost rather than queued, and a task free to run ahead of the dial loop would draw the table's eligible set down for candidates nobody dials. A full buffer stops it asking instead. The loop keeps refilling only when its queue is empty for the same reason: draining every tick would pour the bounded channel into an unbounded `VecDeque` and there would be no backpressure left. `CONTACT_POLL_IDLE` is how often the task re-asks an empty table. At 250ms it stays under ethrex's own lookup interval, so the crawl remains what decides when a peer becomes available. A slower idle would be visible at startup, where the table is empty, the dial loop used to re-ask on every 20ms tick, and this task is now the only thing asking.
* feat(beacon): name the payload status an execution client returns * feat(beacon): let a signed block report its own execution block hash * feat(storage): hold the beacon payload statuses fork choice will read * feat(storage): delete named live chain rows, not just a slot range * feat(beacon): name the optimistic import safety horizon * feat(beacon): read an execution status as the verdict on_block takes * feat(beacon): resolve which block a latestValidHash condemns * feat(beacon): cut an invalidated subtree out of fork choice * feat(beacon): gate optimistic import on the spec's two conditions * feat(beacon): let on_block carry an execution payload verdict Threads tasks 6-9 together: the verdict picks the ExecutionEngine the state transition runs with, an INVALIDATED answer cuts the condemned branch out of fork choice on the way out of the error path, and a successful import records the block's execution hash plus whether the execution layer has vouched for it. Both existing callers pass NotRequired, so every fork_choice fixture case and the chain actor behave exactly as before; the engine client fills the actor's in later. Also folds in the Task 3 review: doc comments on the three undocumented beacon scratch accessors, and the plan's own repeat_byte hash convention in the tests that predated it. * test(beacon): run the optimistic sync spec fixtures * build: add the engine API crate skeleton * feat(engine): mint the JWT an execution client authenticates * feat(engine): shape the engine API's JSON on the wire * feat(engine): call the four methods a follower needs * test(engine): round-trip the wire against a mock execution client * feat(beacon): encode a block's execution requests per EIP-7685 * feat(blockchain): assemble the engine question a beacon block poses * refactor(blockchain): make the import cascade async The engine round trip has to happen between the data-availability gate and fork_choice::on_block, both of which live inside process_block, so that frame and everything above it has to be able to await. No behaviour changes here: the column-release path joins the cascade's frames because it re-enters on_block, and the two production call sites were already inside async handlers. * feat(blockchain): validate a beacon block's payload against the execution client * feat(blockchain): tell the execution client where the head is * feat(cli): pair the beacon follower with an execution client * feat(cli): wire the execution client into the beacon follower * docs(beacon): describe the execution layer pairing and its limits * build: stop tracking a stray .DS_Store, and ignore them Committed by accident alongside the engine crate skeleton. This repo is developed on macOS, so ignore the whole class rather than only this one. * refactor(engine): reuse H256's own hex serde, and check the HTTP status H256 already serializes as 0x-prefixed lowercase hex and already deserializes prefix-optional hex with a 32-byte check, so the two helpers here were a second copy of that logic that could silently drift from it. The payload's own data() calls stay: feeRecipient is an ExecutionAddress, which has no Serialize impl. Also check the response status before parsing the envelope. A rejected JWT answers 401 with a body that is not a JSON-RPC envelope, and parsing it first turned an actionable failure into an opaque decode error. * fix(beacon): read is_execution_block as the spec means it, not as a field check is_optimistic_candidate_block asks whether the parent has execution enabled, and was answering it with "the container has an execution_payload field". A pre-merge bellatrix block carries that field with every byte zero, so its children looked like descendants of a merge block and skipped the age horizon that exists precisely to guard the merge transition. Not reachable today (this follower anchors past electra and only ever asks about electra and fulu blocks), but wrong, and wrong in the direction of importing too readily. The cache's presence is now the predicate: a zero block hash is not stored, which is exactly "the payload is not the fork's own empty one", since a real payload's block hash is a keccak digest. Also adds the metric for the other reason a block is now dropped without a post-state, and records on invalidate_subtree the dangling-vote hazard it creates for get_weight, which stays spec-strict on purpose. * docs(beacon): stop telling operators to disable the availability gate Both paragraphs described a gate that could not follow mainnet, on the strength of which they instructed a mainnet follower to turn it off. The column-fetch work underneath this branch removed that failure mode and the switch they named is gone, so the instruction now points at a flag that does not exist and at a limitation that no longer holds. The engine retry ladder's own paragraph compared itself to that wedge; with nothing left to compare to, it states its shape directly instead. * docs(beacon): correct what the optimistic-import horizon actually gates Two claims here did not survive running the follower against mainnet with a syncing execution client. Condition 1 was described as "a parent with execution enabled". It is `store.beacon_el_block_hash(parent_root).is_some()`, and that hash is written on import, so it asks whether *this node* imported the parent, not whether the parent is post-merge. The difference is invisible until the two disagree. They disagree at exactly one place, which is why the section also claimed condition 2 never fires on a follower anchored past the merge. A checkpoint-sync anchor is never imported and never asked about, so it has no recorded hash and its child fails condition 1; with the execution client answering SYNCING, that child waits for the horizon and the head sits at the anchor until it clears. Measured 2026-09-15: an anchor 85 slots behind cleared in ~9 minutes, not the full 128 slots, because the horizon is counted against the wall clock rather than against the anchor. It happens once, then the backlog cascades on condition 1. Also drops BeaconOptions' claim to have "no flag of its own left", which three flags below it have contradicted since the execution pairing landed. * docs(beacon): point at the optimistic sync spec where it now lives consensus-specs deleted its `dev` branch (the default is now `master`) and moved the document from `sync/optimistic.md` to `specs/bellatrix/optimistic-sync.md`, so the link 404s and the docs link check fails the PR. Bellatrix is the right target rather than `specs/heze/optimistic-sync.md`: the Heze document is a work-in-progress delta covering inclusion list satisfaction, which this follower does not implement, and it extends the Bellatrix one for everything this page describes. The `sync/optimistic` fixture suite named further down keeps its name: that is a consensus-spec-tests directory, not the specification document, and it did not move. * docs(beacon): name the optimistic sync spec file that exists upstream Twenty references named the optimistic sync specification by a path that consensus-specs no longer has, in two stale forms: six said `sync/optimistic.md`, the path from before the document moved, and fourteen said `optimistic.md`, the name from before it was renamed. Upstream the document is `specs/bellatrix/optimistic-sync.md`. They all say `optimistic-sync.md` now, the bare file name, because that is the short form the surrounding comments already use for a specification document whose fork is clear from context: `beacon-chain.md`, `fork-choice.md` and `das-core.md` all appear that way, and the fully qualified form is reserved for where the fork is the point, as in `specs/electra/beacon-chain.md`. The full path is pinned once, in this page's own link, which is also where the ambiguity with `specs/heze/optimistic-sync.md` is resolved: Heze is a work-in-progress delta for inclusion list satisfaction, and Bellatrix is the document every one of these references means. Four references keep `sync/optimistic` unchanged, in `storage/src/store.rs` and three places in `beacon/fork_choice.rs`. Those name the consensus-spec-tests fixture directory, which still exists under that name; they are not references to the specification document. Eight paragraphs are rewrapped because the longer name pushed a line past the width its file wraps at. No wording changed. * fix(beacon): address the engine API review findings Eleven findings from the review of this PR, in one commit because they interleave: `blockchain/src/lib.rs` carries four of them, `fork_choice.rs` four and `storage/src/store.rs` three, and splitting them apart by hand would buy a history whose intermediate commits do not build. Blockers: - A held fulu block was stranded whenever its re-import failed for a reason that was not about columns. `release_block_if_columns_complete` removes the hold before re-importing, which is what makes the re-entry terminate, but an engine failure or a not-yet-an-optimistic-candidate verdict then recorded nothing, so neither `redrive_held_blocks` nor the finality eviction could still see the block and it, and every descendant, waited for a restart. `on_block` reports the handed block's `ImportOutcome` now and the hold goes back on `Held`. An import that failed on the block's own contents deliberately does not come back: it would fail identically every slot until finality evicted it, buying a state transition per slot and nothing else. - `prune_beacon_el_block_hashes` deleted the finalized block's own execution hash. Its bound is a checkpoint's slot, which is that checkpoint's epoch's start, while the checkpoint root is the last block at *or before* that boundary, so a missed proposal at an epoch boundary left the finalized block below the bound. From then on every `forkchoiceUpdated` carried `finalized_block_hash = 0x00..0` and the execution client never advanced its own finalized block. The root is exempted by name now. - `invalidate_subtree` obeyed a verdict against a finalized block. EIP-3675 lets an execution client answer `INVALID` with `latestValidHash = 0x00..0`, and `resolve_invalid_block`'s zero branch then walks as far as the execution hash cache reaches, which finality bounds at the finalized block itself. Obeying that deletes every `LiveChain` row from finality upward, after which `get_head` fails its "block_root in store.blocks" check on every call. It is refused with an error log now: the two layers disagreeing about finalized history is an operator emergency, not something to resolve by emptying fork choice. Majors: - An invalidated head stayed in `KEY_HEAD` and `Table::BlockRoots` until the next tick recomputed it, and the req/resp handlers read both, so `Status` advertised the invalidated root and `BlocksByRange`/`BlocksByRoot` served the invalidated block to peers. That is a stale store, not the stale gauge the comment claimed. The head is recomputed inside the verdict handler now, through a new `update_head_from_fork_choice` that deliberately does not announce it: announcing from there would re-enter the `forkchoiceUpdated` being answered. - `mark_validated` built `Store::block_index` unconditionally, an uncached scan of a table beacon never prunes plus a map build, once per imported block and once per `forkchoiceUpdated`, only to walk a set that is empty whenever the execution client is healthy. It asks `has_beacon_optimistic_roots` first now. - `optimistic_roots` grew without bound. Its own doc claimed an entry left on finality and nothing did that, and the `BeaconScratch` doc called `el_block_hashes` the one bounded exception. An execution client doing a long state sync answers `NOT_VALIDATED` to every block, so the set took one root per import for the life of the process. It carries slots now and is pruned beside the execution hash cache, on the same horizon and at the same two call sites. Both doc claims are corrected, including a note that nothing outside `mark_validated`'s walk reads the set yet. - An `INVALIDATED` verdict was enforced only by `state_transition` happening to fail. That holds only for forks whose `process_execution_payload` consults the engine: bellatrix gates that step on `is_execution_enabled`, and phase0 and altair have no such step, so a condemned block on one of those would transition cleanly and import with nothing recorded. `on_block` states the invariant on the verdict now. The transition still runs first and its own error is still what propagates, which is what keeps the failure arriving from inside `process_execution_payload`, where the specification puts it and where the `sync/optimistic` fixture exercises it. Minors: - `ClientVersionV1.commit` went out without its `0x` prefix. geth decodes that field into `hexutil.Bytes`, which rejects a bare hex string, so `engine_getClientVersionV1` came back an RPC error and the identification handshake never worked, invisibly at default log levels. The `chars().take(8)` is a `get(..8)` now too, since `VERGEN_GIT_SHA` is not guaranteed to be eight or more characters in every build configuration. - `--execution-endpoint` without `--execution-jwt-secret` gave the operator a panic. clap 4 does express "both or neither", with `requires` on each flag, so the parser answers a mismatch with a usage error and the conversion's arm is `unreachable!`. `an_endpoint_without_a_secret_is_refused`, which pinned the panic, is replaced by `execution_flags_come_as_a_pair`, which pins the same invariant one layer earlier and over both halves rather than one. - `get_execution_requests_list` had landed between `process_execution_payload`'s doc comment and its `fn`, so the two blocks concatenated, rustdoc attached them all to the new function, and `process_execution_payload` rendered with no documentation at all. Moved below the function it was dropped into. - `spawn_beacon`'s doc comment opened with six slashes, which is an ordinary comment rather than a doc comment, so its summary line was dropped and the module index showed a truncated description. `docs/beacon_engine.md` gains the finality floor, the two prune bounds, the optimistic-root lifecycle and the immediate head rewrite.
* fix(p2p): stop writing a varint+snappy header for an empty MetaData request The spec's MetaData request has no body at all, but write_request encoded it to an empty Vec and still ran it through write_payload, which always writes a varint length prefix and a bare snappy stream header even for a zero-length slice: eleven bytes where the spec puts none. read_request had the matching problem in reverse, unconditionally calling decode_payload before looking at the protocol, which would block forever on a peer that actually sent the spec-correct zero bytes. Both directions now check for a MetaData protocol first and return before touching write_payload/decode_payload. The existing round-trip test could never have caught this, since our own encoder and decoder agreed with each other on the same wrong framing either way; two new tests pin the actual wire bytes and prove read_request accepts a truly empty stream. * refactor(p2p): one request_response::Behaviour per protocol Two outbound requests on different protocols, fired back to back on one connection (exactly what ConnectionEstablished does: send_status then request_metadata), could get matched to each other's substreams. The shared req_resp: request_response::Behaviour<Codec> field handed every outbound request to one FIFO pair (pending_outbound/requested_outbound in the pinned fork's handler.rs), drained in substream-negotiation completion order, not send order; the fork's own send_request_with_protocol only narrows which protocol string is offered per call; it does not change that shared queue. When two negotiations complete out of send order, write_request is handed the wrong (protocol, request) pair. Splitting the one behaviour into fourteen, one per protocol id (three lean, eleven beacon), makes the crossing structurally unreachable rather than merely less likely: NetworkBehaviour's derive nests each field's ConnectionHandler behind ConnectionHandlerSelect, which routes a negotiated substream back to exactly the field that offered its protocol, so each field's own queue never sees a request sent on another protocol. Every field is built through build_swarm's existing per-wire match, with either its one real protocol registration or none, so a lean node still offers nothing beacon-only and vice versa. send_request now goes through upstream request_response::Behaviour::send_request on the field named by the new ReqRespProtocol enum; the fork's send_request_with_protocol has no remaining call site. Splitting the behaviour also splits its OutboundRequestId sequence: each field now mints its own ids starting at 1, so two different protocols can legitimately produce the same numeric id for two unrelated requests. outbound_requests is keyed on the new composite ReqRespRequestId (protocol + id) instead, threaded through swarm_adapter's SendRequest command and reconstructed in handle_req_resp_message from the BehaviourEvent variant that produced each event. Left at the request-response layer's own default, fourteen fields would each carry a per-connection stream budget sized for one shared behaviour speaking every protocol, multiplying the aggregate budget by fourteen for no added traffic. Retuned per protocol shape instead: HANDSHAKE_MAX_CONCURRENT_STREAMS for the once-per-connection protocols (status, ping, metadata, goodbye) and the larger FETCH_MAX_CONCURRENT_STREAMS for the four protocols that carry real block and data-column fetch traffic. handle_behaviour_event's dispatch is deliberately exhaustive, with no wildcard arm, so a fifteenth protocol added later forces a decision here rather than falling through handle_swarm_event's own catch-all unnoticed. Adds a regression test that fires a real Status and a real MetaData request back to back over a live loopback connection, repeated across many fresh connections since the race is timing-dependent (negotiation completion order, not send order), and asserts every repetition's answer matches the protocol it was sent on. * refactor(p2p): move the req/resp behaviours into their own struct `lib.rs` carried the fourteen `request_response::Behaviour` fields, their fourteen `with_codec` calls, the two stream-budget constants and the three construction helpers, which was most of what `build_swarm` had grown into. All of it now lives in `req_resp::behaviour`, as a `ReqResp` struct that `Behaviour` nests as a single field: the outer `NetworkBehaviour` derive selects into this one and this one selects into its fourteen, so a negotiated substream still routes back to exactly the field that offered its protocol, which is the guarantee the split exists for. Construction is `ReqResp::new(codec, &wire)`, so the `is_lean`/`is_beacon` pair deciding each field's protocol list is no longer computed in `build_swarm` and threaded through fourteen call sites. `handle_behaviour_event` matched fourteen `BehaviourEvent` variants and repeated the same `handle_req_resp_message` call in each with a different `ReqRespProtocol` constant, which the formatter spread over four to six lines an arm. The tag is the whole of what a req/resp arm decides, so the match yields it as a value and the single call that consumes it sits below: each arm is one line, and the function goes from 119 lines to 40. Still exhaustive, with no wildcard arm, for the reason its doc comment gives. The regression test's own tagging match narrows the same way, so it no longer mirrors fourteen `BehaviourEvent` variants. `swarm_adapter`'s exhaustive send dispatch is unchanged beyond the extra `.req_resp` hop; the fields stay `pub(crate)` because picking one by `ReqRespProtocol` is what sending means. Net: `lib.rs` loses 322 lines, the new module is 271.
#17) * feat(blockchain): time a block's import end to end and log the breakdown A block that misses the interval its chain expects attestations at is the symptom; which part of the import spent the time is the question, and until now nothing could answer it. `store::on_block` logged three durations, the mailbox hop between the p2p actor and the chain actor was invisible, and so were `update_head`, the engine round trip, the per-table size estimates every import pays for, and every hold. Instants are captured where a boundary is crossed and returned up the stack; nothing subtracts anything until `BlockImportReport`, which is the sole consumer. That is what lets one report serve both chains: a section that never ran leaves its pair `None` and is simply not printed, so a lean block shows no beacon rows and a beacon block shows no `verify_*`. The three holds carry `_start`/`_end` pairs rather than single instants because a held block does not walk the timeline once, it loops: guards run again for a block held for its parent, the availability check runs again for one held for its columns, and a single instant per boundary would let the second pass overwrite the first and the wait disappear into whichever section straddled it. `held_timings`, taken on release and removed by `discard_pending_subtree`, is what carries a block's original arrival across a hold so its end-to-end time still starts where it really started. `da_complete_on_arrival` is asked before the parent check, which is the only moment it is answerable. Without it an absent `columns_wait` row reads two ways at once: the columns were never missing, or they landed while the block was held for its parent. With it the two are distinguishable, which is the point of separating the parent wait from the availability wait at all. `BlockArrival` rides on `NewBlock` rather than the timings themselves, so `ethlambda-network-api` keeps knowing nothing about the import path that depends on it. The two timing logs `store.rs` already emitted move to debug rather than being deleted: the tree carries the same numbers, and the paths that print no tree (the Hive driver's verify endpoint, the spec-test runner) still have something to turn on. No metrics yet. * feat(metrics): publish a block's import sections as one histogram `lean_fork_choice_block_processing_time_seconds` starts inside `store::on_block`, so everything the previous commit made visible in the log was still invisible to a dashboard: the mailbox hop, the holds, the head update, the engine round trips, and the per-table size estimates every import pays for. One histogram carries all of it, sections and totals alike. They share a unit and a bucket set, and the query these exist for is `rate(..._sum[5m])`, seconds spent per second, which does not divide by an event count and so does not care that some sections are counted per block and others per arrival. Splitting them into separate metrics bought nothing that a `phase` label does not already give, and cost four more families to document, register and keep in step. `lean_block_import_cascade_blocks` is there for when the two counts do need relating, and because the distribution is the part the ratio of two `_count`s cannot give: a mean of 1.2 is a steady 1.2 or a constant 1 with rare bursts of 40, and those are different problems. `total` is written only when the block actually imported. A held block has no total, because its import has not finished, and publishing the span from the wire to wherever it stopped as one would put a two-slot parent wait into the import-cost percentiles. That is what an `outcome` label would have had to exist to filter back out, so there is no `outcome` label. `source` has two values. `Deferred` is not one of them: it says how a block reached the actor this time, not how it reached the node, and a deferred block is a gossip or sync block that waited. Labelling it separately would take it out of the population it belongs to for every section it has left to cross, while duplicating a fact the `defer` section already carries. `BlockArrival` therefore carries the source across the hold, and the handler resolves it before any timing is built. A block this node built itself is not measured at all: it crossed no wire, so its `decode` and `queue` sections are zeroes taken when the import began, and either wire source's percentiles would be understated by them. Both still appear in the log, which has a name for every case because a line costs nothing to write. Series are left to appear as blocks arrive rather than being seeded: a third of the phases are beacon-only, and a lean node can never write to them. A test asserts the tree's sections and the metric's per-block `phase` labels are the same list in the same order, so a section added to one without the other fails the build rather than logging something no dashboard can find. * fix(blockchain): stop charging a hold to the guards section as well `process_or_pend_block` marked `guards_start` with `.or_else`, so a block released from a parent hold kept the start its first pass had taken while `guards_end` was marked on the final pass. The section then spanned the whole wait: a block held eight seconds for its parent printed `parent_wait` of eight seconds and `guards` of eight seconds, and the tree's percentages summed past a hundred. Each pass now marks its own start, which is what the field always meant. Keeping the first mark had a reason, though not a good one: the mailbox handler had already taken it, and overwriting it would have dropped everything the handler does on a block's behalf before the import path sees it. That work is real and occasionally large, since it includes the store-clock catch-up, which can run a whole tick. So it becomes its own `admit` section rather than the head of the first pass's guards, and the guards section goes back to meaning one pass's checks. The regression test asserts what the defect violated: a held block's sections must not sum past its end-to-end span. * refactor(blockchain): drop three import sections that never moved `events`, `evict` and `metrics` each took under a millisecond in all 5248 imports a mainnet follower measured, so they were three rows that never moved in every tree printed and three label values on the phase histogram that never said anything. The work they covered still happens and is still counted, just charged to the section it follows rather than to one of its own: `absorb_tail` moves the end of whichever section ran last, which is `fc_head` on lean and `block_atts` on beacon. Extending the last section rather than the first keeps the rows contiguous, so the tree still sums to end-to-end. * fix(blockchain): scope the decode section to the path that decodes Only gossip decodes a block itself. The req/resp codec has already turned the bytes into a block before any handler sees one, so that path had no decode boundary left to take and reported a zero instead: `BlockArrival` stamped `wire` and `decoded` at the same instant after the fact. A zero is worse than nothing here, because it reads as free work rather than as unmeasured work and drags the decode histogram down with samples that measured nothing. `BlockArrival` now carries `decode_start: Option<Instant>` beside the hand-off, `None` saying the producer did not decode the block, and a fetched block's first section is `queue`. End to end therefore no longer starts at `decode_start`, which not every block has, but at `first_instant`, which falls back to the hand-off. The consequence to keep in mind when reading the numbers is that a gossip block's total and a fetched block's total no longer start at the same point in the block's life, and a fetched one's excludes the request round trip entirely; both are written down. * refactor(blockchain): drop a guards mark nothing could read `process_and_publish_block` is lean's locally-built path and calls `process_block` directly, bypassing `process_or_pend_block`, which is the only place `guards_start` is ever set. So this wrote an end whose start was always `None`, and `Row::new` yields no elapsed unless it has both, so the row never reached the tree or the histogram either way. Removing it also drops the `mut`, leaving the timings built and handed over in one expression.
* test(beacon): add mainnet and devnet network fixtures
* feat(types): accept the two scalar shapes an eth2 config uses
Integers appear quoted or bare depending on the generator, and versions are
hex with or without a prefix. Accepting one form only leaves the field at its
default without saying so.
* feat(types): read Config from an eth2 config.yaml
Missing keys fall back to mainnet rather than failing, which is what every
client surveyed does and what lets a configuration predating a field load.
* feat(types): give every runtime config key a typed home
Networking, deposit contract and custody values were left out because the state
transition never reads them. The spec endpoint does, and a key with no home is
reported as unknown, which would bury a typo among forty valid warnings.
* fix(storage): refuse a directory written before Config widened
Config is SSZ-encoded under KEY_CONFIG, so the added runtime keys change the
on-disk encoding and an old directory would decode into the wrong fields.
* feat(cli): parse a network config, naming the keys it ignores
One warning line rather than one per key: a current config carries the whole
gloas and heze schedule, and per-key lines would bury a typo among them.
Adds the five networking keys mainnet's own published config carries that the
struct still lacked, so a valid configuration reports nothing ignored.
* feat(cli): load a network from a directory of published files
The layout is what eth-clients publishes and kurtosis mounts, so a devnet's
artifact is read unmodified. A missing required file names its own path rather
than failing two layers later as something else.
* feat(cli): take a network name or directory on beacon
A slash is what tells the two apart, so a directory named mainnet cannot shadow
the built-in and a mistyped name says which names exist. A config declaring a
preset this build cannot serve is refused, not warned about: the preset sets
container bounds, so running anyway looks healthy and is not spec compliant.
* feat(beacon): derive the wire from a resolved network
The built-in constants become one arm of the same enum a loaded directory
fills, so mainnet's parameters cannot drift from what they were: everything
downstream reads the source rather than branching on where values came from.
* feat(beacon): anchor a loaded network at its own genesis
A fresh devnet has no checkpoint provider at slot 0, so a genesis anchor is
the only way to join one. The built-in network keeps refusing: mainnet's
genesis is 2020, and this follower would sit there claiming to follow a
live chain.
* fix(beacon): refuse a directory whose chain-defining config changed
A moved fork epoch leaves genesis time and the validators root identical while
putting this node on a different chain at that epoch, so the existing genesis
check cannot see it. Names the field rather than failing later as a decode
error.
* docs(beacon): describe the network flag and the genesis anchor
* docs(cli): say what --checkpoint-sync-url now means on beacon
A loaded network anchors at its own genesis, so the flag is required on a fresh
directory only for the built-in one.
* refactor(types): build an empty execution payload in one place
The test helpers predate BeaconBlockBody::empty and duplicated its payload
construction, which would drift.
* fix(cli): stop claiming beacon has no genesis-sync path
It has one now, for a network loaded with --network <dir>. Only the built-in
network still refuses, and the error should say which case the operator is in
rather than stating a blanket limitation that no longer holds.
* test(types): make the mainnet config.yaml drift tests prove they parse
Five tests read the mainnet config.yaml fixture and asserted values that
Config's own serde default (Config::mainnet()) already supplies, so
replacing the fixture text with an empty document left every one green.
Each now perturbs a field the fixture carries and checks the parsed value
follows the edit rather than the default, so a parser that silently
ignored the file would fail loudly instead of passing by coincidence.
* test(cli): cover a genesis state anchored past phase0
NetworkDir::load documents that a devnet's genesis.ssz decodes through
the fork its own config schedules at epoch 0, but the only fixture
(devnet) keeps mainnet's schedule, so that arm never ran and the fork
choice invariant genesis_anchor_block relies on (its reconstructed empty
body must hash to the state's own latest_block_header) had no coverage
past phase0 either.
Adds a devnet-electra fixture (every fork through electra scheduled at
epoch 0, fulu left unscheduled) plus:
- a NetworkDir::load test that decodes a real electra state, produced by
chaining the actual upgrade_state functions off mainnet's genesis,
against that fixture;
- a test pinning genesis_anchor_block's invariant at every fork, not only
phase0, re-stamping latest_block_header.body_root to each fork's own
empty body since an upgrade (unlike a block import) never touches it.
* refactor(cli): make first_config_difference exhaustive over Config
The hand-written field list compared fork versions/epochs, slot timing
and deposit identity, but skipped every other state-transition field
that changes which blocks are valid: the blob limits and churn/exit
parameters chief among them. Worse, the list was disconnected from
Config itself, so a field added there was silently never compared.
Destructures the full struct with no `..` instead: adding a field to
Config is now a compile error here until it is triaged into the
compare! list or explicitly bound to `_` with a comment saying why it
is operator tuning rather than chain identity. The comparison itself
gains max_blobs_per_block_{deneb,electra}, blob_schedule,
churn_limit_quotient, min/max_per_epoch_churn_limit(_electra),
consolidation_churn_limit_quotient, ejection_balance,
inactivity_score_{bias,recovery_rate}, min_validator_withdrawability_delay
and shard_committee_period.
* fix(cli): apply the minor review fixes (logging, validation, cleanup)
- beacon.rs: the cgc log named chain.custody_requirement, but the ENR
and Status both advertise the compiled-in CUSTODY_REQUIREMENT
constant regardless; a loaded network's differing config value would
make the log lie about what is actually on the wire. Reverted to the
constant, with a comment noting the config value is not yet honored
there.
- beacon.rs / main.rs: two logs said "mainnet" unconditionally
("Derived the mainnet wire parameters", "Resolved mainnet
configuration") even when following a loaded network. Reworded to be
network-agnostic and to name the resolved network via a new
NetworkSource::name() accessor.
- network/dir.rs: SECONDS_PER_SLOT: 0 loaded cleanly and then divided
by zero in epoch_at/milliseconds_per_interval. Rejected at load with
a new NetworkDirError variant naming the field.
- network/mod.rs: an absent PRESET_BASE and a real preset mismatch
produced the same PresetMismatch error, so a missing key rendered as
"declares PRESET_BASE: " with no indication anything was absent.
Split into a PresetCheckError with a distinct Missing variant.
- network/mod.rs: NetworkSpec::BuiltIn(name) resolved straight to
built_in_mainnet() regardless of name, so a second built-in network
would silently dispatch to mainnet. Guarded with a debug_assert
rather than inventing a registry for the one entry that exists today.
- network/mod.rs: dropped a test assertion that only restated
compiled_preset()'s own definition.
- network/config_file.rs: reworded a comment that quoted a warning
count (and the wrong one) rather than naming the mechanism.
* refactor(cli): drop serde_ignored from the network config parse
The crate was pulled in for one line: reporting the `config.yaml` keys no
`Config` field claimed, so a typo or an unprocessable fork gets a warning at
startup. That answer is a set difference, and both of its sides are already
available without a dependency.
The document's keys come from a third pass reading `BTreeMap<String,
IgnoredAny>`, which skips each value whole rather than typing it. That matters:
`TERMINAL_TOTAL_DIFFICULTY` and an unquoted `DEPOSIT_CONTRACT_ADDRESS` both
parse as integers wider than `u64`, which `serde_yaml_ng::Value` has no variant
for, so no untyped map of this document can be built at all. A new test pins
that, since it is also the reason the three passes cannot collapse into one.
The keys `Config` claims come from serde's derive itself: it hands its `FIELDS`
list to `Deserializer::deserialize_struct` and nowhere else, so `CaptureFields`
is a deserializer that exists only to receive that one call, copy the list out
and fail. Asking the derive rather than keeping a list here is what holds the
two in step, and it follows a rename to its wire name for free.
If `Config` ever gains a `#[serde(flatten)]` field the derive stops publishing
the list, the capture comes back empty and every key reads as ignored, which
`a_valid_mainnet_config_reports_nothing_ignored` fails on rather than degrading
quietly.
* fix(p2p): stop identify teaching the swarm peers' loopback addresses
`identify` is registered for interop only and its events are not handled, so it
read as inert. It is not. With an address cache the behaviour pushes every
address a peer reports in its `listenAddrs` to the swarm as
`NewExternalAddrOfPeer`, and `req_resp` dials whatever lands in the address
book. Peers binding the wildcard address report their loopback addresses too,
so this node learned to dial `127.0.0.1` for them instead of the signed ENR
address `dial_addrs` supplies.
Where a peer's advertised QUIC port matches `--gossipsub-port`, that dial
reaches this node's own listener, and libp2p reports it from both ends at once:
`Local peer ID` on the listener, `Unexpected peer ID` on the dialer. Against
lighthouse on a kurtosis devnet, with both sides on the default port, the
follower sat at slot 0 with no peers while the dial loop churned at full rate.
With the cache off it range-syncs from genesis and reaches the tip.
Off is what lighthouse does, for the same reason. Its other defence, a discv5
table filter rejecting non-global addresses, is deliberately not copied: that
one rejects private ranges as well as loopback, and would leave this node
nothing to dial on any devnet whose peers sit behind RFC 1918 addresses.
A live beacon-follower profile shows one thread pinned at 100% of a core while the other 15 sit idle, with 12.5% of total cycles spent in is_valid_indexed_attestation, almost all of it in blst's mulx_mont routines: the per-key subgroup check inside aggregate_verify and fast_aggregate_verify. An Electra block can carry thousands of attesters behind one aggregate, and the state transition only ever runs on a single actor thread, so that loop was leaving a mostly-idle host's other cores untouched on every block import. Each key's validation is independent of every other key's, so nothing about the result depends on running them in order; rayon's par_iter spreads the checks across its pool and still collapses to false the moment any key fails, exactly like the sequential loop's early return did, without skipping or weakening key_validate for a single key. with_min_len keeps small inputs (a devnet or a spec fixture) on one thread instead of paying dispatch overhead for a handful of keys.
…#23) * perf(beacon): rank dial candidates by custody redundancy, not presence A mainnet follower fell ~100 slots behind and could not recover. lean_custody_column_peers showed only 5-8 of 129 connected peers custodying each sampled column, and 78% of its by-root column requests went unanswered (81,073 requests against 17,785 response chunks) once gossip could no longer carry it: off the tip, by-root fetch is the only source, and lean_data_column_fetch_failures_total{reason="max_retries"} hit 5,566 in ~50 minutes, spread evenly across peers rather than one bad one, which points at supply rather than a bad peer. covered_custody_columns collected custodians into a HashSet<u64>, so a single custodian marked a column covered, and uncovered_custody_columns only ever surfaced columns with zero custodians. At 5-8 custodians that set is empty, custody stops contributing to rank_candidates entirely, and dialing optimizes purely for attestation-subnet coverage while columns are the actual binding constraint. Replace presence with a per-column custodian count, and compare it against a redundancy target instead of a zero/nonzero check. The target is tied to MAX_FETCH_RETRIES: handle_column_fetch_failure tracks failed_peers so each retry round wants a custodian it has not already asked, and a column with fewer custodians than the ladder has rounds runs out of fresh peers before it runs out of retries. rank_candidates keeps its existing signature; only how the "wanted" columns handed to it are computed changes. * docs(beacon): describe rank_candidates' custody bar as redundancy, not presence rank_candidates' doc comment still described the "wanted" columns as those no connected peer holds at all, which was true before this branch but is stale now: the caller hands it every column below CUSTODY_REDUNDANCY_TARGET custodians, a much wider set than zero. Rewrite the doc to state what the parameter means today and why thin-but-nonzero supply belongs in it, not just the zero case: a by-root lookup's retry ladder wants a custodian it has not already asked each round, so a column with fewer custodians than the ladder has rounds exhausts its retries re-asking peers that already failed it. That is the failure the measured 5-8-custodians / 78%-unanswered mainnet case actually hit, not merely a column with zero custodians. The zero-custodian case and the supernode-sorts-first observation are kept as the extreme case and still-true observation they are. Also rename covered_custody_columns to custodian_counts_by_column: it has returned a HashMap<u64, usize> since the redundancy-count change, and the old name still reads like the HashSet<u64> presence check it used to be. No behavior change: docs and one function rename only.
* feat(types): write beacon scalars the way the Beacon API reads them
* refactor(types): serialize beacon scalars without allocating
* docs(types): correct the hex_array allocation rationale
The paragraph was adapted from quoted_or_bare's and kept its example: no
Validator field routes through hex_array, so the per-validator cost it
claimed does not exist. The fixed-width byte types it serves appear a
handful of times per state.
* feat(types): add the collection and bitfield serde adapters
* refactor(types): make a mis-annotated ssz_hex field a compile error
The adapter took any SszEncode, so annotating a container field with it
would compile and emit an opaque hex blob where a JSON object belongs.
The guard test planned for the container rollout walks for bare numbers
and cannot see that, so nothing else would catch it across 84 hand
annotations. Bound it to the bitfield and byte-list families instead,
and pin the bitfield encodings the tests only sampled by prefix.
* feat(types): serialize the beacon byte newtypes and uint256
Gives the five byte newtypes and U256 their own Serialize, so container
fields holding them need no attribute at all in the annotation pass.
U256 writes decimal, not hex, which the Beacon API requires and which no
primitive integer can hold: it goes through long division over the
big-endian bytes.
* feat(types): serialize the shared and phase0 containers as JSON
Annotates the first 28 of the 84 containers and adds the guard test the
rest of the pass is verified by: it walks a serialized block and fails on
any bare JSON number, naming the field path, since a forgotten attribute
is valid JSON and silently wrong.
Drops the SszHexEncodable marker trait added alongside ssz_hex: the
adapter takes any SszEncode again, so an ssz_hex annotation on a
non-bitfield field is a hand-checked invariant rather than a compile
error.
* test(types): make the interior-zero-byte case actually interior
The test set only byte 17, so every nonzero byte sat at one end and the
docstring's claim of nonzero bytes above and below the zero run was not
true of the value being asserted. Set the lowest byte too, so sixteen
zero bytes sit between two nonzero ones and the carry genuinely has to
cross them.
* feat(types): serialize the altair containers as JSON
Altair's additions need the three adapters told apart carefully: the
sync committee bits are a bitvector and go out as hex, the participation
lists are ParticipationFlags, a u8 alias, and go out as quoted integers,
and the committee pubkeys are a vector of a type that serializes itself.
* test(types): make the JSON guard reach what it claimed to
The guard built its block from BeaconBlockBody::default(), so every list
serialized as [] and the walker never descended into an element type, and
no test constructed a BeaconState at all. It covered 4 of 28 structs.
Populates every list with real elements, one level into nested lists, and
adds a state test per fork, taking the reach to 20/28 for phase0+shared
and 6/11 for altair; the rest are gossip-only types no block or state can
contain. Also adds a test that the walker fails on an unannotated integer,
so a green suite cannot be confused with a walker that matches nothing.
* feat(types): serialize the bellatrix containers as JSON
Bellatrix brings the execution payload, and with it the first fields the
adapter table does not answer on its own. extra_data and logs_bloom are
byte strings that must go out as one hex string each, though both would
also compile as arrays of numbers. transactions is a sequence of byte
lists, which no single adapter produces: it composes ssz_hex per element
through seq, locally, until a second fork needs the same shape.
* feat(types): serialize capella, and share the byte-string sequence adapter
Capella imports bellatrix's own Transactions type, so the composition
bellatrix kept private became a second call site. Promotes it to
serde_helpers as ssz_hex_seq and repoints bellatrix at it, which is a
net 44 lines out of that file.
The bound is Deref<Target: SszEncode> rather than SszEncode: serde hands
serialize_with a reference, and unlike Display and Serialize, SszEncode
has no blanket impl for &T.
* feat(types): serialize the deneb containers as JSON
The blob commitments are the field worth naming: a sequence of
KzgCommitment, which is a byte newtype that already writes itself as hex,
so it takes seq rather than ssz_hex_seq. Both spellings emit an array of
hex strings, so the guard test cannot tell them apart and the choice has
to come from the element type.
* feat(types): serialize the electra and fulu containers as JSON
Completes the annotation pass: all 84 containers now derive Serialize.
Fulu is where both sequence-of-hex spellings appear side by side. The
data column is a list of Cell, a foreign byte vector with no Serialize
of its own, so it takes ssz_hex_seq; the commitments and proofs beside
it are byte newtypes that write themselves, so they take seq. A single
Cell, as in MatrixEntry, is neither and takes ssz_hex.
Electra turns out to import its execution payload from deneb, and fulu
has no block types at all, so their fixtures reuse rather than
duplicate.
* feat(types): serialize lean signed blocks as JSON
Lets /lean/v0/blocks/finalized answer JSON when a caller asks for it.
Integers stay bare: that is lean's own encoding and the rpc crate's
tests assert it. The proof blob takes the 0x-prefixed convention the
roots and bitlists use, rather than the prefix-less one the fixed-width
XMSS pubkeys use, since it is opaque variable-length data and not an
identity.
* feat(types): serialize the container enums
Neither enum adds a variant tag: the Beacon API carries the fork in the
response envelope's version field and its Eth-Consensus-Version header,
never inside the data object.
The two get there differently, and the asymmetry is the point. The block
enum derives untagged, because its Lean variant really is served as JSON
on /lean/v0/blocks/finalized. The state enum cannot: a derive would
demand a JSON encoding for lean's State, which is deliberately SSZ-only,
so it is hand-written and the lean variant returns a serde error instead.
* feat(rpc): negotiate response encoding on Accept
The Beacon API serves JSON unless the caller asks for
application/octet-stream, which is the order lighthouse serves too. The
lean surface defaults the other way on its two SSZ endpoints, so the
default stays the caller's to pass rather than a constant here.
Ranks the header's entries by their q weight: a client that accepts both
states its preference that way, and ignoring it would serve SSZ to a
caller who merely tolerates it.
base.rs's private ssz_response moves here rather than being duplicated,
since both surfaces now need it.
* feat(types): report a block's body root on either chain
/eth/v1/beacon/headers/{id} answers with a SignedBeaconBlockHeader,
whose body_root is the one field the existing accessors cannot produce:
signed_beacon_block_accessors! hands back a field verbatim, and every
fork stores a different body container, so what the caller wants is the
merkle root of whichever one this is.
Dispatched including lean, since lean's Block has a body too and
/lean/v0 wants the same number.
The phase0 test computes the expected root through the raw libssz_merkle
trait rather than this crate's convenience wrapper, so a body_root that
only worked on the lean arm would fail rather than pass by
construction.
* feat(rpc): parse and resolve beacon block ids
Parsing is chain-agnostic, resolution is not. resolve_beacon reads only
accessors that answer on either chain: Store::head_state and head_slot
panic on a beacon store, so the lean slot lookup stays where it is
rather than being repointed at BlockRoots, which covers the branch
ending at the head rather than everything a lean store was bootstrapped
with.
genesis resolves to the store's anchor rather than to slot 0: a
checkpoint-synced directory has no genesis block, and its anchor is the
earliest block it can answer for at all.
* feat(types): serialize the beacon chain config as JSON
/eth/v1/config/spec echoes the chain configuration in the Beacon API's
encoding: SCREAMING_SNAKE_CASE keys, every integer a quoted decimal,
byte strings 0x-prefixed hex. Config could only be deserialized, which
is how --network <dir> loads a network.
74 fields move from deserialize_with = "m::deserialize" to with = "m",
which resolves to the same function on read, so the existing YAML
parsing tests staying green is the evidence that nothing about loading
a network moved.
Six fields keep their own arrangement: genesis_time stays skipped since
it is not a config.yaml key and /eth/v1/beacon/genesis reports it;
terminal_total_difficulty and terminal_block_hash already have
Serialize on their own types; max_blobs_per_block_deneb keeps its
rename; and blob_schedule pairs its hand-written deserializer with
seq::serialize, since its two halves do not live in one module.
* feat(rpc): add the beacon API envelope and error shape
Every fork-versioned Beacon API payload travels in
{version, execution_optimistic, finalized, data}. The fork sits in the
envelope rather than in the data because the containers serialize
untagged, so version is how a caller knows which shape it just parsed.
The error body is {code, message}, deliberately not the lean surface's
{error}. They are two different APIs and each consumer parses the shape
its own specification names, so unifying them would break one of the
two.
The submodule declarations and routes() land with the files they name,
rather than being written now and commented out.
* feat(rpc): serve beacon blocks by id in JSON and SSZ
/eth/v2/beacon/blocks/{block_id} and its /root sibling, JSON by default
and SSZ on Accept: application/octet-stream, both tagged with
Eth-Consensus-Version so an SSZ caller can tell which fork's container
it received.
genesis is now refused rather than resolved to the anchor slot.
Table::BlockRoots is written only by update_checkpoints, which computes
its delta by walking from the old head to the new one and returns early
when they are the same root. At bootstrap they always are, so the
anchor's own slot is never indexed and the previous spelling could only
ever 404. Saying so is better than a lookup that silently never works.
The fixture moves its head through update_checkpoints rather than
poking the backend, so BlockRoots is populated by its real writer and
the slot-resolution tests exercise the path a running node takes.
get_block_root loads the block rather than only resolving the id: it
needs the slot to answer finalized honestly instead of hardcoding it.
* feat(rpc): serve beacon block headers
/eth/v1/beacon/headers/{block_id}. Every field of the message comes
from SignedBeaconBlock's accessors, which dispatch including lean, plus
body_root(); only the envelope and the signature are beacon-specific,
which is what makes this the one endpoint genuinely shared with the
lean surface.
canonical is asked of BlockRoots rather than assumed, even for a block
reached by its own root, so a sibling or a block at a slot the index
does not cover answers false instead of being taken on trust. The test
pins both directions: the head is canonical, the anchor is not, since
its slot is never indexed.
The quoting comes from ethlambda_types::beacon::serde_helpers rather
than a to_string() per field, so the rule lives in one place.
* feat(rpc): serve beacon states and finality checkpoints
/eth/v2/debug/beacon/states/{state_id} is the path checkpoint_sync.rs
already fetches from other clients, so serving it makes this client
checkpoint-syncable from itself. The SSZ test decodes the body back
through slot_from_ssz, since round-tripping is the whole job.
A 0x... state_id is refused with a message naming the ids that do
work. States are keyed by block root here with no reverse index, so
the alternative to refusing is answering with the wrong state, or
being right only when the two roots happen to coincide.
finality_checkpoints reports the state's own three checkpoints rather
than the store's fork-choice view: the endpoint is defined as a read
of the state the id names, and the state is the only place a previous
justified checkpoint is kept at all.
JSON serializes through the Arc rather than cloning, since a mainnet
state runs to hundreds of megabytes.
* feat(rpc): serve beacon genesis and the spec config
/eth/v1/beacon/genesis reads genesis_validators_root off the finalized
anchor's state rather than looking for slot 0: it is a property of the
chain that every state carries, and a checkpoint-synced directory has
no genesis block to read it from.
/eth/v1/config/spec echoes the Config the store was bootstrapped with,
so a node started with --network <dir> reports that network's constants
rather than mainnet's. GENESIS_TIME is absent from it by design, since
it is not a config.yaml key and the genesis endpoint reports it.
* feat(rpc): serve the beacon node endpoints
syncing reads beacon_head rather than head_slot, which is lean-only and
panics on a beacon store, and computes the wall slot in milliseconds the
way /lean/v0/node/syncing does: slot_duration_ms is the value a loaded
network can actually change, and seconds_per_slot would truncate a
sub-second cadence to zero.
health is 206 while syncing and 200 once caught up. 503 would mean
uninitialized, and a store that answers at all is initialized.
identity reports peer_id and metadata only. enr, p2p_addresses and
discovery_addresses are empty, which is not spec-valid: the ENR is
built for discv5 and owned by the P2P actor, and BuiltSwarm hands
run_node only a local_peer_id, so serving the record means widening
the ethlambda-p2p surface. Deliberate and recorded as a deviation.
* feat(rpc): serve the beacon API from a beacon node
Which HTTP surface a node serves now follows from the store's own chain
tag rather than from the sub-command, so the two cannot disagree.
Until now a beacon node bound --api-port to the lean router off a
beacon store. Those handlers reach for lean state variants and metadata
keys a beacon directory never carries, so calling one panicked that
request. The beacon router replaces it rather than extending it, and
the integration test pins both halves: /eth/v1/node/version answers,
/lean/v0/node/syncing is a 404.
build_beacon_api_router does not merge the metrics router, for the same
reason build_api_router does not: start_http_servers serves it, and
merging a path twice makes axum panic at startup.
start_beacon_rpc_server takes no AggregatorController and no EventBus.
A follower has no aggregator duty to toggle, and the chain-events
stream is part of the lean surface.
* feat(rpc): serve the lean finalized block as JSON on request
SSZ stays the default here, which is the opposite of the beacon
surface and deliberately so: checkpoint_sync.rs reads these bytes, and
other clients' lean checkpoint sync may send no Accept header at all.
Moving the default would break every one of them, so JSON is opt-in.
Not routed through Encoding::from_accept, which defaults to JSON. The
asymmetry is the point, so the branch is spelled out here rather than
hidden behind a shared name that means the other thing.
The existing SSZ test already pinned the default; the new test pins
JSON on request, and that lean integers stay bare rather than picking
up the beacon surface's quoting.
* docs: describe the beacon API surface and its deviations
docs/rpc.md gains the endpoint table, the encoding rules (JSON by
default, SSZ on Accept, quoted integers, Eth-Consensus-Version) and the
three ids that are refused.
docs/spec_deviations.md records two: the partial node identity, and the
ids that cannot be named. The second is a property of Table::BlockRoots
rather than a choice, so it is written down where the reason lives.
CLAUDE.md said a /lean/v0 call on a beacon node panics the request and
that giving the follower its own surface was still a change of its own.
Both stopped being true with this branch.
* fix(rpc): resolve the anchor's own slot, which checkpoint sync asks for
Found by running a mainnet follower and pointing a second one at its
API: the second died with "peer served no block at the anchor slot
15265888" and never started.
checkpoint_sync.rs reads a peer's finalized state, takes the anchor
slot from it, and asks that peer for /eth/v2/beacon/blocks/{slot}.
That is exactly the one slot Table::BlockRoots never holds, because
update_checkpoints is its only writer and writes nothing when the head
does not move, which is the situation at bootstrap. So the endpoint
404'd and this node was unusable as a checkpoint-sync source for any
client, not only for itself.
On an index miss the lookup now tries the roots the store can name and
accepts one only when block_entry confirms its block really sits at
the slot asked for, so a slot the store holds nothing at still 404s
and no request can be answered with the nearest checkpoint instead.
The test that pinned the old 404 asserted a bug, so it now asserts the
anchor resolves, with a sibling holding the line on slots below it.
The docs claimed a slot at or below the anchor was refused; only
genesis and a state-root id are.
* chore: drop the beacon API spec and plans from the tree
These are working notes for building the beacon API, not documentation
of it: a design sketch and two task lists written while the endpoints
were being implemented. They went in with the last fix by accident.
What they describe is already in the tree in a form that stays true as
the code changes: docs/rpc.md for the endpoints, docs/spec_deviations.md
for where we depart from the Beacon API spec. A checklist of tasks that
are now done has nothing left to tell a reader, and would only rot.
The files stay on disk, untracked, for as long as they are still useful
to the people writing the remaining endpoints.
* feat(beacon): add sepolia and hoodi as built-in networks `--network` only knew `mainnet`, so following a public testnet meant checking out its eth-clients repo, fetching Hoodi's 150 MB genesis state through git-lfs, and pointing `--network` at the directory. Both testnets now resolve by name. Each embeds its eth-clients `config.yaml` and `bootstrap_nodes.yaml` unchanged, parsed through the same reader a `--network <dir>` uses, plus `genesis_time` and `genesis_validators_root` as constants. Neither carries its genesis state: a built-in network never anchors at genesis, so those two values are all the state was ever read for. The constants are pinned by tests against fork digests published in each network's own bootnode ENRs, and at runtime checkpoint sync and resume already reject an anchor state whose genesis values disagree. `NetworkSource::BuiltInMainnet` becomes `BuiltIn`, dispatching on a `BuiltInNetwork`. The debug assertion that guarded against a second name silently resolving to mainnet goes away, because each name now has its own dispatch arm and a test checks they resolve to distinct chains. Sepolia's config schedules gloas at epoch 353024, which this build cannot process, so a Sepolia follower stops tracking the chain at that epoch. The ignored-keys warning at startup names it. * refactor(beacon): describe mainnet the way sepolia and hoodi are described Mainnet was the odd one out among the built-in networks: a hardcoded `Config::mainnet()`, a Rust array of bootnode ENRs, and a 5.4 MB genesis state decoded at every startup to read two values off it. Every built-in chain is now one `EmbeddedChain`: its eth-clients `config.yaml` and `bootstrap_nodes.yaml` unchanged, plus `genesis_time` and `genesis_validators_root` as constants. The genesis state leaves the binary (5.4 MB smaller) and becomes a test fixture, because the tests still need a real phase0 state and it is now what the mainnet constants are checked against. The config fixture moves the other way, into `assets/`, since it was already byte-identical to eth-clients' current file. A test pins the parsed config to `Config::mainnet()`, so the file and the Rust constant cannot drift. Taking eth-clients' bootnode file also refreshes the list: upstream replaced four EF ENRs with five newer ones since `MAINNET_BOOTNODES` was copied. * docs(readme): describe the beacon follower as it is now The section still described the first cut: a gossip logger that "keeps no chain, imports nothing" and needs no flags. The follower now anchors at a checkpoint, imports through fork choice, custodies data columns and serves the Beacon API, and a fresh data directory on a built-in network cannot start without --checkpoint-sync-url. The sample output is taken from a live Sepolia run.
The smoke step sat in `Test (node)` on the assumption that the test build had already linked the binary. It had not: `cargo test` never links the plain `ethlambda` binary, and it unifies dev-dependency features in (`bin/ethlambda` enables tokio's `test-util`), which `cargo run` leaves out. tokio is the only crate whose features differ, but everything above it is rebuilt with it: libp2p, hyper, axum, the ethrex crates and the node crates, six minutes of compilation into a target dir the test build leaves at about 1.8 GB free. That pushes the runner to a full disk, so whether `Test (node)` passes comes down to whether the rust-cache post step still finds room to write its log. It fails with `No space left on device` after every test has passed, as on PR #24 and feat/validator-client. A dedicated job builds only the binary's own graph on a fresh runner. It needs no leanSpec fixtures: with `--mock-crypto` the synthetic benchmark generates its own chain and reads no files.
) On a mainnet follower, a block's `stf`, `block_atts`, `queue` and `total` all sit between half a second and a few tens of seconds, which is where `lean_block_import_phase_seconds` doubled its edges and then jumped 4x from 16 to 64 s. Comparing a follower that writes states on its chain actor against one that hands them to a background writer, the mean `stf` fell 16% (4.41 s to 3.69 s), yet both populations sat in the same 2-4 s and 4-8 s buckets, so every percentile the dashboard drew for it was an interpolation. Above 0.5 s the edges now step by 1.5x and 1.33x (0.75, 1, 1.5, 2, 3, 4, 6, 8, 12, 16, 32), which also puts an edge on one mainnet slot. The lowest edge is now 1 ms and the highest 32 s; longer sections land in `+Inf`.
* feat(p2p): count why peers drop us, not just that they did
`lean_peer_disconnection_events_total` is leanMetrics-specified down to its
four `reason` values, and they cannot carry this question: on the mainnet
follower 92% of outbound closes land in `error`, which says only that libp2p
handed back a cause. The reason a peer actually gave was thrown away twice
over, once by bucketing the `ConnectionClosed` cause to a string test and
discarding the cause itself, and once by logging the `goodbye/1` code at
`trace!` on a node that runs at `INFO`.
That mattered because the two readings a follower most needs to tell apart are
identical from the socket alone. A peer at its inbound cap and a peer that has
scored us badly or banned us both just close, and the response to each is the
opposite of the response to the other.
Two counters beside the specified one rather than more values inside it, the
same shape `lean_peer_connections_by_transport_total` already uses to avoid
putting ethlambda off-spec:
- `lean_peer_goodbye_total{reason}`, the only signal here that is not an
inference. No `direction` label, since `goodbye/1` is registered inbound-only
and this node never sends one.
- `lean_peer_disconnect_cause_total{direction,cause}`, read off the
`ConnectionError` variant and an I/O error's `ErrorKind` rather than sniffed
out of a `Display` string. Charged on the same event as the specified
counter, so the two total alike and can be read against each other.
Both label sets are closed, with `other`/`io_other` as the residue: the goodbye
code is a `u64` off the wire, so a remote must not be able to choose this
node's metric cardinality. Only reason codes 1, 2 and 3 are the spec's; it
reserves [4, 127] and leaves 128 up to each client, so the four above 127
follow lighthouse's `GoodbyeReason`, which is what mainnet peers send.
The residue cannot be empty, because a QUIC application close arrives as an
opaque `io::Error`. The close now logs its cause at `DEBUG` so that message is
recoverable, which is the only place it exists.
* fix(p2p): read a close's cause through the muxer error that carries it
Measured on the mainnet follower, `lean_peer_disconnect_cause_total` put every
single close in `io_other`. The label read the outer `io::ErrorKind`, and
`StreamMuxerBox` wraps every muxer error in `io::Error::other`, so the kind is
`Other` for nearly every close and the reason sits one level down. Even a plain
TCP reset arrived that way, as mplex's `ConnectionReset` inside an `Other`.
The inner error is now downcast to the two types it can be, since the swarm
builder boxes each transport on its own:
- `libp2p::quic::Error`. Its `Connection` variant keeps quinn's error in a
private field and forwards only `Display`, so that one is read off the text.
It is safe to match because quinn writes a fixed prefix of its own ("closed
by peer: ", "aborted by peer: ") ahead of anything the peer supplies, so a
close reason cannot pass for another variant; a test pins that case.
- `Either<yamux::Error, io::Error>`, the TCP muxer selection. mplex reports
plain I/O errors. yamux hides its variants but forwards `source` to the I/O
error under an `Io` or `Decode` failure, so the chain is walked for a kind
before falling back to a clean `yamux_closed`.
`either` becomes a direct dependency only to name that type: libp2p-core
builds its muxer error from it and does not re-export it. It is already in the
lockfile through libp2p, so nothing new is fetched.
A mainnet follower spent 23.5% of its CPU deriving the same committees over and over. An Electra attestation names up to MAX_COMMITTEES_PER_SLOT committees, and every one of them was derived from scratch: a scan of the 2.4M-entry validator registry, then a SHUFFLE_ROUND_COUNT-round shuffle *per member* of the committee. A block carries up to MAX_ATTESTATIONS_ELECTRA of those, `process_attestation` walks the committees a second time to check the aggregation-bit lengths, and fork choice walks them a third time when it replays the block's attestations into the latest-message store. Nothing was shared between any of it. Two changes make it one derivation per epoch: `compute_shuffled_indices` computes a whole epoch's permutation in one pass, sharing each round's pivot and window hashes across every position instead of rehashing them per position. `EpochCommittees` stores the active set through that permutation, so a committee becomes a slice of it: no hashing and no allocation per lookup. Ported from feat/mainnet-network, which wrote the same function against a per-(seed, index_count) memo. `CommitteeCache` then keeps those shufflings across calls, keyed by epoch and the block root that last could have changed them: the last slot of epoch E - 2, which is lighthouse's AttestationShufflingId. An epoch's committees depend only on its active set and its seed, the seed is fixed once E - 2 ends, and every write to activation_epoch or exit_epoch lands at least MAX_SEED_LOOKAHEAD epochs out, so two states agreeing on that root agree on the committees whatever else they disagree on. That is what makes sharing sound where a key of epoch or seed alone would not be, since sibling branches can share both while disagreeing on who is active. The cache is owned by the chain actor and threaded down through fork choice and the state transition, rather than kept in a global: how many shufflings are worth keeping resident, at ~19 MB each on mainnet, is the owner's decision and not something a leaf helper can answer. A caller with no cache of its own passes a fresh one and gets exactly the old behaviour, which is what every fixture runner does. Per attestation at mainnet's ~2.4M validators, naming a full slot's 64 committees: 6.75M hashes and 64 registry scans before, 844k hashes and one scan on a cold epoch, none of either on a warm one. `get_beacon_committee` keeps its signature and still builds a fresh `EpochCommittees` per call, so a single-committee caller is unaffected.
* refactor(storage): split the byte-producing half out of insert_state The snapshot-vs-diff decision and the encoding it drives had no name and no test of their own: the only way to assert which table a state lands in was to insert it and read the table back. Extracting is_anchor and the two plan_* functions gives that rule a unit test with no store, no backend and no sequencing, and is the seam a background writer needs. * refactor(storage): give the tagged state codecs a module of their own store.rs and state_writer.rs had begun importing from each other, and the read path landing next would have added five more edges in one direction. The three encode/decode_state_value functions are what both sides actually share, and they depend on nothing in either: BeaconState, ForkName and ssz. * refactor(storage): give state reads one path and a handoff buffer get_state's body moves to a free function over the backend, the cache and a new PendingStates buffer, so something other than a Store can perform a state read: the writer thread needs the parent lookup, and holding a Store there would be a reference cycle through the very Arc that is supposed to join it. PendingStates is inert while writes are synchronous. It exists for the next commit, where a state becomes readable before it is written, and the LRU cannot serve that role because it may evict an entry whose write has not happened yet. * perf(storage): write states on a thread of the Store's own insert_state encoded a whole mainnet BeaconState, diffed it against its parent's bytes and committed, all on the importer's thread. The 2026-09-21 import profile showed that thread saturated with fifteen cores idle. One worker, not a pool: a state's delta is computed against its parent's encoded bytes, so a single thread draining a FIFO is what keeps that chain in order, and it lets the parent memo become thread-local instead of shared. The handoff is PendingStates rather than the LRU because the LRU may evict an entry whose write has not happened yet, which would leave a reader unable to see a state insert_state has already accepted. An entry leaves the buffer only after its commit returns, so no instant has neither answer. The queue holds two states and a full queue blocks the sender, which is what the write already did before it moved threads. * fix(storage): repair a head that outran the state writer on resume crates/blockchain/src/store.rs imports a block, calls insert_state (which since the previous commit only enqueues), then update_head commits KEY_HEAD synchronously. The head pointer, and the canonical BlockRoots index that moves with it, can therefore reach disk before the state they name does: a window as wide as the writer's queue. Before that commit the state write was synchronous and always preceded the head write, so a persisted head implied a persisted state; nothing enforced that invariant explicitly, and an unclean shutdown inside the window left a head naming a state that was never written -- discovered downstream as a panic or a hard failure, not a diagnosed problem. Store::repair_head walks back from the recorded head to the newest ancestor with a persisted state and rewinds there if the head moved, deleting the LiveChain rows of every block it hops over. That deletion is what makes the rewind stick: LiveChain is already this codebase's encoding of "invisible to fork choice" (see insert_pending_block's doc), so without it the very next fork-choice run would walk right back to the stateless tip through its still -live row. Bringing a stateless block back is machinery this codebase already has for other reasons: on_block_core's duplicate check is keyed on has_state rather than the block being on record, range sync starts at head_slot + 1, and the pending-block walk pulls a stateless ancestor back out of storage on its own. The rewind only has to make a block look stateless again for all three of those to take over. The walk is bounded three ways: STATE_WRITE_QUEUE_CAPACITY + 1 blocks (past that it is not this race but corruption, reported as HeadRepairExceededWindow), anchor_slot (below it there was never anything to fall back to; reported as UnexpectedMissingState naming the head an operator actually saw, not an unfamiliar ancestor), and finalized's slot (LiveChain is already pruned below there, so rewinding that far would be worse than the state it is missing; reported as AnchorStateLost). A head naming no block at all is legitimate on a beacon directory mid checkpoint-sync (init_beacon seeds KEY_HEAD before the anchor block/state pair that follows it), so repair_head leaves that alone rather than reporting corruption for a shape from_db_state has always tolerated; the same shape on a lean directory is corruption, since init_store always writes the head's own block synchronously. Justified and finalized are never rewound the same way: they are consensus statements, and inventing an earlier one to paper over a missing state is not something a storage-layer repair may do -- the beacon anchor's own state insertion enqueues like any other now, so an unclean shutdown can catch it too. Store::verify_anchor_states checks both instead, and a resuming caller (fetch_initial_state, fetch_initial_beacon_state) treats a failure the same way it already treats a stale DB: fall back to checkpoint sync if a URL is configured, or fail naming the remedy if not. from_db_state stays read-only, matching its own documented contract: it no longer calls repair_head itself. A resuming caller now runs, in order, the chain check, verify_anchor_states (falling through to checkpoint sync on failure), genesis verification (reusing the state verify_anchor_states already fetched), and only then repair_head -- the one mutation on this path, reached only once both checks have passed. Tests: from_db_state_preserves_block_root_index reverts to its pre-existing form now that from_db_state is read-only again. from_db_state_loads_a_beacon _directory_as_beacon was already covering the beacon early-return case incidentally; it now calls repair_head explicitly and asserts on it, pinning that case on purpose rather than by accident. New: a three-hop rewind with its LiveChain deletions checked directly, the exact STATE_WRITE_QUEUE_CAPACITY + 1 boundary on both sides, a mid-walk broken- parent-chain case, and repair_head exercised on a real beacon chain rather than just init_beacon's bare bootstrap. Also fixes the lean_state_write_queue_depth gauge, which read-then-set from two different threads and could latch one entry too high and stay there; it now increments and decrements atomically instead. And keeps the import report's db_write label as it was -- a live Prometheus label existing dashboards filter on -- fixing only its doc to say what the row covers now that insert_state enqueues rather than commits.
…25) * feat(types): write beacon scalars the way the Beacon API reads them * refactor(types): serialize beacon scalars without allocating * docs(types): correct the hex_array allocation rationale The paragraph was adapted from quoted_or_bare's and kept its example: no Validator field routes through hex_array, so the per-validator cost it claimed does not exist. The fixed-width byte types it serves appear a handful of times per state. * feat(types): add the collection and bitfield serde adapters * refactor(types): make a mis-annotated ssz_hex field a compile error The adapter took any SszEncode, so annotating a container field with it would compile and emit an opaque hex blob where a JSON object belongs. The guard test planned for the container rollout walks for bare numbers and cannot see that, so nothing else would catch it across 84 hand annotations. Bound it to the bitfield and byte-list families instead, and pin the bitfield encodings the tests only sampled by prefix. * feat(types): serialize the beacon byte newtypes and uint256 Gives the five byte newtypes and U256 their own Serialize, so container fields holding them need no attribute at all in the annotation pass. U256 writes decimal, not hex, which the Beacon API requires and which no primitive integer can hold: it goes through long division over the big-endian bytes. * feat(types): serialize the shared and phase0 containers as JSON Annotates the first 28 of the 84 containers and adds the guard test the rest of the pass is verified by: it walks a serialized block and fails on any bare JSON number, naming the field path, since a forgotten attribute is valid JSON and silently wrong. Drops the SszHexEncodable marker trait added alongside ssz_hex: the adapter takes any SszEncode again, so an ssz_hex annotation on a non-bitfield field is a hand-checked invariant rather than a compile error. * test(types): make the interior-zero-byte case actually interior The test set only byte 17, so every nonzero byte sat at one end and the docstring's claim of nonzero bytes above and below the zero run was not true of the value being asserted. Set the lowest byte too, so sixteen zero bytes sit between two nonzero ones and the carry genuinely has to cross them. * feat(types): serialize the altair containers as JSON Altair's additions need the three adapters told apart carefully: the sync committee bits are a bitvector and go out as hex, the participation lists are ParticipationFlags, a u8 alias, and go out as quoted integers, and the committee pubkeys are a vector of a type that serializes itself. * test(types): make the JSON guard reach what it claimed to The guard built its block from BeaconBlockBody::default(), so every list serialized as [] and the walker never descended into an element type, and no test constructed a BeaconState at all. It covered 4 of 28 structs. Populates every list with real elements, one level into nested lists, and adds a state test per fork, taking the reach to 20/28 for phase0+shared and 6/11 for altair; the rest are gossip-only types no block or state can contain. Also adds a test that the walker fails on an unannotated integer, so a green suite cannot be confused with a walker that matches nothing. * feat(types): serialize the bellatrix containers as JSON Bellatrix brings the execution payload, and with it the first fields the adapter table does not answer on its own. extra_data and logs_bloom are byte strings that must go out as one hex string each, though both would also compile as arrays of numbers. transactions is a sequence of byte lists, which no single adapter produces: it composes ssz_hex per element through seq, locally, until a second fork needs the same shape. * feat(types): serialize capella, and share the byte-string sequence adapter Capella imports bellatrix's own Transactions type, so the composition bellatrix kept private became a second call site. Promotes it to serde_helpers as ssz_hex_seq and repoints bellatrix at it, which is a net 44 lines out of that file. The bound is Deref<Target: SszEncode> rather than SszEncode: serde hands serialize_with a reference, and unlike Display and Serialize, SszEncode has no blanket impl for &T. * feat(types): serialize the deneb containers as JSON The blob commitments are the field worth naming: a sequence of KzgCommitment, which is a byte newtype that already writes itself as hex, so it takes seq rather than ssz_hex_seq. Both spellings emit an array of hex strings, so the guard test cannot tell them apart and the choice has to come from the element type. * feat(types): serialize the electra and fulu containers as JSON Completes the annotation pass: all 84 containers now derive Serialize. Fulu is where both sequence-of-hex spellings appear side by side. The data column is a list of Cell, a foreign byte vector with no Serialize of its own, so it takes ssz_hex_seq; the commitments and proofs beside it are byte newtypes that write themselves, so they take seq. A single Cell, as in MatrixEntry, is neither and takes ssz_hex. Electra turns out to import its execution payload from deneb, and fulu has no block types at all, so their fixtures reuse rather than duplicate. * feat(types): serialize lean signed blocks as JSON Lets /lean/v0/blocks/finalized answer JSON when a caller asks for it. Integers stay bare: that is lean's own encoding and the rpc crate's tests assert it. The proof blob takes the 0x-prefixed convention the roots and bitlists use, rather than the prefix-less one the fixed-width XMSS pubkeys use, since it is opaque variable-length data and not an identity. * feat(types): serialize the container enums Neither enum adds a variant tag: the Beacon API carries the fork in the response envelope's version field and its Eth-Consensus-Version header, never inside the data object. The two get there differently, and the asymmetry is the point. The block enum derives untagged, because its Lean variant really is served as JSON on /lean/v0/blocks/finalized. The state enum cannot: a derive would demand a JSON encoding for lean's State, which is deliberately SSZ-only, so it is hand-written and the lean variant returns a serde error instead. * feat(rpc): negotiate response encoding on Accept The Beacon API serves JSON unless the caller asks for application/octet-stream, which is the order lighthouse serves too. The lean surface defaults the other way on its two SSZ endpoints, so the default stays the caller's to pass rather than a constant here. Ranks the header's entries by their q weight: a client that accepts both states its preference that way, and ignoring it would serve SSZ to a caller who merely tolerates it. base.rs's private ssz_response moves here rather than being duplicated, since both surfaces now need it. * feat(types): report a block's body root on either chain /eth/v1/beacon/headers/{id} answers with a SignedBeaconBlockHeader, whose body_root is the one field the existing accessors cannot produce: signed_beacon_block_accessors! hands back a field verbatim, and every fork stores a different body container, so what the caller wants is the merkle root of whichever one this is. Dispatched including lean, since lean's Block has a body too and /lean/v0 wants the same number. The phase0 test computes the expected root through the raw libssz_merkle trait rather than this crate's convenience wrapper, so a body_root that only worked on the lean arm would fail rather than pass by construction. * feat(rpc): parse and resolve beacon block ids Parsing is chain-agnostic, resolution is not. resolve_beacon reads only accessors that answer on either chain: Store::head_state and head_slot panic on a beacon store, so the lean slot lookup stays where it is rather than being repointed at BlockRoots, which covers the branch ending at the head rather than everything a lean store was bootstrapped with. genesis resolves to the store's anchor rather than to slot 0: a checkpoint-synced directory has no genesis block, and its anchor is the earliest block it can answer for at all. * feat(types): serialize the beacon chain config as JSON /eth/v1/config/spec echoes the chain configuration in the Beacon API's encoding: SCREAMING_SNAKE_CASE keys, every integer a quoted decimal, byte strings 0x-prefixed hex. Config could only be deserialized, which is how --network <dir> loads a network. 74 fields move from deserialize_with = "m::deserialize" to with = "m", which resolves to the same function on read, so the existing YAML parsing tests staying green is the evidence that nothing about loading a network moved. Six fields keep their own arrangement: genesis_time stays skipped since it is not a config.yaml key and /eth/v1/beacon/genesis reports it; terminal_total_difficulty and terminal_block_hash already have Serialize on their own types; max_blobs_per_block_deneb keeps its rename; and blob_schedule pairs its hand-written deserializer with seq::serialize, since its two halves do not live in one module. * feat(rpc): add the beacon API envelope and error shape Every fork-versioned Beacon API payload travels in {version, execution_optimistic, finalized, data}. The fork sits in the envelope rather than in the data because the containers serialize untagged, so version is how a caller knows which shape it just parsed. The error body is {code, message}, deliberately not the lean surface's {error}. They are two different APIs and each consumer parses the shape its own specification names, so unifying them would break one of the two. The submodule declarations and routes() land with the files they name, rather than being written now and commented out. * feat(rpc): serve beacon blocks by id in JSON and SSZ /eth/v2/beacon/blocks/{block_id} and its /root sibling, JSON by default and SSZ on Accept: application/octet-stream, both tagged with Eth-Consensus-Version so an SSZ caller can tell which fork's container it received. genesis is now refused rather than resolved to the anchor slot. Table::BlockRoots is written only by update_checkpoints, which computes its delta by walking from the old head to the new one and returns early when they are the same root. At bootstrap they always are, so the anchor's own slot is never indexed and the previous spelling could only ever 404. Saying so is better than a lookup that silently never works. The fixture moves its head through update_checkpoints rather than poking the backend, so BlockRoots is populated by its real writer and the slot-resolution tests exercise the path a running node takes. get_block_root loads the block rather than only resolving the id: it needs the slot to answer finalized honestly instead of hardcoding it. * feat(rpc): serve beacon block headers /eth/v1/beacon/headers/{block_id}. Every field of the message comes from SignedBeaconBlock's accessors, which dispatch including lean, plus body_root(); only the envelope and the signature are beacon-specific, which is what makes this the one endpoint genuinely shared with the lean surface. canonical is asked of BlockRoots rather than assumed, even for a block reached by its own root, so a sibling or a block at a slot the index does not cover answers false instead of being taken on trust. The test pins both directions: the head is canonical, the anchor is not, since its slot is never indexed. The quoting comes from ethlambda_types::beacon::serde_helpers rather than a to_string() per field, so the rule lives in one place. * feat(rpc): serve beacon states and finality checkpoints /eth/v2/debug/beacon/states/{state_id} is the path checkpoint_sync.rs already fetches from other clients, so serving it makes this client checkpoint-syncable from itself. The SSZ test decodes the body back through slot_from_ssz, since round-tripping is the whole job. A 0x... state_id is refused with a message naming the ids that do work. States are keyed by block root here with no reverse index, so the alternative to refusing is answering with the wrong state, or being right only when the two roots happen to coincide. finality_checkpoints reports the state's own three checkpoints rather than the store's fork-choice view: the endpoint is defined as a read of the state the id names, and the state is the only place a previous justified checkpoint is kept at all. JSON serializes through the Arc rather than cloning, since a mainnet state runs to hundreds of megabytes. * feat(rpc): serve beacon genesis and the spec config /eth/v1/beacon/genesis reads genesis_validators_root off the finalized anchor's state rather than looking for slot 0: it is a property of the chain that every state carries, and a checkpoint-synced directory has no genesis block to read it from. /eth/v1/config/spec echoes the Config the store was bootstrapped with, so a node started with --network <dir> reports that network's constants rather than mainnet's. GENESIS_TIME is absent from it by design, since it is not a config.yaml key and the genesis endpoint reports it. * feat(rpc): serve the beacon node endpoints syncing reads beacon_head rather than head_slot, which is lean-only and panics on a beacon store, and computes the wall slot in milliseconds the way /lean/v0/node/syncing does: slot_duration_ms is the value a loaded network can actually change, and seconds_per_slot would truncate a sub-second cadence to zero. health is 206 while syncing and 200 once caught up. 503 would mean uninitialized, and a store that answers at all is initialized. identity reports peer_id and metadata only. enr, p2p_addresses and discovery_addresses are empty, which is not spec-valid: the ENR is built for discv5 and owned by the P2P actor, and BuiltSwarm hands run_node only a local_peer_id, so serving the record means widening the ethlambda-p2p surface. Deliberate and recorded as a deviation. * feat(rpc): serve the beacon API from a beacon node Which HTTP surface a node serves now follows from the store's own chain tag rather than from the sub-command, so the two cannot disagree. Until now a beacon node bound --api-port to the lean router off a beacon store. Those handlers reach for lean state variants and metadata keys a beacon directory never carries, so calling one panicked that request. The beacon router replaces it rather than extending it, and the integration test pins both halves: /eth/v1/node/version answers, /lean/v0/node/syncing is a 404. build_beacon_api_router does not merge the metrics router, for the same reason build_api_router does not: start_http_servers serves it, and merging a path twice makes axum panic at startup. start_beacon_rpc_server takes no AggregatorController and no EventBus. A follower has no aggregator duty to toggle, and the chain-events stream is part of the lean surface. * feat(rpc): serve the lean finalized block as JSON on request SSZ stays the default here, which is the opposite of the beacon surface and deliberately so: checkpoint_sync.rs reads these bytes, and other clients' lean checkpoint sync may send no Accept header at all. Moving the default would break every one of them, so JSON is opt-in. Not routed through Encoding::from_accept, which defaults to JSON. The asymmetry is the point, so the branch is spelled out here rather than hidden behind a shared name that means the other thing. The existing SSZ test already pinned the default; the new test pins JSON on request, and that lean integers stay bare rather than picking up the beacon surface's quoting. * docs: describe the beacon API surface and its deviations docs/rpc.md gains the endpoint table, the encoding rules (JSON by default, SSZ on Accept, quoted integers, Eth-Consensus-Version) and the three ids that are refused. docs/spec_deviations.md records two: the partial node identity, and the ids that cannot be named. The second is a property of Table::BlockRoots rather than a choice, so it is written down where the reason lives. CLAUDE.md said a /lean/v0 call on a beacon node panics the request and that giving the follower its own surface was still a change of its own. Both stopped being true with this branch. * fix(rpc): resolve the anchor's own slot, which checkpoint sync asks for Found by running a mainnet follower and pointing a second one at its API: the second died with "peer served no block at the anchor slot 15265888" and never started. checkpoint_sync.rs reads a peer's finalized state, takes the anchor slot from it, and asks that peer for /eth/v2/beacon/blocks/{slot}. That is exactly the one slot Table::BlockRoots never holds, because update_checkpoints is its only writer and writes nothing when the head does not move, which is the situation at bootstrap. So the endpoint 404'd and this node was unusable as a checkpoint-sync source for any client, not only for itself. On an index miss the lookup now tries the roots the store can name and accepts one only when block_entry confirms its block really sits at the slot asked for, so a slot the store holds nothing at still 404s and no request can be answered with the nearest checkpoint instead. The test that pinned the old 404 asserted a bug, so it now asserts the anchor resolves, with a sibling holding the line on slots below it. The docs claimed a slot at or below the anchor was refused; only genesis and a state-root id are. * refactor(blockchain): move the empty-custody check to the node that configures one The per-block fence in data_availability_for cannot distinguish a misconfigured node from the replay harness, which legitimately supplies every block itself and has no columns to wait on. A node is checked once, at startup, where the custody set is computed. * feat(blockchain): expose an unspawned import entry for offline replay on_block already takes &mut self and no Context, and returns the import verdict, so a caller can drive the real import path without a mailbox. Two public items rather than a benchmark module inside this crate. * feat(cli): add the import benchmark's corpus manifest An SSZ corpus rather than a prepared store directory, so it stays readable and portable across revisions of the store format. Empty slots are recorded as gaps in the manifest rather than treated as a truncated range. * feat(cli): resolve a corpus anchor across an empty slot The state at from-1 is slot-advanced and keyed to no block when that slot is empty, so walk the first block's parent link instead. Written against a trait so the rule is tested without a live server. * feat(cli): stream a beacon block range into an SSZ corpus One state is held at a time and blocks are written as each response lands, so the footprint does not scale with the range and the range needs no cap. * refactor(cli): split the benchmark report by workload A pure move: the shared statistics go to report::common, today's report to report::synthetic, schema_version unchanged. * feat(cli): add the import workload's report and an at-most-once phase timer An import runs a subset of BLOCK_IMPORT_PHASES, so an unobserved phase is omitted rather than recorded as a zero that reads as instant work. * feat(cli): replay a corpus through the real import path Drives BlockChainServer::import_block directly: no mailbox, no tick loop, no p2p, and the clock placed per block from the store's own config so a run is reproducible. A block that does not import aborts the run, since every later block descends from it. * feat(cli): add benchmark import fetch and replay Wires the two phases to the command line and drops the dead_code allows the earlier slices needed while nothing reached this code. * docs: describe the import benchmark workload Records why the corpus exists: a live-follower A/B costs a restart and a moving denominator per leg, and the work being measured is deterministic. * fix(blockchain): publish a replayed import's sections under their own source import_block left ImportTimings.source unset to keep replayed blocks out of the gossip and sync percentiles, but BlockImportReport::observe publishes nothing for a sourceless import. The replay harness reads its per-phase numbers back from exactly those observations, so every phase came back absent and a mainnet replay reported wall time alone. BlockSource::Replay gives them a label no node ever writes: they stay out of the wire sources' percentiles and are still published. * fix(cli): make the import benchmark work against a live mainnet node Run against ethlambda-5's mainnet follower, a corpus whose first block's parent sat mid-epoch failed on that first block with `root in store.blocks`. get_forkchoice_store finalizes (epoch(anchor), anchor_root), and every import walks back to that epoch's first slot, which lies below a mid-epoch anchor. The spec fixtures anchor at genesis, so the tests never reached it. - fetch anchors on the block at the first slot of the epoch holding from - 1, stepping back past empty first slots, and records the blocks up to --from as warm-up that replay imports without sampling. replay refuses a corpus anchored anywhere else and says to re-fetch it. - A 404 means an empty slot, a slot past the head, or one before the source's history. fetch now refuses a --to past the head and checks every block's parent link, so a hole or a reorg mid-fetch stops the fetch instead of failing a replay minutes later. - A failed fetch removes the corpus directory it created. - Progress goes to stderr. fetch's tracing lines sat below the benchmark's WARN filter, and replay printed nothing for a mainnet block's seconds. - The report also samples prune, get_head and fcu, prints its inclusive range as [a, b], and drops the coefficient-of-variation warning: it named a flag replay does not have, and across a corpus it measures how blocks differ, not noise. * fix(blockchain): initialize the committee cache in for_replay Merging beacon-chain-integration brought in #22, which added a `committees` field to `BlockChainServer`. The text merge was clean, but `for_replay` builds the struct by hand and never set the new field, so the branch stopped compiling (E0063). Replay starts from an empty cache, the same as a freshly spawned node. * chore(docs): drop the beacon API spec and plans from the branch They are working notes from the design sessions, not documentation, and were swept into bfdbdc8c alongside an unrelated rpc fix. Nothing links to them. They stay recoverable from that commit. * refactor(cli): stop reading the state cache's occupancy from the benchmark The replay read the store's LRU length to report peak_states_cached and to assert that a long range stays within STATE_CACHE_CAPACITY. An LRU cannot exceed its capacity by construction, and the storage crate's own the_state_cache_is_bounded already pins the cap, so the number could never show anything that test does not. It also cannot see what the replay module doc warns about: an Arc<BeaconState> pinned outside the cache leaves len() unchanged. That is not worth a benchmark-only accessor on Store. * test(blockchain): delete the beacon_replay integration test The benchmark's replay tests in bin/ethlambda already drive BlockChainServer::for_replay and import_block end to end, and this one read consensus-spec-tests fixtures the Test (consensus) job never downloads, so it failed on every CI run. Its snap and serde_yaml_ng dev-dependencies go with it; nothing else in the crate uses them. * test(cli): delete the import benchmark's fixture-backed replay tests They read a consensus-spec-tests case the Test (node) job never downloads, so they failed on every CI run. Two of the five also asserted less than their names claim: the block roots they compared are hashed from the corpus files, so "the head advances" and "two runs agree" held whenever both replays finished. The spec-case corpus builder, ReplayOptions::for_tests and the snap dev-dependency only served these tests and go with them.
…nned eviction (#33) * fix(beacon): harden the committee cache and share it where it was missing Review of #22 found gaps between what the cache's documentation promised and what the code did. - Committee numbers use checked arithmetic. Fork choice's gossip path never bounds `data.index`, so a release build wrapped a hostile index into a real committee where the specification raises. - `EpochCommittees::committee` rejects a slot outside the epoch it was built for. Callers now name that epoch separately and share the result through the cache, so a mismatch would have sliced the wrong shuffle. - Epochs 0 and 1 are keyed on the genesis block, as lighthouse does, instead of rebuilding a whole-epoch shuffle on every lookup. - `get_beacon_committee` is back to the per-member derivation, so a single-committee caller pays what it did before this PR, and the tests holding the sliced form to it compare two independent derivations. - Phase0 epoch processing shares one shuffling per call rather than one whole-epoch shuffle per attestation, and the inclusion-delay deltas derive each attestation's attesters once rather than once per attester. - Pre-Electra `process_attestation` reads the committee count off the cached shuffling instead of scanning the registry per attestation, and reuses the indexed attestation's indices as Electra already did. - The fork-choice and sync fixture runners hold one cache per case, as the chain actor does across imports, so the fixtures' sibling branches test the cache key instead of passing whatever it does. * perf(beacon): shuffle in place and pin the head's shufflings in the committee cache The whole-list shuffle visited every position in every round, each visit a random lookup into that round's window hashes plus a 64-bit `% n`. Lighthouse's `shuffle_list` (protolambda's algorithm, which Prysm and Teku also use) walks each round's swap pairs instead: one decision per pair, in order, rehashing only on crossing into a new window. It shuffles the active set in place, so a build holds one validator-sized list instead of three. Benchmarked standalone on an M3 Pro at 2.4M validators: 1387 ms to 355 ms, peak heap 57.6 MB to 19.2 MB. The cache held two shufflings, exactly one block's worth, so two branches imported in turn evicted each other's entry on every import and rebuilt it on the next. It now holds `COMMITTEE_CACHE_CAPACITY` entries and evicts the way lighthouse does: oldest epoch first, never the canonical head's previous, current, or next shuffling. The chain actor pins those whenever fork choice moves the head, reading the head's post-state from the store's state cache only, so pinning never reconstructs a state on the import thread. `lean_beacon_committee_cache_lookups_total{result}` counts hits, misses, and unkeyable lookups, so the capacity can be checked against live forks.
* refactor(types): move the pure signing helpers off the state transition
They depend on nothing but their arguments and on containers this crate
already owns, so a consumer that only signs a message had to take on
blst, c-kzg and RocksDB to reach them. Re-exported at the old path.
* test(types): restore the domain coverage lost in the move
The relocated tests dropped two assertions: that the domain's tail is
the truncated fork data root, and that it commits to the genesis
validators root. The second is what stops a signature replaying onto
another chain.
* refactor(types): reach the signing helpers by their own path
Importing from `ethlambda_types` directly matches the re-export just
below it and keeps `signing` out of the module list that exists to give
each transitioned type one name. Adds the `compute_signing_root` test
the move never had.
* docs(beacon): name both helpers that stayed behind
The module doc said only merkle branch verification was still implemented
here, which the next function in the file contradicts.
* feat(validator): add the crate skeleton and its error type
* refactor(validator): carry what the callers will need to act on
The duty loop decides retry-or-skip every slot, failover distinguishes a
node that is down from one that is slow, and a signing failure has to name
its validator. Each was a bare string, so each call site would have
re-derived it. Adds the inconsistent-response case a conformant-looking
node can still produce.
* feat(validator): decrypt EIP-2335 keystores
Verified against both specification test vectors, scrypt and pbkdf2,
including the NFKD password normalisation they exercise. The control-code
strip has to run on Unicode scalar values before UTF-8 encoding, not on
the encoded bytes afterwards: byte-level filtering corrupts any multi-byte
character whose continuation byte falls in the same numeric range as a C1
control code, which is exactly what the key-emoji in the vectors exercises.
* fix(validator): reject malformed keystore parameters and scrub key material
A scrypt cost that is not a power of two silently derived the wrong key
and surfaced as a wrong password, blaming the operator for a corrupt
file. The decrypted secret, the derived key and the password buffer now
zeroize on drop rather than living as long as the process.
* feat(validator): load validator definitions and derive signing keys
* fix(validator): make the definitions file survive a crash mid-write
The keymanager API rewrites this file while validators are signing, and an
in-place truncating write would lose every validator rather than the one
being changed. Writes to a sibling file and renames over the target.
* feat(validator): sign attestations under the target epoch's domain
The fork version is resolved per signature rather than cached, so a
signature produced either side of a fork boundary is valid there.
* feat(validator): add the Beacon API wire types
They live here rather than as serde derives on the consensus containers:
quoted integers and hex bytes are a property of this transport, and a
Deserialize on a container is a footgun in the crate that defines it.
* test(validator): catch a signing domain taken from the wrong epoch
Every fixture had the slot's epoch equal to the target's, so swapping one
for the other passed the whole suite. The network would have dropped the
attestations silently.
* test(validator): cover the wire types nothing was parsing
Four DTOs had no test, so a dropped quoted-integer annotation on any of
them would have surfaced first as a beacon node rejecting the request.
* feat(validator): add the Beacon API client behind one trait
The trait is what makes the duty services testable: a reorg invalidating
duties or a node failing mid-slot cannot be exercised against reqwest.
* feat(validator): fail over across beacon nodes in list order
* feat(validator): add the slot clock
Seeded from the beacon node's genesis and then independent of it: the
node's event stream is best-effort, and a dropped event must not mean a
missed duty.
* fix(validator): read the spec response the way nodes actually send it
The fork-version loop asked for PHASE0_FORK_VERSION, which no node sends,
so the genesis fork version silently kept its mainnet value on every
network. A malformed value was indistinguishable from an absent one, both
quietly keeping a default the network would reject every signature under.
Moves the parsing to the crate that owns wire formats, which keeps
serde_json out of ethlambda-types' production dependencies.
* test(validator): let the mock express a node that fails one call
A single blanket failure flag could not describe a node that answers at
startup and then fails every duties poll, which is the shape the duties and
attestation services most need to test against.
* fix(validator): refuse a zero slot duration instead of dividing by it
A beacon node reporting SECONDS_PER_SLOT of 0 reached the clock unchecked
and panicked later inside slot_at, far from the cause. Rejected at the
response boundary, with a constructor precondition behind it. Adds the
tests at the exact attestation instant that a comparison slip would
otherwise have slipped past.
* feat(validator): track attester duties and invalidate them on a reorg
A dependent_root change means the committee shuffling moved, so acting on
the held schedule would attest from the wrong committee.
* feat(validator): subscribe to the attestation subnets its duties need
* feat(validator): sign and publish this slot's attestations
One data fetch is shared across every validator attesting at the slot, and
one validator failing to sign never costs the others their attestation.
* fix(validator): keep a reorg signal a failed lookahead would have eaten
A lookahead fetch is speculative, but its failure was discarding the
current epoch's committee change along with it, so subscriptions were
never re-sent. Adds the tests for the lookahead and for epoch-scoped
lookup, neither of which any mutation would previously have broken.
* feat(validator): add the metrics and keymanager modules
Not yet declared in lib.rs, so nothing compiles them until the next
commit wires them into the duty loop. Split out so that commit stays
readable rather than carrying three subsystems at once.
The keymanager persists an import before inserting it into the live
store: a key that signs before it is durable would vanish on a crash,
and this client has no slashing protection to notice the gap.
* feat(validator): assemble the client and run the duty loop
Wires the slot clock, duties service, subscriptions and attestation
service into one loop, and serves metrics and the keymanager alongside
it. A retryable error keeps the schedule already held and tries again;
anything else stops, rather than hiding its cause behind a warning every
slot.
Also records that attest() must not be retried within a slot: the
signatures already exist, and re-fetching would sign a different message
for the same slot with nothing here to catch it.
Declares tokio's time feature rather than inheriting it through
workspace feature unification.
* feat(cli): run the validator client as ethlambda validator
Added to EXPLICIT so the sub-command is not given the default `node`
token, which would have parsed it as `ethlambda node validator` and
failed with a message about node flags. The historical flat invocation
the Dockerfile, lean-quickstart and the hive shim rely on is unchanged,
and still covered by its own test.
* fix(validator): reject attestation data for a mismatched slot or target epoch
HttpBeaconNode::attestation_data took the response's slot verbatim, so a
stale or buggy beacon node could hand back AttestationData for a slot
other than the one requested and this client would sign it without
complaint. With no slashing-protection database, two such answers
landing under one target epoch is an unnoticed double vote.
Validate in AttestationService::attest, not in the HTTP client, so the
check also covers the mock and any future BeaconNodeApi implementation.
Also compare the target checkpoint's epoch against the epoch the
requested slot belongs to: the signing domain is chosen from that field,
so an internally inconsistent response is a second way a broken node
could get a signature out of this client that it should not.
* fix(validator): stop holding store lock guards across beacon-node awaits
Both call sites into the shared validator store passed
`&*store.read().await` straight into a match/if-let scrutinee. Rust
extends a scrutinee temporary's lifetime across the whole construct, so
the read lock ended up held for the entire refresh_epoch or attest call:
every network round trip they make, not just the store access.
Meanwhile the keymanager's key import takes the write lock across a
batch of deliberately slow EIP-2335 key derivations and disk writes.
tokio::sync::RwLock is write-preferring, so a queued import would jump
ahead of the duty loop's next read and stall attestation signing for as
long as the batch ran.
Fix both sides: the duty loop now takes only the pubkeys it needs as
owned data before calling refresh_epoch, letting the guard drop
immediately; attest takes the RwLock itself and scopes its own read
guard to the synchronous signing loop, dropped before the submission
call; and the keymanager decrypts and persists every key to disk before
ever taking the write lock, which is now held only across the cheap,
synchronous insert_secret calls.
* fix(validator): count partial attestation-pool rejections instead of failing the whole batch
The Beacon API answers POST /eth/v2/beacon/pool/attestations with 400
and per-index detail (IndexedErrorMessage) when part of a submitted
batch is rejected, and still stores and gossips the rest. Collapsing
any non-2xx into one opaque BeaconNodeStatus error, as post_no_content
does for every other endpoint, meant inc_attestations_published never
fired even when most of the batch succeeded, and one bad duty destroyed
the observability of every other validator that slot.
submit_attestations now returns how many of the batch were accepted.
HttpBeaconNode parses the IndexedErrorMessage body on a 400 from this
endpoint specifically, logs which submitted index failed and why, and
reports the true accepted count as a success rather than an error. A
total rejection or an unparsable body still surfaces as Err, unchanged
from before.
* fix(validator): bind metrics before beacon-node startup and round out observability
Three smaller review findings, bundled since they all touch the same
observability surface:
- run() called genesis()/spec() before binding the metrics and health
listener, so a validator that could not reach its beacon node exited
with no HTTP surface at all. Bind metrics first, so /health and
/metrics are up while an operator waits to see whether startup will
succeed.
- ethlambda_validator_beacon_node_available only updated inside
refresh_epoch, once an epoch, so an outage starting right after a
successful refresh kept reporting "available" for up to an epoch
while every attest call failed underneath it. It now also updates
from attest's own failures.
- Added the signing-duration and publication-delay histograms the
design doc promised but never implemented, plus a validators_resolved
gauge so "no validator indices resolved" is visible to monitoring
continuously rather than only in one log line the first time it
happens. All three are registered in init() alongside the existing
series.
Also comments two things a reader could otherwise "clean up" by
mistake: the stopping branch in run()'s refresh_epoch match is
unreachable today (every error that function can produce is
retryable), and Error::Signing is unconstructed today but reserved for
a future remote-signer SigningMethod variant.
* fix(validator): write key material at 0600 and refuse to follow symlinks
Every secret this crate writes (the keymanager API token, imported
keystores, and the plaintext passwords that unlock them) went through
std::fs::write with no explicit mode, landing at 0644 under a standard
umask and readable by any other local account. The password file is
the sharper half of that: it sits right next to a world-readable copy
of the keystore it decrypts, so any other account could recover a
validator's BLS secret.
Route every such write through a new secure_fs helper that opens with
mode 0600 via OpenOptionsExt, matching the 0600 convention this crate
already claims (Lighthouse's) but did not enforce. The keystore and
password paths are additionally derived from the public key and are
therefore predictable, so their writes use O_NOFOLLOW rather than a
plain open: an attacker able to plant a symlink at one of those paths
ahead of a legitimate import must not be able to redirect the write to
a target the operator can write to. O_NOFOLLOW still opens and
truncates an existing regular file, so re-importing an already-known
key continues to overwrite its files exactly as before.
* fix(validator): serialize the definitions file's read-modify-write cycle
import and delete each took their own independent snapshot of
ValidatorDefinitions (open, mutate, save) with nothing serializing that
cycle across concurrent requests. The in-memory store's RwLock did not
help, since the snapshot is taken outside of it. A delete that removed
a key and saved could be undone by a concurrent import whose snapshot
was taken beforehand: it would write the file back with the deleted
key still enabled, silently resurrecting a validator the operator had
just deactivated. With no slashing protection in this crate, that is
exactly the failure this API exists to prevent.
Add a definitions_lock to KeymanagerContext, held by import and delete
across their entire open-mutate-save cycle. It guards only the
definitions file, not the validator store, so holding it across an
import's slow EIP-2335 key derivation cannot stall the duty loop, which
only ever waits on the store lock. Both handlers take definitions_lock
before the store lock, consistently, so the two never deadlock.
Along the way: cap a single import request at 100 keystores
(MAX_KEYSTORES_PER_IMPORT), since each one runs its KDF synchronously
while holding definitions_lock and an unbounded batch could tie it up
indefinitely; and de-duplicate persist_import's definitions entry by
voting_public_key, so re-importing an already-known key updates it in
place instead of appending a second, conflicting entry.
A true concurrent race is not reliably deterministic to assert on in a
test. import_waits_for_an_in_flight_holder_of_the_definitions_lock
proves the lock is actually acquired and enforced (a second request
blocks while it is held); a_delete_survives_a_subsequent_import_of_a_different_key
is the weaker, deterministic fallback: two sequential requests, which
mainly exercises the dedup fix rather than the race itself, since a
genuine interleaving is no longer reachable through the public API
once the lock is real.
* fix(validator): compare the bearer token in constant time and finish hardening
require_bearer compared the presented token with plain String equality,
which short-circuits at the first mismatched byte. This service is
localhost-bound by default, but --http-address is operator-configurable
with nothing enforcing that, so a deployment that moves it off loopback
would leak timing information about how much of the token a caller
guessed right. Compare the bytes with subtle::ConstantTimeEq instead.
Also: make the empty DELETE interchange's fake-empty history deliberate
rather than an accident of KeymanagerContext carrying no real
genesis_validators_root. It stays all-zero (documented in the module,
and pinned by a test) so a conformant importer rejects the file instead
of trusting a validator that "has never signed", and every DELETE now
warns that the interchange it returns cannot be used to migrate the
key elsewhere. Wrap ImportRequest.passwords in Zeroizing<Vec<String>>
so it gets the same scrub-on-drop guarantee Keystore::decrypt's output
already has. And in Keystore::decrypt, construct the Zeroizing<[u8; 32]>
before copying the secret into it, rather than after, so the secret is
never briefly held in a bare, unprotected array on the stack.
* fix(validator): derive the consensus-version header from the target epoch
attest took an opaque fork_name from the caller, computed from the
current epoch and disconnected from the attestation it was actually
describing. The signing domain already keys off data.target.epoch;
now that attest validates that field against the requested slot, the
header can and should be derived from the same, already-consistent
source instead of trusted blindly from outside.
* Refuse to sign an attestation this process already signed for that validator
Two double-vote shapes were reachable through ordinary operation, not through
operator error, and with no slashing-protection record nothing caught either.
A backward wall-clock step: the duty loop derives its slot from
SystemTime::now(), which is not monotonic, so an NTP correction of a few
seconds re-enters a slot already attested. The re-fetched attestation data
differs if a block arrived in between, and two different attestations under one
target epoch is a double vote.
A schedule replaced mid-epoch: a duty refresh that fails at an epoch boundary
and succeeds a few slots later can move a validator to a different slot in the
same epoch, and the loop attests a second time under that target.
AttestationGuard records, per validator, the highest source and target epoch
signed in this run, and refuses anything that does not advance the target or
that reaches back below the source. That is EIP-3076's minimal variant, chosen
because one record per validator cannot evaluate the full surround condition
and requiring monotonic progress makes the question unnecessary. The check runs
before the signature is produced rather than after: a signature that exists is
one that can escape.
This is not slashing protection and does not reverse the scope decision
recorded in lib.rs. It holds nothing on disk, so it knows nothing about a
previous run or about another process holding the same keys, and a restart
empties it. The module documentation opens by saying so.
Keyed by validator, not by slot: validators in one epoch attest at different
slots, so a per-slot or per-epoch record would wrongly refuse the second
validator. A test pins that.
Two existing tests gave two duties the same pubkey under different validator
indices, which the guard now correctly refuses. That shape cannot occur: a
pubkey resolves to exactly one validator index, and ValidatorStore is keyed by
pubkey so a duplicate collapses. Both tests now use two keys, which is what
they were always testing for and leaves their assertions unchanged.
Adds ethlambda_validator_attestations_refused_total, separate from
signing_failures_total: that counts signatures attempted and failed, this
counts signatures deliberately not attempted. It should normally read zero.
* Refuse an empty keymanager API token instead of authenticating everyone with it
An empty api-token.txt authenticated every request. Two facts combined:
load_or_create_token returned Ok("") for any existing file, including a
zero-byte one, and the bearer comparison uses subtle's ct_eq, which reports two
empty slices as equal because its accumulator starts at 1 and the fold body
never runs. A request carrying `Authorization: Bearer ` with a trailing space
strips to Some(""), so empty matched empty and the request was served.
It needed no unusual operator action to reach. secure_fs::write_private opens
with truncate(true) and then writes, so a first boot that died or failed
between those two steps left a zero-byte file that every later boot accepted.
Fixed at both layers. load_or_create_token now refuses a file holding fewer
than MIN_TOKEN_LEN characters, and the middleware independently refuses an
empty configured token: router() takes its token from its caller, so the
comparison cannot rely on the loader being the only source. The length check
sits before the constant-time comparison and leaks nothing, since whether the
configured token is empty is not a secret and no comparison against the
presented value has happened at that point.
Refusing rather than regenerating is deliberate. A token file that exists but
cannot be a token means something went wrong that an operator should see, and
silently minting a new one would change the credential their tooling holds.
The middleware regression test was checked against the unfixed code: it fails
without the guard and passes with it. One test pins subtle's empty-slice
behaviour directly, since the guard's stated reason depends on it.
* Stop a deleted validator key from coming back after a restart
Two paths let an operator believe a key was removed when the definitions file
still held it, so the next restart loaded it and the validator signed again.
With no slashing-protection record, a key returning after the operator moved it
to another host is a double vote.
ValidatorStore::load never checked the definitions entry's voting_public_key
against the key its keystore actually holds. Every signing path uses the
derived key, so a mismatch could not produce a wrong signature, but the
keymanager's delete removes from the store by the derived key and from the
definitions file by the declared one, so a mismatch made retain match nothing
while the response reported "deleted". load now refuses such an entry, which
makes persist_delete's assumption true by construction instead of by hope. It
is fatal rather than skipped, consistent with every other failure in that loop:
a definitions file that does not describe its own keystores is a configuration
error to fix, not one to run half of.
delete_one decided "not_found" from the in-memory store alone. After a failed
delete the in-memory removal has already happened and the entry survives on
disk, so a retry answered not_found, which reads as "already gone". Acting on
that by importing the key elsewhere is what turns a failed delete into two
hosts signing for one validator. It now reads the definitions file first,
through definitions_contain, before the removal makes the store an unreliable
witness. A disabled entry counts as present, and an unparseable declared key
counts as absent, matching exactly what persist_delete will and will not
remove.
Four tests: three on the load-time cross-check including the positive case, and
one driving a real HTTP delete against the state a failed delete leaves. The
first uses `let else` rather than expect_err because ValidatorStore has no
Debug by design, and a test is not a reason to weaken that.
* Roll back a failed import so the next key in the batch does not commit it
One ValidatorDefinitions vector is threaded through every key in an import
request, and each key mutates it and re-saves it. persist_import pushed or
replaced its entry before calling save, and a save failure left that mutation
in place, so the next key's successful save wrote it to disk. A key this
request reported as `error` was therefore enabled on the next restart and
signing. With no slashing-protection record, a validator the operator believes
was never imported is exactly the kind that ends up running in two places.
persist_import now records whether it appended or replaced, and restores the
vector before returning the error. The append case pops rather than removing by
index, which is correct because this function pushes and saves with nothing in
between.
Two tests, one per path. Both make the save fail while the writes preceding it
still succeed, by putting a directory where validator_definitions.yml goes: the
keystore and password writes land, so the mutation is reached, and only the
final rename fails. The replace-path test gives the previous entry
enabled: false so the restored value is distinguishable from what the import
would have written, rather than passing by coincidence.
* Make failover work for a node that answers wrongly rather than not at all
Two bugs, both from an Ok that failover could not see past.
A beacon node stuck on a stale head answered attestation_data successfully with
data for the wrong slot. The slot and target-epoch checks lived in
AttestationService, above FallbackBeaconNode, so try_each had already accepted
that answer and returned by the time they ran. The node was therefore seen as
a success every slot, the next node was never consulted, and the validator
missed every attestation while both nodes looked up. A comment in
attestation.rs claimed rejecting there "lets failover try the next node",
which was the opposite of what happened.
The checks now live in the implementations, behind a contract stated on
BeaconNodeApi::attestation_data, so a wrong answer is an Err that failover
treats like any other. One shared validate_attestation_data rather than two
copies: the implementation calls it so failover can act, and the service calls
it again before signing, because the trait is public and nothing forces an
implementation to honour its contract. MockBeaconNode honours it too, or the
test double would be more permissive than any real node and tests built on it
would stop reflecting production.
is_syncing returning Ok(true) is a node reporting itself unusable, but
try_each treats any Ok as the answer, so a syncing node at the head of the list
returned true and refresh_epoch declined to refresh duties for as long as it
was syncing, with a healthy node behind it unused. It no longer uses try_each:
it looks for a node that is not syncing, reports false as soon as it finds one,
and reports true only when every reachable node said so. A node that is down is
skipped, and only if none answers at all is this an error.
Six tests covering both, including a stale node failed over from, every node
stale, a syncing node masking a healthy one, and one node down with one syncing.
Also corrects two comments that asserted safety properties the code did not
have: try_each's claim to skip syncing nodes, and http.rs's claim that
inconsistent responses are raised by the caller.
* Bound a slot's attestation work by what is left of that slot
Nothing bounded it. HttpBeaconNode's per-request timeout is 8 seconds, failover
tries each node in turn, and attest makes two calls, so one hung node could
carry a 12-second slot's duty past the slot itself and into the next one, whose
duty was then late in turn. An attestation that misses its slot is worth
little; one that also delays the next slot's is worth less than nothing.
SlotClock gains end_of and remaining_in, and the duty loop wraps attest in a
timeout of whatever is left. remaining_in returns zero for a slot already gone
rather than erroring, so a hopeless duty can be passed straight to the timeout
and fail at once instead of needing a branch.
Aborting the future can drop it after signing and before submission. That is
safe only because AttestationGuard records at signing time, so the abandoned
attestation cannot be re-signed next slot under the same target; it is simply
lost, which is what attest's own "do not retry within a slot" rule already
required. Without that guard this timeout would have opened a double-vote path
rather than closed a latency one, so the call site says so.
Adds ethlambda_validator_attestation_deadline_missed_total, distinct from
attestation_failures_total: that counts a duty that failed and returned, this
counts one that never returned in time, which points at a slow beacon node
rather than a rejected attestation.
Also registers attestations_refused_total in metrics::init(), which the earlier
commit adding it missed. That file documents why every series must be forced
there: an alert cannot fire on a series that does not exist, so absent and zero
are not the same thing.
* Document the missing slashing protection where an operator will actually see it
This client's most consequential property was recorded in exactly one place: a
rustdoc comment on the validator crate, pointing at
docs/superpowers/specs/2026-09-11-ethlambda-validator-design.md, which does not
exist in the repository. Nobody running a binary reads either. An operator
could build this branch, point it at mainnet keys and never meet the warning.
Now said in five places.
A startup warning, unconditional and at warn level, before any key is loaded or
any beacon node contacted. Not behind a flag: a warning an operator can silence
is one they will, and the consequence here is losing stake.
A spec_deviations.md entry. That page was scoped to leanSpec, so its
introduction now states that this repository builds for two chains against two
references, and that the validator client is measured against consensus-specs
and the keymanager API. The entry covers what EIP-3076 requires, what exists
instead, what AttestationGuard does and does not cover, what it means for a key
migrated through the keymanager API, and what to do operationally. The
introduction flags it as the only deviation on the page that costs money rather
than performance.
A cli.md section, which previously did not mention the subcommand at all: every
flag, what the client does today, and an explicit list of what is not
implemented, so nobody infers from "validator client" that it proposes blocks.
The ethlambda_validator_* series in metrics.md, none of which were documented,
including why three similar-looking failure counters are separate series.
The crate doc now points at the deviations entry rather than a file that is not
there.
* Refuse to sign a second block for a slot this process already proposed
The proposal-shaped counterpart to AttestationGuard, and the same disclaimer
applies: this is not slashing protection, keeps nothing on disk, and a restart
empties it. It closes the double-proposal shapes reachable inside one run.
Two of them, both reachable without a hostile beacon node or a second instance.
The duty loop takes its slot from SystemTime::now(), which is not monotonic, so
an NTP correction can re-enter a slot already proposed; the block produced the
second time differs from the first, because the node has packed whatever
attestations arrived in between. And proposer duties are only final once the
epoch's randao is fixed, so a refresh landing late can move a validator between
slots inside one epoch.
A proposer slashing is worse than a double vote in one specific way: it needs no
second validator to be caught. The two signed headers are the whole evidence.
The rule is EIP-3076's minimal variant for blocks, per validator: remember the
highest slot proposed and refuse anything that does not strictly advance it.
Equality is refused along with regression, because the same slot proposed twice
is slashable unless the two blocks are byte-identical, and this guard does not
keep enough to tell those apart.
A separate guard rather than a field on AttestationGuard. The two judge
different things, an attestation on its source and target epochs and a block on
its slot, and a validator can legitimately propose in an epoch it also attests
in. One merged record would either refuse a legal proposal or admit an illegal
one depending on which field won.
Tested including the case a HashMap makes easy to get wrong: slot 0 is a real
slot, so a validator that proposed it must not be allowed to propose it again.
* Sign the RANDAO reveal and the block, not just the attestation
A proposer owes two signatures beyond the attestation this crate could already
produce, and neither shares the attestation's domain or its message shape.
The RANDAO reveal signs the epoch itself, not anything about the block. That is
what makes it both unforgeable and unchooseable: a proposer has exactly one
valid signature to offer for its slot's epoch, so it cannot grind the
randomness by trying alternatives. The message is the epoch's merkle root,
which for a uint64 is its eight little-endian bytes zero-padded to thirty-two,
and a test pins that rather than trusting it, since signing the big-endian
bytes or the raw eight would produce a reveal the network rejects with no way
to tell why.
The block signature takes an already-computed root rather than a block, so it
stays independent of which fork's block shape a beacon node produced. Its
domain comes from the epoch containing the block's own slot, which is the one
place a reader might reach for the attestation rule by habit: an attestation's
domain comes from its target epoch, a block's does not.
All three now funnel through one private sign_root, so there is a single place
a signature is produced and a single place the remote-signer method will be
added. It stays private deliberately: a caller that can hand in an arbitrary
root can make this client sign anything, and the domain separation that keeps
an attestation from being read as a block lives in the methods above it.
That separation is stated as a test rather than as three different constants.
One object root signed under the three domains must give three distinct signing
roots. It is not hypothetical for the reveal: its message is a bare uint64
merkle root, which is also a well-formed block root, so without the domain a
reveal for epoch N would be a valid proposer signature for a block whose root
happened to be N's merkle root.
The fork-boundary test crosses a real one. domain() only varies with the epoch
insofar as the epoch selects a different fork version, so two large epochs
inside one fork would assert nothing.
* Wake the duty loop at the slot boundary so a proposal has somewhere to happen
The loop woke once per slot, one third of the way in, because an attestation was
the only thing it had to do. A proposer publishes at the boundary, so a loop
that only ever runs a third of the way into a slot cannot propose for it: by the
time it wakes, the slot it would be proposing for is already a third gone.
So it now wakes at the boundary and sleeps the rest of the way in. Everything a
slot needs is reachable from there, and nothing about the attestation changes:
it still runs at the same instant, still gets the rest of the slot as its
budget.
One thing does improve. The epoch refresh used to run at the attester offset
itself, so every second it spent came straight out of the attestation's
lateness. It now runs at the boundary with a third of a slot of headroom ahead
of it, and if it overruns anyway the remaining sleep is zero and the attestation
is late rather than skipped. It is still unbounded; bounding it to the headroom
would be worse, since the per-request timeout alone is twice that.
next_slot_start replaces next_attestation. Two differences worth naming. It
returns the slot to act on rather than an Option, because before genesis the
answer is slot 0 and the wait until it, so a client started early now sleeps
once, exactly until genesis, rather than waking every slot-length to ask again.
And standing exactly on a boundary returns the following slot, which the loop
depends on: it calls this immediately after finishing a slot's work, and
returning the slot just handled would spin it at zero delay forever.
until_attestation is both the sleep between a slot's proposal and attestation
work and, in the next commit, the proposal's budget. A proposer still waiting on
its beacon node when the attestation is due has already lost the block and must
not also cost this client's attesters their votes.
slot_duration() goes with next_attestation: the before-genesis fallback was its
only caller.
* Decode and republish a produced block as SSZ rather than JSON
To sign a block this client has to compute its hash_tree_root, and a root can
only come from the typed container. The JSON route would mean hand-writing a
field-for-field mapping of a whole BeaconBlockBody, its ExecutionPayload and
every operation list inside them, then trusting that mapping to be exact. One
wrong field order, one missing list, one integer read as decimal that was meant
as hex, and the client signs a root that is not the block's. That failure is
silent where it happens and surfaces as a block the network rejects.
Decoding the specification's own serialization removes the class of bug
outright. The bytes the node sent are the block; the root falls out of them.
Both endpoints support it: Accept on produceBlockV3, Content-Type on
publishBlockV2.
BlockContents lives here rather than in ethlambda-types because it is not a
consensus container. It is defined in beacon-APIs and nowhere else;
consensus-specs never mentions the name and so never states its SSZ encoding
either. The three-field container is what every implementation encodes, but it
is convention, not specification, and ethlambda-types is the authority on what
the chain itself agrees about.
Fulu needs its own pair, which is the part worth reading twice. PeerDAS did not
turn the response back into a bare block, which is the natural guess given that
fulu moves blob distribution to column sampling. The shape is unchanged. What
changed is kzg_proofs: one proof per cell rather than per blob, with a list
limit of FIELD_ELEMENTS_PER_EXT_BLOB * MAX_BLOB_COMMITMENTS_PER_BLOCK.
A draft of this claimed a six-blob fulu block would not fit electra's container
and a guard assertion in the test caught it: six blobs is 768 cell proofs, well
under 4096. The limits agree until a block holds more than thirty-two blobs.
Both facts are now tests, because the harmless one is what makes the other
dangerous: a single shared type appears to work today and would fail silently
later, since fulu made the per-block blob limit depend on the epoch precisely so
later forks can raise it.
Pre-deneb forks are refused rather than decoded. They are reachable only on a
chain that has not reached deneb, and admitting them means four more block
shapes to serve nobody.
Result is deliberately not imported in this module; libssz_derive's generated
code names Result unqualified and means std's, so this crate's one-argument
alias in scope makes every derive in the file fail to compile.
* Ask a beacon node for a block, and publish the signed one back
The two calls block proposal is made of, with the same contract the attestation
path already has and for the same reason.
A produced block is checked against the request inside the implementation, not
above it. FallbackBeaconNode wraps the call, so a check performed by the caller
runs only after one node's answer has been accepted and returned, and a node
stuck on a stale head is never failed over from. Two fields are checked, both
because both end up signed. The slot selects the signing domain and is the whole
of what the proposal guard keys on. The proposer index is not something this
client picks either: a node on a different fork computes a different proposer,
and a block naming someone else can only be signed uselessly while still burning
the guard's record for that slot, which then refuses the real duty.
builder_boost_factor=0 is how this client says it does not do the builder flow.
The parameter is a bid comparison, so zero makes the local execution payload win
unconditionally, and that is the specification's own way to demand an unblinded
block rather than a separate flag. A blinded answer is still rejected rather
than assumed away, because the parameter is a request and signing a block this
client cannot unblind would burn the slot for something that can never be sent.
The fork comes from Eth-Consensus-Version and a missing or unknown value is a
hard error, not a default. SSZ carries no type tag, so guessing the fork means
decoding into the wrong shape and signing whatever root falls out. The match is
case-insensitive: the schema's enum is lowercase, but nothing in the
specification says a client must compare it that way. A node that ignores the
Accept header and answers JSON is caught by content type, so it is reported as
the mismatch it is rather than as a malformed block.
publish_block returns Imported or BroadcastNotImported rather than unit, because
publishBlockV2 answers 202 for something that is neither success nor failure:
broadcast, but not in the node's own database. Flattening that into Ok would
report a proposal as clean when the node that made it cannot follow its own
block, which usually means an unsynced execution layer or a parent that is not
what this client thought.
The body is borrowed rather than owned so failover can offer it to the next node
without copying. A block with blobs runs to megabytes and would otherwise be
cloned once per configured node on every proposal, succeed or fail.
Publishing stays first-wins rather than broadcasting to every node. A node that
accepts a block gossips it, so the others receive it over the network anyway,
and a broadcast makes the outcome ambiguous: three nodes answering 200, 202 and
a timeout have no single answer to report.
The mock answers produce_block with the same fixture block_contents' own tests
decode, rather than a second one that could drift from it, and honours the
contract so a test double behaves like a stale node rather than a more
permissive one than any real implementation.
* Produce, sign and publish a block for a proposer duty
The attestation path's counterpart, shaped like it. What differs is the cost of
each step, and that changes where the checks sit.
The guard is consulted twice. The early check is not about safety, it is about
cost: producing a block makes the beacon node drive an execution-layer payload
build, the most expensive thing this client can ask of it, and finding out
afterwards that the slot was already proposed wastes all of it. The check before
signing is the one that matters, and it is the one that records. Recording there
rather than after publication is what makes the duty loop's deadline safe, since
a proposal abandoned between signing and publishing cannot be re-signed.
A test pins that the early check does not record, using one service and one node
throughout. Two services would carry two empty guards and the test would pass
whether or not the first one recorded anything.
Two facts about forks turned out to be separate, which a test caught by
asserting they were the same. The block is decoded, and must be published, under
the fork the node named in its response header; a re-encoding of what the node
produced has to be announced as what it is. The signing domain comes from this
client's own schedule, fetched from /config/spec at startup. They normally agree
because they came from the same beacon node. When they do not, one of them is
wrong about where a fork boundary sits, and signing anyway yields a block
rejected for a reason nothing in the logs would explain, so they are now
compared and a mismatch refuses before signing.
ProducedBlock therefore carries its fork rather than having it recomputed from
the slot. That is also finer-grained than the payload shape: deneb and electra
share a container but are different forks, and the wire has to be told which.
The store's read lock is taken twice, never across an await, for the reason the
attestation path documents: the keymanager's import holds the write lock across
a batch of slow key derivations, and a write-preferring RwLock would let it
queue ahead of every later duty.
The block's root is taken once from the decoded container, and the same value is
what the guard records against and what the signature covers.
A 202 is logged at warn and counted separately rather than folded into the
proposal counter. The block reached the network, so the proposal may well have
worked; what failed is the node's own import, which points at its execution
layer rather than at this client.
* Propose this slot's block from the duty loop
The last piece: the loop now looks for a proposer duty at the boundary it wakes
on, and runs it before sleeping the rest of the way to the attester offset.
The budget is that offset, not the end of the slot, and it is deliberately
tighter than the attestation's own. The specification defines no
block-production deadline; the nearest thing it does define is the point
attesters stop waiting for a block and vote for the previous head, which is the
same third of a slot. A block published after it has already lost most of its
value, while every second spent there comes straight out of this client's own
attestations, which are due for every validator it holds rather than for the one
proposing. So an overrunning proposal is abandoned and the slot's attesters
still vote.
Dropping that future mid-flight is safe for the same reason it is on the
attestation path, and only for that reason: ProposalService records the slot in
its guard at signing time, so a block abandoned between signing and publication
cannot be signed a second time.
The duty is cloned out of the schedule before the await. It is three small
fields, and holding a borrow across the proposal would stop the next epoch's
refresh from replacing the schedule it came from.
--graffiti is refused rather than truncated when it is too long. The operator
asked for a string, and a silently clipped one would appear in every block they
propose, where they are least likely to look for it. The limit is stated and
checked in bytes because that is what the field holds: seventeen accented
characters are thirty-four bytes, and an operator told "32 characters" would
have no way to work out why theirs was rejected.
It defaults to empty rather than to this client's name and version, which is
what most clients do. Putting it there tells anyone reading the chain which
software built a block, and an operator who wants that can ask for it.
* Tell the beacon node where to pay block rewards, and check that it did
Two halves of one problem, and the second exists because the first is only a
request.
prepare_beacon_proposer is re-sent every epoch rather than once at startup,
because the node forgets. It keeps a preparation for the epoch it arrived in and
two more, and loses all of them when it restarts, so a client that registered
once would silently stop being registered a few minutes later and find out by
proposing a block that paid someone else. A failed registration is logged rather
than propagated: its consequence is a payload built for the wrong address, which
the check below catches, while propagating it would cost this epoch's attester
duties, which is worse and unrelated.
The specification is explicit that a node need not honour the preparation, and
says a client should confirm the fee recipient in a produced block before
signing it. It does not say what to do when they differ, and both answers cost
money. Refusing forfeits the consensus-layer reward as well as the
execution-layer one and costs the network a slot; signing forfeits only the
execution reward, which was already going elsewhere the moment the node built
the payload. So this signs and complains at error level with a counter beside
it, because the thing that actually fixes a misconfigured beacon node is the
operator noticing, not a dropped block.
Failover has a gap here worth naming rather than hiding: a preparation reaches
only the node that answered, so failing over to a second node later in the epoch
finds it unprepared. The check before signing is what catches that.
--suggested-fee-recipient is optional but startup warns loudly when it is
absent, because without it every proposed block pays its execution rewards to an
address the beacon node chose. It stays optional because a client running only
attester duties has no use for one. A malformed address is refused at startup
rather than at first use: a proposal duty can be days away, which is the worst
moment to discover a typo in a flag.
* Document what the validator client does now that it proposes blocks
The three pages that describe this client all said it attests and does nothing
else, which stopped being true.
cli.md gains the two new flags, a walk through what a slot now looks like from
the boundary inward, and the reasoning behind the two choices an operator would
otherwise have to read the source to understand: why a proposal is abandoned at
the attester offset rather than at the end of the slot, and why blocks go over
SSZ when everything else on this path is JSON. Its "not implemented" list is
shorter by one entry and now names the builder flow explicitly, since asking for
an unblinded block and refusing a blinded one is a deliberate limit rather than
a gap.
It also gains a short section on proposer duties being the more fragile
schedule. An attester schedule depends on the block two epochs back and survives
anything shallower; a proposer schedule depends on the block one epoch back, so
a reorg that leaves attester duties untouched can still move a proposer. This
client refetches once per epoch, so that window is real and an operator who
loses a proposal after a reorg deserves to find the explanation written down.
metrics.md gains the six proposal series, and a note on which two should read
zero for the life of a healthy deployment and are therefore worth alerting on at
any value rather than on a rate: the two refusal counters mean this client's own
guards caught something, and the fee-recipient counter means blocks are paying
someone else.
spec_deviations.md now covers proposals in the slashing-protection entry rather
than attestations alone, including the rule the proposal guard enforces and why
a proposer slashing is the worse of the two: a double vote needs a second
validator's attestation to be caught, while two signed headers for one slot are
the whole of the evidence by themselves.
* Say "or a double block proposal" in the startup warning
The client proposes blocks now, so the one message an operator is guaranteed to
read was naming half of what it can lose stake for. A double vote needs a second
validator's attestation to be caught; two signed headers for one slot are the
whole of the evidence by themselves, so if either belongs in a warning it is
that one.
* Work out whether a validator is an aggregator, and sign what that needs
Aggregation's two pure pieces, ahead of the endpoints that use them.
The selection rule is not a choice and the code says why. A validator signs the
slot under a dedicated domain, hashes that signature, and aggregates when the
first eight bytes read as a little-endian integer divide evenly by a modulus
derived from the committee's size. BLS signatures are deterministic, so a
validator gets exactly one answer per slot and cannot search for a better one,
and cannot decline either, since the same computation is what a beacon node
checks the resulting aggregate against.
The modulus floors at one, which is not a guard against division by zero so much
as the intended answer for a small committee: every member of a committee that
cannot supply the target number of aggregators should supply one.
Two tests exist because the obvious one cannot fail. Reading the digest
big-endian would still select about one validator in `modulo`, so a statistical
test looks correct either way; only comparing the two orderings on a signature
where they disagree catches it. The statistical test is kept anyway, with loose
bounds, since the count is binomial and pinning it exactly would be testing
SHA-256 rather than this function.
The two new signatures round out the five this client produces, and the
domain-separation test now asserts the property across all of them rather than
three. One case is sharp enough to state separately: a RANDAO reveal signs an
epoch's merkle root and a selection proof signs a slot's, both bare uint64
roots, so for epoch N and slot N the signed object is byte for byte identical
and only the domain tells them apart. A test asserts the collision first, so it
cannot quietly stop proving anything.
Neither new signature is slashable. That is worth stating for the aggregate one,
which is the only signature here covering an attestation and not being guarded:
the slashing conditions are about a validator's own vote, and an aggregator is
republishing other validators' votes with a wrapper saying who collected them.
TARGET_AGGREGATORS_PER_COMMITTEE stays in this crate rather than moving to
ethlambda-types. It governs how this client behaves, not what the chain agrees
about, and no container's shape depends on it.
* Give the slot clock a second offset, two thirds in, for aggregation
A slot now has three points a duty falls on rather than two: the boundary for a
proposal, a third in for an attestation, two thirds in for an aggregation.
Aggregation has to come after attestation, not merely elsewhere. An aggregator
folds together votes its beacon node has collected, and before the attesters
have voted there is nothing to fold. A test asserts the ordering across several
slots rather than trusting the two constants to stay in the right order.
The offsets now multiply before dividing. Dividing first would put a 10-second
slot's duties at 3 and 6 seconds instead of 3.33 and 6.66, which is a third of a
second of drift on something that is already a deadline. The named constants
went from one divisor to a part-of-three pair, so the arithmetic reads as what
it is rather than as two unrelated fractions.
Duration division keeps nanosecond precision, so the test that pins the uneven
case asserts exact thirds rather than rounded milliseconds, and checks the two
gaps are equal rather than just checking each endpoint.
* Claim the aggregator role in the subscriptions that already go out
The flag was hardcoded false with a comment saying aggregation was out of
scope. It is in scope now, and the flag is not a preference: it is the answer
the selection rule computes from a signature over the slot, and the beacon node
checks the same thing when the aggregate arrives.
What the flag buys is why it has to be sent an epoch ahead rather than
discovered when the aggregation duty runs. It tells the node to hold the subnet
subscribed and collect the votes this client will later ask it to fold. A client
that decided at aggregation time would be asking a node that had not been
listening.
selection_for is one function for both callers because they have to agree.
Computing the two differently would either claim a role this client never
performs, leaving a committee's aggregate to nobody, or perform one it never
claimed, against a node with nothing collected. It returns the proof rather than
a bool, since the caller that acts on it needs the signature itself: it goes
into the published AggregateAndProof, which is what makes the selection
verifiable rather than self-declared.
A duty whose selection cannot be computed is subscribed with the flag clear
rather than dropped. The subscription is what puts the beacon node on the subnet
at all, so losing it would cost that validator its plain attestation as well as
its aggregate.
Two tests, because one is not enough. The first checks the flag against the
selection rule over a range of slots, which a bug that always answered the same
way would also satisfy. The second checks that across 64 slots the role is
claimed for some and not others.
refresh_epoch now takes the store and signing context, since the subscription it
sends needs to sign.
* Fetch and publish the aggregate this client was selected to produce
The half that was missing. Subscriptions already claimed the role; now the
client performs it, so the claim is no longer a promise it breaks.
This client does not build the aggregate. The beacon node does, out of votes it
collected on the subnet the subscription put it on, and hands back the best one
it has. What this client contributes is the wrapper: which of its validators is
publishing, the proof that validator was selected, and a signature over both.
That division is what keeps electra's trap out of reach. A gossiped aggregate
must cover exactly one committee even though the container widened to allow
more, and the multi-committee form exists only on chain, assembled by a
proposer; a client that never constructs an aggregate cannot get that wrong.
The attestation duty now hands its data forward rather than the aggregation
duty re-fetching it. The aggregate to ask for is the one covering the votes
this client's validators just cast, and a head that moved in between would give
different data whose aggregate contains none of them. It is an Option, not a
zeroed default, because a caller handed a default would ask for the aggregate
of an attestation nobody made. It is Some whenever the fetch succeeded even if
nothing was published, since an aggregator collects the whole committee's
votes and still owes the duty when its own signatures were refused.
Only the v2 endpoints exist. The v1 pair was removed, not deprecated, because
electra forces AttestationData.index to zero: the data root no longer
distinguishes one committee's votes from another's in the same slot, so
committee_index had to become a separate required parameter and the old shape
had no way to ask the question. There is no v1 fallback and there should not be.
One fetch per committee rather than per validator. Two of this client's
validators can be selected for the same committee and publish one wrapper each
around the identical aggregate; asking twice would be the same answer at twice
the cost.
A 404 is an ordinary outcome, not a failure. It means the node had nothing to
fold, and one committee's missing aggregate must not stop another's from going
out. It is still worth failing over, since a second node may have been on the
subnet when the votes arrived.
The batch goes out under a single consensus-version header, so all its
aggregates must share a fork. With failover in play they can come from different
nodes, which is the only way they would not.
The aggregation deadline is the end of the slot rather than the attester offset
the proposal uses, because what it competes with differs: a proposal that
overruns eats this client's own attestations, while an aggregation is already
the last duty in its slot. Abandoning it is cheap and safe, since nothing here
is slashable and other aggregators were selected for the same committee.
The aggregate query string is built by a pure function with a test, because it
uses a string continuation: Rust strips the newline and the indentation after
it, and a misplaced one would put a space inside a query parameter, making the
node answer about a different committee with nothing naming the cause.
* Document aggregation, and say where the duty offsets come from
The three pages now cover all three duties. cli.md gains what a slot looks like
with aggregation in it, and why the role is not a choice: a validator signs the
slot, the hash of that signature decides, signatures are deterministic, and the
beacon node checks the same thing when the aggregate arrives.
Both cli.md and the slot clock now say where the offsets come from, because the
specification moved and this code did not. The honest-validator guide used to
express them as fractions of SECONDS_PER_SLOT and now states them as basis
points of a slot duration the beacon node reports: 3333 and 6667. On mainnet's
12-second slot those are 3999 ms and 8000 ms, so the aggregation offset here is
exact and the attestation offset is a millisecond late, well inside the network
latency it competes with. Thirds are kept rather than reading two config values
for a millisecond, but that stops being a rounding question at gloas, which
moves the offsets to 2500 and 5000 basis points. Written down so whoever meets
that fork knows it is a decision and not an oversight.
Also drops a test that asserted Root::ZERO equals Root::ZERO. Its real content
was that both duty aliases name one generic type, which the compiler already
enforces, so it looked like it checked something and did not.
* Stop an overrunning slot from costing the next slot its duties as well
The duty loop asked the clock for "the next slot after now", which is the wrong
question once the previous slot's work has run long.
A slot's duties are bounded by that slot's end, so a hung beacon node returns
the loop to this call at, or just after, the following slot's boundary. The
clock then answered with the slot *after* the one the loop was standing in, and
the loop slept straight past a slot whose attestation was still a third of a
slot away. One hung request cost two slots of duties for every validator held,
which is the cascade the per-slot budget was added to prevent.
Concretely on mainnet: the attestation timeout fires at exactly end_of(S), which
is start_of(S+1). slot_at(now) is then S+1, the old code returned S+2, and slot
S+1 got neither a proposal nor an attestation despite four seconds of its
attestation window remaining.
So the clock now takes the slot just served. If the wall clock has already
entered a later slot, that slot is returned with no delay and served at once;
its own offsets degrade to zero on their own, so its attestation goes out late
rather than not at all. Otherwise the answer is the slot after the one served,
which is also what keeps the loop from spinning on a slot it just finished.
A backwards clock step is handled by the same rule and idles until the stepped
clock catches up, which is right: the earlier slots have been served, and the
duties for a slot that has not happened yet cannot be fetched.
The aggregation work landing alongside this makes it more likely, not less: the
loop body now sleeps to the two-thirds offset before its last duty, so it ends
within a few seconds of the boundary as a matter of course and any hiccup
crosses it.
* Send subscriptions and fee recipients to every node, not the first that answers
try_each is the right rule for a query: there is one right answer and any node
can give it. It is the wrong rule for these two calls, which install state on a
node rather than ask it something. A node that was never told is a node that
cannot serve the duty later.
With two nodes configured, try_each told node 1 which committees this client
attests in and which it will aggregate for, and node 2 heard nothing. When node
1 went down mid-epoch, every later call failed over to the node least prepared
to answer it: not holding those attestation subnets open, and holding none of
the votes an aggregate is folded from, so 404 for every committee this client
asked about. The failover node exists for exactly that moment and was the one
guaranteed to be useless in it.
The fee recipient had the same shape. That one was already documented as a known
limitation with the pre-signing check named as the mitigation, which was true but
is no longer the best available answer.
try_all succeeds if any node accepted rather than requiring all, so one
unreachable node does not stop the reachable ones being told, and logs per node
so a partially registered client is visible rather than silent.
A test pins the opposite rule too, so the two do not drift: a block request
still stops at the first node that answers, because asking every node for a
block would make every node build one.
* Refuse a deneb block by name instead of failing to decode it as electra
Both forks were routed through the same container, which cannot work: a deneb
BeaconBlockBody has twelve fields where electra's has thirteen, so the two have
different fixed-size prefixes and a deneb body fails electra's decoder on its
first offset.
The failure was at least the safe one. It is a hard decode error, not a silent
mis-decode producing a signature over the wrong block, and a post-electra chain
never reaches it because the proposal path already refuses to sign when the
node's fork and this client's schedule disagree. What it cost was honesty: on a
deneb chain the client claimed support and then proposed nothing, reporting a
malformed block each time.
Deneb now joins the refused arm rather than gaining a container pair, because
the pair would serve nobody. The attestations this client submits are electra's
SingleAttestation, which has no pre-electra form, so a chain it cannot propose
on is one it could not attest on either. Refusing by name says that; a decode
error does not.
* Say why the store's read guard is narrow, now that the old reason is stale
Four comments justified scoping the read guard away from every await by saying
the keymanager's import holds the write lock across a batch of slow EIP-2335
derivations. That was true once. The import now does the derivation in
prepare_import, outside the lock, and takes the write lock only for the
synchronous inserts, which its own comment explains.
The scoping is still right; the reason given for it was no longer the reason.
What actually makes a wide read guard costly is that tokio's RwLock is
write-preferring, so a single keymanager writer queuing behind a long-held
reader blocks every reader after it. Keeping these guards narrow bounds that
wait, and is the duty path keeping its half of a bargain the import already
keeps.
* Refuse to sign against a beacon node that is optimistic, not just one syncing
The client asked its node whether it was syncing and signed whenever the answer
was no. A node can answer no and still be tracking a head its execution client
has not validated, and the specification is explicit about that case: an
optimistic validator MUST NOT produce a block and MUST NOT participate in
attestation, naming the proposer, attester, selection and aggregate domains.
The field wa…
* feat(beacon): judge a block by the rules that need no post-state Beacon gossip validation needs the `beacon_block` rules that can be applied before a block's own state transition runs, and the chain actor will want the same rules before it parks a block. They live in one place, `precheck::precheck_block`: a slot after its parent's, the proposer fulu's lookahead already fixed for that slot, and a valid proposer signature. A block whose parent has no post-state yet is judged against a recent state, for the signature alone. `bls::DST` is public and `test_state` gains `secret_key_for`, so a test can sign a block these rules accept. * feat(types): expose a beacon block's blob count and payload timestamp The beacon_block gossip rules bound the commitment count and check the payload timestamp before any state is read; both fields sit behind the fork enum. * feat(beacon): gossip verdict types and seen caches Gossip validation needs a verdict that says both what gossipsub does and whether the chain still wants the object (the spec's MAY-queue cases); fixed reason enums keep metric labels bounded. * build: fetch the spec's gossip validation vectors They first ship in v1.7.0-beta.1, so they come from a tree of their own instead of moving the main fixture pin. Only the fulu block and column handlers are extracted, since those are the topics this node validates. * feat(beacon): beacon_block gossip rules Split by cost so the p2p actor can run the state-free half inline and the rest off its loop; reuses precheck for the proposer and signature rules and reads states only from the cache. * test(beacon): run the spec's beacon_block gossip vectors One vector is skipped by design (a parent without a post-state is queued until a bad-block cache exists); everything else must match. * feat(beacon): data_column_sidecar gossip rules The proposer check reads the parent's lookahead instead of advancing a cloned state, so it is cheap enough for the gossip path; slots outside the window are queued as the spec allows. * test(beacon): run the spec's data_column_sidecar gossip vectors reject_parent_failed_validation is skipped for the same reason as the block runner's parent-not-verified vector: without a bad-block cache we cannot tell a failed parent from one still importing, so it is queued rather than rejected. * fix(beacon): judge the lookahead last and never reject on a failed ancestor walk Code-review fixes to the beacon gossip validation rules: - Block `stateful_checks` ran `precheck::fixed_proposer(..).is_none()` before `precheck_block`, the finalized-ancestry check and the payload-timestamp check. A block whose slot the lookahead cannot place at all (before the parent's own epoch, or a pre-fulu parent) short-circuited into `Queue(ShufflingUnavailable)` after only a head-state signature check, turning spec REJECTs (slot not after parent, finalized not an ancestor, bad timestamp) into IGNOREs. The lookahead gate now runs last, after `precheck_block` already checked slot order, known proposer and signature against the parent's own state. - `descends_from_finalized` collapsed a failed ancestor walk into "not an ancestor", but `fork_choice::get_checkpoint_block` erroring means a row is missing from `LiveChain` (for example after `invalidate_subtree` deletes a late-invalidated parent's rows while its cached state stays), not that the chain has forked away from the finalized checkpoint. Rejecting such a block penalizes peers who haven't yet learned of the invalidation. Replaced the bool with a three-way `FinalizedAncestry` (`Descends` / `Conflicts` / `Unknown`), used by both `gossip::block` and `gossip::column`; `Unknown` queues (`ParentNotReady`) instead of rejecting. - `column::verify_header_proposer` checked the lookahead before the signature, so a slot the lookahead cannot place returned `Queue(ShufflingUnavailable)` before ever checking the signature, even though the signature does not depend on the shuffling. The signature (and the known-proposer check) now run unconditionally; only the proposer-identity question is deferred to a `Queue` when the lookahead has no answer. Tests now cover the head-state signature path for an unknown parent, the lookahead queue path once the parent's own checks have run, and the regression case (a slot before the parent's own lookahead window used to mask a not-after-parent violation as a queue). The spec runner's config fallback now covers every fork up to the case's own rather than hardcoding fulu, and its comment cites the vectors' own README convention instead of a claim the design spec doesn't make. Clock-disparity magic numbers in the gossip unit tests are now derived from `MAXIMUM_GOSSIP_CLOCK_DISPARITY`. * feat(metrics): count beacon gossip verdicts and late verdicts Validation now gates propagation, so how often each rule fires, how long a verdict takes, and whether it beat gossipsub's cache eviction are the three things to watch. * feat(p2p): hold beacon gossip until it has a verdict Gossipsub forwarded every beacon message before anything checked it; lean keeps auto-forwarding because its handlers produce no verdicts. Beacon gossip stops propagating until the verdict plumbing lands; do not deploy this commit alone. * fix(beacon): check a queued column's header signature, and test every rule A queued column is written to the chain actor's PendingDataColumns the same way a queued block is, but only blocks got a signature check before being parked: a forged block is rejected up front, a forged column just sat there. Give columns the same treatment on both queue paths that precede the parent's own state (parent unknown, parent known but uncached), reusing the head-state check block.rs already has. Where the parent's state *is* already in hand but the finalized-ancestry walk itself fails, check the signature against that state instead of the head's, since it is the authoritative one. Also add the tests this rule and the surrounding ones were missing (one per outcome in the design's rule tables for both beacon_block and data_column_sidecar), fold the duplicated `store`/`seen`/`fulu_parent`/clock test helpers shared by block.rs and column.rs into a `test_support` module, and reword two doc comments whose sources had drifted from what they now claim. * feat(p2p): run stateful gossip checks off the actor loop The chain actor's mailbox waits reach seconds at p90 on the followers, past gossipsub's message-cache window, so verdicts come from bounded one-off blocking tasks that report back to the p2p actor instead. * feat(p2p): validate beacon gossip before propagating it Blocks and data column sidecars are judged by the fulu gossip rules and forwarded only on Accept; the other subscribed topics are ignored until they get validators. Undecodable payloads are now rejected instead of relayed. * refactor(p2p): decide a beacon gossip verdict before acting on it Each beacon gossip handler both decided a verdict and acted on it (calling verdict::report or verdict::spawn_stateful_checks inline), which meant the decision could only be exercised through a running actor's Context and never in a unit test. Split the two: the handlers now return a Dispatch describing what to do, and handle_beacon_gossip is the single place that reports or spawns. That single call site is also where a future cheap check that answered Queue would need to forward its object; today no cheap check does, so Report carries no object, and a debug_assert! catches the invariant instead of widening the type for a case that cannot happen yet. The verdict-time seen-cache re-check (first Accept wins) moves out of the actor's Handler impl into its own settle function for the same reason: it needs a &mut P2PServer, not a Context. Both the triage decisions and the settle re-check now have unit tests. A caught validation panic is also logged, since Ignore(Internal) in the metrics previously left no trace of what panicked. * fix(beacon): keep overloaded gossip, and check a column's signature first With every validation permit taken, a message was ignored and dropped. Gossipsub's duplicate cache then hides every other peer's copy, so the chain actor learned of the block only through a child's by-root fetch or range sync. It is still ignored, but now also handed to the chain actor, which judges it itself as it did before gossip validation existed. A column's header signature is now checked right after its parent's state is found, before the finalized-ancestry walk, the inclusion proof and KZG, so a forged header costs one BLS check rather than all three while holding a permit. The lookahead is judged last, as for blocks. The already-stored check read every stored sibling sidecar inline in the p2p actor; it is now a point lookup (`Store::has_data_column`). The p2p test helper ignored its config, so a triage test passed for a different reason than its comment gave; it now honours it. * docs: describe beacon gossip validation Every beacon message now waits for a verdict, so the wire guide, the metrics reference and the chain actor's notes on parked sidecars said things that are no longer true: that columns are deduplicated by a finality-pruned set, that fabricated headers park freely from gossip, and that the import's decode phase ends at the decode. * test(beacon): pass the gossip runner's on_block a committee cache #22 added the parameter, and the merge left the gossip conformance runner calling the old signature, so the spec-test binary no longer compiled. A fresh cache per call, as the fork-choice runner does. * build: derive the gossip fixture configs from the main tree's Each run reads its own preset from both fixture trees, so the two config lists always named the same presets and CI had to set both to agree. The gossip list now follows CONSENSUS_SPEC_TESTS_CONFIGS minus `general`, which has no `networking` runner to extract.
…ctor (#43) The chain actor re-ran, on its single thread, the column checks that gossip validation had already run in p2p: a KZG batch, a BLS verification and a `process_slots` per sidecar. After an empty slot, that `process_slots` merkleizes the full mainnet state once per sidecar (~1.5 s each, 8 custody columns). On the eth-4 follower on 2026-09-23 this held block 15280808 in the actor's mailbox for 7.6 s, and then for another 4.7 s of "columns_wait" that was really the actor checking columns that had already arrived. The checks now live only in `beacon::gossip::column`. A sidecar gossip did not accept (`Queue`, `Ignore(Overloaded)`), every sidecar fetched by root or by range, and parked sidecars replayed after their parent imports all go through the new `chain_checks`, run by `p2p::beacon::column_checks` on blocking threads. The chain actor stores what reaches it without checking it; debug builds re-run `chain_checks` there, so a p2p path that forwards an unchecked sidecar fails a test. `chain_checks` reads the expected proposer from the parent state's proposer lookahead, and only advances a state clone for a slot outside it. The column rejection and KZG metrics move with the checks.
) * feat(rpc): report the preset and constants from /eth/v1/config/spec The endpoint echoed only the network's Config. Lighthouse's validator client reads PRESET_BASE from it, defaults an absent key to "", and refuses any beacon node whose value differs from its own preset, so every lighthouse VC pointed at ethlambda rejected it as incompatible. The Beacon API also asks for the preset and the constants in the same object. The key set now matches lighthouse's, less gloas-only keys (this build cannot process gloas) and three that lighthouse reports but the specification does not define (GAS_LIMIT_ADJUSTMENT_FACTOR, RESP_TIMEOUT, TTFB_TIMEOUT). CONFIG_NAME is not persisted in Config, so it is threaded from the resolved network into the router. TARGET_AGGREGATORS_PER_COMMITTEE and SYNC_COMMITTEE_SUBNET_COUNT move into the types crate's constants, so the endpoint and their existing users name one value. * refactor(types): store PRESET_BASE and CONFIG_NAME in Config /eth/v1/config/spec reported CONFIG_NAME from a string threaded from startup through three RPC signatures, and PRESET_BASE from the compiled preset, while every other value it reports comes off store.config(). Holding both names in Config lets the endpoint read everything from the one Config the store holds, and removes the separate identity pass over config.yaml. Config is SSZ-encoded into Metadata["config"], so the names are bounded byte lists (ConfigName, 64 bytes) and DB_VERSION goes to 4. Existing data directories are refused on start and need a fresh checkpoint sync. An absent PRESET_BASE still parses as empty rather than falling back to mainnet's value like other absent keys, so the startup preset check keeps failing closed. Neither name is compared on resume: the preset is already pinned by check_preset and the directory's preset byte, and a renamed CONFIG_NAME is the same chain. * feat(beacon): refuse to resume under a different config or preset name /eth/v1/config/spec reports the names stored in the directory's Config, so resuming under a config file that names another network or preset would keep reporting the old ones. first_config_difference now compares both, and ConfigName prints as quoted text so the error reads `config_name: directory has "mainnet", config file says "hoodi"`. Also raises the ConfigName bound to 256 bytes. The bound is not part of a list's SSZ encoding, so DB_VERSION stays at 4. * fix(beacon): keep /eth/v1/config/spec truthful and resumes forgiving The spec endpoint reports the custody, subnet, request-limit and message-domain keys off the stored Config, but the node runs on compile-time constants for all of them, some of which size a type. A config.yaml that set any of them differently made the endpoint report values the node does not use, on a node that could not follow that network anyway. check_constants now refuses such a network at startup, next to check_preset, for built-in and directory networks alike, naming every key that differs. A directory built for the other preset used to fail as an SSZ error, because NetworkDir::load decoded genesis.ssz with the compiled preset's container bounds before the preset was checked. Both checks now run before that decode. A changed CONFIG_NAME on resume only warns now. It is a label with no consensus effect, and refusing it cost a renamed devnet's nodes their data directories; PRESET_BASE and every chain value are still refused. ConfigName holds a String, so as_str borrows instead of returning a Cow that re-checks UTF-8 on every call. Its hand-written SSZ impls write the same bytes the byte list did, so DB_VERSION stays at 4. HexPrefixed is public and replaces the copies of format!("0x{}", hex::encode(..)) in the RPC, validator and engine crates.
CI: run jobs on Ubicloud behind the ETHLAMBDA_RUNNER variable
…ot (#48) hold_block_for_columns asked peers for the missing columns the moment it held a block. A block's columns are published alongside it, so at that moment the rest are almost always still in flight on gossip, and the peers asked usually do not have them yet either: they answer empty and the lookup burns its attempts. On the mainnet followers at the tip, gossip completed every held block within 0.3 s (p99), and stored columns equalled gossip-decoded columns, so the by-root fetch contributed nothing while putting hundreds of mostly empty requests an hour on the wire. The per-slot re-drive already re-asks for every held block's missing columns. Make it the only asker: a new hold waits for gossip, and the first tick after it asks for whatever is still missing.
…o attestation subnets (#19) * feat(beacon): the rules for aggregates and attestation subnets Neither is wired to anything yet: this is the pure half, so the gossip handler and the chain actor that follow have something to call. `subnets` is `das`'s counterpart for attestations. A beacon node owes the network `SUBNETS_PER_NODE` long-lived subscriptions chosen from its own node id, because phase 0 has no shard committees and so nothing else gives the attestation subnets a stable membership; a follower holds them for the mesh's sake, not for its own head. Two byte-order traps are what make this worth its own module with fixtures: the prefix is the leading bits of the *big-endian* node id, while the permutation seed is hashed over the *little-endian* period, and either one alone yields a set that agrees with no other client. The prefix width is derived rather than read off `Config`, although #15 gave `ATTESTATION_SUBNET_PREFIX_BITS` a typed home. The specification defines it as a derivation over `ATTESTATION_SUBNET_COUNT`, so deriving it cannot disagree with the count it is taken over, whereas a configuration that narrows the count and omits the prefix key would fall back to mainnet's 6 and shuffle over a space its own count does not match. A test pins the two together for the configurations that do carry both. `aggregate` holds the `beacon_aggregate_and_proof` conditions that need a state. It is deliberately not the whole validator: the conditions needing nothing but a clock belong in the p2p handler, and the two seen-set gates belong in the chain actor, because the specification marks an aggregate seen only *after* its signatures verify. Recording it earlier is a one-message censorship attack, since a garbage aggregate claiming some `(epoch, aggregator)` pair would drop that aggregator's genuine one. Lighthouse splits it the same way, reading its observed-sets early and writing them in `verify_late_checks`. Committees resolve against the aggregate's own target checkpoint state, not `store.block_states[get_head(store).root]` as the pseudocode has it. That is stricter, since the head may sit on a branch the aggregate does not vote for while the target is an ancestor of the attested block by `validate_on_attestation`'s own consistency check, and it means one state serves both these conditions and the `on_attestation` that follows. `SignedAggregateAndProof` moves down from `ethlambda-p2p`, which re-exports it: the gossip path no longer ends at the decode, so the type has to sit where both consumers can reach it. * feat(beacon): apply gossip aggregates to fork choice, and backbone two subnets `beacon_aggregate_and_proof` was subscribed, decoded and dropped, so fork choice learned its votes only from block bodies. It now reaches `on_gossip_aggregate`, and the ENR stops claiming this node serves no attestation subnet when the specification asks every node to serve two. The path is p2p -> `new_beacon_aggregate` -> chain actor, and each layer keeps the checks it can actually answer. p2p runs the conditions needing nothing but a clock, so a stale or malformed aggregate never costs a mailbox slot. The actor runs the rest, because the committee-shaped conditions need a state and the two seen-set gates need the verification verdict: the specification marks an aggregate seen only after its signatures verify, and a set written earlier is a one-message censorship attack, since a garbage aggregate claiming some `(epoch, aggregator)` pair would drop that aggregator's real one for the epoch. The deferral queue is load-bearing, not an optimization. `validate_on_attestation` requires `get_current_slot(store) >= data.slot + 1` and aggregates are published two thirds of the way through the slot they vote for, so every one of them arrives a slot too early. Without the queue this topic would apply approximately nothing. It drains once per beacon tick, between the clock advancing and the head being recomputed, so the votes it releases are in fork choice before the head that tick reports is chosen. The superset-of-bits gate is what makes the volume affordable rather than any amortization: a committee's `TARGET_AGGREGATORS_PER_COMMITTEE` aggregators mostly converge on the same votes, and once the running union covers a later aggregate it is dropped for a hash and two lookups instead of three signature verifications. Per-aggregate costs the gates do not remove are left in place deliberately, one `block_index()` scan and one `EpochCommittees` build each, until measurement says they matter. Which is what the four new metrics are for: decode in p2p, the mailbox hop, the processing span and the end-to-end time. Without them a cost regression here surfaces as an unexplained head lag. The subnet backbone is separate work sharing this branch. Every beacon node should hold `SUBNETS_PER_NODE` long-lived subscriptions chosen from its node id so the attestation subnets have a stable mesh for validators to publish into; what arrives on them is relayed but never applied to fork choice, which is what a lighthouse node with no validators does too. Signatures are not verified there, and that is the one place this stops short of lighthouse on purpose: the relay has already happened by the time a handler sees the payload, since this node runs gossipsub without `validate_messages()`, and nothing downstream reads the verdict. Peer scoring is what should turn it into a real verifier. `SignedAggregateAndProof` moves again, from the state-transition crate to `ethlambda-types`. `ethlambda-network-api` has to name it and depends only on that crate, deliberately, so the container lives there and the one conversion needing fork choice's own `Attestation` stays behind as a `From`. * docs(beacon): describe the aggregate path and the subnet backbone `beacon_wire.md` opened by saying this node "does not read the aggregate topic, so fork choice learns its votes from block bodies", which is now the opposite of what it does. The new section says where each of `validate_beacon_aggregate_and_proof_gossip`'s conditions runs and why that split is forced rather than chosen: the seen-set gates need the verification verdict, so recording on arrival would let one garbage aggregate censor an aggregator's real one for the epoch. Two departures are written down as departures, with their reasoning, so neither reads later as an oversight. Committees resolve against the target checkpoint state rather than the head's, which is stricter. And the subnet backbone is subscribed and relayed but neither applied to fork choice nor signature-verified — the first matching a lighthouse follower, the second stopping short of one, because this node runs gossipsub without `validate_messages()` so the relay has already happened and no verdict is read. `metrics.md` gains the six aggregate metrics with what to watch and why. `mailbox_wait` gets the emphasis: it is the failure mode this path introduces and the only one no other timing can show, since aggregates queueing behind block imports arrive too late to move the head while every per-aggregate timing still looks healthy. * docs(beacon): name the flag that would make subnet verification worth paying for The subnet handler and `beacon_wire.md` both said turning peer scoring on is what should make it verify signatures. That names the wrong trigger: `validate_messages()` is what holds propagation until a verdict is reported back, so it is what gives a verdict somewhere to go. Acting on the old wording would mean enabling scoring and finding nothing had changed. Also says what does *not* differ. Lighthouse sets `validate_messages()` and pairs it with `ValidationMode::Anonymous`, the same mode this node uses, so the mode is not the distinction between the two clients; that one flag is. Worth writing down, since `ValidationMode::Anonymous` is the line a reader of this node's swarm setup actually sees and could easily read as "validation is off". * test(beacon): pin the per-slot aggregate bound per preset MAX_AGGREGATES_PER_SLOT derives from MAX_COMMITTEES_PER_SLOT, which the minimal preset shrinks, so asserting the mainnet answer failed both minimal CI jobs. Assert each preset's own value rather than the definition, which would restate the arithmetic instead of checking it. * refactor(beacon): move the attestation subnet math to the p2p crate Every function in subnets.rs is a pure function of a node id, a slot and Config; none of them touches a BeaconState. What they describe is which topic a message goes on, so they belong with the wire code that consumes them rather than with the state transition. The two primitives they do need, the swap-or-not shuffle and SHA-256, are imported. das.rs is the same shape for columns and stays put: the networking fixture suite has handlers for get_custody_groups and compute_columns_for_custody_group, and that runner lives in the state transition crate. aggregate.rs stays for a different reason, that is_aggregator and the gossip conditions both take a state. * fix(beacon): decode subnet attestations as SingleAttestation from electra on From electra, beacon_attestation_{subnet_id} carries a SingleAttestation, not an Attestation. The decoder chose the fork from a slot read four bytes in, where phase0's Attestation keeps it. On a SingleAttestation those bytes are part of the committee and attester indices, and whichever fork they landed on, the bytes were then decoded as a container they are not. Every subnet vote on mainnet was counted as decode_failed. Relaying was unaffected, since gossipsub forwards a message before the handler sees it. The payload cannot name its own fork on this topic, because electra moved the slot. The fork now comes from the topic instead, the way p2p-interface.md types every topic by its fork digest: BeaconWire carries the fork its digest was computed at, and decode_attestation takes that fork rather than the config. * perf(beacon): resolve gossip aggregate committees through the committee cache The committee cache made EpochCommittees::new shuffle the whole active set up front, which only pays off when callers share the result. The aggregate path still had two uncached get_beacon_committee lookups (is_aggregator and the committee-size check), so every aggregate paid two full shuffles of ~2.4M validators. On the eth-3 mainnet follower that was ~956 ms per applied aggregate, up from 65 ms, which saturated the chain actor and stopped the head. Both lookups now go through the actor's cache, the same one verified_attesting_indices already uses.
…relaying them (#47) * refactor(beacon): keep the committee cache in the Store, shared by both actors Validating aggregates and subnet attestations in p2p needs the same shufflings the chain actor derives, and a shuffle of the mainnet active set is too expensive to run twice per epoch. The actor owned its CommitteeCache as a &mut value, so nothing else could read it. The cache now lives in the Store, which both actors already share, and locks internally: the entry list is locked only to find or insert an entry, never while a shuffle runs, and each entry is a OnceLock so concurrent misses on one shuffling run one derivation and the rest wait for it. Every method takes &self, so the STF and fork choice take &CommitteeCache where they took &mut. storage cannot name state_transition's types, so the pieces split by what they need: EpochCommittees, a plain data type, moves to ethlambda-types; the container, keyed by an opaque ShufflingKey, moves to ethlambda-storage; deriving a shuffling and its key stays in state_transition, behind CommitteeCacheExt so call sites keep calling .committees(state, epoch). No behavior changes. * fix(p2p): refuse a 64-bit attestation subnet prefix The guard accepted a prefix of exactly u64::BITS, and the shift that follows it overflows there: a panic in debug, a wrap to 1 in release. No shipped config reaches it, but a custom --network config can. * feat(beacon): gossip rules for aggregates and subnet attestations beacon_aggregate_and_proof and beacon_attestation_{subnet_id} get the same treatment #38 gave blocks and columns: gossip::aggregate and gossip::attestation hold every condition of the spec's validators, split into cheap checks (message, clock, seen caches) and stateful checks. Three deliberate departures from the spec's text, each documented at the module: - The state is the voted block's cached post-state, not the head's. It is the attested chain's own state, so the shuffling is the one the attesters were assigned even on a fork whose deciding block differs from the head's, and its block_roots answer both ancestry checks without a LiveChain scan. - Pubkey-only signatures run before any committee is derived. A state that cannot key its shuffling derives one uncached, which is a full shuffle of the active set; a forged message must not reach that step. - A voted block with no cached post-state is IGNORE, not the spec's REJECT, for the reason blocks and columns already give: without a bad-block cache, a block that failed is indistinguishable from one still importing. The seen caches follow the spec's semantics rather than a running union: an aggregate is ignored only when one already-accepted aggregate covers its bits, so aggregates the spec says to propagate are not dropped. Both are LRU-bounded, never pruned by finality. The v1.7.0-beta.1 vectors for both topics now run on both presets. The two reject_block_failed_validation cases are skipped for the reason above. The fixture stamp now names its handler list, so a tree extracted before a handler was added re-extracts. * feat(p2p): validate aggregates and subnet attestations before relaying them Since #38 a beacon message is relayed only once p2p reports a verdict. Aggregates were validated in the chain actor instead, after a one-slot deferral, so they were never relayed, and subnet attestations got no verdict at all. Both now run through the verdict path with the rules from the previous commit. Only an aggregate gossip accepted reaches the actor, carrying the attesting indices whose signature p2p verified. The actor no longer derives committees or checks signatures: apply_verified_aggregate runs fork choice's own conditions and records the votes. This closes both blocking review findings on #19. The deferral queue can no longer be filled with unverified aggregates, and no aggregate rebuilds committees on the actor. The actor's applied-bits gate is pruned by the store clock rather than by finality, and a drain scans the block index once, only when something is ready. Aggregates and subnet attestations get a permit pool of their own, so a per-slot burst of them can never make a block or column answer Overloaded. An aggregate is forwarded only on Accept, never on Overloaded, since the actor no longer judges it. Subnet attestations are relayed but never forwarded, as before. The seen-attestation capacity is sized from the per-subnet bound, times this node's backbone subnets.
…re check (#49) Every signature check decompressed and subgroup-checked each signer's public key from its compressed bytes. A mainnet attestation aggregate carries a whole committee of keys, the same active keys sign again every epoch, and that repeated validation was nearly all of an aggregate's verification cost. With aggregates validated in p2p, a slot's burst of them saturated rayon's pool on a 16-core follower and about 15% answered IGNORE(overloaded) unchecked; block import saturates the same pool when a block's attestations are checked. `key_validate` is a pure function of the key's bytes, so its successful answers are now memoized in one process-wide cache keyed by those bytes. Keying by the bytes rather than by validator index is what makes a hit exactly a fresh call's answer on every fork, since an index names whatever key a given state says it does. A key that fails is never cached, and the cache is bounded, so inputs that are not a registry cannot grow it without limit. Hits resolve on the calling thread and only misses are validated in parallel, so concurrent gossip checks no longer queue already-validated keys on rayon. `fast_aggregate_verify` over 512 signers: 4.7 ms with every key a miss (validated across cores), 0.8 ms on one thread with every key a hit.
…st through ethlambda beacon (#34) * Let the Beacon API read a submitted SingleAttestation and compute its gossip subnet, the two pieces POST /eth/v2/beacon/pool/attestations needs before ethlambda beacon can accept attestations from a validator client. SingleAttestation, AttestationData and Checkpoint now derive Deserialize, and BlsPubkey and BlsSignature gain a hex Deserialize matching their existing Serialize. compute_subnet_for_attestation follows phase0's validator.md, and ATTESTATION_SUBNET_COUNT moves from the p2p crate into ethlambda-types (re-exported where it was) so the RPC crate can reach it without depending on p2p. * Give ethlambda beacon a way to gossip a validator client's attestations, which it could not do before: every publisher returned early on the beacon wire. A new RpcToP2P protocol in ethlambda-network-api carries publish_beacon_attestation(subnet_id, SingleAttestation) to the P2P actor, which publishes it on beacon_attestation_{subnet_id} through gossipsub fanout without subscribing to the subnet. The caller validates the attestation and computes its subnet, since both need a state and the P2P actor holds none. Nothing sends the message yet; the Beacon API's pool endpoint will. * Serve GET and POST /eth/v1/beacon/states/{state_id}/validators from ethlambda beacon, which is how a validator client resolves its public keys to registry indices before asking for duties. Ids may be indices or public keys and unknown ones are omitted; statuses filter by the beacon-APIs fine or coarse status names, derived per the beacon-APIs validator-status definitions. To test this against a real registry, ethlambda-state-transition gains a test-utils feature exposing its test_state builder, plus with_signing_validators_at and sign_for so later endpoint tests can produce signatures that verify. * Serve GET /eth/v1/validator/duties/proposer/{epoch} from ethlambda beacon, read straight from the fork-choice head state's fulu proposer_lookahead, so both the head's epoch and the next are answered without advancing a state and any other epoch is refused with 400. dependent_root follows the v1 definition (the block root at the slot before the epoch, the head itself when that slot is past the head), which is the endpoint ethlambda validator calls; v2's earlier dependent slot is left for when a client asks for it. The new beacon/validator.rs holds the /eth/v1/validator routes. * Serve POST /eth/v1/validator/duties/attester/{epoch} from ethlambda beacon. The fork-choice head state answers for its previous, current and next epoch without being advanced, since an epoch's committees depend only on a RANDAO mix fixed an epoch earlier and on activations the registry records ahead of time; other epochs are refused with 400. One EpochCommittees scan per request, then every committee of the epoch is walked to find the requested indices, which is cheap on a devnet and a full shuffle per request on mainnet (a shuffle cache is left for later). dependent_root is the block root at the slot before the previous epoch, per the endpoint's definition. * Serve GET /eth/v1/validator/attestation_data from ethlambda beacon, built as phase0's validator.md describes with the fork-choice head as head_block: beacon_block_root is the head, target is the slot's epoch and its boundary block (the head when nothing has filled the boundary since), index is zero per electra, and source is the current justified checkpoint of the head state advanced to the slot's epoch. That advance only happens when the head sits in an earlier epoch, and goes through fork choice's cached checkpoint_state, which becomes public for it. A slot before the head or past the wall clock is refused with 400. * Accept POST /eth/v2/beacon/pool/attestations on ethlambda beacon and gossip what passes. Each SingleAttestation is checked against the electra beacon_attestation_{subnet_id} conditions this node can evaluate (clock window with MAXIMUM_GOSSIP_CLOCK_DISPARITY, data.index zero, target epoch matching the slot, the voted block known and the target its checkpoint block, committee index in range, attester in the committee, and the BLS signature under the attester domain at the target epoch), then published on compute_subnet_for_attestation's subnet through the P2P actor. Failures come back as the Beacon API's IndexedErrorMessage with each rejected attestation's position, and the valid ones in the same batch are still published. start_beacon_rpc_server now takes the P2P actor's RpcToP2PRef; the seen-attestation cache is the one gossip condition left out. * Acknowledge beacon_committee_subscriptions and prepare_beacon_proposer on ethlambda beacon, and answer 501 on the block production, block publishing and aggregation routes. The two acknowledged endpoints parse their bodies (so a malformed one is still a 400) and return 200 without acting: joining attestation subnets only matters once this node aggregates, publishing needs no subscription, and a fee recipient only matters once it builds payloads. The 501s give a validator client an explicit answer it fails over on, since FallbackBeaconNode moves to its next node on any error per call, which is how a devnet can attest through ethlambda beacon while a second node still proposes. ApiError gains NotImplemented for it. * Drive ethlambda validator's own HttpBeaconNode against ethlambda beacon's Beacon API over a real socket, through one slot of attesting: genesis, spec, the syncing check, index lookup, both duty endpoints, subscriptions, the fee-recipient preparation, attestation data (which the client validates itself before signing) and submitting signed attestations, which reach the P2P stand-in. The per-endpoint tests check each answer against the spec; this checks the two ends agree on field names, quoting and statuses. A second test pins that block production comes back as a retryable 501, which is what lets the client fail over to another node for proposals. RecordingNetwork moves to test_utils so both test modules share it. * Document the validator endpoints ethlambda beacon now serves: the table in docs/rpc.md gains the validators lookup, both duty endpoints, attestation data, the attestation pool and the two acknowledged endpoints, with a section on the window each answers for, what the pool validates, and why the block and aggregate routes answer 501. CLAUDE.md notes the RpcToP2PRef the beacon HTTP server now takes. * Let the kurtosis package run ethlambda beacon as the validator client's first beacon node. With ethlambda_beacon.enabled, it starts ethlambda beacon from the devnet's genesis with a geth of its own over the Engine API (no EL peers needed, since every payload arrives in order from genesis, which also keeps the node out of optimistic mode), and passes the client both nodes, ethlambda's first, so it attests through ethlambda beacon and fails over per call to lighthouse for block production and aggregation. network_params_ethlambda_beacon.yaml runs that shape: two lighthouse participants, because ethereum-package pins a lone lighthouse to --target-peers=0 and lighthouse rejects the flag given twice, plus --subscribe-all-subnets and --import-all-attestations so lighthouse receives and aggregates the attestations ethlambda beacon gossips. --------- Co-authored-by: Tomás Grüner <47506558+MegaRedHand@users.noreply.github.com>
* Serve aggregation from ethlambda beacon: an AttestationPool shared by P2P and the Beacon API holds validated votes per data root and committee, GET /eth/v2/validator/aggregate_attestation builds electra's Attestation from it with the BLS aggregate signature, and POST /eth/v2/validator/aggregate_and_proofs gossips what passes on beacon_aggregate_and_proof. The pool endpoint also inserts the client's own attestations, since gossip never delivers a node its own messages. beacon_committee_subscriptions makes P2P join each aggregator's committee subnet until its slot ends (new Subscribe/Unsubscribe swarm commands, never advertised in attnets), and attestations accepted on those subnets by the gossip verdict path are pooled. Published aggregates are checked with the same gossip::aggregate conditions applied to peers' aggregates, against a fresh seen-cache, so this reuses the base's validators rather than carrying a second copy.
* Teach the Engine client to build payloads, the first piece of block production through ethlambda beacon. forkchoice_updated_with_attributes sends engine_forkchoiceUpdatedV3 with PayloadAttributesV3 (timestamp, prevRandao, suggestedFeeRecipient, withdrawals, parentBeaconBlockRoot) and returns the payloadId the response now carries; get_payload calls Osaka's engine_getPayloadV5 and decodes its answer straight into deneb's ExecutionPayload, the block value, the blobs bundle (cell proofs) and the execution requests, rejecting any field of the wrong width or bound. engine_getPayloadV5 joins the capabilities sent in the handshake, and the crate doc no longer calls payload building absent.
* Record fee recipients from prepare_beacon_proposer instead of dropping them, so block production can put the proposer's into its payload attributes. The addresses go into a FeeRecipients map (validator index to execution address) the beacon HTTP server creates and shares with its handlers; it is in memory only, since a validator client repeats the call every epoch. H160 gains the hex Deserialize the body needs.
* Assemble unsigned electra and fulu blocks in the state transition (beacon::block_production), per validator.md's block proposal. payload_inputs gives what PayloadAttributesV3 needs for a slot (timestamp, prev_randao, expected withdrawals, and the parent execution hash) from the same state process_execution_payload checks against; assemble_block builds the body (the state's own eth1 vote, no deposits, slashings or exits, the empty sync aggregate signed with the G2 point at infinity) and sets state_root by running process_block on a copy, which also rejects a body peers would before anything is signed. pack_attestations keeps what process_attestation accepts at the slot and merges committees voting on the same data into one EIP-7549 attestation, best-covered first, up to MAX_ATTESTATIONS_ELECTRA. parse_execution_requests is the missing inverse of get_execution_requests_list, checking the Engine API's ascending, non-empty rule. G2_POINT_AT_INFINITY becomes public for it.
* Give block production something to pack: the attestation pool now also records the single-committee aggregates this node validated and published (aggregate_and_proofs inserts each one), keeping the best-covered per data root and committee, and block_candidates returns, for every data and committee it holds, whichever covers more of the aggregate built from the pooled votes and the recorded one. Gossip aggregates from other nodes are not recorded yet: they are verified in the chain actor, and packing an unverified one would fail the whole block's process_block. block_production reuses the pool's single_committee helper.
* Serve block production and publishing from ethlambda beacon, so a validator client can propose through it. GET /eth/v3/validator/blocks/{slot} advances the head state to the slot, asks the node's own execution client to build on the head (forkchoiceUpdated with payload attributes from payload_inputs and the proposer's recorded fee recipient, then getPayloadV5), packs the pool's block candidates, and assembles the block with its state root; it answers fulu BlockContents in SSZ or JSON with Eth-Execution-Payload-Blinded: false and the payload value, 503 without an execution client or when the payload carries blobs (whose data columns this node cannot publish yet), and retries without attestations if a packed one breaks the block. POST /eth/v2/beacon/blocks takes SSZ SignedBlockContents, refuses blobs, checks the block is after the head and the proposer's signature against the advanced state, then hands it to P2P through a new RpcToP2P::publish_beacon_block, which gossips it on beacon_block and passes it to the chain actor, since gossip never delivers a node its own block. The execution client reaches the RPC from main.rs, and the beacon server's handles are grouped into BeaconApiHandles. An end-to-end test proposes through the validator client's own HttpBeaconNode against a stand-in execution client.
* Pack only attestations that still count. The first devnet run with ethlambda beacon producing every block showed each block re-including every earlier slot's aggregate, since nothing excluded what was already on chain; with MAX_ATTESTATIONS_ELECTRA slots per block, that would crowd out the newest votes within an epoch and stall finality. pack_attestations now drops any candidate whose attesters the state has all already credited for its target epoch (their participation byte is non-zero), and orders merged attestations by how many new attesters they bring, then newest first.
* Document block production and publishing in docs/rpc.md (produceBlockV3 from the node's own execution client, attestation packing, the empty sync aggregate, publishBlockV2's checks, and blobs refused until data column sidecars are published), and let the kurtosis package point the validator client at ethlambda beacon alone: ethlambda_beacon.fallback: false leaves Lighthouse out of --beacon-nodes, which network_params_ethlambda_beacon.yaml now sets, so every block on that devnet is one ethlambda beacon built. The devnet run shows every block from slot 1 on carrying the ethlambda-vc graffiti and exactly the previous slot's aggregate.
* Pass the committee cache to block production's attestation packing and state root by shared reference, since the base now gives CommitteeCache interior mutability and takes &CommitteeCache throughout the state transition.
…next slot (#52) #48 left every held block's columns to gossip until the next slot's redrive, because at the tip gossip completes a held block within a fraction of a second and an immediate by-root ask only raced it. A block that is already older than the current slot has no gossip left to wait for: its columns went out long before this node held it. Range-synced catch-up is exactly that case. After a fresh checkpoint sync on a mainnet follower, the 72 blocks from before the restart were each held missing every custody column and each waited out a redrive, so the follower imported about one block per slot and its lag stayed flat for eight minutes. A block held at a slot older than the current one now asks by root at hold time. A block at or ahead of the current slot keeps #48's behavior. The condition is the block's age rather than its source, since age is what decides whether gossip can still deliver the columns. Repeated holds of one block cost nothing extra: the p2p side merges a repeat ask into the lookup already in flight for that root.
…t it (#54) A block whose parent has no post-state walks up to its first ancestor that does have a parent state and re-queues that ancestor for import. When the ancestor is held for its custody columns, that import can only hold it again, and a range batch hands the actor its whole chain at once, so every block behind a held one paid for another hold of it. On a mainnet follower's first batch after a checkpoint sync, the 68 blocks behind the first one re-held it 68 times over 54 s, while the columns that would release it had already been delivered. The walk now stops at a block in blocks_awaiting_columns. The child is already registered under its parent in pending_blocks, and the held block's release imports it and cascades down the chain from there.
…peers (#53) A range batch sends its blocks and a DataColumnsByRange for the same span together, and the column request is aimed by custody, which is only known once a peer's metadata/3 answer or ENR cgc has been read. Right after startup that is almost nobody. On a mainnet follower after a fresh checkpoint sync, the first batch went out 5 s after startup with two peers and at most one known custodian per column, and 121 of the 122 holds it caused were missing every custody column; a short or empty range answer is never retried, so each block was left to the by-root path. A batch whose span reaches fulu is now held, blocks included, until every custody column has a known custodian among the connected peers, the way lighthouse's range sync holds its batches. It is re-checked after every metadata answer, on every Status answer, and at a deadline, past which it goes anyway with the old fallback: nothing here searches for a custodian of a specific column, so an open-ended wait could stall sync.
…es (#37) * feat(ssz-tree): add the crate skeleton with packing and depth rules The beacon state keeps its large lists in flat vectors, so every state root rehashes them whole and every cached state holds its own copy. This crate will hold them in a persistent Merkle tree instead, one shaped like the SSZ merkleization so each node can cache its hash and be shared. This first step fixes the two rules the tree's shape follows: how many elements pack into a leaf, and how deep the tree is for a given limit. A basic type whose size does not divide 32 is refused, because SszList packs it across chunk boundaries and a one-chunk-per-leaf tree would produce a different root. * feat(ssz-tree): buffer pending writes in an UpdateMap Writing straight into a persistent tree copies the whole root-to-leaf path on every write, so an epoch that touches every balance would copy each path once per element. Buffering the writes and folding them in one pass copies each shared path once. The buffer is a trait so each list can pick its own: a BTreeMap for the validator registry, where a block writes a handful of large elements, and a dense VecMap for balances, where an epoch writes most of them. Which one wins per field is left to a later benchmark. * feat(ssz-tree): add a persistent Merkle tree that caches node hashes The tree has the exact shape of an SSZ list's merkleization: packed leaves for basic types, one leaf per element for composites, and shared zero subtrees for the unused capacity. Each node keeps its hash once computed, so a later root only rehashes the nodes a write replaced. Writes rebuild only the root-to-leaf paths they touch and share every other subtree with the previous version through Arc, which is what lets several cached states hold one copy of an unchanged registry. Hashes are filled with OnceLock get() then set() rather than get_or_init: hashing runs children under rayon::join, and a worker blocked in get_or_init can steal a job that needs the cell it is filling. Racing workers only duplicate work; they compute the same value. * feat(ssz-tree): iterate a tree in order, pending writes included Scans over the registry (pubkey lookups, epoch sweeps) would cost a full root-to-leaf descent per element if they went through get. The walk keeps a stack of deferred right subtrees instead, so each node is visited once and a scan is linear in the number of elements. The public iterator overlays the pending writes on the walk, so an element written since the last apply_updates reads back its new value, and pending pushes appear after the committed elements. * feat(ssz-tree): add List, a tree-backed drop-in for SszList The beacon state's registry fields are SszLists inside derived containers, so their replacement has to encode, decode and hash exactly as SszList does, and offer the element API the state transition already calls (index, get_mut, push, iter). List does both, with the tree and its pending writes held in an Interface that Vector will share. Fixed-size elements decode straight into tree leaves, so loading a state never holds a second full copy of the registry in a Vec. Every read, encode and root sees pending writes; a root taken with writes still pending is computed on a copy, correct but not cached, so the state transition is expected to apply them first. Equality shortcuts on a shared tree only when nothing is pending, since two clones of one tree can carry different writes. The tests compare encodings, decode rejections and roots against SszList as the oracle. * feat(ssz-tree): add Vector, the fixed-length counterpart of List The state's other large fields (block and state roots, randao mixes, slashings) are SszVectors, so benchmarking them on the tree later needs a Vector that stands in for SszVector the way List stands in for SszList. It shares List's core; it has no push, its length is part of the type, and its root has no length mix-in. * feat(ssz-tree): rebase a tree onto a relative to share its subtrees A state decoded from storage builds its lists from scratch, so it shares nothing with the resident states it mostly equals, and each decode would add a full registry to memory. Rebasing walks the decoded tree beside a resident one and swaps in the resident node wherever the contents match, so what stays unshared is only what actually differs, and the swapped nodes bring their cached hashes with them. Equal cached hashes end the walk only for subtrees wholly inside both lists' shared prefix. At the boundary a shorter list can hash like a longer one, since a trailing zero value in a packed chunk looks like padding, so there only a value comparison counts. * fix(ssz-tree): report too many vector elements the way SszVector does The shared decoder caps a fixed-size element count at N like a list does, so a Vector given too many elements failed with InvalidByteLength where SszVector, which decodes first and counts afterwards, fails with InvalidFixedLength. Both rejected the same inputs; checking the count up front makes the errors match too. * test(ssz-tree): check List and Vector against SszList and SszVector Unit tests pin the cases someone thought of; the tree's failure modes are in combinations of pushes, overwrites and flushes that land on packing and depth boundaries. These properties run random sequences of them and compare every read, encoding and root with libssz's own types, for packed, composite and variable-size elements, including a list at the registry's 2^40 limit. Two more properties cover what those do not reach: decoding arbitrary and mutated bytes must accept or reject exactly as libssz does, and rebasing must keep a list's contents and root while sharing everything it has in common with the base, including across the trailing-zero boundary where hashes cannot be trusted. * feat(types): keep the beacon registry and balances in persistent Merkle trees The validator registry and balances are the bulk of a mainnet state: about 2.4M entries each. As SszLists, every state root rehashes them in full and every cached state holds its own copy, which made state merkleization 11.8% of block import CPU and multiplies memory by the state cache's size. Backed by ethlambda-ssz-tree's List they keep per-node hashes, so a root after a block rehashes only the touched paths, and a state cloned from another shares every unchanged subtree with it. The registry buffers writes in a BTreeMap, since a block writes a few large records; balances use the dense VecMap, since an epoch writes all of them. apply_pending_mutations folds the buffered writes in before a root is taken, and rebase_on lets a state built apart (decoded from storage) share subtrees with a resident one. Both are wired into the state transition and the store in the next commits. * perf(stf): flush registry writes before every state-root computation A tree-backed list hashed with writes still pending computes its root on a throwaway copy: the answer is right, but none of the new hashes are kept, so the next root pays again. Folding the writes in first makes each root cache what it computes in the state's own nodes. The flushes sit where roots are taken: at the top of process_slot and after process_block. process_slots also flushes before returning, because epoch processing runs after the last process_slot of its loop; a state advanced without a block (a checkpoint state, an empty-slot pre-state) would otherwise be cloned and cached with a whole epoch of balance writes still buffered. * perf(storage): rebase a decoded beacon state onto a resident relative A state read from the database is decoded into fresh trees, so every state-cache miss would add a full private copy of the registry beside cached states it nearly equals. Rebasing it onto a resident state right after decode swaps in the resident nodes wherever the contents match, so only what differs stays private. The parent block's state is the closest relative when it is cached; otherwise the most recently used state still shares nearly all of the registry. The rebased state is the decoded one: only the allocations behind it change. * test(types): benchmark the tree-backed registry at mainnet scale Whether the tree pays off depends on numbers no unit test shows: the live heap of one state and of each state derived from it, the rehash after a block's writes, and what the fixed tree depth costs reads and epoch sweeps. This ignored test measures each against SszList at 2.4M validators, with a counting allocator for memory, and asserts on the way that the tree's roots, encoding and rebase agree with SszList. The reads it times include the access patterns the state transition actually has: a block's worth of random attester lookups, and the effective-balance sweep both by index and with zipped iterators. * refactor(ssz-tree): keep Tree private to the crate Tree was exported only so docs could link to it, but none of its methods are public, so outside code could only build arbitrary trees from its variants and name nothing it could use. The public surface is now List, Vector, Iter and the update maps. Also documents UpdateMap::get and get_mut, and stops the docs claiming the benchmark tunes PARALLEL_HASH_HEIGHT: it measures the value. * perf(storage): never cache a beacon state with buffered writes Once a state is behind the cache's Arc it can no longer be flushed, so a buffered write would send every later root taken through it down the slow path, which copies the pending writes and caches no hashes. The store now flushes a state before wrapping it for the cache, on insert and on decode, and cache_state asserts in debug builds that whatever it is handed is already flushed. * docs: describe the tree-backed beacon registry in CLAUDE.md Adds the ssz-tree crate to the codebase map, and notes under the beacon types what changes for code touching validators and balances: no slices or iter_mut, and buffered writes that the state transition flushes before each state root. * test(stf): flush the gossip fixture's registry writes before caching it `fulu_parent`, the gossip tests' shared parent state, sets every validator's pubkey through `validator_mut`. With the registry now in a persistent Merkle tree those writes are buffered, and the block and column tests cache the state through `Store::cache_state`, whose debug assertion refuses an `Arc` holding writes it can no longer flush. The fixture predates the tree (it came in with gossip validation), so nothing flushed it and 14 tests panicked on the assertion. Flushing where the fixture is built hands the tests a state shaped like every one production caches, instead of teaching each call site to flush. * perf(ssz-tree): keep a page of elements per leaf A leaf held one SSZ chunk: one Validator, or four balances. At mainnet scale that is millions of Arc'd nodes, and every lookup walked the tree down to a chunk and then through one more pointer to the value. The profile of the import replay put BeaconState::validator at 1.65 s of CPU per block, most of it that walk, and one state's registry took twice the memory of the Vec it replaced. A leaf now holds the elements of a whole subtree, about 4 KB of them stored contiguously, so a lookup stops at the leaf and indexes into it, iteration walks each leaf as a slice, and the tree has far fewer nodes to allocate. The Merkle shape is unchanged: a leaf hashes its chunks up to its own height, so every root is the same. Rebuilding a leaf now copies its run, so a composite leaf keeps each element's root and a rebuilt leaf carries over the roots of the elements it did not change: rehashing it folds 31 cached roots rather than rehashing 32 validators. * perf(ssz-tree): keep a page of child pointers per inner node Inner nodes were binary, so a lookup still walked one node per level above the leaves: 35 of them for the registry, whose limit sets the depth, most landing on a cold cache line. An inner node now holds up to a page of child pointers and stands for that many binary levels at once, so the same lookup crosses a handful of nodes. The Merkle shape is unchanged. A node folds its children's cached roots up its own levels, padding a missing child with the zero hash of the children's height rather than of a chunk, which is why this does not go through merkleize. Levels are counted up from the leaves, so only the root can span fewer levels than the others. The cost moves to writes: a rebuilt node copies its child pointers and refolds its levels from its children's cached roots, rather than hashing one pair. * docs: describe the tree-backed registry and how to loop over it The page-sized leaves and inner nodes change what a registry read costs, and the import profile showed where that bites: helpers that collect the active indices and then read each one back through validator(i). A contributor writing the next epoch pass needs to know to iterate instead. beacon_stf.md gains a "Registry and balances" section on the layout, the buffered writes and the access pattern; CLAUDE.md points at it. benchmarking.md's replay memory figure described flat states and is replaced by the measured tree-backed one. * docs(ssz-tree): compare the tree to a B+-tree With page-sized leaves and inner nodes the tree reads more like a B+-tree over indices than like the binary Merkle tree it hashes as, so say so, along with the ways it is not one: no keys, no splits, a fixed shape and copy-on-write paths. Also drop two intra-doc links from the public crate docs to private constants, which rustdoc reports as broken.
* Pack other nodes' attestations into blocks ethlambda beacon builds, and make packing safe for them. P2P verifies every gossip aggregate on arrival but only forwarded the accepted ones to fork choice, so the pool block production reads held nothing but this node's own votes and aggregates. Validated::forward now also adds each accepted electra aggregate to the pool, where all three of its signatures have just been verified; pooling on arrival means a slot's aggregates are there when the next slot's block is asked for. pack_attestations now requires an attestation's target root to be the proposal state's own block root for that epoch, because an aggregate made on another branch was checked against that branch's committees, and verifies each merged attestation's signature against the proposal state, dropping one that fails and taking the next, instead of relying on produce()'s retry that dropped every attestation. P2P also prunes the pool on its aggregator-subnet sweep, so a pool nothing is inserted into does not keep stale entries. The pack_attestations fixtures now sign with the real committee members and target the state's own block root (the merge test uses 8192 validators, the fewest that give mainnet's preset two committees a slot), and new tests cover a target on another branch, a signature that does not verify, and forward pooling an accepted electra aggregate but not a phase0 or an unaccepted one. docs/rpc.md describes what the pool holds and the new packing checks. * Let the attestation merge test hold on the minimal preset. It asserted exactly two committees a slot for 8192 validators, which is mainnet's count; minimal's smaller committees give four, so Test minimal preset and Beacon spec tests (minimal) failed on it. The test only uses committees 0 and 1, so it now asserts at least two. --------- Co-authored-by: Tomás Grüner <47506558+MegaRedHand@users.noreply.github.com>
Brings in #613 (per-aggregator subnet window), #616 (logo assets), #606 (leanVM bump: XMSS moves into leanVM, 32-byte pubkeys) and #623 (leanSpec spectests set to `test = false` while fixtures lag leanVM). Resolutions that go beyond picking a side, because this branch reshaped the code main edited: - run_node: main's `init_leanvm(options.prover_arena)` now reads the flag from `LeanOptions` and runs only on `Network::Lean`, since the beacon chain signs with BLS and never reaches leanVM. #613's subnet-id validation, duty-subnet resolution and warning move into the lean arm. - BlockChain::spawn: main's startup key warm-up (`prepare_keys_for`) runs on the built `LeanDuties`, since the server is only assembled later in `start_actor`. The proposer lookup moves to `LeanDuties::our_proposer` so spawn and `get_our_proposer` share it. - on_tick: main reads the validator count once per tick; here that read stays lean-only, because `head_state` panics on a beacon store, and it is passed into `run_interval_duties`. - CI: main's by-name clippy of the skipped spectests joins the split-out `lint` job. - Benchmark reports: #606 dropped `leansig_rev`; ported to the `report/common.rs` and `report/import.rs` split this branch made. - Lean test fixtures in bci-only files (state_writer, beacon containers) move from 52-byte to `PUBLIC_KEY_SIZE` pubkeys.
A new user could only reach a running node by building from source. The quickstart runs the beacon follower from the pre-built ghcr.io/lambdaclass/ethlambda:beacon image and points lean users at the lean-quickstart branch that runs ethlambda, so "Getting started" becomes "Building from source". The steps were test-driven on a fresh mainnet run, which surfaced what the command has to carry: --stop-signal SIGINT, since the node shuts down gracefully on SIGINT only and `docker stop` sends SIGTERM; an advertised IP, since behind Docker's port mapping the ENR otherwise publishes 0.0.0.0; and a warning that is_syncing currently reads false during catch-up. It also states that the follower runs without an execution client, so payloads go unvalidated.
## 🗒️ Description / Motivation `make run-devnet` on this branch does not finalize. The lean-quickstart devnet runs every node with `--network host` and passes no `--discovery.port`, and discv5 is currently always on for both chains. So every node tries to bind the default UDP 9000. The first node gets it, and the others exit at startup: ``` Error: failed to start discv5 discovery 0: failed to bind discovery socket on 0.0.0.0:9000: Address already in use (os error 98) ``` With 3 nodes, only `ethlambda_0` survived. It holds one validator of three, so it saw only its own blocks (the head moved every third slot) and `justified_slot` stayed at 0. The aggregator was one of the nodes that died. Only `beacon` needs discovery on unconditionally, because published mainnet bootnode ENRs carry no `quic` entry and cannot be dialed statically. A lean network has no other discv5 speakers, so a crawl there finds only other ethlambda nodes. This PR gives `node` back the `--discovery.enable` switch that `main` still has, off by default. ## What Changed | Area | Change | |---|---| | `bin/ethlambda/src/cli.rs` | New lean-only `--discovery.enable` flag (`LeanOptions::discovery_enable`, default `false`). `Network::discovery_enabled()` is the single source of truth: the flag on lean, always `true` on beacon. Port validation moves to `Options::validate_ports()`, which applies the discovery rules only when discovery runs | | `bin/ethlambda/src/main.rs` | Builds `DiscoverySpawnConfig` only when discovery is enabled. Rewords the missing `--bootnodes` warning to mention the flag | | `crates/net/p2p/src/lib.rs` | `P2P::spawn` takes `Option<DiscoverySpawnConfig>` and `P2PServer.discovery` is an `Option<DiscoveryState>`, the same shape as `main`. `None` binds no socket and never schedules the first `DiscoverPeers` tick | | `crates/net/p2p/src/discovery/dial.rs` | Dial-loop helpers take `target_peers` as an argument instead of reading it from the discovery state. The refill step is extracted into `refill_candidates` so the borrow of the optional state does not span the dial's `.await`. `forget_discovered_peer` still drops custody when discovery is off | | `docs/`, `CLAUDE.md`, `Dockerfile` | "Always on" becomes "always on for `beacon`, opt-in for `node`" | ## Correctness / Behavior Guarantees - **`beacon` is unchanged.** It always runs discovery and runs the same port checks. It now rejects `--discovery.enable` with a usage error instead of accepting a flag it would ignore. - **`node` without `--discovery.enable`** binds no discovery socket, publishes no ENR and never runs the dial loop. It peers from `--bootnodes` alone, as `node` does on `main`. - Without discovery, `--discovery.port` is not checked against `--gossipsub-port`, and `--gossipsub-port 0` is accepted, since no ENR names it. The TCP clash check against `--api-port`/`--metrics-port` applies either way. It now skips `0` explicitly, because a `0` gossip port can reach it. - `--discovery.enable` has the same name and semantics as on `main`, so launch commands carry over between the two branches. ## Tests Added / Run The port tests in `cli.rs` used to check that discovery was always on for `node`. The ones that test discovery-on rules now pass `--discovery.enable`, and these tests are new: - `discovery_is_opt_in_on_node_and_always_on_on_beacon` - `beacon_has_no_discovery_enable_flag` - `without_discovery_the_discovery_port_is_not_checked` - `gossipsub_port_zero_is_accepted_without_discovery` - `port_zero_never_counts_as_a_clash`, which now covers both settings Commands run: - `make fmt`, `make lint` - `cargo test --profile release-fast -p ethlambda --bins`: 223 passed - `cargo test --profile release-fast -p ethlambda-p2p --lib`: 207 passed This has not been run on a local devnet yet. ## ✅ Verification Checklist - [x] Ran `make fmt`: clean - [x] Ran `make lint` (clippy with `-D warnings`): clean - [ ] Ran `make test`: only the unit tests of the two touched crates were run (see above)
## 🗒️ Description / Motivation A Platåberget follower built with `release-fast` (which keeps `debug-assertions` on) lost its P2P task to a panic inside libp2p request-response. The process, the API and the chain actor kept running, so the head froze for about five hours with nothing but this in the log: ``` panicked at .../rust-libp2p-da8daccbaa8a6b4a/2f14d0e/protocols/request-response/src/lib.rs:708:9: assertion `left == right` failed left: false right: true ``` That line is `debug_assert_eq!(connections.is_empty(), remaining_established == 0)` in `on_connection_closed`: request-response still counted a connection to a peer the swarm said had none left. The cause is the field order of `Behaviour`, where `connection_limits` came last: | Step | What happens | |---|---| | 1 | The swarm calls `handle_established_*_connection` on `Behaviour`; the derive calls each field in declaration order with `?` | | 2 | Every `request_response::Behaviour` in `ReqResp` records the connection in its `connected` map (`preload_new_handler`) | | 3 | `connection_limits` refuses it (per-peer, total, inbound or outbound ceiling) | | 4 | The swarm reports a `ListenFailure`/`DialFailure`, and never a `ConnectionEstablished` or `ConnectionClosed` for it. Request-response ignores both, so the entry stays | | 5 | The peer's last real connection closes with `remaining_established == 0`, while request-response still holds the phantom: the assert fires | Without debug assertions the phantom is worse than a leak. `try_send_request` picks a connection by `request_id % connections.len()`, so some requests to that peer go to `NotifyHandler::One(phantom)`. The swarm drops an event for an unknown connection silently (`swarm/src/lib.rs:1214` in the pinned fork), and the request timeout lives in the handler that was never spawned, so no `OutboundFailure` ever comes back. ethlambda retires in-flight requests on `OutboundFailure`. ## What Changed - `crates/net/p2p/src/lib.rs`: `connection_limits` is now the first field of `Behaviour` (and of its struct literal in `build_swarm`), with a doc comment saying why it has to stay first. A refusal now happens before any other behaviour sees the connection. - `crates/net/p2p/src/lib.rs` (tests): a regression test, below. ## Correctness / Behavior Guarantees - Which connections are refused is unchanged: same limits, same counts. What changes is that no other behaviour sees a refused connection. - `connection_limits` is safe to put first. It records established connections only on `FromSwarm::ConnectionEstablished`, emits no events (`poll` is always `Pending`), and its handler is `dummy`, so the protocols offered and the event dispatch are unchanged. - No field after it refuses connections, so no behaviour can be left holding a phantom. - Lean runs `unlimited_connections()`, which never refuses, so the lean network is unaffected. ## Tests Added / Run - `tests::a_connection_the_limits_refuse_leaves_no_request_response_state` builds the real beacon swarm with `build_swarm` and drives its `Behaviour` the way the swarm does: two connections from one peer (`MAX_CONNECTIONS_PER_PEER`), a third refused and reported as a `ListenFailure`, then both held connections close. Before the fix it fails in both build modes: - with debug assertions (`release-fast`, as CI runs it): the production panic at `request-response/src/lib.rs:708:9` - without (`CARGO_PROFILE_RELEASE_FAST_DEBUG_ASSERTIONS=false`): its own assertion, since the field still reports the peer as connected After the fix it passes in both. - `cargo test -p ethlambda-p2p --lib --profile release-fast`: 208 passed, 1 ignored. ## Related Issues / PRs - libp2p/rust-libp2p#4773: the same assert, open. The maintainers' advice there is to put connection-management behaviours first, which is what this does. - libp2p/rust-libp2p#4870: the underlying design issue (the `handle_*` callbacks take `&mut self`), open. - libp2p/rust-libp2p#6601: fixes request-response itself by recording connections on `ConnectionEstablished`. Open and unreviewed; it could be cherry-picked into `lambdaclass/rust-libp2p` later, and this PR does not depend on it. ## ✅ Verification Checklist - [x] Ran `make fmt` — clean for this change. `cargo fmt --all -- --check` also flags the `mod` order in `bin/ethlambda/src/main.rs`, which is already on `beacon-chain-integration` (from c635524) and left out of this PR - [x] Ran `make lint` (clippy with `-D warnings`) — clean - [ ] Ran `make test` (`test-consensus` plus `test-node`, at `release-fast`) — left to CI; ran the p2p unit tests above
🤖 Codex Code ReviewAutomated review by OpenAI Codex |
🤖 Claude Code ReviewReview of PR 661: "feat: add Beacon Chain support"I did not review the code. The PR is too large for the review tooling: What the PR contains
Concerns
RequestPlease split the PR, or point reviewers to the highest-risk files so the review can be targeted. Once the PR is split or narrowed, I can review those parts in detail. Automated review by Claude (Anthropic) · sonnet · custom prompt |
…ads, keep states in blob files (#658) ## Motivation On the Hoodi follower, the import guard's `has_state(parent)` check costs 0-1 ms on most blocks but **p50 180 ms, p90 332 ms, max 536 ms** on the block after each epoch's first block (48h of import timings, 10-03 to 10-05): | slot % 32 | `guards` p50 | `guards` p90 | |---|---|---| | 0 | 0 ms | 1 ms | | **1** | **180 ms** | **332 ms** | | 2 | 0 ms | 177 ms (when slot 1 is empty) | | 3..31 | 0 ms | 1-8 ms | An epoch-crossing block's state is stored as a full snapshot only, and `has_state` did `get(States)` on it: RocksDB read and copied the whole ~150 MB state to answer a yes/no question, although that state was still in the store's state cache. On Plataberget the same function costs ~30 ms on every block, since `States` has no bloom filter and a miss still reads a snapshot-sized data block per L0 file. ## Changes 1. **`has_state` is an existence check.** It consults the state cache (`peek`, so the LRU order is untouched), then `StateDiffs` before `States`, through `contains`, and never copies a value. A cached `BlockState` always means the state is pending or persisted, and nothing deletes a persisted state, so the answer is unchanged. 2. **`StorageReadView::read` is the one required lookup.** It lends the value to a `&mut dyn FnMut(&[u8])` callback, which keeps the trait usable as `dyn StorageReadView`. `get` and `contains` are default methods on top of it, so both backends implement `read` only. RocksDB uses `get_pinned_cf`. 3. **Decodes read from the backend's buffer.** `StorageReadViewExt::read_with` decodes inside the borrow, and every get-then-decode site uses it. Beacon state reconstruction folds its deltas while the snapshot is still borrowed and hands the result over as a `Cow`: a snapshot root decodes straight from RocksDB's buffer, and the writer's parent-bytes path takes ownership without a second copy. `get(..).is_some()` checks became `contains`. `get` stays where the bytes outlive the read (the beacon walk's delta records, the pending-column take) and for the stored config, which must not be decoded before the version and preset checks. 4. **`States` and `StateDiffs` use RocksDB blob files.** | setting | value | why | |---|---|---| | `min_blob_size` | 4 KiB | one default data block; anything larger gets an oversized block anyway | | blob compression | LZ4 | blob files default to none, while SST blocks get Snappy; the values are raw SSZ | | blob GC | off | nothing deletes or overwrites a state, so GC would only relocate live blobs | | blob cache | none | the store caches decoded states, and a snapshot-sized entry would evict the whole block cache | ## Compatibility No `DB_VERSION` bump. RocksDB applies the blob options to an existing directory as it goes: new writes land in blob files, and inline values move out as compaction rewrites their SSTs. `a_directory_written_without_blob_files_still_reads` opens a directory written with plain options and reads both old inline and new blob values. `estimate_table_bytes` now adds `rocksdb.live-blob-file-size`, which `estimate-live-data-size` leaves out, so the table-size metric keeps counting state bytes. ## Testing - New tests: `has_state` on cached, diff, snapshot-only and absent roots, through a counting backend that proves the cached path reads neither table; `read` on both backends (value, absent key, callback error); a cold-cache beacon reconstruction across a whole snapshot interval; blob placement and the pre-blob directory. - `cargo test --workspace --profile release-fast --lib --bins`: 0 failures. Clippy `-D warnings` clean. - Spec-test suites not run locally. ## Not covered - A bloom filter on `States`: blob files make misses cheap here, but other large-value tables still have none. - The live effect on `guards`: worth re-measuring on a follower after deploy.
MegaRedHand
added this pull request to stack #664
October 5, 2026 22:22
MegaRedHand
removed this pull request from stack #664
October 5, 2026 22:22
## 🗒️ Description / Motivation The libp2p identify reply does not contain the client name. `build_swarm` does not call `identify::Config::with_agent_version`, so rust-libp2p sends its default agent version: `rust-libp2p/0.48.0`. Network crawlers read the identify `agentVersion` to identify the client of a consensus-layer peer. Because of this, ethlambda nodes show as `rust-libp2p`, and crawlers cannot tell them apart from other rust-libp2p peers. ## What Changed - `crates/net/p2p/src/lib.rs`: add `SwarmConfig::agent_version` and give it to the identify config with `with_agent_version`. - `bin/ethlambda/src/main.rs`: set `agent_version` to `version::CLIENT_VERSION`. This is the same string as `/eth/v1/node/version` and `engine_getClientVersionV1`, for example `ethlambda/v0.1.0-<branch>-<sha>/<target>/rustc-v<version>`. - Test `SwarmConfig` literals: set `agent_version: "ethlambda/test"`. - New test `identify_reports_the_configured_agent_version`: two lean swarms connect, and the dialer reads the agent version from the identify reply of the listener. ## Correctness / Behavior Guarantees - The identify `protocolVersion` does not change (`eth2/1.0.0` on beacon, `/ipfs/0.1.0` on lean). - The only change on the wire is the `agentVersion` string. It applies to the lean and the beacon networks. ## Tests Added / Run - `identify_reports_the_configured_agent_version`. Without the fix, it fails with `left: "rust-libp2p/0.48.0"`. - `cargo test --locked -p ethlambda-p2p --lib`: 209 passed, 1 ignored. - `cargo clippy --locked -p ethlambda-p2p -p ethlambda --all-targets -- -D warnings`: clean. - `cargo fmt --all --check`: clean. ## Related Issues / PRs - None. ## ✅ Verification Checklist - [x] Ran `make fmt` — clean - [ ] Ran `make lint` (clippy with `-D warnings`) — clean (ran clippy on the two changed crates, not the full workspace) - [ ] Ran `make test` (`test-consensus` plus `test-node`, at `release-fast`) — all passing (ran the `ethlambda-p2p` unit tests only)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🗒️ Description / Motivation
Merges
beacon-chain-integrationintomain. This adds an Ethereum Beacon Chain client (phase0 through fulu) to the ethlambda binary, alongside the Lean consensus client:ethlambda beacon: a beacon node. It checkpoint-syncs, follows the chain over gossip and req/resp, validates execution payloads through an execution client's Engine API, and serves the Beacon API, including the validator endpoints.ethlambda validator: a beacon validator client (attest, aggregate, propose) that talks to any beacon node over the standard REST API.ethlambda node(lean) keeps its behavior, apart from the items under "Behavior changes for lean" below.Both chains run on the same storage layer, P2P actor, RPC crate and chain actor.
BeaconStatehas aLeanvariant holding lean'sState, and each shared handler dispatches on it once, at the top.Each piece was reviewed as its own PR into the integration branch (listed under "Related Issues / PRs"). This PR merges that branch.
What Changed
ethlambda-types(beacon/)preset-minimalfeature), runtime fork config, and aLeanvariant onForkNameandBeaconStateethlambda-state-transition(beacon/)blst), KZG (c-kzg), gossip validation rules, DAS custody mathethlambda-ssz-tree(new)List/Vectorfor the validator registry and balances, so consecutive states share unchanged subtreesethlambda-storageStorefor both chains. Beacon states are stored as snapshots plus VCDIFF deltas and written on a background thread. Adds a committee cache, anddb_version/chain/presetmetadata that resume checksethlambda-blockchainBlockChain::spawn_beacon): import, holding blocks until their parent or columns arrive, applying gossip aggregates to fork choice, block production and aggregationethlambda-engine(new)newPayload,forkchoiceUpdated,getPayload,getBlobsethlambda-p2pbeacon_blocks_by_{range,root}/2served and requested; range sync; column custody and sampling; attestation subnet backbone; discv5 always on forbeaconethlambda-rpc/eth/v1and/eth/v2: JSON by default, SSZ on request. The router is picked byStore::chain(), so/lean/v0and the Beacon API are never served togetherethlambda-validator(new)docs/spec_deviations.md)bin/ethlambdabeaconandvalidatorsub-commands;--networktakes a built-in name (mainnet,sepolia,hoodi) or a directory of published network files; beacon checkpoint sync;benchmark importreplays a real block corpus offline;run_nodeis one startup path for both chainsmake testsplit intotest-consensusandtest-node(one runner's disk no longer fits both), benchmark-smoke and tooling jobs, disk-cleanup actionbeacon_stf.md,beacon_wire.md,beacon_engine.md,cli.md. Updated:data_storage.md,metrics.md,rpc.md,checkpoint_sync.md,discovery.md,benchmarking.md,spec_deviations.md, README (beacon quickstart with ethrex)Correctness / Behavior Guarantees
beaconmodules that lean code never reads, and those modules never read lean's. The shared crates dispatch on the chain once per handler.lean_state_unreachable/lean_fork_unreachable), so the caller never gets a wrong answer back.ForkName::Leanis left out ofForkName::ALL, so neitherparsenor a fork upgrade can reach it.v1.6.1fixture case passes on both presets: 5705 mainnet and 40009 minimal, including 150 mainnetfork_choicecases. Gossip fixtures are pinned tov1.7.0-beta.1.ethlambda node)mainhas nodb_versionkey, soStore::from_db_statereturnsDbVersionMismatch { found: 0, expected: 4 }. There is no migration: an upgraded node needs a fresh data directory (or checkpoint sync).blstandc-kzgare now dependencies of the lean binary too, since the state-transition crate holds both chains' rules. They are not feature-gated.rev: the beacon containers needDerefMut/IndexMutonSszList(feat: addDerefMuttoSszListandSszVectortypes libssz#33), which no crates.io release has yet.release-fastprofile (used for tests) now setsdebug-assertions = true.P2P::spawnnow runs before the chain actor is spawned, so gossip arriving in that short window is dropped./lean/v0/blocks/finalizedanswers JSON when asked for it by name. SSZ stays the default.Tests Added / Run
make test-beacon(ortest-beacon-mainnet/test-beacon-minimal). Each fixture case is a separately named test, e.g.cargo test -p ethlambda-state-transition --test beacon_spec_tests --features beacon-spec-tests -- electra/attester_slashing. CI runs one job per preset.make teststill needs no consensus-spec download: the beacon spec target is behind thebeacon-spec-testsfeature, and the BLS/KZG vector tests report as ignored without it.Test minimal presetCI job builds and runs theethlambda-typesunit tests withpreset-minimalon.benchmarksub-command end to end.ssz-tree,engine,validator) and in the beacon paths of storage, p2p, rpc and the chain actor.Related Issues / PRs
PRs merged into the integration branch (reviewed on
lambdaclass/ethlambda_private)Types and state transition
beaconnamespaceFollower, sync and storage
ethlambda beaconsub-commandP2P
API and validator
ethlambda beacon/eth/v1/config/specethlambda beaconPerformance and metrics
CI
Open PRs based on
beacon-chain-integration, not included here: #626, #627, #629, #630, #632, #634, #636, #638, #642, #644 to #652, #656, #658.✅ Verification Checklist
make fmt(clean)make lint(clippy with-D warnings, clean)make test(test-consensusplustest-node, atrelease-fast), all passingmake test-beacon(both presets), all passing