[renovate]

Note

This PR body was truncated due to platform limits.

This PR contains the following updates:

Package	Update	Change
ghcr.io/streamingfast/firehose-ethereum	minor	v2.17.3 → v2.24.1

---

Release Notes

streamingfast/firehose-ethereum (ghcr.io/streamingfast/firehose-ethereum)

v2.24.1

Compare Source

Changed

• With continuous authentication (?continuous=true on the grpc auth plugin), each continuous Authenticate response now refreshes the stream's trusted headers. A stream whose first authentication returned no headers (for example an authenticator that failed open) is metered with its identity from the next continuous call on, instead of without it for its whole lifetime.

• Substreams: Tier2 skips loading dependency stores and compiling WASM modules for a segment once its cached block index proves every module excludes all of the segment's blocks. It still writes the same (empty) output files as before.

• Bump firehose-core to v1.21.1, dauth to 75940c1 and substreams to 5ed941a.

v2.24.0

Compare Source

Added

• Store URLs, such as --common-merged-blocks-store-url, accept compression_config to tune how files are written, matching the store's compression: a zstd level with an optional window in MiB (best, better/32), or a gzip level from 1 to 9. For example gs://bucket/merged-blocks?compression_config=best/32. Files written with any setting are read back without configuration. An invalid value makes opening the store fail.

• On zstd stores, compression_config also takes decoder settings, comma separated after the level or alone: lowmem=false gives each decoder a history buffer of twice the window, and pool=<name> reuses decoders across reads and across every store using that name, in place of the default pool. pool=none gives each file a decoder of its own. For example --common-merged-blocks-store-url=gs://bucket/merged-blocks?compression_config=best/32,lowmem=false,pool=blocks and --substreams-state-store-url=gs://bucket/states?compression_config=better/16,pool=cache. Substreams tier2 gets them through the URLs tier1 sends. Reading 100 MiB best/32 merged blocks files 10 at a time, adding lowmem=false to the default pool used 70% less CPU and 57% more peak heap. lowmem=false only helps files larger than their window plus about 1 MiB.

• Call.input_truncated and Call.return_data_truncated in sf.ethereum.type.v2. They flag the calls whose input was cut to its 4-byte selector, or whose return data was left out, by a tracer that limits how much call input and return data a transaction or block records. A trace without such limits never sets them.

• New fireeth tools convert-to-v5 <src> <dest> <start> <stop>, which rewrites merged-blocks files of version 2, 3 or 4 as version 5 blocks. For each block it:

  • removes every gas change and account creation;
  • removes the balance, nonce, code and storage changes whose old and new values are equal;
  • keeps in Call.keccak_preimages only the preimages of at most 256 bytes that explain a storage slot written by the same transaction or system call, like the tracers do;
  • gives an ordinal to the calls recorded with a begin_ordinal of 0 (the root call of every transaction in version 3 and below), and to the genesis root call recorded with an end_ordinal of 0;
  • sets the caller of the DELEGATE calls of version 2 blocks to the address of the closest ancestor that is not a DELEGATE call;
  • on blocks of version 3 and below, moves the ordinals of the transactions above those of the system calls when both use the same ordinal (blocks before Prague only), copies the root call's return data to TransactionTrace.return_data, and sets the input of a root CREATE call to the transaction's input;
  • removes the single empty topic of a log without topics emitted by a reverted call;
  • sets Log.index to the position of the log among all the logs of its transaction, logs of reverted calls included;
  • renumbers the ordinals of the block as 1, 2, 3, ... in their existing order;
  • applies the call input and return data limits of the tracers, setting Call.input_truncated and Call.return_data_truncated.

  With --target-bundle-size, it writes merged-blocks files of a different number of blocks than the source files (--merged-blocks-bundle-size), like resize-merged-blocks, in the same pass.

  With --rpc-endpoint, it also fetches Block.withdrawals on the blocks whose header withdrawals_root announces withdrawals that the block does not hold. Chains and block ranges without withdrawals make no RPC call.

  When the source store ends before <stop>, it checks for the next source file every 5 seconds until it reaches <stop>, so it can run ahead of the chain and catch up as new files are merged.

  The known issues of version 3 that the block alone cannot fix are left as they are, so the output differs from tracer-produced version 5 blocks on those fields. See the command's help for the list.

  On Google Cloud Storage, each merged-blocks file it writes carries the same annotation the merger writes (data size, block count and first block time), read by firecore tools stats-merged-blocks. The other fireeth tools commands that rewrite merged-blocks files write it too.

• Substreams: --substreams-tier1-allow-pruned-chain lets tier1 serve modules (mappers and stores) whose initial block is below --common-first-streamable-block, after that block was raised (the chain was pruned). They run from the first streamable block on, and their stores resume from the snapshots taken before the pruning. A request is refused when a store has no snapshot at the first streamable block to resume from, or when a store has no initial block: every store must set one, at or above the first streamable block. Tier2 must run this version before the flag is set on tier1.

• Substreams eth_call and eth_getBalance calls are now sent in the metering events as external_calls_eth_call and external_calls_eth_getBalance. A batch counts for as many calls as it contains, and retries are not counted.

• fireeth tools resize-merged-blocks --fix-misplaced-blocks rewrites a store where merged-blocks files hold blocks outside of their range, as the empty files of a gap in block numbers can (ex: near-testnet around block 5346171). Each block is written to the file of its own number, with its parent number set to the last block written. A block at or below the last block written is skipped, and one that does not have it as parent stops the process with an error. Without the flag, such a store fails with contains block ..., beyond the configured bundle size.

• fireeth tools substreams prune-states, prune-outputs and purge: --delete-rate (default 500) caps deletions per second, 0 removes the cap.

Fixed

• Firehose and substreams-tier1 now return the sessions they still hold to the session server before exiting, whatever --common-system-shutdown-signal-delay is set to. Sessions of requests cut by the shutdown used to be released in the background while the process exited, so they stayed counted against the organization until they expired on the session server, and a client reconnecting right away could be refused with Concurrent stream limit exceeded. This applies to session plugins implementing Close(ctx) error, which the tgm:// plugin does.

• Substreams: Fix tier1 requests rejected before their body is read (authentication, compression enforcement) sometimes failing at a load balancer with HTTP 502 or INTERNAL instead of returning their error.

• fireeth tools compare-blocks-rpc/compare-oneblock-rpc no longer report false differences for header.slot_number (EIP-7843) and the EIP-4844 blob fields (blob_gas, blob_gas_fee_cap, blob_hashes, receipt blob_gas_used/blob_gas_price) on Amsterdam-fork blocks. The RPC side of the comparison silently dropped these fields when rebuilding a block from eth_getBlockByNumber/eth_getTransactionReceipt, because eth-go's RPC types had no fields to decode them into; bumped eth-go to pick up the fix and mapped the new fields in block.RpcToEthBlock.

• fireeth tools compare-blocks-rpc/compare-oneblock-rpc no longer report false differences on set_code_authorizations (EIP-7702) for transactions carrying a SetCode (type 4) authorization list. eth-go's RPC Transaction type had no field to decode authorizationList; bumped eth-go to add it and mapped the tuples in block.RpcToEthBlock, recovering authority from the tuple's signature the same way the Firehose tracer does. discarded stays Firehose-only (it depends on execution-time account state the RPC doesn't expose) and is excluded from the comparison, same as Calls/ReturnData.

• Substreams: Fix a tier1 panic (INTERNAL: runtime error: invalid memory address or nil pointer dereference) on a production-mode request starting past the chain's final block, when that final block falls within the first segment of the request's stores.

• Firehose, substreams-tier1 and the relayer no longer exit with received 5 consecutive unlinkable blocks or cannot link block after reconnection when they start while the merger is deleting the one-block files of a bundle it just merged.

• fireeth tools compare-blocks-rpc no longer panics with a nil pointer dereference when a Firehose transaction trace has no receipt (observed on reth for an Amsterdam transaction that runs out of gas before its root frame). A missing receipt is now reported as a diff instead of crashing the tool.

• BlockHeader.block_access_list_hash and block_access_list_rlp (added to the proto in v2.23.0) were never actually present in the generated Go code, since types/pb wasn't regenerated after the proto change. This made every compare-blocks-rpc/compare-oneblock-rpc comparison on a block carrying either field fail with an empty, uninformative diff. Bumped firehose-ethereum/types to pick up the regenerated code and compare block_access_list_hash against RPC's eth_getBlockByNumber; block_access_list_rlp stays Firehose-only and is excluded from the comparison, same as Calls/ReturnData.

• fireeth tools compare-blocks-rpc/compare-block-store-rpc no longer report a withdrawals diff on every block carrying EIP-4895 validator withdrawals. block.RpcToEthBlock only filled Header.withdrawals_root and never populated Block.withdrawals from the RPC block, while the Firehose side always has them, so the comparison flagged a difference on every such block.

• index-builder no longer silently skips a block's index write on a transient index store error, nor restarts indexing from the configured start block afterwards. Bumped bstream and firehose-core so BlockIndexer.Add returns the write error instead of only logging it, and EthCombinedIndexer.ProcessBlock now propagates it. (firehose-core#288)

• Substreams: Fix requests failing with snapshot "…" does not exist after --common-first-streamable-block was raised, for stores without an initial block. Their snapshots named after the previous first streamable block are now ignored and the store is rebuilt.

• Substreams: Fix dev-mode requests reading blocks below the first streamable block when that block is not a multiple of the segment size.

• Substreams: Fix requests with a cursor being refused with invalid start block …, must be >= … when their start block is below the first streamable block. The block the cursor resolves to is what is checked now.

• Substreams: Fix the live back filler (started after a quickload) stopping for the rest of the request when it rewound below the first streamable block, or to a snapshot named after another initial block.

• index-builder now serves its health check, over gRPC and HTTP /healthz, on --index-builder-grpc-listen-addr, which it never listened on.

• Startup no longer lowers the open-files limit (RLIMIT_NOFILE) to 24,576 when it can't raise it to 1,000,000 and the current limit is already higher, as in containers without CAP_SYS_RESOURCE (k0s: 999,999, containerd 2.x: 524,288). It now keeps the current limit.

• Substreams: Fix wasm calls that fail because the host ran out of file descriptors or memory (for example Too many open files while creating an instance) being reported as deterministic errors, which returned InvalidArgument and were cached for 1h. They now return Internal, and the shared cache runs the call again instead of handing the same error to every stream on that block.

• The tools writing merged-blocks files (resize-merged-blocks, download-from-firehose and others) now fail instead of writing a block to a file whose range does not hold it. The previous bundle's last block carried into the files of a gap is still written.

• Merger no longer moves canonical one-block files to the forked-blocks store when readers write the same block with a different LIB, as happens on Polygon PoS / Amoy. Only blocks with another ID at the same height are moved.

• The reader now always clamps a decoded block's lib_num down to its own block number when the node/plugin reports lib_num greater than the block number (invalid; lib_num equal to the block number is still valid), logs an error and increments reader_node_invalid_libnum_clamped_count, instead of letting the bad value reach the relayer where it moved LIB past head and silently stalled it forever.

• zstd stores (merged blocks, one-block files, substreams states) now close the object's HTTP stream or file when a reader is closed before the end of the file. Streams stopping mid-bundle, at their stop block or when the client goes away, used to leave it open.

• fireeth tools substreams prune-states, prune-outputs and purge no longer fail deletions on GCS with 429 rateLimitExceeded ("request distribution is too uneven across the key-ranges" or "request-rate on the bucket has ramped up too fast"). They now delete in random order instead of listing order, which sent every concurrent deletion to the same part of the bucket, are capped by the new --delete-rate flag, and retry a failed deletion 5 times over several seconds instead of twice over 50ms.

Changed

• Substreams: production-mode requests disconnected for being more than 2 segments behind the last final block now get back-processing ended too far behind the last final block of the chain, please reconnect to process the gap in parallel or stream fell too far behind the last final block of the chain, please reconnect to process the gap in parallel instead of endpoint is shutting down, please reconnect. Both are still Unavailable.

• Apps running in the same process now share one session pool, created from --common-session-plugin, instead of each creating their own. With the local:// plugin, firehose and substreams-tier1 running together now count against the same max_sessions and max_sessions_per_organization limits.

• Bump firehose-networks to v0.3.0.

• Subscribers to live blocks (relayer, firehose and substreams subscriptions to live) are no longer disconnected as soon as 100 blocks are waiting for them. On fast chains, a live source pausing for a few seconds and then sending its backlog at once filled those 100 slots before a subscriber could send the first block, disconnecting every subscriber at the same time. A subscriber is now disconnected when it has not caught up for --common-live-subscriber-catch-up-timeout (default 30s), which catches one that is stuck or slower than the chain, or when --common-live-subscriber-max-buffered-blocks (default 10000) blocks are waiting for it.

• The relayer, firehose and substreams-tier1 no longer look up one-block files, nor log block not linkable after one-block lookup, when a live source sends a block they already have below LIB. With several --relayer-source, this happened every few seconds.

• Substreams: tier2 logs refusing Substreams ProcessRange request and the gRPC finished streaming call with code ResourceExhausted line at Debug instead of Info. The substreams_tier2_rejected_request_counter metric still counts the refusals by reason.

• Substreams: tier1 handles Ethereum partial blocks about 20x faster, with almost no allocations. It reads the transaction traces straight from the encoded block and drops the ones already sent by copying bytes, instead of decoding and re-encoding the whole block for every partial.

• Merged blocks are read without copying their payload, which halves the memory allocated per merged-blocks file read (25.8 MiB instead of 50.8 MiB for 100 blocks of 256 KiB).

• Go 1.27 or newer is now required to build.

• fireeth tools resize-merged-blocks now accepts equal --source-bundle-size and --target-bundle-size to rewrite merged-blocks files with different settings (ex: compression), after a confirmation prompt (--force skips it). It also clears the deprecated payload_kind and payload_buffer fields from every block written; the block data stays in payload.

• firehose and substreams-tier1 launched in the same process now share one forkable hub instead of each holding its own copy of the live blocks; on Solana that saves about 0.5 GiB per process. Firehose still streams complete blocks only.

• The firehose hub keeps as many final blocks as the substreams-tier1 hub: two merged-blocks files worth, at least 200, instead of at least 500. Clients resuming between 200 and 500 blocks below the last irreversible block now read those blocks from merged-blocks files.

• zstd stores (merged blocks, one-block files, substreams states) now read with decoders kept in a process-wide default pool when compression_config names no pool, and those decoders work on the reading goroutine instead of 4 background ones. Reading 100 MiB best/32 merged blocks files 10 at a time, that used 6% less CPU and 53% less peak heap; on 2 MiB files, 42% less CPU. compression_config=pool=none brings back a decoder per file.

• Firehose, substreams-tier1 and the relayer now detect a relayer or reader that disappears without closing the connection (for example a host reboot) in about 40s instead of 5 minutes. They send gRPC keepalive pings after 30s without data and drop the connection after 10s without an answer.

• fireeth tools poller prints each FIRE BLOCK line 11-15x faster and no longer allocates several times the block size to do it: the line is built once in a reused buffer, with SIMD base64 encoding.

• Bump firehose-core to v1.21.0, bstream to 71724f9 and substreams to d18ca66.

Removed

• Removed the --firehose-discard-partial-blocks flag. Firehose clients only receive complete blocks, and the hub firehose builds when it runs without substreams-tier1 never receives partial blocks from the relayer, so the flag no longer did anything. Operators setting it must drop it, as an unknown flag stops the process from starting.

Deprecated

• The SOURCE_CHAN_SIZE environment variable, replaced by --common-live-subscriber-max-buffered-blocks. It is still used when the flag is not set, with a warning at startup.

Security

• Bump golang.org/x/net to v0.61.0 to fix CVE-2026-78663 (critical), CVE-2026-78669, CVE-2026-78660 and CVE-2026-78659 (high).

• Bump OpenTelemetry to v1.47.0 to fix known vulnerabilities.

v2.23.0

Compare Source

Added

• New FIREETH_COMPARE_IGNORE_KECCAK_PREIMAGES_ABOVE_256 (alias FIREETH_TOOLS_COMPARE_IGNORE_KECCAK_PREIMAGES_ABOVE_256), which drops Call.keccak_preimages entries whose preimage exceeds 256 bytes from both blocks before comparing. It lets a node running a tracer that caps the map be compared against one that still emits the large entries, without hiding every other preimage the way FIREETH_COMPARE_IGNORE_KECCAK does. Applies to fireeth tools compare-blocks and to a reader node in --reader-node-test-mode.

• New fireeth tools remove-large-keccak-preimages <src> <dest> <start> <stop>, which rewrites merged-blocks files dropping every Call.keccak_preimages entry whose preimage exceeds 256 bytes. It brings a range written before the tracer cap in line with one written after it. Preimages carry no ordinal, so nothing else in the block moves and the block version is unchanged.

• New BlockHeader.block_access_list_hash and BlockHeader.block_access_list_rlp fields for EIP-7928 block access lists, scheduled for the Amsterdam hard fork. Both are ignored in legacy headers; only the hash is committed to by the header itself. block_access_list_rlp is experimental, populated for measurement purposes only, and must not be depended on until StreamingFast declares it stable.

• Added Morph specific fields to the Ethereum block model. All of them are unset on every other chain.

  • New TransactionTrace.Type values TRX_TYPE_MORPH for MorphTx (0x7f) and TRX_TYPE_MORPH_L1_MESSAGE for L1 message transactions (0x7e). The L1 message gets its own value because it shares its type byte with Optimism's deposit transaction but has different semantics.

  • New TransactionTrace.morph_tx_config holding the MorphTx fields version, fee_token_id, fee_limit, reference and memo.

  • New TransactionTrace.morph_l1_message_config holding the L1 message queue_index.

  • New TransactionReceipt.morph_receipt_config holding l1_fee, fee_rate and token_scale.

  • New BlockHeader.morph_next_l1_msg_index recording the queue index at which the next block resumes processing L1 messages.

• --relayer-source accepts a retry_interval=<duration> query parameter (e.g. my.source:12345?retry_interval=120s) setting the minimum time between two connection attempts to that source. Use it for a rescuer or fallback endpoint that is expected to be down most of the time, so the relayer does not dial it (and log the failure) every 5s. retry_interval must be at least 5s, and since sources are checked every 5s it is rounded up to the next 5s increment (e.g. 12s behaves as 15s). Without it, a source is retried every 5s.

• gRPC clients receive responses up to 2 GiB instead of 1 GiB, and servers send responses up to 2 GiB (bumped dgrpc), so a block up to 2 GiB goes from the reader through the relayer to Firehose and Substreams. Request limits are unchanged. External clients need their own receive limit raised to get blocks that big.

• New --substreams-tier1-cpu-eviction-order flag (default dev,prod-cached,prod-catchup) listing the request classes the CPU eviction may cancel, least important first. A class left out is never cancelled, so live production requests are no longer cancelled unless prod-live is added to the order. See the substreams bump below for the new prod-cached class.

• New --substreams-tier1-max-request-duration flag (default 0, no limit). When set, a Substreams request that has run for that long is ended gracefully: its stores are quick-saved (with --substreams-tier1-quicksave-store) and the client is told to reconnect. Set it a bit under the stream duration limit of the load balancer serving requests.

• New --substreams-tier1-squasher-plugin flag so fireeth start can point substreams-tier1 at a remote store-merge (squasher) process. Empty or local:// keeps in-process squashing. Production uses grpcs://host?secret=<token>: TLS, port defaults to 443, secret is sent as the authorization header. Plaintext local/dev uses grpc://host:port. Extra TLS/plaintext toggles stay on the DSN query string (insecure=true, plaintext=false).

• Reader: the base64 payload of a FIRE BLOCK line (Firehose protocol 3.x) is decoded while the line is read, instead of after the whole line was copied into a string, using the firehose-core line splitter, which decodes with github.com/emmansun/base64 like fireeth did before. Reading a block needs about 1.5 times its line size in memory instead of about 3 times. With --reader-node-debug-firehose-logs, a FIRE BLOCK line is logged as FIRE BLOCK <header> <payload: N bytes decoded> instead of its full base64 text.

Changed

• Documented in Call.keccak_preimages that only preimages of 256 bytes or less are recorded. The map is there so a consumer can walk a storage slot back to the expression that produced it, and Solidity's slot derivations are small: 32 bytes for a dynamic array or a long bytes/string, 64 bytes for a mapping with a value-type key, and 32 bytes plus the key for a mapping(string => V). 256 bytes covers all of those, with room for a 224-byte dynamic key. A larger preimage is a contract hashing its own data, and is dropped rather than truncated, since a truncated preimage no longer hashes back to its key.

• --reader-node-line-buffer-size is now the normal size of the line buffer (default 100 MiB, at least 32 KiB). The buffer grows past it in pieces of that size for a longer line, up to 2861913428 bytes: the longest line whose block, with 1 MiB for the message around it, fits in a 2 GiB response. Each time 50 blocks in a row used less than half of it, it shrinks by half, keeping whole pieces and never going below the normal size. A grown buffer stays allocated until it shrinks back.

• --reader-node-line-buffer-size outside of 32768 to 2861913428 bytes is now refused at startup instead of being used as is. A reader configured above that range does not start until the value is lowered.

• Bumped firehose-core to v1.20.1, which needs the github.com/ShinyTrinkets/overseer replace directive moved to github.com/streamingfast/overseer v0.2.1-0.20260917150444-9ebead8ffdef.

• Bumped bstream to v0.0.2-0.20260921191230-ea57fcbd4fbb for the per-source retry interval backing retry_interval on --relayer-source.

• Bumped substreams to v1.23.0:

  • Server: production requests that fall far behind the chain are disconnected so they reconnect and back-process the gap in parallel. The main case is a long back-processing: when it finishes, the request would otherwise process everything the chain produced in the meantime linearly on tier1, which can take hours. A production request is disconnected when it is more than 2 segments behind the last final block (rounded down to a segment), checked when back-processing finishes and at every segment boundary while streaming final blocks. It gets the same Unavailable "endpoint is shutting down, please reconnect" error as a tier1 restart, so clients reconnecting from their cursor pick up where they left off. Set the SUBSTREAMS_MAX_LINEAR_HANDOFF_LAG_SEGMENTS environment variable on tier1 to change the number of segments.

  • Server: the CPU eviction order is configurable. Classes are cancelled in the configured order, highest burn first within a class and oldest first on a tie. A new prod-cached class covers production requests that have not processed a block on tier1 yet, only streaming outputs cached by tier2. They run no wasm on tier1, so --substreams-tier1-cpu-eviction-min-burn-cores does not apply to them (--substreams-tier1-cpu-eviction-min-age still does), and since their CPU cost is unknown, a round of eviction stops right after cancelling one; the next round, after --substreams-tier1-cpu-eviction-cooldown, measures what it freed. The substreams_tier1_evicted_requests_counter metric gains the prod-cached class.

  • Server: per-store lines are logged at Debug instead of Info: using mmap KV store, using in-memory KV store, flushing store at boundary, merged partial into full store, deleting partial store. squashing time metrics stays at Info.

  • Server: the tier1 block hub logs processing block at Debug, except for one line every 10 seconds kept at Info to show progress. linking live block using one blocks moves up to Info, so a one-block store lookup after an unlinkable live block shows in production logs.

  • Server: tier1 can merge stores on a remote squasher selected by --substreams-tier1-squasher-plugin. Empty or local:// stays in-process. Each remote run is one RPC per store module; if the remote is unreachable that run is squashed locally and later runs stay local until one succeeds. A live remote that returns an application error still fails the request.

Security

• Bumped google.golang.org/grpc to v1.85.0-dev.0.20260825072537-93e31b48545e, the commit that fixes CVE-2026-84445. v1.84.0 is still in the affected range; this pin comes in with substreams v1.23.0.

v2.22.0

Compare Source

Added

• New --substreams-tier1-cpu-eviction-* flags letting substreams-tier1 shed requests when its own cgroup reports the CPU saturated. --substreams-tier1-cpu-eviction-mode (default off, so nothing changes until you set it) selects what it does: observe logs the requests it would cancel without cancelling any, dev-only cancels development-mode requests, full cancels production ones as well. See the bump above for what a round of eviction does.

  The rest are tunables, each defaulting to the value the eviction was designed around:

  Flag	Default	Meaning
  --substreams-tier1-cpu-eviction-threshold	0.90	CPU usage above which the instance counts as overloaded
  --substreams-tier1-cpu-eviction-sustain	15s	how long that must hold before it acts
  --substreams-tier1-cpu-eviction-recover-threshold	0.75	CPU usage under which it stops counting as overloaded
  --substreams-tier1-cpu-eviction-recover-sustain	30s	how long that must hold before it advertises itself as ready again
  --substreams-tier1-cpu-eviction-target-ratio	0.75	CPU usage a round of eviction cuts down to, which sizes the round
  --substreams-tier1-cpu-eviction-interval	5s	how often it looks at the CPU
  --substreams-tier1-cpu-eviction-cooldown	15s	minimum delay between two rounds
  --substreams-tier1-cpu-eviction-drain-delay	8s	wait between going unready and the first cancellation, covering the load balancer's routing lag
  --substreams-tier1-cpu-eviction-min-age	90s	requests younger than this are never cancelled, a request being at its most expensive while it loads its stores
  --substreams-tier1-cpu-eviction-min-burn-cores	0.05	requests burning less than this are never cancelled, since cancelling them frees nothing and costs a reconnect
  --substreams-tier1-cpu-eviction-quota-cores-override	0	CPU budget to measure usage against instead of the cgroup's cpu.max
  --substreams-tier1-cpu-eviction-nominal-capacity	0	requests a full instance carries; scales the new substreams_tier1_effective_active_requests metric and nothing else. 0 takes --substreams-tier1-active-requests-soft-limit

  Thresholds and ratios are fractions of the CPU budget. That budget is the CPU limit the cgroup carries, so an instance running under cgroup v2 with no limit set — cpu.max reading max — has nothing to compare usage to and logs a warning at startup, leaving the eviction off; --substreams-tier1-cpu-eviction-quota-cores-override names the budget yourself in that case, and should stay at or under whatever limit the kernel does enforce, since above it the instance is throttled before the eviction ever fires. Reading the cgroup CPU files failing outright, cgroup v1 included, also logs a warning and leaves the eviction off. Every one of these warnings is only emitted when the mode is not off.

• Added fireeth tools annotate-merged-blocks <gs-store-url>, which writes the datasize, itemcount and timestamp metadata entries above on merged-blocks files written before the merger did it. Files a previous run already annotated are skipped straight from the listing, so a rerun over a mostly-done store costs one listing and nothing else. --parallelism (32 by default) sets how many files are read at once; --start-block, --stop-block, --overwrite and --dry-run are supported. Google Cloud Storage only.

• Added fireeth tools stats-merged-blocks <gs-store-url>, which reports a merged-blocks store's total compressed and uncompressed size, block count, compression ratio and bytes per block, broken down by month. Nothing is downloaded: every number comes from those three metadata entries, which come back with the listing, so the whole report costs one listing however large the range is. Google Cloud Storage only.

• Added fireeth tools last-oneblock <oneblocks-store> which prints the highest block number found among the store's one-block files, as a bare number on stdout, exiting non-zero when the store cannot be listed, holds no one-block file, or a filename does not parse as one.

• Added fireeth tools substreams purge <state-url> which deletes substreams module caches that have not been used recently, reading the last_used*.zst markers that substreams-tier1 refreshes on every request it serves. Retention can be per billing plan (--retention default=30d,pro=30d,scaling=14d,free=3d), a module folder being kept as soon as one of its markers is within its own plan's retention. Direct tier1 layouts (<state-url>/<hash>, <state-url>/<tag>/<hash>) and shared network roots (<state-url>/<network>/substreams-states/<tag>/<hash>) are both recognized. Supports --dry-run, --scan-only, --keep glob patterns, --read-marker-contents for stores whose objects were copied (which resets last-write times), and --daemon --interval 12h for unattended purging. See the firehose-core changelog for the full details.

• Added fireeth tools substreams prune-states <store-url> --keep-every <blocks> --truncate-below-block <block> to delete intermediate store snapshots (<end>-<start>.kv files) of modules still in use, keeping the first snapshot of each --keep-every-aligned window. Only snapshots whose end block is at or below --truncate-below-block are thinned, the most recent snapshot of each module is always kept, and --minimum-age additionally spares recently written ones. substreams-tier1 rebuilds stores from the last remaining snapshot before the requested block (see the bump above), so pruning trades disk space for reprocessing time. Supports --dry-run.

• Added fireeth tools substreams prune-outputs <store-url> --network <network> --truncate-below-block <block> --minimum-age <age> to delete execution output files (<start>-<end>.output under a module's outputs/ folder) that are BOTH fully below --truncate-below-block AND last modified longer than --minimum-age ago, the modification time coming straight out of the listing. Module folders carrying an spkg (directly-queried output modules) are kept untouched unless --output-module-minimum-age is set. Supports --dry-run and --force.

Changed

• Bumped bstream to enable parallel one-blocks downloading upon bootstrap or reconnect (very useful on fast chains)

• Bumped golang.org/x/crypto to v0.56.0, clearing CVE-2026-78662 and CVE-2026-56855 (both HIGH), which the Docker Scout scan of the published image fails on.

• Bumped google.golang.org/grpc to v1.83.2, which fixes CVE-2026-84445.

• Bumped substreams to v1.22.1-0.20260903162505-4035f21109ec:

  • Server: substreams-tier1 scheduling no longer slows down as a large backprocessing range progresses. Picking the next tier2 job walked every segment between the squasher and the job frontier on every call, re-checking dependencies that could not have changed, so a run over N segments cost O(N²) in scheduling. The scheduler now keeps, per stage, the lowest segment that may still be pending and the highest segment completed so far, and only looks at the handful of segments those point at. On a 3-stage graph with 4 workers and a squasher three times slower than the jobs, scheduling 8000 segments went from 1.18s to 2.7ms, and now grows linearly with the range. Job order is unchanged.

  • Server: substreams-tier1 now downloads the next cached execution output files while it streams the current one to a production-mode client. Before, each 1000-block segment was opened, decompressed and sent before the next one was even requested from the object store, so every segment paid a full store round trip on the critical path. Prefetching is bounded per request, with no flag to set: at most 4 segments ahead, holding at most 64 MiB of decompressed data. No size is ever asked of the store, the decompressed size of the last downloaded segment being the estimate for the next ones, and a file bigger than the whole budget turns prefetching off for the rest of the request. Missing files are left to the existing retry loop.

  • Server: substreams-tier1 now sends each batch of cached execution output on its own goroutine, so decoding the next batch overlaps with compressing and writing the current one. At most one batch is being built while one is sent, and a segment is only reported done once every batch is out, so message order is unchanged. Decoding a batch also copies one less time, the item now aliasing the buffer it was read into.

  • Server: substreams-tier1 writes the substreams.spkg and last_used cache markers in the background instead of before the pipeline starts, so those object store round trips no longer delay the first block sent. They run detached from the request with a 30 second timeout: the request neither starts nor exits waiting for them, and a client that disconnects early still leaves its usage marker behind.

• Bumped substreams to v1.22.1-0.20260908190533-360e41fbc42f:

  • CLI: a manifest can now import sf/substreams/sink/sql/schema/v1/schema.proto without vendoring a copy of it. The file is a system protobuf, but protoparse needs the source on disk to honour its extensions, so an import previously failed with no such file. It is now served from an embedded copy, the same way sf/substreams/options.proto already was.

  • Server: fixed partial-blocks (flashblocks) streams on a tier1 that is shutting down. The stream now ends with Unavailable like a full-block stream does, so the client reconnects elsewhere. It used to stay open but silent, then send an undo signal at each block boundary naming a block the client had never received. An undo signal is also no longer sent for partial-block state whose outputs were never sent.

  • Server: substreams_tier1_effective_active_requests could read below substreams_active_requests, the metric it is meant to replace as the horizontal autoscaler input. Requests still setting up were counted by one and not the other, so a tier1 instance with requests queued in setup looked emptier to the autoscaler than it was.

• Bumped substreams to v1.22.1-0.20260911133942-1b7d09c2de7b:

  • Server: substreams-tier1 squashes store partials in runs of up to 1000 segments or 30 s of work per command instead of one segment per command, which capped squashing at about 15 segments per minute on busy backprocessing requests. Segments whose partials are all empty get a copy of the previous full store instead of a merge, server-side on object stores that support it.

  • Server: substreams-tier1 no longer closes the squasher's cached stores while a squash is still running. When the scheduler stopped early (a tier2 job failed, or the instance was shutting down), the in-flight merge could panic the process or write an empty full store to storage.

  • Server: substreams-tier1 asks the relayer for every block from its own LIB when it connects or reconnects, instead of the last 2 blocks, so a gap left by a disconnect is filled from the relayer's memory rather than from the one-block store.

• Bumped dstore to v0.2.4-0.20260911133316-3b0685e87595: S3 CopyObject is done server-side (multipart above 5 GiB) instead of downloading and uploading the object back, falling back to the old behaviour on backends answering NotImplemented or MethodNotAllowed.

• Bumped firehose-core to v1.18.1-0.20260911145235-fa9c1e143386 (untagged, released as v1.19.0):

  • Merger: no longer spins when a walk over one-block files keeps hitting the unlinkable-blocks limit. A merger stuck behind a gap in one-block files re-walked and logged too many unlinkable blocks, continuing to next loop about ten times per second; it now waits --merger-time-between-store-lookups like any other iteration, and the line is logged at Warn.

  • Merger: the unlinkable-blocks limit (bundleSize*4) can be overridden with the MERGER_MAX_UNLINKABLE_BLOCKS env var, for deployments where the default is too tight (e.g. several one-block-file writers per chain).

  • Block poller: no clients have been working for over 1 minute, still retrying is only logged when fetches have actually been failing for over a minute, and the per-block lines (about to fetch block, processing block, saved cursor, ...) are logged at Debug instead of Info.

  • Firehose: the Blocks handler logs an incoming firehose Blocks request line when a request starts, and the firehose process completed line gains duration, time_to_first_data and first_sent_block. Both lines carry trace_id and are emitted for every outcome, early rejections and client disconnects included.

  • reader-node-firehose no longer re-emits STEP_UNDO responses from its upstream endpoint as new blocks; it logs and skips them.

  • Tools: fireeth tools compare-blocks --fields prints the protobuf field paths that differ for each mismatched block, and fireeth tools stats-merged-blocks gains --json and shows the first block of each month.

• Bumped firehose-core to v1.18.1-0.20260902155646-475a571f0fe2 and substreams to v1.22.1-0.20260902153244-5658911b40ce:

  • Server: store snapshots (fullKV files) can now be pruned to save disk space: substreams-tier1 no longer assumes that a fullKV at block x implies that every earlier fullKV still exists. At request start it walks backwards from the first segment needing work, in growing listing windows, until it finds the last block where every store module still has a snapshot, and rebuilds the stores from there. Only snapshots actually seen are reused, and a job is only scheduled once the previous segment of every lower stage is done, so a pruned file is never read. The new fireeth tools substreams prune-states command (below) is what does the pruning.

  • Server: a store snapshot deleted while a request that still needs it is running now fails that request with FailedPrecondition: ... does not exist (it may have been pruned); restart the request instead of retrying the tier2 job forever. Likewise, a mapper output file deleted after its job completed is re-produced after 30 seconds instead of being waited on forever, and a scheduler deadlock on partials left by an interrupted run (a unit shadowed under one merging a partial from disk) is fixed.

  • Server: the initial store lookup no longer falls back to listing a store's whole history when its last snapshot is far behind: it lists at most a handful of bounded windows.

  • Server: progress messages are sent far less often, the cadence widening with the age of the request — every second for the first minute, every 10 seconds up to 5 minutes, every 30 seconds up to 10 minutes, then every minute — including in the linear phase, which previously sent one every 200ms. Progress messages count as egress like any other response, so a long-running or live request paid for a steady stream of them. Request.progress_messages_interval_ms is now honoured (it was validated and then ignored): setting it pins the cadence for the whole request, the 500ms minimum unchanged.

  • Server: substreams-tier1 now names the usage marker it writes in every module cache folder after the request's plan tier: last_used_<plan> (lowercase, e.g. last_used_pro), still plain last_used when unauthenticated. fireeth tools substreams purge reads the plan back from that name to apply a retention per plan.

  • Server: substreams-tier1 can now evict requests when its CPU is saturated, which the new --substreams-tier1-cpu-eviction-* flags below turn on. When its own cgroup reports CPU usage above 90% of quota for 15 seconds, the instance advertises itself as unready to the load balancer, refuses new requests, waits 8 seconds for the balancer to stop routing to it, then cancels enough of the heaviest requests with Unavailable to bring usage back under 75% of quota, so their clients reconnect elsewhere. Development-mode requests are cancelled first, then production requests on live blocks, then production requests still catching up from files — a live request burning two cores keeps burning them for as long as it stays connected, while a catching-up one is spending them on work that ends, and cancelling it throws away the segment progress it would have to redo. New substreams_tier1_cpu_* gauges and a substreams_tier1_evicted_requests_counter report what it sees and does, in observe mode included.

  • Server: new substreams_tier1_effective_active_requests gauge, meant to replace substreams_active_requests as the horizontal autoscaler input on tier1: the higher of the plain active-request count and the number of requests the CPU budget is being spent at. An instance full of expensive requests, or one holding its CPU down by cancelling requests, then reports itself at capacity and the autoscaler adds instances instead of leaving the count looking healthy.

  • Server: the jobs of a tier1 request now reach the tier2 fleet in the order the client will read their output, instead of racing each other for whatever instance has room. A job asks a per-request launch queue for its turn before every request it sends to a tier2, first attempt and retries alike, and leaves the queue as soon as a tier2 takes it. The queue is ordered by lowest segment first and, within a segment, by highest stage, and only its first 20% may dial at all — at least two jobs, at most 15 — the ones behind sending nothing until a job ahead gets in and moves the window up. A job with nothing queued ahead of it dials immediately, so a fleet with room is paced no differently than before. Without this, the segment the client reads first was no more likely to land than one it would only read minutes later, and a whole request could idle behind a single unlucky low segment. Tunable with SUBSTREAMS_WORKER_LAUNCH_WINDOW_PERCENT (default 20) and SUBSTREAMS_WORKER_LAUNCH_WINDOW_MAX (default 15).

  • Server: a tier2 job that fails is no longer retried on a growing 1s-to-5s backoff of its own; every reason to dial again goes through that same queue, so retries keep the request's reading order. A job that could not get in at all (ResourceExhausted: service currently overloaded, a refused connection, Unavailable: no healthy upstream) waits 100ms, and a job that got in and then failed waits 5 seconds once — if a tier2 then turns it away for capacity, it is back on the 100ms delay with its failure already counted. The counters that end a hopeless request are unchanged: five failures, or two execution timeouts, and neither is charged for a job that never got in. Tunable with SUBSTREAMS_WORKER_OVERLOADED_RETRY_DELAY (default 100ms, was 300ms), SUBSTREAMS_WORKER_FAILED_RETRY_DELAY (default 5s) and SUBSTREAMS_WORKER_OVERLOADED_RETRY_JITTER (default 100ms, added to a wait so jobs turned away at the same instant do not redial in lockstep).

  • Server: jobs are now scheduled up to twice the request's worker count ahead of the blocks the client is reading, instead of 1.5 times, so a client that reads slowly still keeps the workers busy.

  • Server: substreams-tier2 logs far less per segment. A large backfill fans out into tens of thousands of ProcessRange calls, each of which was writing about ten Info lines with no steady-state diagnostic value, enough to raise an instance's log volume by an order of magnitude. The duplicate auth-info log in the response handler is gone, the store-size and exec-output-file-open logs are now Debug, and so are the four shutdown-lifecycle logs of the per-request dmetering event emitter, which is opened and torn down on every ProcessRange call. The benign http2: server: error reading preface ...: connection reset by peer logged whenever a client drops a connection mid-handshake against the plaintext/h2c tier2 port is suppressed, like the existing TLS-handshake ones.

  • The merger now records what a merged-blocks file holds on the object itself, as three custom metadata entries written as it uploads each bundle: datasize, the file's size once decompressed, itemcount, the number of blocks it holds, and timestamp, the time of its first block. A listing then tells what a file holds without reading it. Google Cloud Storage only: no other backend keeps custom object metadata a listing reads back, so elsewhere nothing is written and nothing changes. An annotation that cannot be written never fails the merge.

  • fireeth tools substreams prune-states and prune-outputs delete much faster: deletions now run on their own --delete-parallelism (250 by default) instead of sharing the listing's --parallelism (16 and 64), each attempt is bounded at 5s instead of 30s, and a failed deletion is retried once after 50ms instead of four times over 7.5s. A deletion that still fails is reported as before and picked up by the next run.

  • fireeth tools substreams purge, prune-states and prune-outputs all skip any module folder carrying a DO_NOT_PRUNE file at its root, next to the last_used*.zst markers, and report how many folders that spared.

  • fireeth tools substreams logs connection now takes its time range via a --since flag instead of a positional [<date-range>] argument, matching logs connections. Both commands' --since now accepts the same grammar as logs reexec's date-range argument: a relative duration (30m, 2h, 1d, 1w, "1 day ago", …), a single timestamp, or a <start>/<end> range.

  • Dependencies: google.golang.org/grpc moves from v1.83.0 to v1.83.1, which clears GHSA-vp52-pcj8-j9qc, reported as HIGH: a peer could exhaust server heap by fragmenting HTTP/2 DATA frames.

Removed

• Removed the --reader-node-firehose-compression flag. It has never had any effect: the connection to the upstream endpoint always uses zstd. Operators setting it must drop it, as an unknown flag stops the process from starting.

v2.21.0

Compare Source

Added

• Added Ethereum Hoodi testnet (hoodi, eip155:560048) StreamingFast Firehose and Substreams endpoints (hoodi.eth.streamingfast.io:443).

• New substreams/spkgs/eth-methods-v0.1.0.spkg, a diagnostic Substreams package exercising the rpc::eth_call and rpc::eth_get_balance WASM extensions against the endpoints an instance was started with. Each extension gets its own module, map_eth_call and map_eth_get_balance, so a setup configuring only one of --substreams-rpc-endpoints / --substreams-rpc-get-balance-endpoints can still be tested, map_eth_methods runs both. Those modules also get one WASM binary each, map_transfers and map_eth_methods sharing a third importing neither extension: a module resolves its imports when it is instantiated rather than when it calls them, so a single binary mentioning both extensions is refused wholesale (unknown import: 'rpc::eth_get_balance' has not been defined) by an instance registering only one of them, whatever the module being run. Every block performs at least one single-request invocation and one batched one per method, tunable through the call_ratio, balance_ratio, call_batch_size and balance_batch_size parameters. Call targets and queried accounts are resolved out of each block, so the package runs unchanged against any EVM chain; --network pins a per chain contract for mainnet, base, arbitrum-one, optimism, matic, bsc, sepolia and hoodi, and the call_to parameter pins an arbitrary one. On a chain quiet enough that a block carries no transfer, or no transaction at all, which regularly happens on a testnet, the target resolution degrades in steps and each report says through status, target_source and hints what it could and could not do, so a quiet chain never reads as a failing endpoint. The packed artifact is committed and is what tools and operators should use, sources live under substreams/modules/eth_methods.

• Block comparison: new FIREETH_COMPARE_IGNORE_WITHDRAWALS=true which removes every trace of beacon chain withdrawals from both sides before comparing: th

[!IMPORTANT]
✂ PR body was truncated to here.

---

Configuration

📅 Schedule: (UTC)

• Branch creation
  • At any time (no schedule defined)
• Automerge
  • At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.

[renovate]

Edited/Blocked Notification

Renovate will not automatically rebase this PR, because it does not recognize the last commit author and assumes somebody else may have edited the PR.

You can manually request rebase by checking the rebase/retry box above.

⚠️ Warning: custom changes will be lost.