[renovate]

Note

This PR body was truncated due to platform limits.

This PR contains the following updates:

Package	Update	Change
ghcr.io/streamingfast/firehose-core	minor	v1.14.4 → v1.21.1

---

Release Notes

streamingfast/firehose-core (ghcr.io/streamingfast/firehose-core)

v1.21.1

Compare Source

Changed

• Bumped dauth: with continuous authentication (?continuous=true on the grpc auth plugin), each continuous Authenticate response now refreshes the stream's trusted headers. A stream whose first authentication returned no headers (for example an authenticator that failed open) is metered with its identity from the next continuous call on, instead of without it for its whole lifetime.

• Substreams: Tier2 skips loading dependency stores and compiling WASM modules for a segment once its cached block index proves every module excludes all of the segment's blocks. It still writes the same (empty) output files as before.

v1.21.0

Compare Source

Fixed

• index-builder now serves its health check, over gRPC and HTTP /healthz, on --index-builder-grpc-listen-addr, which it never listened on.

• The tools writing merged-blocks files (resize-merged-blocks, download-from-firehose and others) now fail instead of writing a block to a file whose range does not hold it. The previous bundle's last block carried into the files of a gap is still written.

• index-builder no longer restarts indexing from --index-builder-start-block when the index store errors at startup (for example, a transient S3 failure). It now retries, then fails startup so the process restarts, instead of treating the store as empty. A failed index write now stops the indexer instead of being logged and skipped, which left a gap that made every later restart rewind to it (requires chains to propagate the error from their indexer's ProcessBlock) (#​288).

• Merger no longer moves canonical one-block files to the forked-blocks store when readers write the same block with a different LIB, as happens on Polygon PoS / Amoy. Only blocks with another ID at the same height are moved.

• The reader now always clamps a decoded block's lib_num down to its own block number when the node/plugin reports lib_num greater than the block number (invalid; lib_num equal to the block number is still valid), logs an error and increments reader_node_invalid_libnum_clamped_count, instead of letting the bad value reach the relayer where it moved LIB past head and silently stalled it forever.

• zstd stores (merged blocks, one-block files, substreams states) now close the object's HTTP stream or file when a reader is closed before the end of the file. Streams stopping mid-bundle, at their stop block or when the client goes away, used to leave it open.

• Firehose and substreams-tier1 now return the sessions they still hold to the session server before exiting, whatever --common-system-shutdown-signal-delay is set to. Sessions of requests cut by the shutdown used to be released in the background while the process exited, so they stayed counted against the organization until they expired on the session server, and a client reconnecting right away could be refused with Concurrent stream limit exceeded. This applies to session plugins implementing Close(ctx) error, which the tgm:// plugin does.

• Substreams: Fix tier1 requests rejected before their body is read (authentication, compression enforcement) sometimes failing at a load balancer with HTTP 502 or INTERNAL instead of returning their error.

• Substreams: Fix a tier1 panic (INTERNAL: runtime error: invalid memory address or nil pointer dereference) on a production-mode request starting past the chain's final block, when that final block falls within the first segment of the request's stores.

• Firehose, substreams-tier1 and the relayer no longer exit with received 5 consecutive unlinkable blocks or cannot link block after reconnection when they start while the merger is deleting the one-block files of a bundle it just merged.

• tools substreams prune-states, prune-outputs and purge no longer fail deletions on GCS with 429 rateLimitExceeded ("request distribution is too uneven across the key-ranges" or "request-rate on the bucket has ramped up too fast"). They now delete in random order instead of listing order, which sent every concurrent deletion to the same part of the bucket, are capped by the new --delete-rate flag, and retry a failed deletion 5 times over several seconds instead of twice over 50ms.

• Startup no longer lowers the open-files limit (RLIMIT_NOFILE) to 24,576 when it can't raise it to 1,000,000 and the current limit is already higher, as in containers without CAP_SYS_RESOURCE (k0s: 999,999, containerd 2.x: 524,288). It now keeps the current limit.

• Substreams: Fix wasm calls that fail because the host ran out of file descriptors or memory (for example Too many open files while creating an instance) being reported as deterministic errors, which returned InvalidArgument and were cached for 1h. They now return Internal, and the shared cache runs the call again instead of handing the same error to every stream on that block.

Added

• tools resize-merged-blocks --fix-misplaced-blocks rewrites a store where merged-blocks files hold blocks outside of their range, as the empty files of a gap in block numbers can (ex: near-testnet around block 5346171). Each block is written to the file of its own number, with its parent number set to the last block written. A block at or below the last block written is skipped, and one that does not have it as parent stops the process with an error. Without the flag, such a store fails with contains block ..., beyond the configured bundle size.

• Store URLs, such as --common-merged-blocks-store-url, accept compression_config to tune how files are written, matching the store's compression: a zstd level with an optional window in MiB (best, better/32), or a gzip level from 1 to 9. For example gs://bucket/merged-blocks?compression_config=best/32. Files written with any setting are read back without configuration. An invalid value makes opening the store fail.

• On zstd stores, compression_config also takes decoder settings, comma separated after the level or alone: lowmem=false gives each decoder a history buffer of twice the window, and pool=<name> reuses decoders across reads and across every store using that name, in place of the default pool. pool=none gives each file a decoder of its own. For example --common-merged-blocks-store-url=gs://bucket/merged-blocks?compression_config=best/32,lowmem=false,pool=blocks and --substreams-state-store-url=gs://bucket/states?compression_config=better/16,pool=cache. Substreams tier2 gets them through the URLs tier1 sends. Reading 100 MiB best/32 merged blocks files 10 at a time, adding lowmem=false to the default pool used 70% less CPU and 57% more peak heap. lowmem=false only helps files larger than their window plus about 1 MiB.

• tools substreams prune-states, prune-outputs and purge: --delete-rate (default 500) caps deletions per second, 0 removes the cap.

• reader-node and reader-node-stdin (not reader-node-firehose, which doesn't parse console lines) now decode the FIRE INIT <version> <node_variant> <node_version> line printed by every EVM Firehose tracer (geth and forks: Injective, BSC, Polygon/bor, etc) and map it to sf.ethereum.type.v2.Block, instead of writing an invalid Payload.TypeUrl.

• reader-node and reader-node-stdin also decode FIRE INIT <version> <protobuf_fully_qualified_type> <node_name> <node_version>, trusting the given type and treating the trailing fields as informational, for any chain.

• Substreams: external_calls_<kind> metering metrics (for example external_calls_eth_call), counting the calls made by WASM extensions. A batch counts for as many calls as it contains.

• Substreams: --substreams-tier1-allow-pruned-chain lets tier1 serve modules (mappers and stores) whose initial block is below --common-first-streamable-block, after that block was raised (the chain was pruned). They run from the first streamable block on, and their stores resume from the snapshots taken before the pruning. A request is refused when a store has no snapshot at the first streamable block to resume from, or when a store has no initial block: every store must set one, at or above the first streamable block. Tier2 must run this version before the flag is set on tier1.

Changed

• Go 1.27 or newer is now required to build.

• Substreams: production-mode requests disconnected for being more than 2 segments behind the last final block now get back-processing ended too far behind the last final block of the chain, please reconnect to process the gap in parallel or stream fell too far behind the last final block of the chain, please reconnect to process the gap in parallel instead of endpoint is shutting down, please reconnect. Both are still Unavailable.

• tools resize-merged-blocks now accepts equal --source-bundle-size and --target-bundle-size to rewrite merged-blocks files with different settings (ex: compression), after a confirmation prompt (--force skips it). It also clears the deprecated payload_kind and payload_buffer fields from every block written; the block data stays in payload.

• firehose and substreams-tier1 launched in the same process now share one forkable hub instead of each holding its own copy of the live blocks; on Solana that saves about 0.5 GiB per process. Firehose still streams complete blocks only.

• The firehose hub keeps as many final blocks as the substreams-tier1 hub: two merged-blocks files worth, at least 200, instead of at least 500. Clients resuming between 200 and 500 blocks below the last irreversible block now read those blocks from merged-blocks files.

• Apps running in the same process now share one session pool, created from --common-session-plugin, instead of each creating their own. With the local:// plugin, firehose and substreams-tier1 running together now count against the same max_sessions and max_sessions_per_organization limits.

• Bump firehose-networks to v0.3.0.

• RPC pollers (blockpoller) print each FIRE BLOCK line 11-15x faster and no longer allocate several times the block size to do it: the line is built once in a reused buffer, with SIMD base64 encoding.

• Subscribers to live blocks (relayer, firehose and substreams subscriptions to live) are no longer disconnected as soon as 100 blocks are waiting for them. On fast chains, a live source pausing for a few seconds and then sending its backlog at once filled those 100 slots before a subscriber could send the first block, disconnecting every subscriber at the same time. A subscriber is now disconnected when it has not caught up for --common-live-subscriber-catch-up-timeout (default 30s), which catches one that is stuck or slower than the chain, or when --common-live-subscriber-max-buffered-blocks (default 10000) blocks are waiting for it.

• The relayer, firehose and substreams-tier1 no longer look up one-block files, nor log block not linkable after one-block lookup, when a live source sends a block they already have below LIB. With several --relayer-source, this happened every few seconds.

• zstd stores (merged blocks, one-block files, substreams states) now read with decoders kept in a process-wide default pool when compression_config names no pool, and those decoders work on the reading goroutine instead of 4 background ones. Reading 100 MiB best/32 merged blocks files 10 at a time, that used 6% less CPU and 53% less peak heap; on 2 MiB files, 42% less CPU. compression_config=pool=none brings back a decoder per file.

• Firehose, substreams-tier1 and the relayer now detect a relayer or reader that disappears without closing the connection (for example a host reboot) in about 40s instead of 5 minutes. They send gRPC keepalive pings after 30s without data and drop the connection after 10s without an answer.

• Substreams: tier2 logs refusing Substreams ProcessRange request and the gRPC finished streaming call with code ResourceExhausted line at Debug instead of Info. The substreams_tier2_rejected_request_counter metric still counts the refusals by reason.

Performance improvements

• Substreams: tier1 handles Ethereum partial blocks about 20x faster, with almost no allocations. It reads the transaction traces straight from the encoded block and drops the ones already sent by copying bytes, instead of decoding and re-encoding the whole block for every partial.

• Merged blocks are read without copying their payload, which halves the memory allocated per merged-blocks file read (25.8 MiB instead of 50.8 MiB for 100 blocks of 256 KiB).

Removed

• Removed the --firehose-discard-partial-blocks flag. Firehose clients only receive complete blocks, and the hub firehose builds when it runs without substreams-tier1 never receives partial blocks from the relayer, so the flag no longer did anything. Operators setting it must drop it, as an unknown flag stops the process from starting.

Deprecated

• The SOURCE_CHAN_SIZE environment variable, replaced by --common-live-subscriber-max-buffered-blocks. It is still used when the flag is not set, with a warning at startup.

Security

• Bump golang.org/x/net to v0.61.0 to fix CVE-2026-78663 (critical), CVE-2026-78669, CVE-2026-78660 and CVE-2026-78659 (high).

• Bump OpenTelemetry to v1.47.0 to fix known vulnerabilities.

v1.20.1

Compare Source

Changed

• The reader decodes the base64 payload of FIRE BLOCK lines about 2.4 times faster (about 10 GiB/s instead of 4 GiB/s, measured on an Apple M4 Max), by using github.com/emmansun/base64 instead of the standard encoding/base64.

v1.20.0

Compare Source

Added

• New --substreams-tier1-max-request-duration flag (default 0, no limit). When set, a Substreams request that has run for that long is ended gracefully: its stores are quick-saved (with --substreams-tier1-quicksave-store) and the client is told to reconnect. Set it a bit under the stream duration limit of the load balancer serving requests.
• Reader: the base64 payload of a FIRE BLOCK line is decoded while the line is read, instead of after the whole line was copied into a string. A block no longer crashes the reader when it is bigger than the old 200 MiB line buffer, and reading a block needs about 1.5 times its line size in memory instead of about 3 times (measured on a 512 MiB block: 1.1 GiB peak instead of 2.2 GiB). This applies to chains using the default firecore.NewConsoleReader; chains with their own console reader keep receiving FIRE BLOCK lines as text.
• gRPC clients receive responses up to 2 GiB instead of 1 GiB, and servers send responses up to 2 GiB (bumped dgrpc), so a block up to 2 GiB goes from the reader through the relayer to Firehose and Substreams. Request limits are unchanged. External clients need their own receive limit raised to get blocks that big. Substreams modules cannot take a block that big: wasm32 has 4 GiB for the block, its decoded form and the output.
• --relayer-source accepts a retry_interval=<duration> query parameter (e.g. my.source:12345?retry_interval=120s) setting the minimum time between two connection attempts to that source. Use it for a rescuer or fallback endpoint that is expected to be down most of the time, so the relayer does not dial it (and log the failure) every 5s. retry_interval must be at least 5s, and since sources are checked every 5s it is rounded up to the next 5s increment (e.g. 12s behaves as 15s). Without it, a source is retried every 5s.
• New --substreams-tier1-cpu-eviction-order flag (default dev,prod-cached,prod-catchup) listing the request classes the CPU eviction may cancel, least important first. A class left out is never cancelled, so live production requests are no longer cancelled unless prod-live is added to the order. See the substreams bump below for the new prod-cached class.
• Added --substreams-tier1-squasher-plugin so fire{chain} start can point substreams-tier1 at a remote store-merge (squasher) process. Empty or local:// keeps in-process squashing. Production uses grpcs://host?secret=<token> — TLS, port defaults to 443, secret is sent as the authorization header. Plaintext local/dev uses grpc://host:port. Extra TLS/plaintext toggles stay on the DSN query string (insecure=true, plaintext=false).
• New rpc.RollingStrategySequential, a rolling strategy that walks the client pool in order from a configurable starting client, rolling forward through the rest for failover. rpc.WithSpreadStart() spreads that starting client round-robin across successive WithClients/WithClientsContext calls, so concurrent block fetches (e.g. blockpoller's batched polling) don't all hammer the same provider first; failover order after the starting point is unchanged. Without the option, behavior matches the now-deprecated RollingStrategyAlwaysUseFirst. StickyRollingStrategy is unaffected either way.

Changed

• --reader-node-line-buffer-size is now the normal size of the line buffer (default 100 MiB, at least 32 KiB). The buffer grows past it in pieces of that size for a longer line, up to 2861913428 bytes: the longest line whose block, with 1 MiB for the message around it, fits in a 2 GiB response. Each time 50 blocks in a row used less than half of it, it shrinks by half, keeping whole pieces and never going below the normal size. Reading the longest line takes up to 5.4 GiB of memory, and a grown buffer stays allocated until it shrinks back.

• --reader-node-line-buffer-size outside of 32768 to 2861913428 bytes is now refused at startup instead of being used as is. A reader configured above that range does not start until the value is lowered.

• The reader now reads the node output directly, instead of through a channel of lines holding up to 1000 of them. A reader that falls behind stops reading the node's stdout sooner, which blocks the node instead of buffering more lines in the reader.

• With --reader-node-debug-firehose-logs, a FIRE BLOCK line is logged as FIRE BLOCK <header> <payload: N bytes decoded> instead of its full base64 text, so a big block no longer writes hundreds of megabytes into the logs.

• Bumped bstream for MultiplexedSourceWithRetryIntervals, the option backing the new retry_interval on --relayer-source.

• Bumped substreams to v1.23.0:

  • Server: production requests that fall far behind the chain are disconnected so they reconnect and back-process the gap in parallel. The main case is a long back-processing: when it finishes, the request would otherwise process everything the chain produced in the meantime linearly on tier1, which can take hours. A production request is disconnected when it is more than 2 segments behind the last final block (rounded down to a segment), checked when back-processing finishes and at every segment boundary while streaming final blocks. It gets the same Unavailable "endpoint is shutting down, please reconnect" error as a tier1 restart, so clients reconnecting from their cursor pick up where they left off. Set the SUBSTREAMS_MAX_LINEAR_HANDOFF_LAG_SEGMENTS environment variable on tier1 to change the number of segments.

  • Server: the CPU eviction order is configurable. Classes are cancelled in the configured order, highest burn first within a class and oldest first on a tie. A new prod-cached class covers production requests that have not processed a block on tier1 yet, only streaming outputs cached by tier2. They run no wasm on tier1, so --substreams-tier1-cpu-eviction-min-burn-cores does not apply to them (--substreams-tier1-cpu-eviction-min-age still does), and since their CPU cost is unknown, a round of eviction stops right after cancelling one; the next round, after --substreams-tier1-cpu-eviction-cooldown, measures what it freed. The substreams_tier1_evicted_requests_counter metric gains the prod-cached class.

  • Server: per-store lines are logged at Debug instead of Info: using mmap KV store, using in-memory KV store, flushing store at boundary, merged partial into full store, deleting partial store. squashing time metrics stays at Info.

  • Server: the tier1 block hub logs processing block at Debug, except for one line every 10 seconds kept at Info to show progress. linking live block using one blocks moves up to Info, so a one-block store lookup after an unlinkable live block shows in production logs.

  • Server: tier1 can merge stores on a remote squasher selected by --substreams-tier1-squasher-plugin. Empty or local:// stays in-process. grpc:// / grpcs:// are registered from github.com/streamingfast/substreams/squash/grpc. Each remote run is one RPC per store module; if the remote is unreachable that run is squashed locally and later runs stay local until one succeeds. A live remote that returns an application error still fails the request.

Deprecated

• rpc.RollingStrategyAlwaysUseFirst / rpc.NewRollingStrategyAlwaysUseFirst, use rpc.RollingStrategySequential / rpc.NewRollingStrategySequential instead. The old name is now a type alias with an unchanged, no-option constructor, so existing callers keep compiling as-is.

Security

• Bumped google.golang.org/grpc to v1.85.0-dev.0.20260825072537-93e31b48545e, the commit that fixes CVE-2026-84445. v1.84.0 is still in the affected range; this pin comes in with substreams v1.23.0.

v1.19.0

Compare Source

Fixed

• The ready{app="firehose"} Prometheus metric now remains 0 while the firehose waits to read the first streamable block and only changes to 1 after initialization succeeds, including when serving from merged blocks without a live source.

Added

• tools compare-blocks: --fields prints the protobuf field paths that differ for each mismatched block (e.g. header.chain_id, txs[2], unknown_field(12)). Use it when --diff prints nothing: blocks are marked different by proto.Equal (which includes unknown fields), while --diff compares JSON with unknown fields stripped by default.

• The merger's maxUnlinkableBlocks circuit breaker (bundleSize*4) can now be overridden with the MERGER_MAX_UNLINKABLE_BLOCKS env var, for deployments where the default is too tight (e.g. several one-block-file writers per chain) to tolerate an ordinary reorg or brief writer hiccup. Unset or invalid values keep the existing bundleSize*4 default.

• The Blocks request handler now logs an "incoming firehose Blocks request" line as soon as a request starts, carrying trace_id, organization_id, api_key_id, real_ip, start_block, stop_block, final_blocks_only and cursor. The existing "firehose process completed" line gains duration (total request time), time_to_first_data and first_sent_block (zero-valued if no block was ever sent). Both lines are now emitted for every request outcome, including early rejections (session denied, rate limited, unimplemented transforms) and client disconnects, so every request can be paired up downstream by trace_id. A client-initiated cancellation is logged with error: "context canceled"; a server-initiated one (e.g. a revoked session) logs its real cause instead of collapsing into the same bucket.

• The merger now records what a merged-blocks file holds on the object itself, as three custom metadata entries written as it uploads each bundle: datasize, the file's size once decompressed, itemcount, the number of blocks it holds, and timestamp, the time of its first block written as 2025-10-12 10:23:12 in UTC. A listing then tells what a file holds without reading it.

  Google Cloud Storage only: no other backend keeps custom object metadata a listing reads back, so elsewhere nothing is written and nothing changes. The size is counted as the bundle streams to the store, and the first block's time is read from the head of the first one-block file, far enough to reach the timestamp field and no further, so a chain with large blocks costs no more than one with small blocks. An annotation that cannot be written never fails the merge: the bundle is written and correct, only its description is missing, and firecore tools annotate-merged-blocks fills that back in.

• Added firecore tools annotate-merged-blocks <gs-store-url>, which writes those same three entries on merged-blocks files written before the merger did it. Google Cloud Storage only, for the same reason.

  Each file is streamed through a zstd decoder and dropped as it decodes, one block at a time into a scratch buffer reused from block to block and from file to file, so memory stays flat whatever the block size; of the first block only the head is looked at, far enough to reach its timestamp. Only the block's metadata fields are ever decoded, its chain-specific payload being skipped, so no chain block type is needed. --parallelism (32 by default) sets how many files are read at once.

  Files a previous run already annotated with all three entries are skipped straight from the listing, which carries their metadata, so a rerun over a mostly-done store costs one listing and nothing else; --overwrite recomputes them. --start-block and --stop-block bound the listing service-side, --dry-run reports what would be annotated without reading anything.

  Reads and metadata writes are both pinned to the generation and metageneration the listing returned, so a file the merger rewrites mid-run is reported as failed rather than annotated with values from a version that is gone. --grpc switches to the Cloud Storage gRPC API, which on GKE takes the DirectPath route straight to the storage backend, spread over --grpc-connection-pool connections (one per 32 of --parallelism by default).

  The store URL takes the same two query parameters dstore reads off a gs:// URL: ?project= bills the reads to that project on a requester-pays bucket, and ?client_protocol=grpc selects the same transport as --grpc. Setting a project turns DirectPath off, since DirectPath does not carry the x-goog-user-project header requester-pays billing needs.

• Added firecore tools stats-merged-blocks <gs-store-url>, which reports a merged-blocks store's total compressed and uncompressed size, block count, compression ratio and bytes per block, broken down by month and totalled over --start-block to --stop-block.

  Nothing is downloaded: every number comes from the three metadata entries above, which come back with the listing, so the whole report costs one listing however large the range is. Files missing any of the three are counted and reported, and contribute to nothing. Google Cloud Storage only, where the annotation lives.

  The report is labelled with a chain name, which defaults to the second-to-last part of the store path, the bucket excluded (eth-mainnet for both gs://mybucket/something/eth-mainnet/v1 and gs://mybucket/eth-mainnet); --chain-name overrides it.

  Pass --json to get the same report as a single JSON object on stdout instead of the table: the chain name, the requested range, the first and last block seen, one entry per month, the total, and the count of files carrying no complete annotation.

• firecore tools substreams purge, prune-states and prune-outputs all skip any module folder carrying a DO_NOT_PRUNE file at its root, next to the last_used*.zst markers, and report how many folders that spared.

• Added firecore tools last-oneblock <oneblocks-store> which prints the highest block number found among the store's one-block files, as a bare number on stdout, exiting non-zero when the store cannot be listed, holds no one-block file, or a filename does not parse as one.

• Added firecore tools substreams purge <state-url> which deletes substreams module caches that have not been used recently, reading the last_used*.zst markers that substreams-tier1 refreshes on every request it serves. The required state URL matches direct tier1 layouts (<state-url>/<hash> and <state-url>/<tag>/<hash>) and shared network roots (<state-url>/<network>/substreams-states/<tag>/<hash>); --network is optional for restricting shared network roots. No database involved, the store is the only source of truth.

  Retention can be per billing plan (--retention default=30d,pro=30d,scaling=14d,free=3d), a module folder being kept as soon as one of its markers is within its own plan's retention, so a folder last used 15 days ago by a pro request survives a 3 days free retention. Plans with no explicit retention fall back to the default one, and a folder carrying no marker at all is never touched unless --purge-without-last-used is set.

  A condemned folder is emptied completely, substreams.spkg.zst and substreams.partial.spkg.zst included, unless --keep names glob patterns to spare. The scan itself downloads nothing: substreams-tier1 overwrites a marker on every request it serves, so the object's last-write time is the usage date and the listing already carries it (--read-marker-contents downloads each marker and reads the date tier1 stored inside it instead, which is what you want on any backend once the objects have been copied, migrated or rsynced, since that resets the last-write time and would otherwise make every module look freshly used). Finding the folders costs one small request per module folder, all independent, so it scales with --scan-workers (256 by default, worth raising on a large network); the state and output files of a folder are only enumerated once it is condemned. --scan-only reports what is past retention without listing or deleting anything.

  Storage goes entirely through dstore, so every store it supports works (gs://, s3://, az://, file://, a bare local path), with no backend-specific code: names, sizes and modification times all come out of the listings it was already making. The command accepts direct tier1 paths ([tag/]module-hash) and shared roots (network/substreams-states/[tag/]module-hash). A folder listing is paged one round trip at a time however few folders come back, so on the stores that bound a listing service-side (gs://, s3://) each level is cut into 17 slices listed concurrently, which takes the module-hash level of a network holding 39k folders from 8.2s to 1.6s.

  Scopes are processed one at a time and fully isolated: a shared network or direct state-store scope that cannot be listed, or whose files refuse to be deleted, is reported and the run moves on to the next one, the command exiting non-zero with a per-scope summary. Use --dry-run (-n) to see what would go, --force (-f) to skip the confirmation prompt, --network to restrict shared-root networks (direct tier1 layouts always scan all cache tags), and --daemon --interval 12h to keep purging unattended (a failed pass is retried at the next interval instead of killing the daemon).

• Added firecore tools substreams prune-states <store-url> --keep-every <blocks> --truncate-below-block <block> to delete intermediate store snapshots (<end>-<start>.kv files) and reclaim disk space on long-lived Substreams. The first snapshot of each --keep-every-aligned window is kept, windows starting on multiples of --keep-every whatever the module's initial block is (the first window is simply shorter). Only snapshots whose end block is at or below --truncate-below-block are thinned — everything above stays untouched, so requests in flight and the live head keep their nearby resume points — and the most recent snapshot of each module is always kept. --minimum-age, if set, additionally spares any snapshot last modified more recently than that (the modification time comes straight out of the listing). substreams-tier1 rebuilds stores from the last remaining snapshot before the requested block, so pruning trades disk space for reprocessing time. Deletion asks for confirmation unless --force is given; supports --dry-run. Module folders are discovered the same way purge does — direct tier1 layouts, cache tags and shared network/substreams-states/ roots — and only each module's states/ prefix is then listed; a URL pointing inside a single module folder falls back to walking the whole tree.

• Added firecore tools substreams prune-outputs <store-url> --truncate-below-block <block> --minimum-age <age> to delete execution output files (<start>-<end>.output under a module's outputs/ folder) that are BOTH fully below --truncate-below-block AND last modified longer than --minimum-age ago (the modification time comes straight out of the listing, nothing is downloaded). The URL may point at a network folder, at its substreams-states folder, at a cache tag under it, or directly at a single module folder; cache tags are discovered the same way purge does, and nothing outside outputs/ folders is ever touched. Folders carrying an spkg (substreams.spkg.zst or substreams.partial.spkg.zst), which marks a module requested directly rather than an intermediate dependency, are kept untouched by default; --output-module-minimum-age, if set, prunes them too using that (typically longer) age. Supports --dry-run and --force.

• New --substreams-tier1-cpu-eviction-* flags letting substreams-tier1 shed requests when its own cgroup reports the CPU saturated. --substreams-tier1-cpu-eviction-mode (default off, so nothing changes until you set it) selects what it does: observe logs the requests it would cancel without cancelling any, dev-only cancels development-mode requests, full cancels production ones as well. See the substreams bump below for what a round of eviction does.

  The rest are tunables, each defaulting to the value the eviction was designed around:

  Flag	Default	Meaning
  --substreams-tier1-cpu-eviction-threshold	0.90	CPU usage above which the instance counts as overloaded
  --substreams-tier1-cpu-eviction-sustain	15s	how long that must hold before it acts
  --substreams-tier1-cpu-eviction-recover-threshold	0.75	CPU usage under which it stops counting as overloaded
  --substreams-tier1-cpu-eviction-recover-sustain	30s	how long that must hold before it advertises itself as ready again
  --substreams-tier1-cpu-eviction-target-ratio	0.75	CPU usage a round of eviction cuts down to, which sizes the round
  --substreams-tier1-cpu-eviction-interval	5s	how often it looks at the CPU
  --substreams-tier1-cpu-eviction-cooldown	15s	minimum delay between two rounds
  --substreams-tier1-cpu-eviction-drain-delay	8s	wait between going unready and the first cancellation, covering the load balancer's routing lag
  --substreams-tier1-cpu-eviction-min-age	90s	requests younger than this are never cancelled, a request being at its most expensive while it loads its stores
  --substreams-tier1-cpu-eviction-min-burn-cores	0.05	requests burning less than this are never cancelled, since cancelling them frees nothing and costs a reconnect
  --substreams-tier1-cpu-eviction-quota-cores-override	0	CPU budget to measure usage against instead of the cgroup's cpu.max
  --substreams-tier1-cpu-eviction-nominal-capacity	0	requests a full instance carries; scales the new substreams_tier1_effective_active_requests metric and nothing else. 0 means default to --substreams-tier1-active-requests-soft-limit

  Thresholds and ratios are fractions of the CPU budget. That budget is the CPU limit the cgroup carries, so an instance running under cgroup v2 with no limit set — cpu.max reading max — has nothing to compare usage to and logs a warning at startup, leaving the eviction off; --substreams-tier1-cpu-eviction-quota-cores-override names the budget yourself in that case, and should stay at or under whatever limit the kernel does enforce, since above it the instance is throttled before the eviction ever fires. Reading the cgroup CPU files failing outright, cgroup v1 included, also logs a warning and leaves the eviction off. Every one of these warnings is only emitted when the mode is not off.

Changed

• Regenerated well-known protobuf descriptors from the Buf Schema Registry. The Cosmos block model (sf.cosmos.type.v2.Block) now includes CometBFT v1 consensus params (abci, synchrony, feature), a bls12381 public-key variant, and a local Int64Value for feature heights, which were previously unknown fields.

• Bumped bstream to enable parallel one-blocks downloading upon bootstrap or reconnect (very useful on fast chains)

• Bumped substreams: substreams-tier1 squashes store partials in runs of up to 1000 segments or 30 s of work per
  command instead of one segment per command, which capped squashing at about 15 segments per minute on busy
  backprocessing requests. Segments whose partials are all empty get a copy of the previous full store instead of a
  merge, server-side on object stores that support it.

• Bumped substreams: substreams-tier1 asks the relayer for every block from its own LIB when it connects or
  reconnects, instead of the last 2 blocks, so a gap left by a disconnect is filled from the relayer's memory rather
  than from the one-block store.

• Bumped dstore: S3 CopyObject is done server-side (multipart above 5 GiB) instead of downloading and uploading
  the object back, falling back to the old behaviour on backends answering NotImplemented or MethodNotAllowed.

• Block poller per-block lines (about to fetch block, requesting block, optimistically fetching block, block was optimistically polled, fetching block with hash, processing block, saved cursor) are now logged at Debug; they fired several times per block at Info.

• Bumped golang.org/x/crypto to v0.56.0, clearing CVE-2026-78662 and CVE-2026-56855 (both HIGH), which the Docker Scout scan of the published image fails on.

• firecore tools substreams prune-states and prune-outputs delete much faster: deletions now run on their own --delete-parallelism (250 by default) instead of sharing the listing's --parallelism (16 and 64), each attempt is bounded at 5s instead of 30s, and a failed deletion is retried once after 50ms instead of four times over 7.5s. A deletion that still fails is reported as before and picked up by the next run.

• Updated the embedded Solana block protobuf definitions to the latest ones published on the Buf registry: Message.version, Message.transaction_config (with the new TransactionConfig message holding priority_fee, compute_unit_limit, loaded_accounts_data_size_limit and heap_size) and the DeactivatedStake reward type are now decoded by firecore tools print and friends.

• Bumped substreams to v1.22.1-0.20260903162505-4035f21109ec:

  • Server: substreams-tier1 scheduling no longer slows down as a large backprocessing range progresses. Picking the next tier2 job walked every segment between the squasher and the job frontier on every call, re-checking dependencies that could not have changed, so a run over N segments cost O(N²) in scheduling. The scheduler now keeps, per stage, the lowest segment that may still be pending and the highest segment completed so far, and only looks at the handful of segments those point at. On a 3-stage graph with 4 workers and a squasher three times slower than the jobs, scheduling 8000 segments went from 1.18s to 2.7ms, and now grows linearly with the range. Job order is unchanged.

  • Server: substreams-tier1 now downloads the next cached execution output files while it streams the current one to a production-mode client. Before, each segment was opened, decompressed and sent before the next one was even requested from the object store, so every segment paid a full store round trip on the critical path. Prefetching is bounded per request, with no flag to set: at most 4 segments ahead, holding at most 64 MiB of decompressed data. No size is ever asked of the store, the decompressed size of the last downloaded segment being the estimate for the next ones, and a file bigger than the whole budget turns prefetching off for the rest of the request. Missing files are left to the existing retry loop.

  • Server: substreams-tier1 now sends each batch of cached execution output on its own goroutine, so decoding the next batch overlaps with compressing and writing the current one. At most one batch is being built while one is sent, and a segment is only reported done once every batch is out, so message order is unchanged. Decoding a batch also copies one less time, the item now aliasing the buffer it was read into.

  • Server: substreams-tier1 writes the substreams.spkg and last_used cache markers in the background instead of before the pipeline starts, so those object store round trips no longer delay the first block sent. They run detached from the request with a 30 second timeout: the request neither starts nor exits waiting for them, and a client that disconnects early still leaves its usage marker behind.

  • Server: store snapshots (fullKV files) can now be pruned to save disk space — which is what firecore tools substreams prune-states does: substreams-tier1 no longer assumes that a fullKV at block x implies that every earlier fullKV still exists. At request start it walks backwards from the first segment needing work, in growing listing windows, until it finds the last block where every store module still has a snapshot, and rebuilds the stores from there. Only snapshots actually seen are reused, and a job is only scheduled once the previous segment of every lower stage is done, so a pruned file is never read.

  • Server: a store snapshot deleted while a request that still needs it is running now fails that request with FailedPrecondition: ... does not exist (it may have been pruned); restart the request instead of retrying the tier2 job forever. Likewise, a mapper output file deleted after its job completed is re-produced after 30 seconds instead of being waited on forever, and a scheduler deadlock on partials left by an interrupted run (a unit shadowed under one merging a partial from disk) is fixed.

  • Server: the initial store lookup no longer falls back to listing a store's whole history when its last snapshot is far behind: it lists at most a handful of bounded windows.

  • Server: progress messages are sent far less often, the cadence widening with the age of the request — every second for the first minute, every 10 seconds up to 5 minutes, every 30 seconds up to 10 minutes, then every minute — including in the linear phase, which previously sent one every 200ms. Progress messages count as egress like any other response, so a long-running or live request paid for a steady stream of them. Request.progress_messages_interval_ms is now honoured (it was validated and then ignored): setting it pins the cadence for the whole request, the 500ms minimum unchanged.

  • Server: substreams-tier1 now names the usage marker it writes in every module cache folder after the request's plan tier: last_used_<plan> (lowercase, e.g. last_used_pro), still plain last_used when unauthenticated. firecore tools substreams purge reads the plan back from that name to apply a retention per plan.

  • Server: substreams-tier1 can now evict requests when its CPU is saturated, which the new --substreams-tier1-cpu-eviction-* flags above turn on. When its own cgroup reports CPU usage above 90% of quota for 15 seconds, the instance advertises itself as unready to the load balancer, refuses new requests, waits 8 seconds for the balancer to stop routing to it, then cancels enough of the heaviest requests with Unavailable to bring usage back under 75% of quota, so their clients reconnect elsewhere. Development-mode requests are cancelled first, then production requests on live blocks, then production requests still catching up from files — a live request burning two cores keeps burning them for as long as it stays connected, while a catching-up one is spending them on work that ends, and cancelling it throws away the segment progress it would have to redo. New substreams_tier1_cpu_* gauges and a substreams_tier1_evicted_requests_counter report what it sees and does, in observe mode included. It needs cgroup v2 CPU accounting, and a CPU budget to measure usage against — the cgroup's own limit, or --substreams-tier1-cpu-eviction-quota-cores-override where the cgroup carries none.

  • Server: new substreams_tier1_effective_active_requests gauge, meant to replace substreams_active_requests as the horizontal autoscaler input on tier1: the higher of the plain active-request count and the number of requests the CPU budget is being spent at (nominal_capacity * cpu_usage_ratio / cpu_eviction_target_ratio). An instance full of expensive requests, or one holding its CPU down by cancelling requests, then reports itself at capacity and the autoscaler adds instances instead of leaving the count looking healthy.

  • Server: the jobs of a tier1 request now reach the tier2 fleet in the order the client will read their output, instead of racing each other for whatever instance has room. A job asks a per-request launch queue for its turn before every request it sends to a tier2, first attempt and retries alike, and leaves the queue as soon as a tier2 takes it. The queue is ordered by lowest segment first and, within a segment, by highest stage, and only its first 20% may dial at all — at least two jobs, at most 15 — the ones behind sending nothing until a job ahead gets in and moves the window up. A job with nothing queued ahead of it dials immediately, so a fleet with room is paced no differently than before. Without this, the segment the client reads first was no more likely to land than one it would only read minutes later, and a whole request could idle behind a single unlucky low segment. Tunable with SUBSTREAMS_WORKER_LAUNCH_WINDOW_PERCENT (default 20) and SUBSTREAMS_WORKER_LAUNCH_WINDOW_MAX (default 15).

  • Server: a tier2 job that fails is no longer retried on a growing 1s-to-5s backoff of its own; every reason to dial again goes through that same queue, so retries keep the request's reading order. A job that could not get in at all (ResourceExhausted: service currently overloaded, a refused connection, Unavailable: no healthy upstream) waits 100ms, and a job that got in and then failed waits 5 seconds once — if a tier2 then turns it away for capacity, it is back on the 100ms delay with its failure already counted. The counters that end a hopeless request are unchanged: five failures, or two execution timeouts, and neither is charged for a job that never got in. Tunable with SUBSTREAMS_WORKER_OVERLOADED_RETRY_DELAY (default 100ms, was 300ms), SUBSTREAMS_WORKER_FAILED_RETRY_DELAY (default 5s) and SUBSTREAMS_WORKER_OVERLOADED_RETRY_JITTER (default 100ms, added to a wait so jobs turned away at the same instant do not redial in lockstep).

  • Server: jobs are now scheduled up to twice the request's worker count ahead of the blocks the client is reading, instead of 1.5 times, so a client that reads slowly still keeps the workers busy.

  • Server: substreams-tier2 logs far less per segment. A large backfill fans out into tens of thousands of ProcessRange calls, each of which was writing about ten Info lines with no steady-state diagnostic value, enough to raise an instance's log volume by an order of magnitude. The duplicate auth-info log in the response handler is gone (the incoming request already logs it once per segment), the store-size and exec-output-file-open logs are now Debug, and so are the four shutdown-lifecycle logs of the per-request dmetering event emitter, which is opened and torn down on every ProcessRange call. The benign http2: server: error reading preface ...: connection reset by peer logged whenever a client drops a connection mid-handshake against the plaintext/h2c tier2 port is suppressed, like the existing TLS-handshake ones.

Removed

• Removed the --reader-node-firehose-compression flag. It has never had any effect: the connection to the upstream endpoint always uses zstd. Operators setting it must drop it, as an unknown flag stops the process from starting.

Fixed

• The merger no longer spins when a walk over one-block files keeps hitting the unlinkable-blocks limit: that path skipped the merger-time-between-store-lookups delay, so a merger stuck behind a gap in one-block files re-walked and logged too many unlinkable blocks, continuing to next loop about ten times per second. It now waits like any other iteration, the line is logged at Warn instead of Info, and the wait is interrupted by shutdown.

• Bumped google.golang.org/grpc to v1.83.2, which fixes CVE-2026-84445.

• reader-node-firehose no longer acts on undo signals from its upstream endpoint. It treated a STEP_UNDO response like a new block and re-emitted it, now it just logs and skips them. (a reader does not take decisions on reorgs)

• The block poller no longer warns no clients have been working for over 1 minute, still retrying on slower chains like Bitcoin/Litecoin with block rate exceeding 1 minute. Instead it only warns if fetches have been actually failing for over a minute.

• Bumped substreams: substreams_tier1_effective_active_requests could read below substreams_active_requests, the
  metric it is meant to replace as the horizontal autoscaler input. Requests still setting up were counted by one and
  not the other, so a tier1 pod with requests queued in setup looked emptier to the autoscaler than it was.

• Bumped substreams: fixed handling of partial-blocks (flashblocks) streams on a tier1 that is shutting down. The
  stream now ends with Unavailable like a full-block stream does, so the client reconnects elsewhere. It used to
  stay open but silent, then send an undo signal at each block boundary naming a block the client had never
  received. An undo signal is also no longer sent for partial-block state whose outputs were never sent.

• Bumped substreams: substreams-tier1 no longer closes the squasher's cached stores while a squash is still
  running. When the scheduler stopped early (a tier2 job failed, or the pod was shutting down), the in-flight merge
  could panic the process or write an empty full store to storage.

• proto/generator: FileDescriptorSet JSON from Buf is unmarshalled with unknown fields discarded, so go generate in proto/ works against current BSR descriptor extensions.

v1.18.0

Compare Source

Added

• Added flag --validate-blocks (-b) on firecore tools check merged-blocks, ensures that individual block sequence is checked (reading the content of the merged blocks, instead of just checking by file name). Flag is implied by --print-stats and --print-full

• Bumped firehose-networks to v0.2.3: Added Ethereum Hoodi testnet (hoodi, eip155:560048) StreamingFast Firehose and Substreams endpoints (hoodi.eth.streamingfast.io:443).

• Bumped substreams to v1.21.1-0.20260820161654-1cffa6c10a8d:

  • Server: substreams-tier1 emits a periodic substreams request progress log per request (after 1 minute, then every 5 minutes) meant to answer "why is my substreams slow?" while the request is still running: phase, per-stage module and job progress, external call cost, last job error, and time spent blocked writing to the consumer. It ends with a short hints list naming the likely bottleneck when one is detected. Rates and deltas are suffixed _5m and cover a fixed trailing 5 minutes whatever the emission interval is; cadence is tunable with SUBSTREAMS_PROGRESS_LOG_FIRST_DELAY and SUBSTREAMS_PROGRESS_LOG_INTERVAL.

  • Server: substreams-tier2 reports its progress every 10 seconds while a block is being processed, instead of only once the block completes, and ExternalCallMetric gained failed_count, in_flight_count, oldest_in_flight_ms and oldest_in_flight_block. An eth_call retrying against an unreachable endpoint is a single wasm extension call that can last minutes: it used to be completely invisible to tier1 until the segment timed out.

  • Server: substreams-tier1 now restarts when its block hub can no longer link incoming live blocks, the same fix as the firehose app one below.

  • Server: the incoming Substreams Blocks request log of substreams-tier1 now carries a parallelism object (requested, granted by the authentication layer, effective count and which of the two capped it, plan tier, stage layer executors) plus parallel_segment_count and stage_count. A client asking for 300 parallel workers and getting 15 previously left no trace of the negotiation in the logs.

  • Server: the same worker accounting is now also reported once the request ends and while it runs. The substreams request stats log of substreams-tier1 gained a workers object (requested, granted, effective, peak, `pool_

[!IMPORTANT]
✂ PR body was truncated to here.

---

Configuration

📅 Schedule: (UTC)

• Branch creation
  • At any time (no schedule defined)
• Automerge
  • At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.

[renovate]

Edited/Blocked Notification

Renovate will not automatically rebase this PR, because it does not recognize the last commit author and assumes somebody else may have edited the PR.

You can manually request rebase by checking the rebase/retry box above.

⚠️ Warning: custom changes will be lost.