refactor(dash-spv): drop committed range sweep by ZocoLini · Pull Request #1016 · dashpay/rust-dashcore

ZocoLini · 2026-09-12T12:32:14Z

Drops de final sweep, syncs goes from 21–27 min → 6–9 min, gives up the #846 case. This can be done now, because all (known) bugs that were losing funds are solved and it doesn't discover anything new

PR Hygiene · `1674cd2`

Bots — coderabbitai ✓
Self-review — post /self-reviewed
Within your 5 open PRs
Build green
Approvals
- dash-spv (dash-spv/src/sync/filters/batch.rs, dash-spv/src/sync/filters/block_match_tracker.rs, dash-spv/src/sync/filters/coinjoin_gap_discovery_tests.rs and 2 more) — QuantumExplorer or xdustinface

When every box is checked the PR Hygiene check passes and this can merge.

coderabbitai · 2026-09-12T12:32:23Z

Warning

Review limit reached

Next included review available in 18 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: dashpay/rust-dashcore/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 2ace7df7-a4c6-455d-b999-03c39db0b316

📥 Commits

Reviewing files that changed from the base of the PR and between 59dd41b and 1674cd2.

📒 Files selected for processing (5)

dash-spv/src/sync/filters/batch.rs
dash-spv/src/sync/filters/block_match_tracker.rs
dash-spv/src/sync/filters/coinjoin_gap_discovery_tests.rs
dash-spv/src/sync/filters/manager.rs
dash-spv/src/sync/filters/sync_manager.rs

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

codecov · 2026-09-12T12:53:09Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.09%. Comparing base (59dd41b) to head (1674cd2).

Additional details and impacted files

@@            Coverage Diff             @@
##              dev    #1016      +/-   ##
==========================================
- Coverage   77.16%   77.09%   -0.08%     
==========================================
  Files         329      329              
  Lines       83847    83577     -270     
==========================================
- Hits        64699    64430     -269     
+ Misses      19148    19147       -1

Flag	Coverage Δ
core	`78.18% <ø> (ø)`
ffi	`49.29% <ø> (-0.01%)`	⬇️
rpc	`20.00% <ø> (ø)`
spv	`91.98% <100.00%> (-0.10%)`	⬇️
wallet	`80.33% <ø> (ø)`

Files with missing lines	Coverage Δ
dash-spv/src/sync/filters/batch.rs	`97.56% <ø> (-0.32%)`	⬇️
dash-spv/src/sync/filters/block_match_tracker.rs	`99.35% <ø> (-0.02%)`	⬇️
dash-spv/src/sync/filters/manager.rs	`97.83% <100.00%> (+0.02%)`	⬆️
dash-spv/src/sync/filters/sync_manager.rs	`100.00% <ø> (ø)`

... and 3 files with indirect coverage changes

With the committed-height release gone, the segment cache itself has to bound the resident set. Ten 50 000-item segments per cache kept most of a restore in memory; two keep the peak below what the release achieved, without costing time. Mainnet restore, mainnet.100mbi.100ms, #1015/#1016/#1014 applied, jemalloc heap profiling, wallet identical in every run (14114383 sat, 7112 records, 13389 addresses): segments peak RSS time segment loads from disk (headers/filter headers/filters/blocks) 10 + #946 982 MiB 8.2 min - 10 1490 MiB 8.8 min 84 / 2 / 28 / 0 2 930 MiB 8.8 min 212 / 103 / 74 / 0 1 1000 MiB 7.7 min 427 / 742 / 156 / 33 Each cache served ~2.34M requests from memory in every run. One segment starts reloading block segments of up to 87 MB from disk, which raises the peak again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017DruChNTWXwJoWPartZwCf

A dirty segment leaving the resident set used to wait in `evicted`, in memory, until the next 5 s storage tick wrote it. The resident limit was therefore not a limit: every segment evicted between two ticks stayed in memory. Now eviction writes the segment first and only then drops it; if the write fails the segment stays resident and the error is returned. The `evicted` map is gone, and the tick only persists the resident segments. Mainnet restore, mainnet.100mbi.100ms, #1015/#1016/#1014 applied, two resident segments: 7.7 and 7.9 min, peak RSS 943 and 948 MiB, wallet identical (14114383 sat, 7112 records, 13389 addresses). No cache misses served from `evicted` any more; header, filter header and filter segments are reloaded from disk about as often as they used to come back from `evicted` (~270 / ~150 / ~90), block segments never. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017DruChNTWXwJoWPartZwCf

… one buffer `Segment::persist` encoded the whole segment into a `Vec` before handing it to `atomic_write`. For a block segment near the tip (up to 87 MB on disk) that `Vec` grew to 128 MiB, and since dirty segments are now written on eviction this happened in the middle of the sync. `atomic_write_items` encodes one item at a time through a 1 MiB `BufWriter` into the temporary file, then syncs and renames it as before. `atomic_write` keeps its behaviour and shares the temporary file and rename logic. Mainnet restore, mainnet.100mbi.100ms, #1015/#1016/#1014 applied, two resident segments, wallet identical in every run (14114383 sat, 7112 records, 13389 addresses): whole-segment buffer peak RSS 943 / 948 MiB 7.7 / 7.9 min streamed peak RSS 845 / 855 / 856 MiB 9.4 / 7.7 / 6.5 min The persist buffer, 68–133 MiB in the earlier heap snapshots, no longer shows up. The 9.4 min run lost time to peers in the header and filter header phases. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017DruChNTWXwJoWPartZwCf

Every segment cache held 50 000 items, whatever an item costs. Near the tip a block segment holds tens of MB of decoded blocks, while a header segment spanning the same heights is 5.6 MB and a filter header one 1.6 MB. `Persistable::ITEMS_PER_SEGMENT` lets each type choose: headers 10 000 (~1.1 MB per segment), filter headers 50 000 (~1.6 MB), filters 2 000 (~2 MB near the tip) and blocks 1 000. This changes the on-disk layout of the header, filter and block segments. Storage written with 50 000-item segments is misread by this layout and has to be deleted until a migration or a versioned folder lands. Mainnet restore, mainnet.100mbi.100ms, #1015/#1016/#1014 applied, two resident segments, jemalloc heap profiling, wallet identical in every run (14114383 sat, 7112 records, 13389 addresses): segment items time peak RSS segment caches 50 000 for every type 7.0 min 818 MiB 332 MiB 5 000 for every type 7.3-8.3 min 554-687 MiB 23-106 MiB 1 000 for every type 9.2-12.2 min 453-577 MiB 2-66 MiB per type (this commit) 7.5 min 538 MiB 38 MiB With 1 000 items everywhere, the header and filter header phases paid an fsync per evicted segment (105-141 s and 254-332 s instead of ~60 s and ~150 s). Blocks at 500 items saved ~18 MiB of cache but reloaded 50 % more block segments. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017DruChNTWXwJoWPartZwCf

HashEngineering · 2026-09-23T15:12:13Z

I tested this with an android build and found no negative effects.

**Speed —Two clean restores of the same
wallet, both from scan_start=0, same device:

build	sync to SYNCED
before #1016	7 min 04 s
with #1016	3 min 07 s

In my test I was using a modified (not pushed) #979.

github-actions · 2026-09-23T17:19:09Z

This PR has merge conflicts with the base branch. Please rebase or merge the base branch into your branch to resolve them.

…scripts rescan The #846 sweep re-tested every late-derived script against the whole stored filter history, from the wallet's birth height. On a mainnet restore of the bench wallet that was one 63 s walk over 200 000..tip matching 41 687 blocks, overwhelmingly compact-filter false positives, then about 17.7 min downloading and applying them, out of a 21–27 min sync. The notification-driven path (`collect_new_scripts`, `FiltersBatch::collected_scripts`, the current-and-later batch rescan at commit) duplicated `reconcile_untested_scripts`, which asks the wallet what it watches, a superset of any `new_scripts`, and re-tests whatever the batch never matched. `rescan_complete` was set immediately before the batch left `active_batches`, so it was never read as true. Trade-off: the #846 case, a CoinJoin output paying an index derived only after its batch committed, is no longer recovered. Its repro, `coinjoin_gap_limit_stall_across_committed_batch`, fails without the sweep and is removed with it, as is the sweep-cost test. The bench wallet does not hit that case. Validated on top of the spend re-application fix: 5 consecutive mainnet restores at 100 Mbit / 100 ms, then 100/50 Mbit x 100/500 ms, all ending at 7112 records, 14 114 383 sat and 13 389 addresses, in 6.0–17.2 min. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017DruChNTWXwJoWPartZwCf

ZocoLini · 2026-09-23T18:48:25Z

@coderabbitai approve

coderabbitai · 2026-09-23T18:48:36Z

✅ Action performed

Comments resolved and changes approved.

github-actions · 2026-09-23T18:49:04Z

Bots are done — your move: post /self-reviewed.
Full checklist in the description.

A dirty segment leaving the resident set used to wait in `evicted`, in memory, until the next 5 s storage tick wrote it. The resident limit was therefore not a limit: every segment evicted between two ticks stayed in memory. Now eviction writes the segment first and only then drops it; if the write fails the segment stays resident and the error is returned. The `evicted` map is gone, and the tick only persists the resident segments. Mainnet restore, mainnet.100mbi.100ms, #1015/#1016/#1014 applied, two resident segments: 7.7 and 7.9 min, peak RSS 943 and 948 MiB, wallet identical (14114383 sat, 7112 records, 13389 addresses). No cache misses served from `evicted` any more; header, filter header and filter segments are reloaded from disk about as often as they used to come back from `evicted` (~270 / ~150 / ~90), block segments never. Claude-Session: https://claude.ai/code/session_017DruChNTWXwJoWPartZwCf Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

… one buffer (#1088) `Segment::persist` encoded the whole segment into a `Vec` before handing it to `atomic_write`. For a block segment near the tip (up to 87 MB on disk) that `Vec` grew to 128 MiB on every write of the segment, in the middle of the sync. `atomic_write_items` encodes one item at a time through a 1 MiB `BufWriter` into the temporary file, then syncs and renames it as before. `atomic_write` keeps its behaviour and shares the temporary file and rename logic. Mainnet restore, mainnet.100mbi.100ms, #1015/#1016/#1014 applied, two resident segments written on eviction, wallet identical in every run (14114383 sat, 7112 records, 13389 addresses): whole-segment buffer peak RSS 943 / 948 MiB 7.7 / 7.9 min streamed peak RSS 845 / 855 / 856 MiB 9.4 / 7.7 / 6.5 min The persist buffer, 68–133 MiB in the earlier heap snapshots, no longer shows up. The 9.4 min run lost time to peers in the header and filter header phases. Claude-Session: https://claude.ai/code/session_017DruChNTWXwJoWPartZwCf Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor(dash-spv): drop the committed-height segment release Reverts ae2a5d6 (#946). Segments were released from memory only when a sync manager remembered to call `set_committed_height` on the block and filter storages, and only on the next timed persist. The resident set should be bounded by the segment cache itself, not by callers reporting progress to storage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017DruChNTWXwJoWPartZwCf * perf(dash-spv): keep at most two segments resident per storage cache With the committed-height release gone, the segment cache itself has to bound the resident set. Ten 50 000-item segments per cache kept most of a restore in memory; two keep the peak below what the release achieved, without costing time. Mainnet restore, mainnet.100mbi.100ms, #1015/#1016/#1014 applied, jemalloc heap profiling, wallet identical in every run (14114383 sat, 7112 records, 13389 addresses): segments peak RSS time segment loads from disk (headers/filter headers/filters/blocks) 10 + #946 982 MiB 8.2 min - 10 1490 MiB 8.8 min 84 / 2 / 28 / 0 2 930 MiB 8.8 min 212 / 103 / 74 / 0 1 1000 MiB 7.7 min 427 / 742 / 156 / 33 Each cache served ~2.34M requests from memory in every run. One segment starts reloading block segments of up to 87 MB from disk, which raises the peak again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017DruChNTWXwJoWPartZwCf --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Every segment cache held 50 000 items, whatever an item costs. Near the tip a block segment holds tens of MB of decoded blocks, while a header segment spanning the same heights is 5.6 MB and a filter header one 1.6 MB. `Persistable::ITEMS_PER_SEGMENT` lets each type choose: headers 10 000 (~1.1 MB per segment), filter headers 50 000 (~1.6 MB), filters 2 000 (~2 MB near the tip) and blocks 1 000. This changes the on-disk layout of the header, filter and block segments. Storage written with 50 000-item segments is misread by this layout and has to be deleted until a migration or a versioned folder lands. Mainnet restore, mainnet.100mbi.100ms, #1015/#1016/#1014 applied, two resident segments, jemalloc heap profiling, wallet identical in every run (14114383 sat, 7112 records, 13389 addresses): segment items time peak RSS segment caches 50 000 for every type 7.0 min 818 MiB 332 MiB 5 000 for every type 7.3-8.3 min 554-687 MiB 23-106 MiB 1 000 for every type 9.2-12.2 min 453-577 MiB 2-66 MiB per type (this commit) 7.5 min 538 MiB 38 MiB With 1 000 items everywhere, the header and filter header phases paid an fsync per evicted segment (105-141 s and 254-332 s instead of ~60 s and ~150 s). Blocks at 500 items saved ~18 MiB of cache but reloaded 50 % more block segments. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017DruChNTWXwJoWPartZwCf

* perf(dash-spv): size storage segments per item type Every segment cache held 50 000 items, whatever an item costs. Near the tip a block segment holds tens of MB of decoded blocks, while a header segment spanning the same heights is 5.6 MB and a filter header one 1.6 MB. `Persistable::ITEMS_PER_SEGMENT` lets each type choose: headers 10 000 (~1.1 MB per segment), filter headers 50 000 (~1.6 MB), filters 2 000 (~2 MB near the tip) and blocks 1 000. This changes the on-disk layout of the header, filter and block segments. Storage written with 50 000-item segments is misread by this layout and has to be deleted until a migration or a versioned folder lands. Mainnet restore, mainnet.100mbi.100ms, #1015/#1016/#1014 applied, two resident segments, jemalloc heap profiling, wallet identical in every run (14114383 sat, 7112 records, 13389 addresses): segment items time peak RSS segment caches 50 000 for every type 7.0 min 818 MiB 332 MiB 5 000 for every type 7.3-8.3 min 554-687 MiB 23-106 MiB 1 000 for every type 9.2-12.2 min 453-577 MiB 2-66 MiB per type (this commit) 7.5 min 538 MiB 38 MiB With 1 000 items everywhere, the header and filter header phases paid an fsync per evicted segment (105-141 s and 254-332 s instead of ~60 s and ~150 s). Blocks at 500 items saved ~18 MiB of cache but reloaded 50 % more block segments. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017DruChNTWXwJoWPartZwCf * refactor(dash-spv): keep Persistable inside the storage module Nothing outside `storage` implements or names the trait, and the `segments` module it lives in is private to `storage` already. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017DruChNTWXwJoWPartZwCf * feat(dash-spv): migrate segment storage to per-type segment sizes Storage version 2. Segment files move from 50_000 items each to the per-type sizes (block headers 10_000, filter headers 50_000, filters 2_000, blocks 1_000), and segment ids in file names are padded to 6 digits instead of 4, since the smaller segments push ids past 9999. The migration keeps its own frozen copy of the legacy and new layouts (names, items per segment, item encodings and sentinels) and copies raw item bytes, so it does not depend on the storage code as it evolves. Each folder is rebuilt in <storage>/tmp/<folder>, swapped in through tmp/<folder>.old, and tmp/ is removed before the next folder, so the extra disk space at any time is one folder. An interrupted run either finishes the swap or rebuilds the folder from the untouched legacy files. New segments holding only sentinels are not written: an all-sentinel segment with the highest id would make load_or_new report no tip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(dash-spv): simplify segment migration recovery and sentinel check Recovery no longer tracks a swap state: the migration deletes <storage>/tmp on start, migrates every folder that still holds 4-digit segment files and skips folders that only hold 6-digit ones. Staged segments are moved into the folder file by file and the legacy files are deleted last, so every 6-digit file in a folder is complete and rebuilding from the remaining legacy files reproduces the same files. The sentinel check looks at one field per item type (version i32::MAX for headers and blocks, an all-zero filter header, an empty filter). It is still needed: on a dev V1 mainnet storage the last new filters segment is all sentinels, and writing it made the client load a filter tip of 0 and download every filter again from height 200000. Checked on that storage (synced by dev to 2547714): the client resumes at the tip, every new file equals its slice of the legacy file and every skipped slice is only sentinels, and a SIGKILL mid-migration followed by a restart yields identical files. Blocks skip 771 of 2350 segments (523 -> 443 MiB). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(dash-spv): migrate segment folders in parallel and treat mixed folders as corruption Each folder is migrated by its own tokio task, and inside it WORKERS_PER_FOLDER spawn_blocking workers split the legacy segments (worker n takes legacy ids n, n + 4, ...). The split stays the streaming, blocking code: reading whole legacy files through tokio::fs would hold up to ~600 MB at once (filters files reach 61 MB, blocks 88 MB). Recovery is now: delete <storage>/tmp on start, skip folders that only hold 6-digit segment files, and return Corruption for a folder holding both 4-digit and 6-digit files, which wipes the storage. On a dev V1 mainnet storage (cold page cache, this server) the migration took 41-46 s sequentially and 4-17 s now. Almost all of it is the per-file fsync, which stays: without it the whole migration takes ~2 s because writes only reach the page cache, and the legacy files are deleted right after. Deferring the fsyncs until all files are written was slower. Bytes were verified against the legacy files, a SIGKILL mid-migration followed by a restart migrates correctly, and a mixed folder wipes the storage and syncs again from the checkpoint. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dash-spv): treat a legacy segment ending mid-item as corruption `read_item` took every `UnexpectedEof` as the clean end of a legacy segment, including one hit after part of an item was read. The truncated segment was then migrated and its legacy file deleted, losing the tail silently. Only an EOF before the first byte of an item ends the segment now; anything else is `MigratorError::Corruption`, which clears the storage for a resync. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

ZocoLini mentioned this pull request Sep 15, 2026

perf(dash-spv): reduce ram usage #1024

Closed

ZocoLini force-pushed the refactor/drop-committed-range-sweep branch 2 times, most recently from 5156e07 to 53c8f98 Compare September 22, 2026 16:04

ZocoLini closed this Sep 22, 2026

ZocoLini reopened this Sep 22, 2026

github-actions Bot added the merge-conflict The PR conflicts with the target branch. label Sep 23, 2026

ZocoLini force-pushed the refactor/drop-committed-range-sweep branch from 53c8f98 to 1b6d1c8 Compare September 23, 2026 18:04

github-actions Bot removed the merge-conflict The PR conflicts with the target branch. label Sep 23, 2026

ZocoLini changed the title ~~Refactor/drop committed range sweep~~ refactor(dash-spv): drop committed range sweep Sep 23, 2026

ZocoLini force-pushed the refactor/drop-committed-range-sweep branch 2 times, most recently from 1b6d1c8 to 1674cd2 Compare September 23, 2026 18:22

ZocoLini marked this pull request as ready for review September 23, 2026 18:23

github-actions Bot added the waiting-bots Waiting for the review bots to report on this head label Sep 23, 2026

coderabbitai Bot approved these changes Sep 23, 2026

View reviewed changes

github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed and removed waiting-bots Waiting for the review bots to report on this head labels Sep 23, 2026

ZocoLini merged commit 342cb29 into dev Sep 23, 2026
66 of 68 checks passed

ZocoLini deleted the refactor/drop-committed-range-sweep branch September 23, 2026 18:51

ZocoLini mentioned this pull request Sep 23, 2026

fix(dash-spv): make backward coverage durable by rewinding synced_height instead of sweeping in memory #1002

Closed

5 of 13 tasks

Navigation Menu

Repository navigation

refactor(dash-spv): drop committed range sweep - #1016

refactor(dash-spv): drop committed range sweep#1016
ZocoLini merged 1 commit into
devfrom
refactor/drop-committed-range-sweep

ZocoLini commented Sep 12, 2026 •

edited by github-actions Bot

Loading

Uh oh!

coderabbitai Bot commented Sep 12, 2026 •

edited

Loading

Review limit reached

Uh oh!

codecov Bot commented Sep 12, 2026 •

edited

Loading

Uh oh!

HashEngineering commented Sep 23, 2026

Uh oh!

github-actions Bot commented Sep 23, 2026

Uh oh!

ZocoLini commented Sep 23, 2026

Uh oh!

coderabbitai Bot commented Sep 23, 2026

Uh oh!

github-actions Bot commented Sep 23, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ZocoLini commented Sep 12, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Hygiene · 1674cd2

Uh oh!

coderabbitai Bot commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

codecov Bot commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

HashEngineering commented Sep 23, 2026

Uh oh!

github-actions Bot commented Sep 23, 2026

Uh oh!

ZocoLini commented Sep 23, 2026

Uh oh!

coderabbitai Bot commented Sep 23, 2026

Uh oh!

github-actions Bot commented Sep 23, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ZocoLini commented Sep 12, 2026 •

edited by github-actions Bot

Loading

PR Hygiene · `1674cd2`

coderabbitai Bot commented Sep 12, 2026 •

edited

Loading

codecov Bot commented Sep 12, 2026 •

edited

Loading