Repository navigation
Parallel output writer and optional LCP array - #13
Merged
Merged
Conversation
jebrosen
reviewed
Oct 5, 2026
jebrosen
left a comment
Member
There was a problem hiding this comment.
This looks generally good, with two caveats here I noticed that we might still deem acceptable. Both are adjacent to some future cleanups I have in mind anyway.
Port of the parallel-writer branch from my attempts to revise main (PR #13: 4cd61e9, 93e5e32, 022761b) onto dev. The writer no longer streams the file through one BufWriter. open_output writes the header and the text; each partition's suffix array and LCP are written in parallel at precomputed offsets in 64 MiB chunks; finish_output appends the sequence names and patches the header. write_at sits next to read_exact_at in file_access.rs with the same cfg(unix)/cfg(windows) split. The LCP array is optional and off by default (SufrBuilderArgs.write_lcp, `sufr create --write-lcp`). lcp_pos = 0 in the header means no LCP array; SufrMetadata.has_lcp reports it, summarize prints it, and `list --show-lcp` exits with an error when it is absent. Without an LCP, the reader serves a max-query-len shorter than the build-time value by comparing text slices (care positions for a seed mask, the first mql bytes otherwise), streaming the suffix array in 4M-element blocks. FileAccessIter refills 16 MiB at a time instead of 1 GiB and reuses its buffer. Implemented with assistance from Claude (Fable 5.1)
traviswheeler
force-pushed
the
parallel-writer
branch
from
October 8, 2026 05:33
022761b to
75d19d3
Compare
jebrosen
approved these changes
Oct 8, 2026
jebrosen
left a comment
Member
There was a problem hiding this comment.
LGTM! I validated identical output for the --write-lcp enabled path on a few sample files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebuilt on
dev(2026-10-06). The three earlier commits were based onmain; this branch is now one commit on top of dev 503e3ca, and the PR
base is dev.
Two changes to how
sufr createwrites its output.Positioned parallel writes. Write the header, text, suffix array and
LCP array into the output file at precomputed offsets from several
threads (64 MiB chunks) instead of streaming them through one BufWriter.
write() drains each partition's scratch buffer in parallel and still
patches the LCP at each partition boundary. The write helper sits next
to the positional read helper in file_access.rs with the same
cfg(unix)/cfg(windows) split. The whole workspace type-checks for
x86_64-pc-windows-msvc, aarch64-apple-darwin and x86_64-apple-darwin
(with needletail's compression features off, which is otherwise what
stops a cross-compile here); nobody has run it on Windows.
LCP array optional, off by default. Adds
write_lcptoSufrBuilderArgsandsufr create --write-lcp. Searching never readsthe LCP array.
list --show-lcpfails on a file without one and says torebuild with
--write-lcp. For a query-time max query length shorterthan the build-time value, the subsampling step read the LCP array; on a
file without one it now compares each suffix with its predecessor in the
text, streaming the suffix array in 4M-element blocks. A file without an
LCP array stores 0 as the LCP offset; older files, which always have one,
open as before.
summarizereports whether a file has an LCP array.FileAccessIter refills 16 MiB at a time instead of 1 GiB.
Timing on 300 MB random DNA, mask 11011, 16 threads, 16 partitions, two
runs each: dev 8.1 s; this branch 6.5 s with the LCP array, 6.0 s
without. Peak memory 4.5-4.7 GB in all three. The file is 4 bytes per
suffix smaller without the LCP array.
Tests: dev's tests pass with
write_lcp: true. A new test checks thatsubsampling with and without the LCP array gives the same result in
seed-mask and MQL mode and matches an independent definition. With
--write-lcpthe output is byte-identical to dev's on data/inputs andthe 300 MB file across default,
-n 3,--seed-mask 11011,--max-query-len 20and--dnabuilds, except--max-query-len 20without
--dnaon the 300 MB file, where two runs of unmodified devalso differ from each other at the same byte.
Found while testing, present on unmodified dev, not addressed here:
locate -m 3andextract -m 3panic at an assertion infile_access.rs when a query is longer than the max query length.
Implemented with assistance from Claude (Fable 5.1)