Skip to content

Parallel output writer and optional LCP array - #13

Merged
jebrosen merged 1 commit into
devfrom
parallel-writer
Oct 8, 2026
Merged

jebrosen merged 1 commit into
devfrom
parallel-writer

Conversation

@traviswheeler

@traviswheeler traviswheeler commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Rebuilt on dev (2026-10-06). The three earlier commits were based on
main; this branch is now one commit on top of dev 503e3ca, and the PR
base is dev.

Two changes to how sufr create writes its output.

Positioned parallel writes. Write the header, text, suffix array and
LCP array into the output file at precomputed offsets from several
threads (64 MiB chunks) instead of streaming them through one BufWriter.
write() drains each partition's scratch buffer in parallel and still
patches the LCP at each partition boundary. The write helper sits next
to the positional read helper in file_access.rs with the same
cfg(unix)/cfg(windows) split. The whole workspace type-checks for
x86_64-pc-windows-msvc, aarch64-apple-darwin and x86_64-apple-darwin
(with needletail's compression features off, which is otherwise what
stops a cross-compile here); nobody has run it on Windows.

LCP array optional, off by default. Adds write_lcp to
SufrBuilderArgs and sufr create --write-lcp. Searching never reads
the LCP array. list --show-lcp fails on a file without one and says to
rebuild with --write-lcp. For a query-time max query length shorter
than the build-time value, the subsampling step read the LCP array; on a
file without one it now compares each suffix with its predecessor in the
text, streaming the suffix array in 4M-element blocks. A file without an
LCP array stores 0 as the LCP offset; older files, which always have one,
open as before. summarize reports whether a file has an LCP array.
FileAccessIter refills 16 MiB at a time instead of 1 GiB.

Timing on 300 MB random DNA, mask 11011, 16 threads, 16 partitions, two
runs each: dev 8.1 s; this branch 6.5 s with the LCP array, 6.0 s
without. Peak memory 4.5-4.7 GB in all three. The file is 4 bytes per
suffix smaller without the LCP array.

Tests: dev's tests pass with write_lcp: true. A new test checks that
subsampling with and without the LCP array gives the same result in
seed-mask and MQL mode and matches an independent definition. With
--write-lcp the output is byte-identical to dev's on data/inputs and
the 300 MB file across default, -n 3, --seed-mask 11011,
--max-query-len 20 and --dna builds, except --max-query-len 20
without --dna on the 300 MB file, where two runs of unmodified dev
also differ from each other at the same byte.

Found while testing, present on unmodified dev, not addressed here:
locate -m 3 and extract -m 3 panic at an assertion in
file_access.rs when a query is longer than the max query length.

Implemented with assistance from Claude (Fable 5.1)

@jebrosen jebrosen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks generally good, with two caveats here I noticed that we might still deem acceptable. Both are adjacent to some future cleanups I have in mind anyway.

Comment thread libsufr/src/sufr_builder.rs Outdated
Comment thread libsufr/src/sufr_file.rs Outdated
Port of the parallel-writer branch from my attempts to revise main
(PR #13: 4cd61e9, 93e5e32, 022761b) onto dev.

The writer no longer streams the file through one BufWriter. open_output
writes the header and the text; each partition's suffix array and LCP are
written in parallel at precomputed offsets in 64 MiB chunks; finish_output
appends the sequence names and patches the header. write_at sits next to
read_exact_at in file_access.rs with the same cfg(unix)/cfg(windows) split.

The LCP array is optional and off by default (SufrBuilderArgs.write_lcp,
`sufr create --write-lcp`). lcp_pos = 0 in the header means no LCP array;
SufrMetadata.has_lcp reports it, summarize prints it, and `list --show-lcp`
exits with an error when it is absent. Without an LCP, the reader serves a
max-query-len shorter than the build-time value by comparing text slices
(care positions for a seed mask, the first mql bytes otherwise), streaming
the suffix array in 4M-element blocks.

FileAccessIter refills 16 MiB at a time instead of 1 GiB and reuses its
buffer.

Implemented with assistance from Claude (Fable 5.1)
@traviswheeler
traviswheeler changed the base branch from main to dev October 8, 2026 05:33

@jebrosen jebrosen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I validated identical output for the --write-lcp enabled path on a few sample files.

@jebrosen
jebrosen merged commit 8dd9a5b into dev Oct 8, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants