[cyberspace-cs]

What this fixes

NgramHashDeduplicateFilter splits each text into n_gram chunks with a single integer chunk size:

gram_length = len(text) // self.n_gram
ngrams = [text[i*gram_length:(i+1)*gram_length] for i in range(self.n_gram)]

That slice covers n_gram * (len(text) // n_gram) characters, which is only the whole text when the length is an exact multiple. Two silent data-loss cases follow:

1. The tail is never hashed. With n_gram=3, "abcdefghij" and "abcdefghix" both reduce to abc / def / ghi. The last character is invisible to the comparison, so the pair is judged a duplicate and the second row is dropped. The larger the remainder (len % n_gram), the more of the text is ignored.

2. Short rows collapse onto one hash. When len(text) < n_gram, gram_length is 0, every chunk is "", and the hash set is {hash("")} for every short row. ["a", "b", "c"] deduplicates down to ["a"].

Neither case is reported: the operator logs Total unique items: 1 and moves on. MinHashDeduplicateFilter already carries a fix for the same short-text collapse (test_minhash_short_texts.py); this brings the n-gram variant in line.

The fix

_split_chunks places the boundaries at round(i * len(text) / n_gram) instead of using a uniform integer chunk size. The chunk count stays at n_gram, but the chunks now tile the text exactly, so every character contributes to at least one hash. Texts no longer than n_gram become a single chunk, so they still hash to something that distinguishes them.

Second change: the uniqueness check was quadratic

if all(len(hash_value & hash) < self.diff_size for hash in seen_hashes):

Every row intersected its chunk set with every kept row. The predicate is equivalent to "no earlier kept row shares at least diff_size chunks", which an inverted index from chunk hash to kept rows answers without touching rows that share nothing:

rows (all distinct, n_gram=1)	before	after
8,000	2.91 s	0.04 s
20,000	20.12 s	0.08 s

Both runs keep all rows, so the outputs are identical. The 20k case is ~250× faster and the cost is no longer quadratic in the row count.

diff_size < 1 used to make every row after the first a duplicate (len(...) < 0 is never true). It is now rejected in the constructor instead of dropping data silently.

Tests

New file test/cpu_only/test_ngramhash_chunk_coverage.py, 15 cases:

• test_texts_differing_only_in_the_tail_are_not_duplicates — 10/3, 7/4 and 11/5 characters per chunk, with diff_size == n_gram so the tail is the only distinguishing signal.
• test_texts_shorter_than_n_gram_stay_distinct — ["a", "b", "c"] for n_gram 2/3/5/8.
• test_chunk_overlap_below_diff_size_is_kept — the diff_size threshold still separates 2 shared chunks from 3.
• test_exact_duplicates_are_still_removed, test_empty_texts_are_deduplicated_into_a_single_row, test_n_gram_one_hashes_the_whole_text, test_multiple_input_keys_cover_the_tail_of_the_joined_text, test_large_distinct_batch_keeps_every_row.
• test_invalid_chunk_configuration_is_rejected — n_gram=0 and diff_size=0.

On main, 10 of the 15 fail; with the patch all 15 pass, and the existing test_dedup_single_input_keys.py + test_minhash_short_texts.py still pass (28 passed together).

$ python -m pytest test/cpu_only/test_ngramhash_chunk_coverage.py -q
15 passed in 0.69s

Happy to drop the inverted index if you would rather review the correctness fix on its own — the chunking change does not depend on it.