[cyberspace-cs]

What breaks today

SemDeduplicateFilter has two problems, one that stops a run and one that quietly changes the output.

1. Memory. The operator materialises the full N x N cosine matrix:

cos_sim_matrix = compute_cos_sim_matrix(embeddings)   # N x N float32
cos_sim_matrix.fill_diagonal_(0)
cos_sim_matrix = torch.triu(cos_sim_matrix, diagonal=1)   # second N x N copy
similar_pairs = torch.where(cos_sim_matrix >= (1 - self.eps))

Measured on this machine:

rows	one cosine matrix	with the triu copy
20,000	1.5 GB	~3 GB
100,000	37.3 GB	~75 GB

Embedding has the same shape problem: tokenizer(texts, padding=True, ...) pads every text in the dataframe to the longest one and runs a single forward pass over all of them.

2. A chain of near-duplicates collapses into one row. The operator drops the second element of every similar pair, including pairs whose first element was already dropped:

labels = [1] * len(dataframe)
for idx in similar_pairs[1].tolist():
    labels[idx] = 0

With alpha ~ beta, beta ~ gamma, alpha !~ gamma and the default threshold of 0.95:

main:  kept ['alpha']            <- gamma is not similar to alpha, but is dropped anyway

Seven rows 10° apart on the unit circle (each neighbour at 0.985, each second neighbour at 0.940) collapse to ['t0'] on main; only three of the six drops are real duplicates.

What changed

• get_text_embedding takes a batch_size (default 256) and concatenates the per-batch embeddings. run passes it through.
• find_duplicate_labels scores block_size rows at a time (default 1024), so peak memory is O(block_size ** 2) — 4 MB — instead of O(N ** 2). Only the rows of one block against one column block are ever in memory.
• A row is dropped only when it is similar to a row that is still kept. This is the standard greedy definition, and it keeps the tail of a chain. Within a block the rows are resolved in index order against the block's own precomputed sub-matrix, so the result is identical to a row-by-row scan (asserted in the tests for block_size 1, 3, 64 and 1024).
• batch_size and block_size are documented in both get_desc variants.
• An empty dataframe is written back instead of calling the model with no texts.
• Removed the leftover _compute_hash, which referenced self.hash_func_dict, an attribute that is never defined.

compute_cos_sim_matrix is kept as-is for callers that already hold a small number of embeddings.

Measurements

20,000 rows, 2,000 tight clusters, eps=0.05, CPU:

	time	survivors
before	1.42 s	2000
after	1.46 s	2000

Same result on clustered data, same runtime, but the matrix is gone: tracemalloc peak drops from 1526 MB (a single full matrix) to 41 MB for the same 20k rows, and the cost no longer grows quadratically with the frame size.

Tests

test/cpu_only/test_sem_dedup_blocked_scoring.py — 10 cases, no model download (the embedder is stubbed, the operator is built without __init__ so CI does not hit the Hugging Face hub):

• test_a_chain_of_near_duplicates_keeps_the_tail — the alpha/beta/gamma case above.
• test_a_chain_is_not_collapsed_into_a_single_row — the seven-row chain.
• test_blocked_scoring_matches_a_row_by_row_reference[1|3|64|1024] — blocked scorer equals a naive reference.
• test_eps_is_the_margin_below_one — eps still means 1 - eps.
• test_texts_are_embedded_in_batches, test_embedding_helper_splits_the_input — batching.
• test_an_empty_dataframe_is_written_back_untouched.

On main, 9 of the 10 fail:

assert ['alpha'] == ['alpha', 'gamma']
assert ['t0'] == ['t0', 't2', 't4', 't6']
assert [(3, 256)] == [(3, 2)]

With the patch, all 10 pass, and test/cpu_only/test_dedup_single_input_keys.py plus test/cpu_only/test_word_number_filter_missing_values.py still pass (25 tests together).