[rchauhan9]

Closes #962.

Reading a text CIF file spends most of its time on tokenizing atom_site in Python: a loop over each value, then np.asarray() on lists of str (see the measurements in #962). This PR moves the CIF tokenizer to Rust. Rust splits the file into blocks and the blocks into categories, and turns category text directly into NumPy unicode arrays. The public API and the lazy loading do not change.

Performance

Median seconds, release build, both branches on 49273958. Apple M4 Max, macOS 27.0.1, Python 3.12.0. Medians of 20 runs (files under 2 MB), 5 runs (under 120 MB) or 3 runs (3J3Q).

Entry	atom_site rows	atom_site deserialize, main → PR	get_structure(model=1), main → PR
1AKI	1,079	0.0025 → 0.0006	0.0047 → 0.0020
4HHB	4,779	0.0109 → 0.0024	0.0184 → 0.0074
5J7L	295,261	1.21 → 0.16	1.78 → 0.52
1HTQ	978,720	3.41 → 0.50	4.15 → 0.88
3J3Q	2,440,800	9.96 → 1.12	13.04 → 3.41

Peak memory when deserializing the 3J3Q atom_site (/usr/bin/time -l): 4.14 GB → 2.48 GB.

Behaviour

The output equals main in 235,044 comparisons, with 0 differences. These include every category, block split and file split of the 34 CIF files under tests/ and of the 5 entries above, 40,000 random malformed category texts (half with indented lines), and every code point up to U+3100 except surrogates. The error cases (64,195) raise the same error type with the same message.

One deliberate change: text with lone surrogates now raises a DeserializationError (details on the line).

Trade-offs

• Exact parity, also for odd input. Line breaks follow str.splitlines(), whitespace follows str.isspace(), and broken input raises the same errors. I did not measure the cost separately, but it is a quick check per character.
• Two passes over atom_site. The first pass counts and measures the values, and the second pass writes them directly into the arrays. One pass was faster (3J3Q: 0.92 s instead of 1.14 s, measured before the rebase), but it kept every value until all widths were known, which needed about 0.6 GB more.

What is left, and what it would take

For 3J3Q, parsing now takes about 1.6 s of the 3.4 s. get_structure() spends most of the rest on converting string columns to numbers: converting one float column takes 0.22 s.

1. Parse numeric columns directly to typed arrays in Rust. This removes the unicode arrays and the second parse, but CIFColumn holds strings today. It would need a typed fast path in get_structure(), or a change to what CIFCategory returns.
2. Do not copy the text at each lazy level. For 3J3Q, splitting blocks and splitting categories each take about 0.2 s, and each level copies its text into a new string. Storing offsets into one shared text would remove these copies, but it changes how CIFFile and CIFBlock store unparsed elements.
3. One pass over atom_site, if speed matters more than memory.

As a reference point, #962 measured BinaryCIF at 0.23 s and gemmi at 1.19 s for 3J3Q.

This PR keeps the current handling of blank lines inside ; text values (see #962). If you want, I can change that in a separate PR.

Benchmark script

import gc
import statistics
import sys
import time
from pathlib import Path

import biotite
import biotite.structure.io.pdbx as pdbx


def timed(fn, repeats):
    times = []
    for _ in range(repeats):
        gc.collect()
        start = time.perf_counter()
        fn()
        times.append(time.perf_counter() - start)
    return statistics.median(times)


data_dir = Path(sys.argv[1])
print(f"biotite {biotite.__version__}")
for pdb_id in sys.argv[2:]:
    cif = data_dir / f"{pdb_id}.cif"
    size_mb = cif.stat().st_size / 1e6
    n = 20 if size_mb < 2 else 5 if size_mb < 120 else 3
    raw = pdbx.CIFFile.read(cif).block._categories["atom_site"]
    split = timed(lambda: pdbx.CIFFile.read(cif).block, n)
    deserialize = timed(lambda: pdbx.CIFCategory.deserialize(raw), n)
    structure = timed(lambda: pdbx.get_structure(pdbx.CIFFile.read(cif), model=1), n)
    print(f"{pdb_id}: read + block split {split:.4f} s, atom_site deserialize {deserialize:.4f} s, get_structure {structure:.4f} s")

[rchauhan9]

This is the one deliberate change in behaviour. PyString::to_str() needs valid UTF-8, and a lone surrogate (for example from a handle opened with errors="surrogateescape") is not valid UTF-8. main parses such text, and this PR raises a DeserializationError that says why. CIFFile.read(path) opens files in strict mode, so this only affects callers who pass their own text handle. Exact support would need a second path over bytes. I think the clear error is enough, but I can add that path if you prefer.

[rchauhan9]

Two passes over the values. The first pass counts the values and finds the widest one per column, and the second writes each value straight into its array. One pass is about 20% faster on atom_site (3J3Q: 0.92 s instead of 1.14 s, measured before the rebase), but it must keep all values (16 bytes each as &str) until the widths are known, so peak memory was 3.07 GB instead of 2.46 GB. If you prefer speed, the change back is small.

[rchauhan9]

str::lines() would be shorter, but it splits only at \n and \r\n. Python's str.splitlines() also splits at \r, \x0b, \x0c, \x1c–\x1e, \x85, 
 and 
. Block splitting reads the raw file text, so this function copies splitlines() exactly to keep the current behaviour.

[rchauhan9]

This copies Python's IndexError("list index out of range"), so that broken input fails exactly as on main. A DeserializationError that names the problem ("category has no lines", "key has no '.'") would be clearer. CIFBlock.__getitem__ wraps every error anyway, so only direct callers of CIFCategory.deserialize() would see a different type. Happy to switch.

[rchauhan9]

Rust returns pairs, and dict() on the Python side handles a repeated key: the key keeps its first position and gets the last value. That is the same result as the old category_dict[key] = ... assignment.

[rchauhan9]

split_one_line stays a #[pyfunction] only so that this test keeps all its cases. The case '"\t\n"' cannot go through CIFCategory.deserialize(), because the \n would split the line.