[settylab-dotto-bot]

Summary

find_landmarks → build_graph built a pynndescent index and then queried that index for every training point, although the build had already produced a neighbor graph. On large data with low intrinsic dimension and strongly varying density, which is the geometry of diffusion components and other trajectory-like embeddings, that query takes orders of magnitude longer than the build, and its cost grows much faster than n. This PR makes the build-only graph the default and adds an exact, order-invariant option for users who need landmarks reproducible across machines and row orders.

Related: #34 (the exact centroid snap, the previous change to find_landmarks). There is no existing issue for the slowdown.

The slowdown, reproduced on synthetic data

19 dimensions, 8 CPUs, k = 15, one run per cell on several cluster nodes, so read these as orders of magnitude:

data	cells	build only (new default)	build + query (0.9.0)	exact kd-tree
branched curve, varying density	200 000	1.5 s	217 s	5.9 s
branched curve, varying density	500 000	4.7 s	1 133 s	20 s
branched curve, varying density	1 100 000	10 s	> 44 min (stopped at a 45-min cap)	130 s
isotropic Gaussian mixture	200 000	3.2 s	6.6 s	32 s
isotropic Gaussian mixture	500 000	5.1 s	22 s	102 s
isotropic Gaussian mixture	1 100 000	12 s	89 s	837 s

• Trigger. Five 1-d curves embedded nonlinearly in 19 dims, with cells concentrated toward one end (t = u**4). Similar curves at uniform density are slower than the mixture but far less extreme (query 53 s at 200k, 98 s at 500k). Exact duplicate rows (2% or 20%) and 100× tighter clusters do not slow the query.
• Exact kNN is the mirror image. A kd-tree is fast at low intrinsic dimension and slow at high: 837 s on the 1.1M mixture, and 1 893 s at 1.1M × 50. That rules it out as the default.
• Instruments. The 0.9.0 column times pynndescent's build and query directly, except at 1.1M mixture cells, where it is the 0.9.0 build_graph end to end.
• Recall vs exact neighbors, 200k cells. Build-only graph 0.956 against 0.940 for the 0.9.0 query (mixture); 0.992 against 0.990 (curve). The new default is not a loss of accuracy.

What changes

• knn_method="nndescent" (default): NNDescent(...).neighbor_graph, with no re-query.
• knn_method="exact": scipy.spatial.cKDTree. It depends only on the data: not on the seed, the thread count or (under order_invariant) the row order.
• knn_method="nndescent_query": the 0.9.0 graph, kept so that old landmarks can be reproduced.
• find_landmarks(..., order_invariant=True):
  • lexsorts the rows into a canonical order, runs the exact graph and a seeded Leiden, and maps the indices back to the caller's order;
  • raises without an int random_state, since Leiden is otherwise unseeded, and raises with a non-exact knn_method, since the pynndescent graph changes with the numba thread count;
  • the guarantee holds across machines, thread counts and row orders, given the same numpy, scipy and python-igraph versions;
  • the docstring says it buys reproducibility, not stability: a seed change moves landmarks about as much as a permutation.
• Edge list: vectorised, and returned as an (n_edges, 2) array instead of a list of tuples. pynndescent's -1 padding entries are dropped.
• Approximate snap: exact_snap=False raises with knn_method="exact", which builds no index to snap with.
• Logging: the method used is logged.

Compatibility

This changes which landmarks are selected relative to 0.9.0 (CHANGELOG, [Unreleased]).

On a 200k-cell mixture with n_clusters=200 and random_state=0, the new default and the 0.9.0 graph share 65 landmarks (of 178 and 175). For scale, changing only the seed shares 61 (of 178 and 192), and a same-seed rerun reproduces the landmarks exactly.

• Reproduce 0.9.0: knn_method="nndescent_query" with the same seed. Verified bit-identical against the 0.9.0 code (edges, landmark indices and coordinates), including above 1000 cells per landmark.
• Reproduce 0.8.0: additionally exact_snap=False.

da(), de() and the differential classes choose landmarks with mellon, not find_landmarks, so they are unaffected. This follows the convention of #34: the default changes, the old behaviour stays one keyword away, and the CHANGELOG says so.

Tests

tests/test_find_landmarks_graph.py (12 tests):

• the exact graph equals a brute-force kNN edge for edge, and is identical at 1 and 4 threads;
• the default equals neighbor_graph, and nndescent_query equals the 0.9.0 graph;
• the vectorised edges equal the old loop, including padding;
• landmarks under 3 row permutations, indices pointing into the caller's order, and the seed and method refusals.

Every new test fails on main. The exact-graph test fails on main on a 12.6% edge mismatch, not on a type difference.

Mutation checks:

• replacing the canonical sort with the identity fails the permutation test;
• dropping the index map-back fails the indices test.

Full suite locally (Python 3.12, numpy 2.5.3, scipy 1.18.1, pynndescent 0.6.0, python-igraph 0.11.9): 2 215 passed, 36 skipped, 0 failed. One existing test was updated: the approximate-snap opt-out now passes knn_method="nndescent_query".

Timing check (not added to CI)

import time, numpy as np, pynndescent
from kompot.utils import build_graph

def curve(n, d=19, seed=0):  # branched 1-d curve, density concentrated toward one end
    rng = np.random.default_rng(seed)
    t = rng.random(n) ** 4; b = rng.integers(0, 5, n)
    W = rng.normal(size=(5, 3, d))
    basis = np.stack([np.sin((k + 1) * np.pi * t) for k in range(3)], 1)
    return np.einsum("nk,nkd->nd", basis, W[b]) + rng.normal(size=(n, d)) * 0.001

X = curve(200_000)
build_graph(X[:2000])  # numba warm-up
for m in ("nndescent", "exact", "nndescent_query"):
    t = time.perf_counter(); build_graph(X, knn_method=m)
    print(m, round(time.perf_counter() - t, 1), "s")

Not merged by me; for the next release.