[pnuu]

New attempt at speeding up gradient search, now with caching. I asked Claude to start with adding a generic zarr-backed caching and implement caching for target area's projection coordinates for gradient search. That part is in the first commit. Then I asked about other caching possibilities, and we changed the caching so that the actual xy indices are stored. That is in the second commit, and matches how caching for kdtree-based nearest and bilinear resamplers work, so that's consistent. So for every source and target area combination there's a separate cache.

Timing comparisons (seconds):

chunk	target	main	branch, cache off	branch, cold cache	branch, warm cache	warm vs main
4096	nordic_stere_2000	1.70	1.68	2.05	0.45	3.8× faster
4096	euro_laea_3000	2.47	2.44	3.09	0.81	3.1× faster
4096	ceu_latlon_6000	3.91	3.88	5.24	1.77	2.2× faster
1024	nordic_stere_2000	0.65	0.65	0.85	0.33	2.0× faster
1024	euro_laea_3000	1.19	1.22	1.58	0.60	2.0× faster
1024	ceu_latlon_6000	2.86	2.92	4.01	1.59	1.8× faster

Memory usage comparisons (RSS, in MB):

chunk	target	main	branch, cache off	branch, cold cache	branch, warm cache	cold vs main	warm vs main
4096	nordic_stere_2000	588	588	607	594	+3 %	+1 %
4096	euro_laea_3000	998	998	1001	1017	+0 %	+2 %
4096	ceu_latlon_6000	2221	2210	2466	2492	+11 %	+12 %
1024	nordic_stere_2000	614	618	811	630	+32 %	+3 %
1024	euro_laea_3000	851	878	1213	958	+43 %	+13 %
1024	ceu_latlon_6000	1174	1122	1492	1343	+27 %	+14 %

Comparison between caching only the target projection coordinates vs. full indices vs. main:

chunk	target	coordinates cache, warm vs main	indices cache, warm vs main	coordinates cache, warm max RSS vs main	indices cache, warm max RSS vs main
4096	nordic_stere_2000	3.3×	3.8×	+2 %	+1 %
4096	euro_laea_3000	2.5×	3.1×	+1 %	+2 %
4096	ceu_latlon_6000	1.7×	2.2×	+8 %	+12 %
1024	nordic_stere_2000	1.6×	2.0×	+2 %	+3 %
1024	euro_laea_3000	1.7×	2.0×	+7 %	+13 %
1024	ceu_latlon_6000	1.4×	1.8×	+29 %	+14 %

Claude's summary:

• Caching off: the branch performs the same as main. Time is within ±3 % and peak RSS within ±5 % in all
  six cases, with differences in both directions.

• Warm cache, time: 1.8–3.8× faster end to end.

  • The search is skipped. What remains is reading the indices, the interpolation, and about 0.1 s in
    precompute() for hashing both areas and opening the Zarr store.
  • Warm precompute() stays lazy, with no dask computation.

• Warm cache, memory:

  • Peak RSS is +1–3 % for the two smaller targets at chunk 4096 and the 2000² target at chunk 1024.
  • It is +12–14 % for the 6000² target (both chunk sizes) and the 3000² target at chunk 1024.
  • That's an extra 107–270 MB. The memory used while computing goes up by a similar amount, so the most
    likely cause is reading and decompressing the float64 indices for each block. This wasn't profiled further.

• Cold cache:

  • The first run is 21–40 % slower than main (+0.2 to +1.3 s), because the indices for the whole target
    area are computed and written during precompute(), before the interpolation starts.
  • Peak RSS ranges from the same as main to +43 %.
  • The worst precompute() peak is 1.4 GB above baseline, for the 36 Mpix target at chunk 4096.
  • This is a one-time cost per combination of source area, target area and chunk size. With FCI or SEVIRI,
    each channel resolution pays it once.

• Disk: about 14.3–14.7 bytes per target pixel, almost the 16 bytes of the raw float64 x/y index pair.
  That's 56 MB, 125 MB and ~490 MB per source resolution for the three targets.

• Results: cached results are bit-identical to uncached ones, which the tests check.

• Indices vs coordinates cache:

  • Caching the indices is faster in every case: 1.8–3.8× vs 1.4–3.3× over main.
  • Disk use is the same for both.
  • The indices cache gives up reuse across source areas that share a projection. Each source resolution or
    extent now needs its own entry and its own cold run.

• Closes #xxxx

• Tests added

• Tests passed

• Fully documented

[codecov]

Codecov Report

❌ Patch coverage is 99.56897% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 94.02%. Comparing base (4a46786) to head (cf0df1b).
⚠️ Report is 17 commits behind head on main.

Files with missing lines	Patch %	Lines
pyresample/_caching.py	98.38%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #759      +/-   ##
==========================================
+ Coverage   93.90%   94.02%   +0.12%     
==========================================
  Files          89       90       +1     
  Lines       13924    14205     +281     
==========================================
+ Hits        13075    13356     +281     
  Misses        849      849              

Flag	Coverage Δ
unittests	94.02% <99.56%> (+0.12%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[pnuu]

Here are some benchmarks with FCI data where I load 16 composites (channels with 3 different resolutions), resample to a 5624 x 5020 pixel LAEA (EPSG:3035 with extended extents) area and do da.compute(lcl_scene). So no writing the files.

• main: 77.3 seconds
• branch, no caching: 77.9 seconds
• branch, cache init: 41.7 seconds
• branch, cache used: 31.9 seconds

I'm really surprised that even the initial run with the caching turned on is so fast compared to main / no caching run.

And dask graphs.

The graphs for main and branch with no caching pretty much identical, so here's only one:

Cache init:

Cache used:

[pnuu]

Oh, of course the initial cache run is faster than main! After the indices are computed for the first channel of a given resolution, the next channels already benefit from the cache.

Timings when actually writing the images to COG.

• main: 118.2 seconds
• branch, no caching: 115.5 seconds
• branch, cache init: 79.2 seconds
• branch, cache used: 69.5 seconds

[pnuu]

I went through the code, and LGTM. I just added optional dependencies for caching.

I also ran some more tests with FCI data when saving the images to disk. This time I reduced the PYTROLL_CHUNK_SIZE from the default 4096 to 1024 and got these timings:

main: 87.9 s
branch, no caching: 85.5 s
branch, cache init: 69.1 s
branch, cache used: 64.7 s

Also memory usage dropped from 10 GB close to, or just below, 8 GB for every case. 2024 was pretty similar both in processing time and memory use, 1-2 seconds faster or slower and maybe 200 - 400 MB more memory used than 1024. I'd say within measurement error.

With 512 that matches the tiling in the images the timing got worse and memory usage rose by 0.5 - 1 GB, must be too small chunks and create too much overhead.

[pnuu]

Couple of additional suggestions Claude gave me yesterday evening but forgot to think about:

Possible improvements

  1. Store attributes in each Zarr group (cache version, pyresample version, and a readable description
     of the key inputs), and check the version on load. That makes entries self-describing and catches
     a mismatch even if directories get copied around.
  2. Add a test that pins a small known indices result, so a change to the search output fails CI until
     someone looks at it and bumps the version. This turns the "remember to bump" rule into an
     enforced one.