Read ShapePipe v1 and v2 products side by side

[cailmdaley]

Closes #340. Closes #342. Closes #361.

Where to look

82 files change, but most are mechanical. The design is in four files; read them first:

1. src/sp_validation/io.py: the one catalogue reader (file format from contents, lazy or column-restricted).
2. src/sp_validation/grammar.py: v1 → v2 column names and units, plus column_map.
3. config/columns/shapepipe_v2.yaml: every column sp_validation reads, with its meaning.
4. src/sp_validation/galaxy.py: the mask cut and the v2 selection choices (NGMIX_N_EPOCH > 0, mask bits).

Skim:

• Call sites: catalog_builders.py, rho_tau.py, cosmo_val/*, calibration.py, masks.py and scripts/ now read through the above, and most of their diffs are that routing.
• Configs: config/calibration/mask_v1.X.* renames mask columns.
• Tests: test_grammar, test_io, test_grammar_calibration and test_release_regression are the behavioural evidence.

Skip:

• Deleted files (patch-era scripts, survey.py, compute_theory_cov.py): about 1,700 lines, nothing replaces them.
• catalog.py's removed readers, which moved to io.py.

What changes

sp_validation now reads both generations of ShapePipe products through one code path written against the v2 grammar:

• v2: the Snakemake campaign products (final_cat_<campaign>.hdf5, full_starcat_<campaign>.hdf5, per-bit mask columns, no patches).
• v1: every existing catalogue, v1.3–v1.6, including the fiducial v1.4.6.3.

Only two things know about v1: the reader and the adapter. Everything else — builders, workflow, configs, scripts — speaks the v2 campaign model.

One reader, any container. sp_validation.io is the only place that opens catalogue files, and every call site goes through it, cosmo_val, ρ/τ and the calibration scripts included.

• open_catalogue opens a file as one lazy table without reading data up front. FITS is memory-mapped; HDF5 columns are read when indexed.
• read_catalogue reads a restricted set of columns into memory and, given a key_column, records which chunk each row came from (EXPID on the star catalogue).
• Format from contents: the reader tells FITS from HDF5 by the file's contents, not its extension. It handles a FITS table HDU, a single HDF5 dataset, the comprehensive data+data_ext, and ShapePipe's campaign files of one dataset per tile or exposure, at any group nesting (checked against n_tiles/n_exposures).
• Memory: the campaign merge streams, so peak memory is the merged array plus one tile.

Column names are configuration. The v2 names are the canonical schema, and config/columns/shapepipe_v2.yaml lists every column sp_validation reads by fixed name, each with its meaning.

• column_map: any catalogue config (cat_config blocks, calibration params) may carry column_map: {canonical name: name in the file}, with one-* patterns allowed ({"NGMIX_*": "MYFIT_*"}), so a catalogue in another naming convention needs a few lines of YAML.
• Role keys: the cat_config *_col keys still pick which column plays a role (e.g. w_iv vs w_des).
• Guard test: test_column_schema checks code and schema against each other in both directions.
• Docs: the docs build renders the schema as the "Catalogue columns" page.

Adapter (v1). grammar.adapt presents a v1 table in v2 names and units, and reads the generation from the file's columns.

• It renames E1_PSF_HSM → HSM_G1_PSF, NGMIX_ELL_* → NGMIX_G1/G2_*, and so on.
• It converts SIGMA_*_HSM = σ → HSM_T_* = 2σ².
• It corrects one v1 defect: the no-shear reconvolved-PSF size is taken from the 1P column. Remove vestigial FHP/MK NOSHEAR←1P Tpsf substitution (guaranteed no-op) #267 removed this substitution as a no-op, which holds only for v2 products.
• It applies a config column_map ahead of its v1 rules; a map entry overrides only the rule for its own name.
• It works lazily over numpy, FITS_rec and h5py. A row selection on h5py reads each dataset once.
• The map is in docs/ngmix_psf_column_migration.md.

Patches retired (#340). No code branches on, loops over, or names P1–P8. JointCat merges a list of campaign files and adds a campaign column. Patch-only scripts are deleted: prepare_patch_for_spval.sh, plot_rho_stats_patches.py, survey_stats_all.sh, star_match_stats.py, check_tile_IDs_SP_LF.py, combine_results.py and compute_area.py (the last two fed a create_joint_shape_cat.py that no longer exists), and cosmo_val/compute_theory_cov.py. That last one had hard-coded home paths and version keys that are not in cat_config; rho_tau builds the same CovTauTh from cat_config, and the workflow drives that path. merge_psf_cat.py and stats_tile_id_gal_counts.py take campaigns. Released v1 merged catalogues are flat (data / data_ext) and keep their int8 patch column, which passes through untouched.

Masks (#342). v1 and v2 share one mask vocabulary: one boolean column per UNIONS mask bit, named MASK_{bit}_{Label} (MASK_1_Faint_star_halos, MASK_4_Stars, MASK_1024_Maximask, …). These are the bit/label pairs ApplyHspMasks has always written into v1 data_ext, now prefixed.

• grammar.MASK_LABELS is the single table, and ApplyHspMasks writes through it.
• The adapter presents v1 {bit}_{Label} and the pre-release MASK_n{bit} of the v2 integration campaigns under the same names.
• ShapePipe #886 writes the same names (eleven columns; bit 512 has no map).
• CalibrateCat.read_cat returns one table (data joined with data_ext). Every mask config is a single dat: list, and a config with a dat_ext: section raises an error.
• galaxy.mask_cut ORs the six reason bits {1, 2, 4, 8, 64, 1024} and drops NaN rows with a warning.
• v1 configs keep their IMAFLAGS_ISO == 0 cut, whose bits mean something different from the mask columns.

ρ/τ reads catalogues through the adapter and passes tables to shear_psf_leakage (#44 there). cat_config column names were corrected to match their files, and a candide test now checks every entry. Dropping a scalar mask= argument fixes the DES and jackknife ρ/τ paths, which built empty catalogues.

Breaking changes for existing scripts and notebooks:

• catalog.read_campaign_catalogue, read_star_catalogue, iter_campaign_tiles and read_shape_catalog are replaced by io.open_catalogue / io.read_catalogue; metacal loses its prefix argument (use column_map).
• read_cat() returns one table, not (dat, dat_ext).
• get_masks_from_config(config, dat, ...) takes that table.
• Mask configs name MASK_{bit}_{Label}.
• extract_info no longer runs on v1 per-patch final_cat files, and v1 per-patch FITS can no longer be re-merged; the merged v1 releases are what the readers open.
• The nine SExtractor-only columns DR6 catalogue mode lacks (shapepipe #924) are no longer passed through.

Verification.

• Release regression: a candide test recalibrates rows 200M–201M of the v1.4 comprehensive catalogue and matches released v1.4.6.3 exactly: the same 131,526 objects in the same order, and 17 per-object columns bit-identical. On a separate 5M-row / 240-tile subset, the selection is identical (684,385 objects) and so is every per-object column. The weights and leakage correction are not bit-identical to the release, because the release fed its weight and leakage binning the uncorrected PSF size while this branch uses the corrected size throughout.
• ρ/τ on SP_v1.4.6.3 through the adapter is identical to the earlier run on a hand-translated PSF file (≤1e-13σ with the same patch seed).
• v2: exercised on the smk-g7 campaign (64 tiles; see comment).
• Masks: a table written with each of the three spellings gives identical columns and identical cuts.
• Generality: a catalogue under made-up column names reads to the same table from FITS, flat HDF5, data+data_ext and chunked HDF5. A metacal catalogue renamed NGMIX_*→MYFIT_* and given a column_map calibrates to bit-identical R and g.
• Memory: reading the v1.4.6.3 shear and star catalogues through the leakage path peaks at 15.4 GiB RSS, the same as the upstream reader, with c1/c2 bit-identical.
• Unit tests: the fast suite passes (448), and the slow suite (release regression, additive bias) and the candide config tests pass at b1fbf84.

— Claude (Opus) on behalf of Cail

🤖 Generated with Claude Code

[github-actions]

✅ ruff is clean — nothing to fix here.

[cailmdaley]

Exercised against the first real v2 products (smk-g7, 64 tiles): galaxy reader 1,851,100 rows / 77 columns in 11 s at 1.2 GB RSS; star reader 53,264 stars over 127 exposures; default mask cut keeps 72.98%; full selection 1,105,851 rows with mean e1 = −0.00020 ± 0.00019, e2 = −0.00015 ± 0.00019. Three follow-up commits from that run: an explicit NGMIX_N_EPOCH > 0 guard (CosmoStat/shapepipe#889 — 1.03% of objects are never fit but carry MCAL_FLAGS = 0 with −10 sentinels; the old != −10 PSF comparison happened to catch them), a NaN-safe mask cut, and n_exposures validation + an EXPID column on the star reader. One decision left open on purpose: config/calibration/mask_v2.0.yaml has no NGMIX_MCAL_FLAGS cut — adding one changes row counts, so it is Cail's call.

— Claude (Fable) on behalf of Cail

[martinkilbinger]

Very nice, I like the campaign tag superceding the patch IDs. And the adopt mechanism.

[cailmdaley]

this is now ready for review @sachaguer!

[sachaguer]

I am confused by the new fits to hdf5 catalog loading. Until this PR, sp_validation was able to read .fits file given some configuration giving the name of the columns. Now my understanding of the PR is that there are two branches to read the ShapePipe output. One that reads the catalog from hdf5 files and one that reads from FITS files.

What is puzzling me is whether the catalog loading is now kind of hard coded on UNIONS and I could not find my answer in the many modifications that were made. For example, in galaxy.py the star columns written by ShapePipe v2 is hardcoded in the script. My worry is if someone wants to run our pipeline and its data and, say, the star catalog is a FITS file or a hdf5 file with a different name convention for the columns. Is it possible to run sp_validation with this updated API.

I would be happy to have clarification on that matter. I was not able to disentangle it myself just going through the many modified files.

[cailmdaley]

Replying to Sacha's question about hard-coded catalogue loading. That was a fair worry, and it pointed at something real. Column names were still configurable, but the file format was decided separately at each call site, and cosmo_val could only open FITS. Since b1fbf84, both the file format and the column names are configuration:

• File format: sp_validation.io is the only code that opens catalogues, and everything goes through it. It tells FITS from HDF5 by the file's contents, and reads a FITS table, a single HDF5 dataset, or ShapePipe's campaign files of one dataset per tile or exposure. A star catalogue can be FITS or HDF5 under any name.
• Column names: the code reads only the names listed in config/columns/shapepipe_v2.yaml, which gives each column's meaning and is rendered as the "Catalogue columns" docs page. A catalogue with other names adds column_map: {canonical name: name in your file} to its config entry, and patterns like {"NGMIX_*": "MYFIT_*"} work. A test fails if the code reads a name the schema doesn't list.
• Your galaxy.py example: the mask columns there are defaults for the mask_columns parameter, and they can be renamed through the map like everything else.

A test reads one catalogue with made-up column names from each of these layouts and gets identical tables. Another renames a metacal catalogue's columns and recovers bit-identical R and calibrated g through the map. The PR description now opens with a "Where to look" section: the four files that carry the design, which diffs are mechanical, and what can be skipped, so the 82 files are less of a wall.

— Claude (Opus) on behalf of Cail