From ae5264bf92cd98be6ddb7ca0254ea946a2621e8d Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 30 Sep 2026 05:03:34 +0200 Subject: [PATCH 1/7] Clarify detection decisions after catalogue switch --- astra.yaml | 75 +++++++++++-------- .../modules/sextractor_package/__init__.py | 3 +- workflow/config/cfis/config_exp_psfex.ini | 1 + workflow/config/cfis/config_tile_Sx.ini | 3 +- workflow/config/cfis/config_tile_Uc.ini | 2 +- 5 files changed, 50 insertions(+), 34 deletions(-) diff --git a/astra.yaml b/astra.yaml index 24dd350d3..a77b49e06 100644 --- a/astra.yaml +++ b/astra.yaml @@ -331,9 +331,11 @@ analyses: description: >- Object detection on r-band tiles (the galaxy sample) and on single-exposure CCDs (PSF-star candidates, always SExtractor). The tile - sample is Gwyn's UNIONS tile catalogue or ShapePipe's own SExtractor run - (tile_detection); the latter follows the MegaPipe parameters of that - catalogue, while exposures keep ShapePipe's stock values. + sample is Gwyn's UNIONS tile catalogue (committed for data) or + ShapePipe's own SExtractor run (image simulations), per tile_detection. + Every default_tile.sex / config_tile_Sx.ini setting below applies only + to the SExtractor run, which follows the MegaPipe parameters of that + catalogue; exposures keep ShapePipe's stock values. inputs: - id: tile_stack type: data @@ -376,20 +378,25 @@ analyses: detection_threshold_policy: label: Detection significance, minimum area, matched filter rationale: >- - Tiles use the MegaPipe tile-catalogue threshold, minimum area and - filter, so ShapePipe's galaxy sample matches the catalogue UNIONS - adopts; the 7x7 Gaussian of FWHM 3 px is near the CFIS average seeing - of 0.65 arcsec (about 3.5 px at 0.187 arcsec/px). Exposures only feed - star selection and keep stock values with the 3x3 FWHM 2 px kernel. - The tiles differ from Guinot+22 in all three. Matching image - simulations must use the same prescription. + Tile SExtractor (tile_detection = sextractor, i.e. image + simulations) uses the MegaPipe tile-catalogue threshold, minimum + area and filter, so that its galaxy sample reproduces the UNIONS + catalogue committed data adopts; the 7x7 Gaussian of FWHM 3 px is + near the CFIS average seeing of 0.65 arcsec (about 3.5 px at + 0.187 arcsec/px). Exposures only feed star selection and keep stock + values with the 3x3 FWHM 2 px kernel. The tiles differ from Guinot+22 + in all three. Values: + config_tile_Sx.ini#SEXTRACTOR_RUNNER.DOT_SEX_FILE = + $SP_CONFIG/default_tile.sex; default_tile.sex#DETECT_THRESH = 1.0; default_tile.sex#ANALYSIS_THRESH = 1.0; default_tile.sex#DETECT_MINAREA = 3; default_tile.sex#FILTER = Y; config_tile_Sx.ini#SEXTRACTOR_RUNNER.DOT_CONV_FILE = $SP_CONFIG/gauss_3.0_7x7.conv; + config_exp_psfex.ini#SEXTRACTOR_RUNNER.DOT_SEX_FILE = + $SP_CONFIG/default_exp.sex; default_exp.sex#DETECT_THRESH = 1.5; default_exp.sex#ANALYSIS_THRESH = 1.5; default_exp.sex#DETECT_MINAREA = 5; @@ -543,17 +550,18 @@ analyses: saturation_level: label: Saturation level read from each image's header rationale: >- - Both passes take each image's saturation level from its SATURATE - header card and set FLAGS bit 4 on objects with saturated pixels. - Star selection requires FLAGS == 0 at every step, so the level decides - which bright stars leave the PSF sample; tile FLAGS reach the - catalogue for downstream cuts. Where the card is absent, SExtractor - falls back to its built-in SATUR_LEVEL (50000 ADU per its - documentation), since neither .sex file sets one. Delivered data - carry the card: a sampled tile (CFIS.186.307) has SATURATE 9558.7 - and every CCD of a sampled exposure (2114045p) 65535, and split_exp - keeps each CCD's header. Changing the card or pinning a fixed level requires - checking the resulting bright-star selection. + Each SExtractor pass takes the image's saturation level from its + SATURATE header card and sets FLAGS bit 4 on objects with saturated + pixels. Neither .sex file sets SATUR_LEVEL, so where the card is + absent SExtractor falls back to its built-in level (50000 ADU per + its documentation). Delivered exposures carry the card (every CCD + of 2114045p: 65535) and split_exp keeps each CCD's header; a sampled + tile (CFIS.186.307) has 9558.7, which applies under tile_detection = + sextractor. The PSF-star masks require FLAGS == 0, so the level can + remove bright stars from the PSF sample. In committed data the final + catalogue's FLAGS column is the UNIONS catalogue's own, not this + pass's. Changing the card or pinning a fixed level requires checking + the resulting bright-star selection. Values: default_exp.sex#SATUR_KEY = SATURATE; default_tile.sex#SATUR_KEY = SATURATE; @@ -571,12 +579,15 @@ analyses: photometry_parameters: label: Kron and aperture photometry definitions rationale: >- - Stock Kron, aperture and flux-fraction settings on both passes; no - rationale recorded. MAG_AUTO is the axis of the star-selection - magnitude window and the catalogue magnitude; FLUX_AUTO is PSFEx's - photometric normalisation. A different Kron factor shifts magnitudes - and so every magnitude-based cut. The reported apertures and flux- - fraction measurements must also match the simulated catalogue. + Stock Kron, aperture and flux-fraction settings; no rationale + recorded. On the exposure pass MAG_AUTO is the axis of the + star-selection magnitude windows and FLUX_AUTO is PSFEx's + photometric normalisation, so a different Kron factor moves stars + across those cuts. The tile values apply under tile_detection = + sextractor; in committed data the catalogue's MAG_AUTO is the UNIONS + catalogue's own. The reported apertures and flux-fraction + measurements must match the simulated catalogue they are compared + with. Values: default_tile.sex#PHOT_AUTOPARAMS = 2.5,3.5; default_exp.sex#PHOT_AUTOPARAMS = 2.5,3.5; @@ -619,10 +630,11 @@ analyses: `all_world2pix(..., 0)` returns 0-based pixels, and the strict test accepts `33 < x < 2080`. In 1-based FITS coordinates this is x=35 through x=2080 inclusive: it trims DATASEC columns 33 and 34 but - admits its upper endpoint, column 2080. A WCS inversion failure skips - that CCD, lowering N_EPOCH; this sets how many exposures enter each - galaxy's multi-epoch fit. - Both tile_detection options run this post-processing. + admits its upper endpoint, column 2080. [HARDCODED] The comparisons + are strict in make_post_process; CCD_SIZE sets only the bounds. A + WCS inversion failure skips that CCD, lowering N_EPOCH, which caps + how many exposures enter each galaxy's multi-epoch fit. Both + tile_detection options run this post-processing. Values: SEXTRACTOR_RUNNER.CCD_SIZE = 33,2080,1,4612; SEXTRACTOR_RUNNER.MAKE_POST_PROCESS = True; @@ -634,6 +646,7 @@ analyses: label: "x in (33,2080), y in (1,4612), strict" inclusive_bounds: label: Same bounds, inclusive + description: Not implemented; make_post_process uses strict comparisons. catalogue_neighbour_marking: label: Neighbour pixels in the catalogue path's VIGNET rationale: >- diff --git a/src/shapepipe/modules/sextractor_package/__init__.py b/src/shapepipe/modules/sextractor_package/__init__.py index 40f40ca91..386985d49 100644 --- a/src/shapepipe/modules/sextractor_package/__init__.py +++ b/src/shapepipe/modules/sextractor_package/__init__.py @@ -64,7 +64,8 @@ WORLD_POSITION : list, optional List of world coordinates to use to match objects CCD_SIZE : list, optional - Size of a CCD in pixels ``[nx, ny]`` + Accepted CCD pixel bounds ``[xmin, xmax, ymin, ymax]``, 0-based and + compared strictly """ diff --git a/workflow/config/cfis/config_exp_psfex.ini b/workflow/config/cfis/config_exp_psfex.ini index 2c573e652..05382bb16 100644 --- a/workflow/config/cfis/config_exp_psfex.ini +++ b/workflow/config/cfis/config_exp_psfex.ini @@ -77,6 +77,7 @@ NUMBERING_SCHEME = -0000000-0 EXEC_PATH = source-extractor # SExtractor configuration files +# @sc [decision:detection.detection_threshold_policy] DOT_SEX_FILE = $SP_CONFIG/default_exp.sex DOT_PARAM_FILE = $SP_CONFIG//default.param # @sc [decision:detection.detection_threshold_policy,label:effective-filter] exposure-dot-conv-overrides-filter diff --git a/workflow/config/cfis/config_tile_Sx.ini b/workflow/config/cfis/config_tile_Sx.ini index 0c05c38b5..aa5db396f 100644 --- a/workflow/config/cfis/config_tile_Sx.ini +++ b/workflow/config/cfis/config_tile_Sx.ini @@ -72,6 +72,7 @@ NUMBERING_SCHEME = -000-000 EXEC_PATH = source-extractor # SExtractor configuration files +# @sc [decision:detection.detection_threshold_policy] DOT_SEX_FILE = $SP_CONFIG/default_tile.sex # @sc [decision:detection.detection_source_mode,label:override] tile-parameter-file-overrides-column-list # DOT_PARAM_FILE overrides PARAMETERS_NAME; keep it pointed at default_noimaflags.param so tile detections do not request IMAFLAGS_ISO. @@ -124,6 +125,6 @@ MAKE_POST_PROCESS = True # @sc [decision:preparation.object_position_columns] WORLD_POSITION = XWIN_WORLD,YWIN_WORLD -# Number of pixels in x,y of a CCD. Format: Nx,Ny +# Accepted CCD pixel bounds xmin,xmax,ymin,ymax (0-based, strict) # @sc [decision:detection.epoch_membership_ccd_bounds] CCD_SIZE = 33,2080,1,4612 diff --git a/workflow/config/cfis/config_tile_Uc.ini b/workflow/config/cfis/config_tile_Uc.ini index ab9d04742..861abb287 100644 --- a/workflow/config/cfis/config_tile_Uc.ini +++ b/workflow/config/cfis/config_tile_Uc.ini @@ -91,6 +91,6 @@ MAKE_POST_PROCESS = True # World coordinate keywords, SExtractor output. Format: KEY_X,KEY_Y WORLD_POSITION = ALPHA_J2000,DELTA_J2000 -# Number of pixels in x,y of a CCD. Format: Nx,Ny +# Accepted CCD pixel bounds xmin,xmax,ymin,ymax (0-based, strict) # @sc [decision:detection.epoch_membership_ccd_bounds] CCD_SIZE = 33,2080,1,4612 From 839a59fb963562d47f03e566e6058aef7f4dbf34 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 30 Sep 2026 05:04:02 +0200 Subject: [PATCH 2/7] Correct masking decision scope --- astra.yaml | 31 +++++++++++++++++-------------- 1 file changed, 17 insertions(+), 14 deletions(-) diff --git a/astra.yaml b/astra.yaml index a77b49e06..4f459460c 100644 --- a/astra.yaml +++ b/astra.yaml @@ -202,12 +202,12 @@ analyses: # ═════════════════════════════════════════════════════════════════════════ masking: description: >- - Which masks reach the measurement, and where. ShapePipe generates no - masks; the instrument flag image delivered with each exposure is the - only one that reaches pixels. Sky-fixed masks (star halos and bodies, - manual regions, missing bands) are healsparse maps built and designed - outside ShapePipe, which only queries them at object positions into - catalogue columns that no stage cuts on. + Which external masks reach the measurement, and where. The instrument + flag image delivered with each exposure is the only one that reaches + pixels. Sky-fixed masks (star halos and bodies, manual regions, missing + bands) are healsparse maps built and designed outside ShapePipe, which + only queries them at object positions into catalogue columns that no + stage cuts on. inputs: - id: exposure_flags type: data @@ -227,18 +227,21 @@ analyses: decisions: [pixel_mask_source, psf_star_mask_veto, sky_mask_application] decisions: pixel_mask_source: - label: Only the instrument flag image reaches pixels + label: >- + Of the external masks, only the instrument flag image reaches pixels rationale: >- On exposures SExtractor reads the split flag image, producing IMAFLAGS_ISO, which the PSF star selection requires to be zero. The multi-epoch vignet run cuts flag stamps from the same image; ngmix - gives flagged pixels weight 0 and drops mostly-flagged epochs - (shape_measurement.defect_fill, + gives flagged pixels weight 0 and drops epochs more than a third + flagged (shape_measurement.defect_fill, shape_measurement.epoch_masked_fraction_cut). Tiles have no flag - image, so tile detection runs unflagged - (detection.detection_source_mode). Sky-fixed masks never touch - pixels: an object inside a star halo is measured from the same - unmodified pixels as one outside it. + image, and ShapePipe's tile SExtractor run requests no IMAFLAGS_ISO + (detection.detection_source_mode). Neighbour pixels are masked + separately, from the tile segmentation map + (detection.catalogue_neighbour_marking, + shape_measurement.blend_handling). Sky-fixed masks never touch + pixels: a star halo changes no stamp. Values: config_exp_psfex.ini#SEXTRACTOR_RUNNER.FLAG_IMAGE = True; VIGNETMAKER_RUNNER_RUN_2.ME_IMAGE_PATTERN = flag, image, weight, @@ -317,7 +320,7 @@ analyses: claim: >- The published ShapePipe pipeline generated its own masks, including Messier objects and CCD borders, and applied them to the images; the - current pipeline generates none. + current pipeline applies no sky-fixed mask to pixels. created_at: "2022-06-01T00:00:00Z" evidence: - id: ev_farrens22_masks From a0ed81d78dd55fe153c9cabc64f630823d7a2ae1 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 30 Sep 2026 05:05:01 +0200 Subject: [PATCH 3/7] Clarify astrometry and epoch provenance record --- astra.yaml | 29 +++++++++++-------- .../sextractor_package/sextractor_script.py | 2 ++ .../modules/split_exp_package/split_exp.py | 6 ++-- 3 files changed, 22 insertions(+), 15 deletions(-) diff --git a/astra.yaml b/astra.yaml index 4f459460c..bc1a22945 100644 --- a/astra.yaml +++ b/astra.yaml @@ -749,11 +749,13 @@ analyses: astrometric_solution_source: label: Astrometry taken verbatim from delivered per-CCD headers rationale: >- - split_exp builds WCS(header) from each raw CCD header and - merge_headers stores them; every downstream world-to-pixel transform - (stamp placement, epoch membership, position seeding) uses that - solution. [HARDCODED] no astrometric re-derivation or refinement - exists; a joint re-fit would move every stamp centre and position seed. + split_exp builds WCS(header) from each delivered CCD header, and + merge_headers stores these per-tile. Every committed world-to-pixel + step on exposures uses that solution: epoch membership, + multi-epoch stamp placement and OFFSET, tile-side PSF interpolation + positions, and the per-epoch ngmix Jacobian. [HARDCODED] No + astrometric re-derivation or refinement exists. A re-fit would shift + stamp centres and Jacobians and could change epoch membership. default: delivered_headers options: delivered_headers: @@ -780,13 +782,16 @@ analyses: epoch_provenance_from_tile_history: label: Epoch sets parsed from tile FITS HISTORY cards rationale: >- - The coadd's own provenance is the epoch list: the file names in - column COLNUM of each HISTORY line, stripped of their full extension - and deduplicated. A tile whose header cannot be read fails rather - than yielding an empty list. Names keep their trailing p (2243881p), - an epoch letter rather than a prefix, so EXP_PREFIX is blank; - downstream code drops the p when it needs the bare exposure ID. A - mis-parse changes N_EPOCH and which exposures are fit. + The tile's primary-header HISTORY cards are its exposure list, and + two readers parse the same cards. find_exposures takes the file name + in column COLNUM, strips the extension and deduplicates. EXP_PREFIX + is blank, so 2243881p keeps its p. This list decides which exposures + are retrieved, split and merged into the tile's WCS log. + make_post_process re-parses the HISTORY copied into the catalogue + (the digits before an optional p) and builds the EPOCH HDUs and + N_EPOCH that the stamps and ngmix use. It raises if a HISTORY + exposure is missing from the WCS log. A tile whose header or HISTORY + cannot be read fails; it never gets an empty exposure list. Values: FIND_EXPOSURES_RUNNER.COLNUM = 3. default: history_parse diff --git a/src/shapepipe/modules/sextractor_package/sextractor_script.py b/src/shapepipe/modules/sextractor_package/sextractor_script.py index 8faf240ab..10143739d 100644 --- a/src/shapepipe/modules/sextractor_package/sextractor_script.py +++ b/src/shapepipe/modules/sextractor_package/sextractor_script.py @@ -123,6 +123,8 @@ def make_post_process(cat_path, f_wcs_path, pos_params, ccd_size, w_log=None): objects near it, never the tile. CCD_N is the 0-based index that split_exp gives the CCD file and its header entry. + @sc [decision:preparation.epoch_provenance_from_tile_history] + The columns will be: - ``NUMBER``: same as SExtractor NUMBER diff --git a/src/shapepipe/modules/split_exp_package/split_exp.py b/src/shapepipe/modules/split_exp_package/split_exp.py index 5c08b8163..dcdbc24bf 100644 --- a/src/shapepipe/modules/split_exp_package/split_exp.py +++ b/src/shapepipe/modules/split_exp_package/split_exp.py @@ -79,9 +79,9 @@ def create_hdus(self, exp_path, output_suffix, transf_int, save_header): @sc [decision:preparation.astrometric_solution_source,label:convention] wcs-from-delivered-header The stored WCS is ``WCS(header)`` of the delivered CCD header, - unmodified. Every downstream world-to-pixel transform (epoch - membership, stamp placement, position seeding) uses it, so a refit or - header edit here moves every stamp centre and epoch assignment. + unmodified. Epoch membership, multi-epoch stamp placement, PSF + interpolation positions and the ngmix Jacobian all use it; a refit or + header edit here shifts them. Parameters ---------- From 29bf6aee22d28e646a27dba8d131884e9ca64c33 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 30 Sep 2026 05:06:06 +0200 Subject: [PATCH 4/7] Record the MCCD workflow gate --- astra.yaml | 25 +++++++++++++++---------- workflow/README.md | 8 +++++--- workflow/Snakefile | 1 + 3 files changed, 21 insertions(+), 13 deletions(-) diff --git a/astra.yaml b/astra.yaml index bc1a22945..59625994a 100644 --- a/astra.yaml +++ b/astra.yaml @@ -990,16 +990,18 @@ analyses: psf_modelling_software: label: PSF model, PSFEx per CCD or MCCD over the focal plane rationale: >- - psf_model in workflow/config.yaml's input_types: table (image sims - use fake) selects the exposure and tile config pair; the committed value is psfex, which fits each CCD - independently. MCCD (Liaudat+2021) fits one hybrid local+global - model over the focal plane; the completeness table treats its - counts as warnings because no campaign has run it. Up to the model, - its exposure chain matches PSFEx's: SExtractor reads the split - image, weight and instrument flag directly, and mask_query sits - between SExtractor and setools. The selected model name also flows - through SP_PSF in downstream inputs, so producer and consumer choices - must move together. + psf_model in workflow/config.yaml's input_types: table selects the + exposure and tile config pair and is exported as SP_PSF to the tile + interpolation. The committed value is psfex, which fits each split + CCD independently (image sims default to fake, the true PSF with no + exposure fit). MCCD (Liaudat+2021) fits one hybrid local+global + model per exposure over the focal plane. Its ShapePipe chain exists + and matches PSFEx's up to the model: SExtractor reads the split + image, weight and instrument flag, then mask_query, then setools. + However, the Snakefile refuses psf_model=mccd at parse time, because + exp_persist and star_cat_merge read only PSFEx products. Choosing + MCCD therefore needs MCCD-aware persistence and star-catalogue + merging first. Values: INSTANCE.N_COMP_LOC = 8; INSTANCE.D_COMP_GLOB = 8; @@ -1020,6 +1022,9 @@ analyses: insights: [guinot22_psfex_software, farrens22_two_psf_methods] mccd_focal_plane: label: MCCD hybrid local+global focal-plane model + description: >- + Refused by workflow/Snakefile (refuse_unpersistable_psf) until + persistence and star-catalogue merging read MCCD products. insights: [farrens22_two_psf_methods] psf_model_complexity: label: PSFEx pixel basis with degree-2 spatial variation per CCD diff --git a/workflow/README.md b/workflow/README.md index 6e91b344f..a1654fcb4 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -25,8 +25,9 @@ uv pip install 'snakemake>=9,<10' 'snakemake-executor-plugin-slurm>=2.7,<3' # Write a run config (see Run configuration below) that sets at least `run:`, # the campaign's name; workflow/config.yaml's input_types: and machines: tables supply the rest. -# `psf_model` is `psfex` or `mccd`. psfex is exercised by smk-g4 through smk-g6; mccd has run the full chain on -# an image-sim star tile (one focal-plane model per exposure, ~1.5 CPU-hours each). +# `psf_model` is `psfex` for data (`fake` for image sims). `mccd` is refused +# until `persist_exp.py` and `merge_star_cat.py` read MCCD products. PSFEx is +# exercised by smk-g4 through smk-g6. # `tile_detection` is `unions_catalogue` (the input_types default for data: # the UNIONS per-tile catalogue at `inputs.catalogues` is fetched and # converted in place, keeping its NUMBER, and its segmentation map sets @@ -70,7 +71,8 @@ SExtractor (for the background maps the vignets read), and `tile_vignets` runs `fake_interp_runner`, which writes the `galaxy_psf` product from `psf_dict`. With no PSF model there is nothing to persist per exposure, so `exp_persist` and `star_cat_merge` do not run and `clean_exposure` does not wait on them. -Simulations that contain stars can run `psfex` or `mccd` exactly as the data do. +Simulations that contain stars can run `psfex` as the data do. `mccd` is +refused until `persist_exp.py` and `merge_star_cat.py` read MCCD products. One campaign per shear branch, each with its own run config: diff --git a/workflow/Snakefile b/workflow/Snakefile index c9124fcd2..6aed81cff 100644 --- a/workflow/Snakefile +++ b/workflow/Snakefile @@ -602,6 +602,7 @@ PERSIST_EXP = list(config.get("persist_exp") or []) PERSISTS_PSF = PSF_MODEL != "fake" +# @sc [decision:star_selection_psf.psf_modelling_software] def refuse_unpersistable_psf(psf_model): """Refuse a PSF model whose products the persistence path cannot read. From 7fbf375df5ce4ba470952604155334d005393a4d Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 30 Sep 2026 05:07:12 +0200 Subject: [PATCH 5/7] Scope PSF likelihood noise and sentinel tags --- astra.yaml | 7 +++++-- src/shapepipe/modules/ngmix_package/ngmix.py | 2 +- src/shapepipe/modules/ngmix_runner.py | 2 +- tests/module/test_ngmix.py | 10 +++++----- 4 files changed, 12 insertions(+), 9 deletions(-) diff --git a/astra.yaml b/astra.yaml index 59625994a..c2c2bbc32 100644 --- a/astra.yaml +++ b/astra.yaml @@ -1377,8 +1377,11 @@ analyses: psf_likelihood_noise: label: Flat PSF-observation weight rationale: >- - [HARDCODED] the PSF observation carries a flat weight 1/PSF_NOISE^2; - without it the g-prior swamps the PSF likelihood. The recovered PSF + [HARDCODED] each PSF observation carries a flat weight + 1/PSF_NOISE^2. Without it, ngmix's unit weight lets the GPriorBA(0.4) + prior swamp the original-PSF prefit and pull the *_psf_orig + ellipticity to zero; the metacal shears are unchanged + (test_psf_weight_leaves_metacal_shear_invariant). The recovered PSF shape and size are flat for PSF_NOISE from 1e-4 to 1e-6 on the digital twin. Values: diff --git a/src/shapepipe/modules/ngmix_package/ngmix.py b/src/shapepipe/modules/ngmix_package/ngmix.py index 6c05ef62e..4adc8227b 100644 --- a/src/shapepipe/modules/ngmix_package/ngmix.py +++ b/src/shapepipe/modules/ngmix_package/ngmix.py @@ -71,7 +71,7 @@ def get_type_flags(fit): Fit flags of one metacal type, reading absence of evidence of success as failure. - @sc [label:convention] mcal-flags-zero-means-measured + @sc [decision:catalogue_assembly.failure_sentinels,label:convention] mcal-flags-zero-means-measured A flag of 0 means the fit ran, reported success and returned a finite shear; no default or fallback may produce 0. FLAGS_, MCAL_FLAGS (OR) and MCAL_TYPES_FAIL (count) all derive from this function, and diff --git a/src/shapepipe/modules/ngmix_runner.py b/src/shapepipe/modules/ngmix_runner.py index 75d26f60a..5817d75a8 100644 --- a/src/shapepipe/modules/ngmix_runner.py +++ b/src/shapepipe/modules/ngmix_runner.py @@ -44,7 +44,7 @@ def ngmix_runner( ): """Define The Ngmix Runner. - @sc [decision:shape_measurement.blend_handling,decision:shape_measurement.centroid_source,decision:shape_measurement.defect_fill,decision:shape_measurement.metacal_scheme] + @sc [decision:shape_measurement.blend_handling,decision:shape_measurement.centroid_source,decision:shape_measurement.defect_fill,decision:shape_measurement.metacal_scheme,decision:shape_measurement.galaxy_pixel_weights] @sc [label:operations] empty-tile-product A tile whose PSF or galaxy vignette store is entirely empty never diff --git a/tests/module/test_ngmix.py b/tests/module/test_ngmix.py index ed87b4be2..1824e49c0 100644 --- a/tests/module/test_ngmix.py +++ b/tests/module/test_ngmix.py @@ -128,11 +128,11 @@ def _metacal_noshear_g(seed): def test_metacal_is_reproducible_with_fixed_seed(): """Same seed -> identical metacal shear. - The module seeds ``self._rng = RandomState(seed)`` per tile precisely so a - rerun reproduces. This guards the noise-image and masked-pixel draws in - ``prepare_ngmix_weights`` against silently falling back to the unseeded - global ``numpy.random`` state, which would make shear estimates - irreproducible from one run to the next. + Production seeds a fresh RandomState per object from its position + (position_seed), so a rerun reproduces. This guards the noise-image and + masked-pixel draws in ``prepare_ngmix_weights`` against silently falling + back to the unseeded global ``numpy.random`` state, which would make + shear estimates irreproducible from one run to the next. """ npt.assert_array_equal(_metacal_noshear_g(42), _metacal_noshear_g(42)) From e7a858c24c451559b05e7b15558802d30848ecd6 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 30 Sep 2026 05:13:11 +0200 Subject: [PATCH 6/7] Update catalogue and completeness decision record --- astra.yaml | 92 +++++++++++-------- .../read_ext_sexcat.py | 5 +- workflow/config/cfis/config_tile_Uc.ini | 1 + workflow/config/cfis/final_cat.param | 4 +- workflow/scripts/merge_final_cat.py | 11 +-- 5 files changed, 66 insertions(+), 47 deletions(-) diff --git a/astra.yaml b/astra.yaml index c2c2bbc32..d7f1fb5cc 100644 --- a/astra.yaml +++ b/astra.yaml @@ -83,18 +83,20 @@ decisions: per_unit_completeness: label: Per-unit completeness gate rationale: >- - Every shapepipe_run rule checks its products against a nominal - per-runner count (per chunk for tile_ngmix). A runner short of its count - fails the unit (exposure or tile), so a partial unit never enters the - catalogue and a failed unit's objects are absent. The one tolerated - shortfall is the exposure-side psfex_interp VALIDATION output: a CCD - whose model fails the acceptance gate - (star_selection_psf.psf_acceptance_thresholds) produces nothing and the - unit only warns. The MCCD chain, never run in a campaign, warns on every - runner; its counts follow config_exp_mccd.ini (one mask_query catalogue - per CCD; preprocessing merges an exposure's stars into one train and one - test catalogue). Science-path PSF rejection bypasses this table: - psfex_interp drops the epoch per object inside the tile run. + Every shapepipe_run rule checks its products against a per-runner + nominal count (per chunk for tile_ngmix). The unit fails if a + non-warning runner falls short or shapepipe_run exits nonzero, and a + failed unit publishes no manifest for later rules to build on. A tile + needs the exp_psf manifest of every exposure it lists, so one failed + exposure keeps every tile it contributes to out of the catalogue; the + tile does not run with fewer epochs. The one tolerated shortfall is the + exposure-side psfex_interp VALIDATION output. A CCD whose model fails + the acceptance gate (star_selection_psf.psf_acceptance_thresholds) + produces nothing there, and the unit only warns. Science-path PSF + rejection bypasses this table: psfex_interp drops that epoch per object + inside the tile run. The MCCD table (one mask_query catalogue per CCD; + one train and one test catalogue per exposure) warns on every runner, + and the workflow refuses psf_model=mccd. Values: COMPLETENESS[exp_psf.psfex.psfex_interp_runner.warn] = True; COMPLETENESS[exp_psf.mccd.mask_query_runner.expect] = 40; @@ -121,14 +123,18 @@ decisions: postage_stamp_size: label: Postage-stamp size shared by vignets, ngmix stamps and PSF models rationale: >- - One size pins three coupled apertures: the SExtractor VIGNET around each - detection, the vignetmaker stamps ngmix fits, and the PSFEx model - stamp. The stamp bounds the measurable galaxy size and truncates the - wings of large galaxies. No rationale for 51 px (about 9.5 arcsec) is - recorded. + Every stamp width is 51 px (about 9.5 arcsec): the tile VIGNET (cut + from the tile image by read_ext_sexcat in committed data, by SExtractor + under tile_detection = sextractor), the multi-epoch vignetmaker stamps + ngmix fits, the exposure SExtractor VIGNET PSFEx trains on, and the + PSFEx model stamp. Only the first two are coupled: ngmix lays the tile + VIGNET's -1e30 neighbour markers over each epoch stamp pixel for pixel, + so they must be the same size. The stamp truncates the wings of large + galaxies. No rationale for 51 px is recorded. Values: default_noimaflags.param#VIGNET = 51; default.param#VIGNET = 51; + READ_EXT_SEXCAT_RUNNER.VIGNET_SIZE = 51; VIGNETMAKER_RUNNER_RUN_1.STAMP_SIZE = 51; VIGNETMAKER_RUNNER_RUN_2.STAMP_SIZE = 51; PSF_SIZE = 51. @@ -139,20 +145,22 @@ decisions: larger_adaptive: label: Larger or size-adaptive stamps description: >- - Not implemented; the vignet, stamp and PSF sizes must change - together. + Per-object sizes are not implemented. A larger fixed size is a + config change, provided the tile VIGNET and the multi-epoch stamp + change together. photometric_zeropoint: label: Magnitude zero-point convention rationale: >- - Tiles use a fixed zero-point of 30 (ZP_FROM_HEADER=False), repeated in - ngmix's MAG_ZP; exposures read the per-image header PHOTZP. The fixed - value assumes the MegaPipe stacks are calibrated to 30; nothing in the - repo checks it, but on a sampled exposure (2114045p) FSCALE equals - 10^(-0.4 (PHOTZP - 30)) to 0.02%. Exposure epochs are rescaled by FSCALE, which must agree - with header PHOTZP as well as tile MAG_ZEROPOINT and ngmix MAG_ZP. - Magnitude cuts (the star-selection window, downstream galaxy cuts) - inherit their stage's convention. + Exposures read the per-image header PHOTZP, so the PSF-star MAG_AUTO + window is on each exposure's own scale. Epochs are multiplied by header + FSCALE before the joint fit, which puts them on zero-point 30 only if + FSCALE = 10^(-0.4 (PHOTZP - 30)). Nothing in the repo checks this, but it + held to 0.02% on a sampled exposure (2114045p). ngmix magnitudes use + MAG_ZP = 30. In committed data the catalogue's tile MAG_AUTO is the + UNIONS catalogue's own; the fixed tile zero-point of 30 + (ZP_FROM_HEADER=False) applies only under tile_detection = sextractor, + and assumes the MegaPipe stacks are calibrated to 30. Values: MAG_ZEROPOINT = 30.0; config_tile_Sx.ini#SEXTRACTOR_RUNNER.ZP_FROM_HEADER = False; @@ -166,8 +174,9 @@ decisions: header_everywhere: label: Per-image header zero-points on tiles too description: >- - ZP_FROM_HEADER=True on tiles; changes magnitudes only where a tile's - header zero-point differs from 30. + ZP_FROM_HEADER=True for tile SExtractor (tile_detection = + sextractor only); changes tile magnitudes only where a tile's header + zero-point differs from 30. ngmix MAG_ZP stays 30. prior_insights: des_psf_blacklist: @@ -2009,16 +2018,21 @@ analyses: tile_overlap_handling: label: Tile-overlap duplicates neither removed nor flagged rationale: >- - Adjacent tiles overlap, and objects in the overlap are measured in - both. [HARDCODED] make_cat attaches only TILE_ID; with no - unique-object rule or overlap flag, duplicates are left to - downstream selection. NUMBER_LIST selects input tiles; it does not - carve disjoint sky regions, and no config key makes these catalogues - unique or flags their overlap rows. + Adjacent tiles overlap, and the UNIONS per-tile catalogue spans the + whole tile image, so an object in an overlap is measured once in + each tile. [HARDCODED] make_cat keeps every row and adds TILE_ID and + TILE_UNIQUE_ID (tile_id * 10**6 + NUMBER). The second is unique per + tile row, so an overlap object gets a different ID in each tile. No + config key deduplicates or flags overlap rows. sp_validation removes + the duplicates by keeping each object only inside its own tile's + nominal box (classification_galaxy_overlap_ra_dec, from TILE_ID and + XWIN_WORLD/YWIN_WORLD). default: no_dedup_in_pipeline options: no_dedup_in_pipeline: - label: Ship duplicates; TILE_ID is the only handle + label: >- + Ship overlap duplicates; sp_validation cuts each tile to its + nominal box overlap_flagging: label: Flag overlap objects in the catalogue description: Not implemented. @@ -2026,8 +2040,10 @@ analyses: label: Keep each object only in its nearest tile excluded: true excluded_reason: >- - Requires cross-tile coordination at assembly time, which the - per-tile DAG deliberately avoids. + Not needed in the pipeline: the cut needs only each object's + TILE_ID and position, and sp_validation applies it downstream. + Leaving it out keeps every UNIONS catalogue row in the shape + catalogue. failure_sentinels: label: Objects without shape measurements kept, with sentinel values rationale: >- diff --git a/src/shapepipe/modules/read_ext_sexcat_package/read_ext_sexcat.py b/src/shapepipe/modules/read_ext_sexcat_package/read_ext_sexcat.py index ff4f58cc8..de9af6511 100644 --- a/src/shapepipe/modules/read_ext_sexcat_package/read_ext_sexcat.py +++ b/src/shapepipe/modules/read_ext_sexcat_package/read_ext_sexcat.py @@ -63,7 +63,10 @@ def _build_ldac_imhead(img_header): def _centre_pixels(x_pos, y_pos): - """0-based (column, row) of the pixel holding each 1-based position.""" + """0-based (column, row) of the pixel holding each 1-based position. + + @sc [decision:preparation.stamp_positioning_and_padding] + """ col = np.rint(np.asarray(x_pos, dtype=float)).astype(np.int64) - 1 row = np.rint(np.asarray(y_pos, dtype=float)).astype(np.int64) - 1 return col, row diff --git a/workflow/config/cfis/config_tile_Uc.ini b/workflow/config/cfis/config_tile_Uc.ini index 861abb287..906ddef02 100644 --- a/workflow/config/cfis/config_tile_Uc.ini +++ b/workflow/config/cfis/config_tile_Uc.ini @@ -75,6 +75,7 @@ SUFFIX = sexcat # Side length of the square postage stamp (vignet) extracted from the tile # image, in pixels (must be odd). Default: 51 +# @sc [decision:postage_stamp_size] VIGNET_SIZE = 51 # The third input is the catalogue's r-band segmentation map: set neighbours' diff --git a/workflow/config/cfis/final_cat.param b/workflow/config/cfis/final_cat.param index cc5feaabf..88a831b95 100644 --- a/workflow/config/cfis/final_cat.param +++ b/workflow/config/cfis/final_cat.param @@ -3,8 +3,8 @@ XWIN_WORLD YWIN_WORLD -# tile ID, for plot of tile-dependent additive bias. -# Can maybe be removed. +# Tile of origin: sp_validation's overlap cut keeps each object only inside +# its own tile's box (catalogue_assembly.tile_overlap_handling). # @sc [decision:catalogue_assembly.tile_overlap_handling] TILE_ID diff --git a/workflow/scripts/merge_final_cat.py b/workflow/scripts/merge_final_cat.py index f4be98e06..afa05373a 100644 --- a/workflow/scripts/merge_final_cat.py +++ b/workflow/scripts/merge_final_cat.py @@ -82,14 +82,13 @@ hard error here, not a skip — under the DAG it cannot happen, since every one of them is a declared input of this job. -@sc [label:selection] never-fit-rows-pass-through +@sc [decision:catalogue_assembly.failure_sentinels,label:selection] never-fit-rows-pass-through Every row of every tile catalogue reaches the merged file, unchanged, including objects ngmix never fit. Those carry `NGMIX_N_EPOCH == 0` with -sentinel values (`NGMIX_MCAL_FLAGS == 0`, ellipticities `-10`, `T == 0`), so -`NGMIX_MCAL_FLAGS == 0` is not a validity cut: consumers select fitted objects -with `NGMIX_N_EPOCH > 0`. The merge neither fills these rows nor drops them; -that selection belongs to the consumer. Enforced by -tests/unit/test_final_cat_merge_invariants.py. +sentinel values (ellipticities `-10`, `T == 0`), `NGMIX_MCAL_FLAGS` nonzero +(`LM_FUNC_NOTFINITE`) and `NGMIX_MCAL_TYPES_FAIL == 5`, so the consumer's +`NGMIX_MCAL_FLAGS == 0` cut rejects them. The merge neither fills these rows +nor drops them. Enforced by tests/unit/test_final_cat_merge_invariants.py. """ import argparse From ac1a696f19a6f0f0c30c6ebe0fadb40506e77f1a Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 30 Sep 2026 06:53:59 +0200 Subject: [PATCH 7/7] Record that the tile SExtractor config reproduces DR6 detection (#929) Co-Authored-By: Claude Opus 5.5 --- astra.yaml | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/astra.yaml b/astra.yaml index d7f1fb5cc..e566a1253 100644 --- a/astra.yaml +++ b/astra.yaml @@ -393,7 +393,12 @@ analyses: Tile SExtractor (tile_detection = sextractor, i.e. image simulations) uses the MegaPipe tile-catalogue threshold, minimum area and filter, so that its galaxy sample reproduces the UNIONS - catalogue committed data adopts; the 7x7 Gaussian of FWHM 3 px is + catalogue committed data adopts. Run on the DR6 image of tile + 186.307, it reproduces the DR6 catalogue to 99.89% completeness + and 100% purity with identical segmentation pixels; the 45 misses + are faint deblended children of bright parents (#929). This + validates the configuration on data pixels, not detection on + simulated images. The 7x7 Gaussian of FWHM 3 px is near the CFIS average seeing of 0.65 arcsec (about 3.5 px at 0.187 arcsec/px). Exposures only feed star selection and keep stock values with the 3x3 FWHM 2 px kernel. The tiles differ from Guinot+22