Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
264 changes: 157 additions & 107 deletions astra.yaml

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion src/shapepipe/modules/ngmix_package/ngmix.py
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ def get_type_flags(fit):
Fit flags of one metacal type, reading absence of evidence of success
as failure.

@sc [label:convention] mcal-flags-zero-means-measured
@sc [decision:catalogue_assembly.failure_sentinels,label:convention] mcal-flags-zero-means-measured
A flag of 0 means the fit ran, reported success and returned a finite
shear; no default or fallback may produce 0. FLAGS_<SHEAR>, MCAL_FLAGS
(OR) and MCAL_TYPES_FAIL (count) all derive from this function, and
Expand Down
2 changes: 1 addition & 1 deletion src/shapepipe/modules/ngmix_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ def ngmix_runner(
):
"""Define The Ngmix Runner.

@sc [decision:shape_measurement.blend_handling,decision:shape_measurement.centroid_source,decision:shape_measurement.defect_fill,decision:shape_measurement.metacal_scheme]
@sc [decision:shape_measurement.blend_handling,decision:shape_measurement.centroid_source,decision:shape_measurement.defect_fill,decision:shape_measurement.metacal_scheme,decision:shape_measurement.galaxy_pixel_weights]

@sc [label:operations] empty-tile-product
A tile whose PSF or galaxy vignette store is entirely empty never
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,10 @@ def _build_ldac_imhead(img_header):


def _centre_pixels(x_pos, y_pos):
"""0-based (column, row) of the pixel holding each 1-based position."""
"""0-based (column, row) of the pixel holding each 1-based position.

@sc [decision:preparation.stamp_positioning_and_padding]
"""
col = np.rint(np.asarray(x_pos, dtype=float)).astype(np.int64) - 1
row = np.rint(np.asarray(y_pos, dtype=float)).astype(np.int64) - 1
return col, row
Expand Down
3 changes: 2 additions & 1 deletion src/shapepipe/modules/sextractor_package/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,8 @@
WORLD_POSITION : list, optional
List of world coordinates to use to match objects
CCD_SIZE : list, optional
Size of a CCD in pixels ``[nx, ny]``
Accepted CCD pixel bounds ``[xmin, xmax, ymin, ymax]``, 0-based and
compared strictly

"""

Expand Down
2 changes: 2 additions & 0 deletions src/shapepipe/modules/sextractor_package/sextractor_script.py
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,8 @@ def make_post_process(cat_path, f_wcs_path, pos_params, ccd_size, w_log=None):
objects near it, never the tile. CCD_N is the 0-based index that split_exp
gives the CCD file and its header entry.

@sc [decision:preparation.epoch_provenance_from_tile_history]

The columns will be:

- ``NUMBER``: same as SExtractor NUMBER
Expand Down
6 changes: 3 additions & 3 deletions src/shapepipe/modules/split_exp_package/split_exp.py
Original file line number Diff line number Diff line change
Expand Up @@ -79,9 +79,9 @@ def create_hdus(self, exp_path, output_suffix, transf_int, save_header):

@sc [decision:preparation.astrometric_solution_source,label:convention] wcs-from-delivered-header
The stored WCS is ``WCS(header)`` of the delivered CCD header,
unmodified. Every downstream world-to-pixel transform (epoch
membership, stamp placement, position seeding) uses it, so a refit or
header edit here moves every stamp centre and epoch assignment.
unmodified. Epoch membership, multi-epoch stamp placement, PSF
interpolation positions and the ngmix Jacobian all use it; a refit or
header edit here shifts them.

Parameters
----------
Expand Down
10 changes: 5 additions & 5 deletions tests/module/test_ngmix.py
Original file line number Diff line number Diff line change
Expand Up @@ -128,11 +128,11 @@ def _metacal_noshear_g(seed):
def test_metacal_is_reproducible_with_fixed_seed():
"""Same seed -> identical metacal shear.

The module seeds ``self._rng = RandomState(seed)`` per tile precisely so a
rerun reproduces. This guards the noise-image and masked-pixel draws in
``prepare_ngmix_weights`` against silently falling back to the unseeded
global ``numpy.random`` state, which would make shear estimates
irreproducible from one run to the next.
Production seeds a fresh RandomState per object from its position
(position_seed), so a rerun reproduces. This guards the noise-image and
masked-pixel draws in ``prepare_ngmix_weights`` against silently falling
back to the unseeded global ``numpy.random`` state, which would make
shear estimates irreproducible from one run to the next.
"""
npt.assert_array_equal(_metacal_noshear_g(42), _metacal_noshear_g(42))

Expand Down
8 changes: 5 additions & 3 deletions workflow/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,9 @@ uv pip install 'snakemake>=9,<10' 'snakemake-executor-plugin-slurm>=2.7,<3'
# Write a run config (see Run configuration below) that sets at least `run:`,
# the campaign's name; workflow/config.yaml's input_types: and machines: tables supply the rest.

# `psf_model` is `psfex` or `mccd`. psfex is exercised by smk-g4 through smk-g6; mccd has run the full chain on
# an image-sim star tile (one focal-plane model per exposure, ~1.5 CPU-hours each).
# `psf_model` is `psfex` for data (`fake` for image sims). `mccd` is refused
# until `persist_exp.py` and `merge_star_cat.py` read MCCD products. PSFEx is
# exercised by smk-g4 through smk-g6.
# `tile_detection` is `unions_catalogue` (the input_types default for data:
# the UNIONS per-tile catalogue at `inputs.catalogues` is fetched and
# converted in place, keeping its NUMBER, and its segmentation map sets
Expand Down Expand Up @@ -70,7 +71,8 @@ SExtractor (for the background maps the vignets read), and `tile_vignets` runs
`fake_interp_runner`, which writes the `galaxy_psf` product from `psf_dict`.
With no PSF model there is nothing to persist per exposure, so `exp_persist` and
`star_cat_merge` do not run and `clean_exposure` does not wait on them.
Simulations that contain stars can run `psfex` or `mccd` exactly as the data do.
Simulations that contain stars can run `psfex` as the data do. `mccd` is
refused until `persist_exp.py` and `merge_star_cat.py` read MCCD products.

One campaign per shear branch, each with its own run config:

Expand Down
1 change: 1 addition & 0 deletions workflow/Snakefile
Original file line number Diff line number Diff line change
Expand Up @@ -602,6 +602,7 @@ PERSIST_EXP = list(config.get("persist_exp") or [])
PERSISTS_PSF = PSF_MODEL != "fake"


# @sc [decision:star_selection_psf.psf_modelling_software]
def refuse_unpersistable_psf(psf_model):
"""Refuse a PSF model whose products the persistence path cannot read.

Expand Down
1 change: 1 addition & 0 deletions workflow/config/cfis/config_exp_psfex.ini
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ NUMBERING_SCHEME = -0000000-0
EXEC_PATH = source-extractor

# SExtractor configuration files
# @sc [decision:detection.detection_threshold_policy]
DOT_SEX_FILE = $SP_CONFIG/default_exp.sex
DOT_PARAM_FILE = $SP_CONFIG//default.param
# @sc [decision:detection.detection_threshold_policy,label:effective-filter] exposure-dot-conv-overrides-filter
Expand Down
3 changes: 2 additions & 1 deletion workflow/config/cfis/config_tile_Sx.ini
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ NUMBERING_SCHEME = -000-000
EXEC_PATH = source-extractor

# SExtractor configuration files
# @sc [decision:detection.detection_threshold_policy]
DOT_SEX_FILE = $SP_CONFIG/default_tile.sex
# @sc [decision:detection.detection_source_mode,label:override] tile-parameter-file-overrides-column-list
# DOT_PARAM_FILE overrides PARAMETERS_NAME; keep it pointed at default_noimaflags.param so tile detections do not request IMAFLAGS_ISO.
Expand Down Expand Up @@ -124,6 +125,6 @@ MAKE_POST_PROCESS = True
# @sc [decision:preparation.object_position_columns]
WORLD_POSITION = XWIN_WORLD,YWIN_WORLD

# Number of pixels in x,y of a CCD. Format: Nx,Ny
# Accepted CCD pixel bounds xmin,xmax,ymin,ymax (0-based, strict)
# @sc [decision:detection.epoch_membership_ccd_bounds]
CCD_SIZE = 33,2080,1,4612
3 changes: 2 additions & 1 deletion workflow/config/cfis/config_tile_Uc.ini
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ SUFFIX = sexcat

# Side length of the square postage stamp (vignet) extracted from the tile
# image, in pixels (must be odd). Default: 51
# @sc [decision:postage_stamp_size]
VIGNET_SIZE = 51

# The third input is the catalogue's r-band segmentation map: set neighbours'
Expand All @@ -91,6 +92,6 @@ MAKE_POST_PROCESS = True
# World coordinate keywords, SExtractor output. Format: KEY_X,KEY_Y
WORLD_POSITION = ALPHA_J2000,DELTA_J2000

# Number of pixels in x,y of a CCD. Format: Nx,Ny
# Accepted CCD pixel bounds xmin,xmax,ymin,ymax (0-based, strict)
# @sc [decision:detection.epoch_membership_ccd_bounds]
CCD_SIZE = 33,2080,1,4612
4 changes: 2 additions & 2 deletions workflow/config/cfis/final_cat.param
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
XWIN_WORLD
YWIN_WORLD

# tile ID, for plot of tile-dependent additive bias.
# Can maybe be removed.
# Tile of origin: sp_validation's overlap cut keeps each object only inside
# its own tile's box (catalogue_assembly.tile_overlap_handling).
# @sc [decision:catalogue_assembly.tile_overlap_handling]
TILE_ID

Expand Down
11 changes: 5 additions & 6 deletions workflow/scripts/merge_final_cat.py
Original file line number Diff line number Diff line change
Expand Up @@ -82,14 +82,13 @@
hard error here, not a skip — under the DAG it cannot happen, since every one of
them is a declared input of this job.

@sc [label:selection] never-fit-rows-pass-through
@sc [decision:catalogue_assembly.failure_sentinels,label:selection] never-fit-rows-pass-through
Every row of every tile catalogue reaches the merged file, unchanged,
including objects ngmix never fit. Those carry `NGMIX_N_EPOCH == 0` with
sentinel values (`NGMIX_MCAL_FLAGS == 0`, ellipticities `-10`, `T == 0`), so
`NGMIX_MCAL_FLAGS == 0` is not a validity cut: consumers select fitted objects
with `NGMIX_N_EPOCH > 0`. The merge neither fills these rows nor drops them;
that selection belongs to the consumer. Enforced by
tests/unit/test_final_cat_merge_invariants.py.
sentinel values (ellipticities `-10`, `T == 0`), `NGMIX_MCAL_FLAGS` nonzero
(`LM_FUNC_NOTFINITE`) and `NGMIX_MCAL_TYPES_FAIL == 5`, so the consumer's
`NGMIX_MCAL_FLAGS == 0` cut rejects them. The merge neither fills these rows
nor drops them. Enforced by tests/unit/test_final_cat_merge_invariants.py.
"""

import argparse
Expand Down
Loading