Questions and uncertainties about Stephen Gwyn's official DR6 SExtractor run

[cailmdaley]

Goal: the official DR6 tile catalogues carry everything the shape measurement reads, so ShapePipe can use Stephen Gwyn's products directly, including his segmentation maps. We'd run his exact build on the image sims. Deblending assigns pixels at random (see 1), so reproducing DR6 means exact agreement for unblended objects and agreement in distribution for deblended ones; that's enough for calibration, because the same binary on sims gives the same statistics.

Not true yet, so for now we run SExtractor ourselves (#933) with MegaPipe's public parameters (#896), on data and sims alike. Doing so (a) keeps data detection consistent with the image sims, and (b) gives us the windowed centroids DR6 doesn't have. DR6 still defines which objects exist, their IDs and their photometry, through #933's positional match.

What stands between us and the goal:

1. Reproducing DR6's quantities: we reproduce detection exactly, and deblended photometry cannot be reproduced. #933 pairs our detections with DR6 to 4e-5 px. MAG_AUTO differs by more than 0.01 mag for 9% of paired objects on 216.292, 3.5% on 381.259, 2% on 092.301 and 0.2% on 186.307.

The cause is SExtractor's deblender, not a configuration setting. gatherup() in src/refine.c (l. 330) gives each parent pixel no child claims to a child chosen by rand(), and SExtractor never seeds it. One random stream runs through the whole image. Once any object draws a different number of times than in Gwyn's run, every later deblended object gets different pixels. Positions are unchanged, but ISOAREA, A, FLUX_RADIUS and MAG_AUTO move, typically by 0.8–1.25× in flux.

Evidence (memtest, 2 Oct):

• On each tile, deblended objects match DR6 exactly up to one point in processing order. After it, about 30–60% differ, against about 2% of unblended objects.
• Re-seeding (srand(2)) on 186.307, the tile that otherwise matches, raises its mismatch from 0.03% to 7% (20 < m < 24.5).
• Varying MEMORY_OBJSTACK/BUFSIZE/PIXSTACK and DEBLEND_NTHRESH, or using DR6's exact .param, changes nothing. SExtractor 2.25.0 through 2.28.2 (gcc builds) are bit-identical. An Intel-compiled build desyncs differently.
• Inference: Gwyn's build differs from ours by a floating-point detail that changes when the stream desyncs.

So the bar for "DR6 reproduces" is exact agreement for unblended objects, plus agreement in distribution for deblended ones. Unblended objects nearly pass already. 092.301 has a residual of about 0.7% of unblended objects that differ throughout; zero-weight interpolation is the untested suspect. The one remaining question for Stephen is his build: SExtractor version, compiler and platform. Data and sims run the same binary, so for parity this randomness is per-object noise, not a bias.

2. Columns DR6 would need to ship.

• Windowed centroids and flags: XWIN_IMAGE, YWIN_IMAGE, FLAGS_WIN. DR6 has none. The old converter substituted the isophotal barycentre, which is 0.16 px off at the median and 1.6 px at p99.
• MAG_WIN, MAGERR_WIN, SNR_WIN, FLUX_AUTO, FLUXERR_AUTO, FLUX_APER, FLUXERR_APER, FWHM_IMAGE, FWHM_WORLD. Under Detect tiles with SExtractor on data too; take NUMBER from the UNIONS catalogue #933 these come from our own run.

3. Segmentation maps. UberSeg (#925, being ported onto #933) uses our own run's segmentation on data, as on sims, because blending calibration needs the same deblending on both sides. Deblended pixels are assigned at random (see 1), so DR6's maps and ours disagree at the borders of blends however well the configurations match.

Claude Opus 5.5 on behalf of Cail