Skip to content

Clamp out-of-range alpha-scale radius in validate_config - #679

Merged
solidpixel merged 3 commits into
ARM-software:mainfrom
17krishna8:clamp-alpha-scale-radius
Oct 21, 2026
Merged

solidpixel merged 3 commits into
ARM-software:mainfrom
17krishna8:clamp-alpha-scale-radius

Conversation

@17krishna8

Copy link
Copy Markdown
Contributor

Description

The a_scale_radius field is a public astcenc_config tuning field, and the CLI also sets it directly from the -a <radius> option using atoi() with no bounds check (astcenccli_toplevel.cpp line 741). It can therefore hold any unsigned value, including 0xFFFFFFFF for a negative argument such as -a -1.

init_compute_averages() sizes the averaging pre-pass work memory as a product of padded block extents that grow linearly with the radius:

size_t max_padsize_xy = max_blk_size_xy + kerneldim;   // kerneldim = 2r + 1
size_t max_padsize_z  = max_blk_size_z + kerneldim;
ag.work_memory_size = 2 * max_padsize_xy * max_padsize_xy * max_padsize_z;

This size_t multiplication overflows for large radii, and even where it does not overflow the requested allocation grows without bound. compute_averages() performs the allocation with a bare new vfloat4[...] inside astcenc_compress_image(), so a large radius causes an uncaught std::bad_alloc, which terminates the process instead of returning an error code like every other invalid input to the codec:

$ astcenc -cl tiny.png out.astc 6x6 -fastest -a -1
terminate called after throwing an instance of 'std::bad_alloc'
Aborted

The same happens for moderate values such as -a 1000000, and the equivalent library-level config (a_scale_radius = 1000000) hits the same path.

Changes

  • Clamp a_scale_radius in validate_config() to a new TUNE_MAX_A_SCALE_RADIUS constant (64), following the existing pattern used for the other tunable config fields. The work memory after the clamp is bounded to around 100 MB even for a 3D image, and practical alpha-scale use cases use single-digit radii (the test suite uses -a 1 and -a 2).
  • Add a unit test compressing an image with a_scale_radius = 0xFFFFFFFF, which previously died with std::bad_alloc (or an ASAN allocation-size-too-big report) and now compresses normally.

17krishna8 and others added 3 commits October 20, 2026 15:20
The a_scale_radius field is a public astcenc_config tuning field, and is
also set directly by the CLI '-a <radius>' option via atoi() with no
bounds check, so it can hold any unsigned value.

init_compute_averages() sizes the averaging pre-pass work memory as a
product of padded block extents that grow linearly with the radius:

    work_memory_size = 2 * (32 + 2r + 1)^2 * (16 + 2r + 1)

This size_t multiplication overflows for large radii, and even where it
does not overflow the allocation grows without bound. A config with a
large radius (e.g. the value 0xFFFFFFFF resulting from atoi() of a
negative CLI argument, or a radius of 1000000) therefore causes
astcenc_compress_image() to fail with an uncaught std::bad_alloc, which
terminates the process rather than returning an error code.

Clamp the radius in validate_config() to the largest supported value,
following the existing pattern for the other tunable config fields.
Worst-case work memory after the clamp is around 100 MB for a 3D image,
and any practical alpha-scale use case uses single-digit radii.
@solidpixel
solidpixel self-requested a review October 21, 2026 11:43
@solidpixel
solidpixel merged commit 1b181e7 into ARM-software:main Oct 21, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants