Repository navigation
Clamp out-of-range alpha-scale radius in validate_config - #679
Merged
solidpixel merged 3 commits intoOct 21, 2026
Merged
Conversation
The a_scale_radius field is a public astcenc_config tuning field, and is
also set directly by the CLI '-a <radius>' option via atoi() with no
bounds check, so it can hold any unsigned value.
init_compute_averages() sizes the averaging pre-pass work memory as a
product of padded block extents that grow linearly with the radius:
work_memory_size = 2 * (32 + 2r + 1)^2 * (16 + 2r + 1)
This size_t multiplication overflows for large radii, and even where it
does not overflow the allocation grows without bound. A config with a
large radius (e.g. the value 0xFFFFFFFF resulting from atoi() of a
negative CLI argument, or a radius of 1000000) therefore causes
astcenc_compress_image() to fail with an uncaught std::bad_alloc, which
terminates the process rather than returning an error code.
Clamp the radius in validate_config() to the largest supported value,
following the existing pattern for the other tunable config fields.
Worst-case work memory after the clamp is around 100 MB for a 3D image,
and any practical alpha-scale use case uses single-digit radii.
solidpixel
self-requested a review
October 21, 2026 11:43
solidpixel
approved these changes
Oct 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The
a_scale_radiusfield is a publicastcenc_configtuning field, and the CLI also sets it directly from the-a <radius>option usingatoi()with no bounds check (astcenccli_toplevel.cpp line 741). It can therefore hold any unsigned value, including0xFFFFFFFFfor a negative argument such as-a -1.init_compute_averages()sizes the averaging pre-pass work memory as a product of padded block extents that grow linearly with the radius:This
size_tmultiplication overflows for large radii, and even where it does not overflow the requested allocation grows without bound.compute_averages()performs the allocation with a barenew vfloat4[...]insideastcenc_compress_image(), so a large radius causes an uncaughtstd::bad_alloc, which terminates the process instead of returning an error code like every other invalid input to the codec:The same happens for moderate values such as
-a 1000000, and the equivalent library-level config (a_scale_radius = 1000000) hits the same path.Changes
a_scale_radiusinvalidate_config()to a newTUNE_MAX_A_SCALE_RADIUSconstant (64), following the existing pattern used for the other tunable config fields. The work memory after the clamp is bounded to around 100 MB even for a 3D image, and practical alpha-scale use cases use single-digit radii (the test suite uses-a 1and-a 2).a_scale_radius = 0xFFFFFFFF, which previously died withstd::bad_alloc(or an ASAN allocation-size-too-big report) and now compresses normally.