Skip to content

Edition guard misses encoders in private candidate lists (Sparse patch-index cascade offers fastlanes.delta) #440

Description

@dfa1

Bug

The writer's edition guard filters out-of-edition encodings only among the encoders VortexWriter itself lists (encodings + cascadeCodecs). VortexWriter.editionExcluded builds its set from those two lists alone.

Some encoders keep a private candidate list that never goes through those lists. SparseEncodingEncoder.INDEX_CASCADE_CANDIDATES, used to compress a sparse validity bitmap's patch indices, offers fastlanes.delta, which is an unstable-family encoding. Its id never lands in editionExcluded, so CascadingCompressor's per-candidate ctx.excluded() check lets it compete under the default core edition.

Impact

While plain (terminal) delta always lost that competition, the bug stayed latent. Once delta cascades its children (#410 / #439), it wins on regular patch indices, and a default-edition write fails the backstop guard:

VortexException: fastlanes.delta: outside the configured edition(s) [core2026.08.0];
it joins unstable2025.05.0 — enable that edition via WriteOptions.withEdition(...)

Reproduce

A nullable Utf8 column of 50k rows (8 categories, a null every 10th row), written with WriteOptions.cascading(3).withGlobalDict(false) and default editions. The periodic nulls make the validity sparse with perfectly regular patch indices. This was hit by FileSizeComparisonIntegrationTest (nullableLowCardinalityUtf8_javaVsJni, nullableLowCardinalityUtf8_multiChunk_globalDict_javaVsJni, globalDict_nullableMultiChunkUtf8_smallerThanPerChunkDict).

Fix

editionExcluded covers every EncodingId.WellKnown id outside the enabled editions, plus custom encoders the writer holds, so private candidate lists obey the same policy. This is safe by construction: anything newly excluded would have failed the backstop guard anyway. The regression test WriterEditionGuardTest#cascading_defaultEditions_privateCandidateListsHonorTheEdition fails without the fix. Fixed in #439.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions