Skip to content

fix(storage): make cassandra index readiness probe actually wait - #810

Merged
lakhansamani merged 1 commit into
mainfrom
fix/cassandra-index-readiness-probe
Oct 14, 2026
Merged

lakhansamani merged 1 commit into
mainfrom
fix/cassandra-index-readiness-probe

Conversation

@lakhansamani

Copy link
Copy Markdown
Contributor

Closes #803.

waitForCassandraSecondaryIndex existed to block startup until a ScyllaDB secondary index became queryable. Its probe was:

probe := fmt.Sprintf("SELECT id FROM %s.%s WHERE %s='' LIMIT 1 ALLOW FILTERING", keyspace, table, column)

ALLOW FILTERING is exactly the hint that lets Cassandra answer a query on a non-indexed column. The probe therefore succeeded whether or not the index existed, returned on its first attempt, and never waited for anything. All 13 call sites kept the startup race the helper was written to close — an intermittent index-unavailable error on the first query against a freshly provisioned keyspace.

waitForCassandraIndexes, added in #802 for the audit-log columns, already omitted the hint and carried a comment explaining why. This brings the other 13 in line.

Changes

  • Drop the hint, so the probe fails until the index is ready. One line, but it makes 14 waits genuinely block for the first time.
  • Consolidate all 17 probes into one call at the end of NewProvider, against a single 90s budget split per probe. This is the part that matters: left as 13 sequential 30s waits plus the audit-log wait, making them genuinely block turns a no-op into up to 13 × 30s = 6.5 min of serial startup delay. One shared budget bounds worst-case startup regardless of how many indexes are added later.
  • Deferring to the end costs nothing: nothing inside NewProvider queries an indexed column, and the boot-time reserved-client seed runs in cmd/root.go:592, after the provider is constructed. Indexes created earliest also get the longest to settle before being probed.
  • Hoist the list to cassandraIndexProbes and extract indexProbeQuery, so both are testable without a live cluster.

All 17 probed columns were confirmed to have a matching CREATE INDEX, so no probe can hang on a condition that is never satisfiable. Columns indexed as bigint (audit_log.created_at, session_token.expires_at, mfa_session.expires_at) are deliberately absent — the probe restricts on '', which only type-checks against a text column.

What the probe does and does not detect

It fails until the index schema propagates. For a fresh keyspace that is the same moment the (empty) index is usable, which is the race this closes.

It is not a view-build gate. An index added to a table that already holds rows answers the probe while its backfill is still running. Backfill readiness is system_distributed.view_build_status, deliberately not consulted here — a cold start must not block on a backfill it does not need. This is documented on the function so the fix is not mistaken for a guarantee that a filtered audit query is authoritative mid-backfill (see the #802 upgrade notes in CHANGELOG.md).

Testing

  • TestIndexProbeQueryOmitsAllowFiltering — the regression guard. Confirmed it fails with the hint restored and passes without it, so it genuinely pins the bug.
  • TestIndexProbesCoverAuditLogFilters — keeps the wait list in step with the columns ListAuditLogs filters on, and rejects duplicates (a duplicate would consume two shares of the budget).
  • make test-scylladb against a cold ScyllaDB container: green, 25/25 storage subtest groups, 0 failures. internal/storage ran in 3.59s, so the now-blocking probes add no measurable startup cost on a fresh keyspace.
  • go build ./..., go vet ./... clean.

make lint-go reports 4 pre-existing issues (1 gofmt in oauth_sso_verify_test.go, 3 reflect.Ptr govet inlining hints) that reproduce identically on main — local golangci-lint version drift, untouched here.

The probe carried ALLOW FILTERING, which is precisely the hint that
lets Cassandra answer a query on a non-indexed column. It therefore
succeeded whether or not the index existed, returned on its first
attempt, and never waited for anything — leaving all 17 call sites
exposed to the startup race the helper was written to close. Symptom
is an intermittent index-unavailable error on the first query against
a freshly provisioned keyspace.

- drop the hint, so the probe fails until the index is ready
- consolidate the 13 per-table waits and the audit-log wait into one
  call at the end of NewProvider against a single 90s budget split
  per probe. Without this, making the waits genuinely block turns a
  no-op into up to 13 x 30s = 6.5 min of serial startup delay.
  Nothing inside NewProvider queries an indexed column, so deferring
  is free: the reserved-client seed runs in cmd/root.go afterwards.
- hoist the probe list to cassandraIndexProbes and extract
  indexProbeQuery, so both are testable without a live cluster

The probe detects index SCHEMA propagation, not view-build
completion. For a fresh keyspace those coincide. An index added to a
populated table answers the probe while its backfill still runs —
view_build_status is deliberately not consulted, since a cold start
must not block on a backfill it does not need.

Closes #803
@lakhansamani
lakhansamani force-pushed the fix/cassandra-index-readiness-probe branch from 4aa5f17 to e1f46c4 Compare October 14, 2026 08:58
@lakhansamani
lakhansamani merged commit 904f459 into main Oct 14, 2026
4 checks passed
lakhansamani added a commit that referenced this pull request Oct 14, 2026
Sixteen PRs merged since the 2.4.1 tag; [Unreleased] documented three
of them and none of the five advisory fixes. A security release whose
notes omit its security fixes gives nobody a reason to upgrade, which
is the operator-facing half of the failure SECURITY.md triage exists
to prevent.

Renames [Unreleased] to 2.4.2 and documents all of it:

- Security: the six advisories closed (#814, #799, #798, #796 x2,
  #815) and the three reachable dependency advisories (#797, #807,
  #811), plus the Go 1.27 builder (#800).
- Fixed: the audit-filter parity bugs (#802) and the Cassandra index
  probe that never waited (#810).
- Changed: direct-gRPC x-authorizer-url now yields to --url (#814),
  and the dropped Cassandra audit_log(created_at) index (#812).
- Added: the audit plumbing (#806), stated as writing no new records
  and changing no behaviour, plus two deployment entries.
- Upgrade notes: eight items, including the three that were missing
  entirely — that ANY two audit filters now scan on Cassandra (not
  only the timestamp range, which the previous note implied), the
  longer first start on ScyllaDB now that probes genuinely block, and
  the manual DROP INDEX for the removed index.

Section order leads with Security, matching 2.4.1 — the file orders by
what matters for the release at hand rather than a fixed sequence.

Deliberately omitted: #791 (scorecard action bump) and #809 (the
previous changelog edit). Neither is user-facing; Keep a Changelog
asks for notable changes, not a commit log.

The ScyllaDB view-build note is scoped to ScyllaDB: global secondary
indexes are materialized views there, while Apache Cassandra builds
local 2i with no view-build step and no nodetool viewbuildstatus.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

waitForCassandraSecondaryIndex never waits: ALLOW FILTERING defeats its own probe

1 participant