Repository navigation
fix(storage): make cassandra index readiness probe actually wait - #810
Merged
Merged
Conversation
This was referenced Oct 14, 2026
The probe carried ALLOW FILTERING, which is precisely the hint that lets Cassandra answer a query on a non-indexed column. It therefore succeeded whether or not the index existed, returned on its first attempt, and never waited for anything — leaving all 17 call sites exposed to the startup race the helper was written to close. Symptom is an intermittent index-unavailable error on the first query against a freshly provisioned keyspace. - drop the hint, so the probe fails until the index is ready - consolidate the 13 per-table waits and the audit-log wait into one call at the end of NewProvider against a single 90s budget split per probe. Without this, making the waits genuinely block turns a no-op into up to 13 x 30s = 6.5 min of serial startup delay. Nothing inside NewProvider queries an indexed column, so deferring is free: the reserved-client seed runs in cmd/root.go afterwards. - hoist the probe list to cassandraIndexProbes and extract indexProbeQuery, so both are testable without a live cluster The probe detects index SCHEMA propagation, not view-build completion. For a fresh keyspace those coincide. An index added to a populated table answers the probe while its backfill still runs — view_build_status is deliberately not consulted, since a cold start must not block on a backfill it does not need. Closes #803
lakhansamani
force-pushed
the
fix/cassandra-index-readiness-probe
branch
from
October 14, 2026 08:58
4aa5f17 to
e1f46c4
Compare
This was referenced Oct 14, 2026
lakhansamani
added a commit
that referenced
this pull request
Oct 14, 2026
Sixteen PRs merged since the 2.4.1 tag; [Unreleased] documented three of them and none of the five advisory fixes. A security release whose notes omit its security fixes gives nobody a reason to upgrade, which is the operator-facing half of the failure SECURITY.md triage exists to prevent. Renames [Unreleased] to 2.4.2 and documents all of it: - Security: the six advisories closed (#814, #799, #798, #796 x2, #815) and the three reachable dependency advisories (#797, #807, #811), plus the Go 1.27 builder (#800). - Fixed: the audit-filter parity bugs (#802) and the Cassandra index probe that never waited (#810). - Changed: direct-gRPC x-authorizer-url now yields to --url (#814), and the dropped Cassandra audit_log(created_at) index (#812). - Added: the audit plumbing (#806), stated as writing no new records and changing no behaviour, plus two deployment entries. - Upgrade notes: eight items, including the three that were missing entirely — that ANY two audit filters now scan on Cassandra (not only the timestamp range, which the previous note implied), the longer first start on ScyllaDB now that probes genuinely block, and the manual DROP INDEX for the removed index. Section order leads with Security, matching 2.4.1 — the file orders by what matters for the release at hand rather than a fixed sequence. Deliberately omitted: #791 (scorecard action bump) and #809 (the previous changelog edit). Neither is user-facing; Keep a Changelog asks for notable changes, not a commit log. The ScyllaDB view-build note is scoped to ScyllaDB: global secondary indexes are materialized views there, while Apache Cassandra builds local 2i with no view-build step and no nodetool viewbuildstatus.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #803.
waitForCassandraSecondaryIndexexisted to block startup until a ScyllaDB secondary index became queryable. Its probe was:ALLOW FILTERINGis exactly the hint that lets Cassandra answer a query on a non-indexed column. The probe therefore succeeded whether or not the index existed, returned on its first attempt, and never waited for anything. All 13 call sites kept the startup race the helper was written to close — an intermittent index-unavailable error on the first query against a freshly provisioned keyspace.waitForCassandraIndexes, added in #802 for the audit-log columns, already omitted the hint and carried a comment explaining why. This brings the other 13 in line.Changes
NewProvider, against a single 90s budget split per probe. This is the part that matters: left as 13 sequential 30s waits plus the audit-log wait, making them genuinely block turns a no-op into up to 13 × 30s = 6.5 min of serial startup delay. One shared budget bounds worst-case startup regardless of how many indexes are added later.NewProviderqueries an indexed column, and the boot-time reserved-client seed runs incmd/root.go:592, after the provider is constructed. Indexes created earliest also get the longest to settle before being probed.cassandraIndexProbesand extractindexProbeQuery, so both are testable without a live cluster.All 17 probed columns were confirmed to have a matching
CREATE INDEX, so no probe can hang on a condition that is never satisfiable. Columns indexed asbigint(audit_log.created_at,session_token.expires_at,mfa_session.expires_at) are deliberately absent — the probe restricts on'', which only type-checks against a text column.What the probe does and does not detect
It fails until the index schema propagates. For a fresh keyspace that is the same moment the (empty) index is usable, which is the race this closes.
It is not a view-build gate. An index added to a table that already holds rows answers the probe while its backfill is still running. Backfill readiness is
system_distributed.view_build_status, deliberately not consulted here — a cold start must not block on a backfill it does not need. This is documented on the function so the fix is not mistaken for a guarantee that a filtered audit query is authoritative mid-backfill (see the #802 upgrade notes inCHANGELOG.md).Testing
TestIndexProbeQueryOmitsAllowFiltering— the regression guard. Confirmed it fails with the hint restored and passes without it, so it genuinely pins the bug.TestIndexProbesCoverAuditLogFilters— keeps the wait list in step with the columnsListAuditLogsfilters on, and rejects duplicates (a duplicate would consume two shares of the budget).make test-scylladbagainst a cold ScyllaDB container: green, 25/25 storage subtest groups, 0 failures.internal/storageran in 3.59s, so the now-blocking probes add no measurable startup cost on a fresh keyspace.go build ./...,go vet ./...clean.make lint-goreports 4 pre-existing issues (1 gofmt inoauth_sso_verify_test.go, 3reflect.Ptrgovet inlining hints) that reproduce identically onmain— local golangci-lint version drift, untouched here.