Repository navigation
fix: Don't poll profile deletion while clusters are paused - #2023
Merged
gianlucam76 merged 2 commits intoOct 6, 2026
Merged
Conversation
The ClusterSummary controller already reports a failed deployment at info
level ("failed to deploy"). The Helm, Kustomize and PolicyRefs handlers
repeated it, so each expected waiting state (request is queued, feature is
still being provisioned) was logged twice more at error level.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A ClusterSummary for a paused cluster is never removed: the ClusterSummary controller returns without doing anything until the cluster is unpaused. The ClusterProfile/Profile being deleted has no way to know that. It does not watch ClusterSummaries, so it polls: `cleanClusterSummaries` returns `clusterSummaries still present` and the profile is requeued every 10 seconds, for as long as the cluster stays paused. On a management cluster with several paused clusters this produced about 2,100 `failed to clean ClusterSummaries` errors in 35 minutes (ten profiles, one reconcile and one error every 10 seconds each), all with the same outcome. When a ClusterProfile/Profile is being deleted and every ClusterSummary still present is for a paused cluster, it now stops polling: - `reconcileDeleteCommon` returns a new `WaitingForPausedClustersError` (listing the clusters) instead of the generic error. Both the ClusterProfile and the Profile controllers log it once at info level and requeue after 10 minutes (`pausedClustersRequeueAfter`). The 10 minutes is only a safety net. - Before stopping, both controllers register the profile in `ClusterMap` for the paused clusters (`trackClusters`). A profile that is being deleted is only in the maps if it went through the normal reconcile, so after a restart an unpause would otherwise never reconcile it. Unpausing a cluster now reconciles the profile, which then resumes the usual 10 second polling until the ClusterSummaries are gone. - If at least one remaining ClusterSummary is for a cluster that is not paused, or whose cluster is not found, nothing changes: the profile keeps polling as before. The extra work in that case is one cached list plus, at most, a cached get per ClusterSummary, with an early exit on the first cluster that is not paused. - `cleanClusterSummaries` now returns the `errClusterSummariesStillPresent` sentinel with the same message as before. - `allClusterSummariesGone` and the new check share `getClusterSummaryListOptions`. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
gianlucam76
force-pushed
the
reduce-deploy-log-noise
branch
from
October 6, 2026 06:39
9b59684 to
569e8fa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A ClusterSummary for a paused cluster is never removed: the ClusterSummary controller returns without doing anything until the cluster is unpaused. The ClusterProfile/Profile being deleted has no way to know that. It does not watch ClusterSummaries, so it polls:
cleanClusterSummariesreturnsclusterSummaries still presentand the profile is requeued every 10 seconds, for as long as the cluster stays paused.On a management cluster with several paused clusters this produced about 2,100
failed to clean ClusterSummarieserrors in 35 minutes (ten profiles, one reconcile and one error every 10 seconds each), all with the same outcome.When a ClusterProfile/Profile is being deleted and every ClusterSummary still present is for a paused cluster, it now stops polling:
reconcileDeleteCommonreturns a newWaitingForPausedClustersError(listing the clusters) instead of the generic error. Both the ClusterProfile and the Profile controllers log it once at info level and requeue after 10 minutes (pausedClustersRequeueAfter). The 10 minutes is only a safety net.ClusterMapfor the paused clusters (trackClusters). A profile that is being deleted is only in the maps if it went through the normal reconcile, so after a restart an unpause would otherwise never reconcile it. Unpausing a cluster now reconciles the profile, which then resumes the usual 10 second polling until the ClusterSummaries are gone.cleanClusterSummariesnow returns theerrClusterSummariesStillPresentsentinel with the same message as before.allClusterSummariesGoneand the new check sharegetClusterSummaryListOptions.