server: reserve cluster capacity for HA failover (HA admission control) - #14042
nagaboinaramgopal wants to merge 2 commits into
Conversation
Adds cluster.ha.failover.capacity.reservethreshold (default 1.0 = disabled). When set below 1.0, FirstFitPlanner also excludes any cluster whose allocated+requested cpu/memory would cross the reserve from new deployments, so headroom stays free for HA-triggered restarts. The threshold is per-cluster scope aware.
|
@DaanHoogland Could you pls take a look at this |
| ConfigKey.Scope.Global); | ||
| // Reserve spare cluster capacity for HA failover. | ||
| static final ConfigKey<Float> ClusterHAFailoverReserveThreshold = | ||
| new ConfigKey<Float>( |
There was a problem hiding this comment.
| new ConfigKey<Float>( | |
| new ConfigKey<>( |
| // Exclude clusters that would cross the HA failover reserve threshold, reserving | ||
| // capacity for HA-triggered restarts. Threshold is Cluster-scoped, resolved per-cluster by the DAO. | ||
| long haRequested = (capacity == Capacity.CAPACITY_TYPE_CPU) ? cpu_requested : ram_requested; | ||
| List<Long> clustersCrossingHAReserve = capacityDao.listClustersCrossingThreshold( | ||
| capacity, plan.getDataCenterId(), ClusterHAFailoverReserveThreshold.key(), haRequested); | ||
| if (clustersCrossingHAReserve != null && !clustersCrossingHAReserve.isEmpty()) { | ||
| avoid.addClusterList(clustersCrossingHAReserve); | ||
| clusterListForVmAllocation.removeAll(clustersCrossingHAReserve); | ||
| logger.warn(String.format( | ||
| "HA admission control: excluding clusters %s from new deployments; their %s allocation would cross the HA failover reserve threshold [%s], reserving capacity for HA failover", | ||
| clustersCrossingHAReserve, CapacityVO.getCapacityName(capacity), | ||
| ClusterHAFailoverReserveThreshold.key())); | ||
| } |
There was a problem hiding this comment.
please extract a method and use the comment as javadoc
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #14042 +/- ##
======================================
Coverage 3.53% 3.53%
======================================
Files 487 487
Lines 41863 41863
Branches 7912 7912
======================================
Hits 1479 1479
Misses 40170 40170
Partials 214 214
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Use the diamond operator for the ClusterHAFailoverReserveThreshold ConfigKey, and extract the HA reserve exclusion into excludeClustersCrossingHAReserve with the former inline comment as its javadoc.
|
Thanks @DaanHoogland. Done in 41c7210: used the diamond for the ConfigKey, and pulled that block into its own method (excludeClustersCrossingHAReserve) with the comment as the javadoc. |
| logger.warn(warnMessageForClusterReachedCapacityThreshold); | ||
| } | ||
|
|
||
| excludeClustersCrossingHAReserve(capacity, cpu_requested, ram_requested, plan, avoid, clusterListForVmAllocation); |
There was a problem hiding this comment.
does this also block ha restarts? cluster.threshold.enabled is on by default, so this check runs on every start, not just new deploys
There was a problem hiding this comment.
Good question. The reserve check lives inside removeClustersCrossingThreshold, so it does run on the first HA attempt, since HA starts the VM with its original planner. But if the VM can't fit outside the reserve, HA falls back to SkipHeuresticsPlanner, which overrides removeClustersCrossingThreshold to a no-op ("Deploying vm during HA process, so skipping disable threshold check"). That skips the reserve exclusion too, so the restart lands in the reserved capacity. So the reserve keeps normal deployments out but stays available to HA restarts through that fallback.
This branch has not been deployed
3 participants
Description
Adds optional HA admission control: an operator can reserve a fraction of each cluster's CPU/memory so new deployments leave headroom for HA-triggered restarts when a host fails.
A new cluster-scoped setting
cluster.ha.failover.capacity.reservethreshold(Float, default 1.0 = disabled) controls it. Below 1.0, FirstFitPlanner excludes any cluster whose allocated + requested CPU or memory would cross that fraction from new deployments, using the samelistClustersCrossingThresholdpath the capacity disable threshold already uses. HA restarts are not subject to the reserve, so the headroom is available exactly when failover needs it. Off by default, so existing behaviour is unchanged until an operator opts in per cluster.For example, 0.8 keeps about 20% of a cluster's CPU and memory free for failover. It is meant to be set below the corresponding
cluster.*.allocated.capacity.disablethreshold.Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
How Has This Been Tested?
Added unit tests in
FirstFitPlannerTest:checkHAFailoverReserveDisabledByDefault: at the default 1.0 the cluster selection is unchanged.checkHAFailoverReserveExcludesClusterOnDeploy: a cluster whose allocation would cross the reserve is excluded from a new deployment.Also built the standard packages and deployed on a KVM advanced zone.