Repository navigation
Implement failover host controller - #1417
Open
PhilippMatthes wants to merge 15 commits into
Open
PhilippMatthes wants to merge 15 commits into
PhilippMatthes wants to merge 15 commits into
Conversation
![cortex-ai-agents[bot]](https://avatars.githubusercontent.com/in/3521331?s=60&v=4){kind=small}
Add an optional PartitionDescription field to the FailoverHostReservation spec: a short, human-readable label describing what distinguishes the hosts of a failover partition from those of other partitions (e.g. "spec.aggregates=[hana]"). It carries no scheduling semantics and exists purely to make a reservation understandable at a glance. Regenerate the reservations CRD to include the new field. Signed-off-by: Philipp Matthes <p.matthes@sap.com>
Introduce the failoverhost controller, which keeps whole compute hosts reserved as gang-evacuation targets. It groups hosts into failover partitions (sets of hosts whose workloads can be evacuated onto one another, discovered via the nova external scheduler) and keeps a configurable share of each partition empty and reserved, so when a host fails its entire workload can be moved onto a reserved host in one step. The package is split into: - partitions.go: discover and characterize failover partitions - pathfinding.go: determine which hosts can evacuate onto which - planning.go: decide how many hosts to reserve per partition - controller.go: reconcile FailoverHostReservation resources to match Wire the controller into the manager behind the "failover-host-reservations-controller" entry in EnabledControllers, and add its configuration (ratio, minimum, generationInterval, externalSchedulerURL) under failover.hosts in the cortex-nova bundle values. Signed-off-by: Philipp Matthes <p.matthes@sap.com>
Add a page on failover host reservations describing how whole hosts are kept reserved as gang-evacuation targets, and slot it into the reservations-and-inventory section as chapter 04. Renumber the following chapters accordingly (concurrency/in-flight 04->05, placement API shim 05->06) and update the overview, readme, and cross-references that point at them. Signed-off-by: Philipp Matthes <p.matthes@sap.com>
Add tools/mirror/apply-with-status.sh, a development helper that replicates a snapshot of Kubernetes resources into another cluster with their .status subresource intact: it server-side applies the objects, then patches each object's status onto the freshly created resource. Before applying it strips origin-cluster metadata (notably ownerReferences) that would otherwise cause the target cluster's garbage collector to delete the resources. Useful for testing the failover host reservation feature against captured cluster state. Signed-off-by: Philipp Matthes <p.matthes@sap.com>
PhilippMatthes
force-pushed
the
implement-failover-partitioning-algorithm
branch
from
October 26, 2026 10:42
426b14f to
24177e5
Compare
Signed-off-by: Philipp Matthes <p.matthes@sap.com>
Maximal cliques from Bron-Kerbosch can overlap: a host with a mutual edge into two otherwise-disjoint cliques appears in both partitions. Since reservations are keyed by target host name, such a host planned as a failover target by two partitions produced two Reservation objects with the same name but conflicting SourceHosts, and the last write into the desired map silently dropped the other partition's coverage. Assign each shared host to exactly one partition (the largest, ties broken deterministically) before planning, so the planner input is disjoint and the name collision cannot arise. Signed-off-by: Philipp Matthes <p.matthes@sap.com>
…h KPIs The failover host controller previously emitted no metrics, so operators could not tell whether the calculation loop was running, whether the external scheduler was reachable, or whether reservations could be reconciled. Add a Monitor (mirroring the sibling failover controller) covering: - calculation-loop runs, duration and aborts (by reason) - partitions / desired / existing reservation counts - external scheduler request count and duration (by status) - reservation create/update/delete and error counts (by op) Add KVM-gated alerts on the actionable subset: the loop stalling, the loop aborting, reconcile errors, and a high scheduler error rate. Expose the failover partition graph as a reservation-reading KPI (cortex_failover_partition_graph_node / _edge), derived from the failover host reservation specs rather than the controller. Signed-off-by: Philipp Matthes <p.matthes@sap.com>
Rewrite the failover host reservations page to explain the concept and algorithm at a high level rather than mirroring the code structure. - Replace implementation-internal references (struct, method and file names, the Bron-Kerbosch algorithm and its internals, field names) with plain-language explanations of what each step does and why. - Reframe the controller as a periodic planner plus a continuous reconciler, and the pipeline as four narrowing steps. - Add diagrams and prose for the edge cases a human needs to reason about: intersection collapse, one-way links, empty-host recruiting, the overlap dedupe, and the reserve-count rules (with a worked table). - Add a troubleshooting table mapping 'host has no failover target' symptoms back to the step that caused them. Signed-off-by: Philipp Matthes <p.matthes@sap.com>
Previously only failover target hosts were emitted as graph nodes, so source hosts appeared solely as edge endpoints. Emit every source host as a node too, carrying the same az and reserved capacity as the target it fails over onto. Signed-off-by: Philipp Matthes <p.matthes@sap.com>
Derive partition-graph nodes from the hypervisor list rather than from failover reservations, so every hypervisor appears as a node (reserved or not) carrying its own az and capacity. The node value is 1 when the hypervisor has a failover path and 0 when it has none; reservations now only supply the edges. Signed-off-by: Philipp Matthes <p.matthes@sap.com>
When overlapping maximal cliques share a host, assign it to the smallest partition it belongs to rather than the largest. A small partition only survives if it keeps its shared hosts, while a large one usually stays above the two-host minimum after giving one up, so this preserves failover coverage for as many partitions as possible. Also refine the failover-host reservations docs: correct the per-vm lookup behaviour (an empty candidate set is tolerated, but a lookup error aborts the whole pass), explain why Bron-Kerbosch is used for partitioning, the vm-level self-partitioning rationale, bidirectional rescue links, and the candidate ordering that retains existing reservations before creating new ones. Signed-off-by: Philipp Matthes <p.matthes@sap.com>
Empty hosts were recruited into a partition using only the inbound reachability check (member -> empty), with no size guard. A large empty host (e.g. a 4TB host) could therefore be folded into a partition of smaller hosts (e.g. 1TB) that could reach it, and then be recorded as a SourceHost of the smaller reserved failover target -- a host it could never actually be gang-evacuated onto, violating the whole-host fit rule that Step 1 enforces for every other edge. Having no VMs, an empty host has no rescue links to prove it can be rescued by its new peers, so recruitment now falls back to the same gang-fit rule and requires equal effective capacity (memory then CPU, rounded to GiB). A larger empty host stays on its own instead. Signed-off-by: Philipp Matthes <p.matthes@sap.com>
Signed-off-by: Philipp Matthes <p.matthes@sap.com>
The evacuation probe asked the scheduler whether a peer's vms fit on a candidate given its current free capacity. But a failover target is reserved empty -- an evacuation never lands on an occupied host. So a busy host accepted no one, gained no inbound rescue link, formed no mutual edge, and stayed orphaned, even though reserving it would drain it. Set AssumeEmptyHosts on the probe so candidates are judged by total capacity instead of current free capacity. Partition compatibility becomes occupancy-independent; which host is cheapest to actually empty is still decided by the planner's candidate ordering. The whole-host fit check already compares total capacities, so it is unaffected. Signed-off-by: Philipp Matthes <p.matthes@sap.com>
Signed-off-by: Philipp Matthes <p.matthes@sap.com>
Contributor
Test Coverage ReportTest Coverage 📊: 71.7% |
PhilippMatthes
marked this pull request as ready for review
October 26, 2026 16:31
PhilippMatthes
requested review from
SoWieMarkus,
auhlig,
juliusclausnitzer,
mblos and
umswmayj
as code owners
October 26, 2026 16:31
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This implements the failover host reservation system which supersedes the failover slot reservation logic. The key motivation to implement a host based failover system is better resource utilization. Instead of optimizing slot reservations until we have an acceptably low amount of reserved resources, we solve this over reserving entire hosts. This simplifies the original problem, because hosts are easier to match.
Selecting the correct hosts inside the infrastructure is solved through a self-partitioning algorithm. Rather than hand-crafting rules that re-encode which hosts are mutually compatible, Cortex asks the real scheduler, per VM, where each VM could go, and lets the failover partitions fall out of those answers. This way the compatibility rules are never duplicated: whatever the placement pipeline does, the partitions inherit automatically, including future changes to filters and weighers.
A planner runs on a fixed interval. It discovers which hosts can rescue which others by probing the scheduler per VM, keeps only the mutual (reversible) rescue links, groups them into partitions of fully interchangeable hosts via maximal-clique enumeration, resolves hosts that fall into several partitions, and finally reserves a small fraction of each partition as empty landing zones. A separate reconciler continuously nudges the actual reservations toward the planner's desired picture. The planner always publishes a complete picture computed from a complete snapshot, so reservations are never recomputed from partial data and stale ones are pruned for free.
The change adds the controller for this, a PartitionDescription field on the reservation spec to make a reservation understandable at a glance, Prometheus metrics, alerts and partition-graph KPIs, documentation of the algorithm and lifecycle, and a small mirror helper tool. The slot-based failover controller is deprecated and retained only for migration.