Implement failover host controller

api/v1alpha1/reservation_types.go

@@ -179,6 +179,14 @@ type FailoverHostReservationSpec struct {
 	// See Spec.TargetHost for the target host of the failover reservation.
 	// +kubebuilder:validation:Optional
 	SourceHosts []string `json:"sourceHosts,omitempty"`
+
+	// PartitionDescription is a short, human-readable label describing what
+	// distinguishes the hosts of this failover partition from those of other
+	// partitions, e.g. "spec.aggregates=[hana]". It carries no scheduling
+	// semantics and exists purely to make a reservation understandable at a
+	// glance. See characterizePartitions in the failoverhost controller.
+	// +kubebuilder:validation:Optional
+	PartitionDescription string `json:"partitionDescription,omitempty"`
 }
 
 // ReservationSpec defines the desired state of Reservation.

cmd/manager/main.go

@@ -59,6 +59,7 @@ import (
 	"github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/commitments"
 	commitmentsapi "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/commitments/api"
 	"github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/failover"
+	"github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/failoverhost"
 	"github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/inflight"
 	"github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/quota"
 	"github.com/cobaltcore-dev/cortex/pkg/cache"
@@ -791,6 +792,30 @@ func main() {
 			"maxVMsToProcess", failoverConfig.MaxVMsToProcess,
 			"vmSelectionRotationInterval", failoverConfig.VMSelectionRotationInterval)
 	}
+	if slices.Contains(mainConfig.EnabledControllers, "failover-host-reservations-controller") {
+		setupLog.Info("enabling controller", "controller", "failover-host-reservations-controller")
+		failoverHostConfig := conf.GetConfigOrDie[failoverhost.Config]().Failover.Hosts
+		failoverHostMonitor := failoverhost.NewMonitor()
+		if err := metrics.Registry.Register(failoverHostMonitor); err != nil {
+			setupLog.Error(err, "failed to register failover host monitor metrics, continuing without metrics")
+			failoverHostMonitor = nil
+		}
+		controller := &failoverhost.Controller{
+			Client:  multiclusterClient,
+			Config:  failoverHostConfig,
+			Monitor: failoverHostMonitor,
+		}
+		if err := controller.SetupWithManager(mgr); err != nil {
+			setupLog.Error(err, "unable to create controller",
+				"controller", "failover-host-reservations-controller")
+			os.Exit(1)
+		}
+		setupLog.Info("failover-host-reservations-controller registered",
+			"ratio", failoverHostConfig.Ratio,
+			"minimum", failoverHostConfig.Minimum,
+			"generationInterval", failoverHostConfig.GenerationInterval,
+			"externalSchedulerURL", failoverHostConfig.ExternalSchedulerURL)
+	}
 	if slices.Contains(mainConfig.EnabledControllers, "capacity-controller") {
 		setupLog.Info("enabling controller", "controller", "capacity-controller")
 		capacityConfig := conf.GetConfigOrDie[capacity.Config]()

docs/02-external-scheduler-api/02-nova-smart-scheduling.md

@@ -122,7 +122,7 @@ pay-as-you-go placement and is not slotted. This is how a committed reservation
 > confirmed location — and several Nova requests can be in flight at once. How Cortex reserves for its
 > candidates before the next request scores, and how it reconciles against what Nova actually did, is the
 > subject of
-> [Concurrency and in-flight reservations](../03-reservations-and-inventory/04-concurrency-and-in-flight-reservations.md).
+> [Concurrency and in-flight reservations](../03-reservations-and-inventory/05-concurrency-and-in-flight-reservations.md).
 
 ## Closing the loop: descheduling
 

docs/03-reservations-and-inventory/01-reservations-overview.md

@@ -46,7 +46,9 @@ are checked arithmetically against headroom. This is the subject of
 These reserve **headroom for evacuation**: if a host fails, its workloads need somewhere to land. Cortex
 pre-clears space to absorb up to `m` simultaneous host failures, shared across the protected workloads
 rather than duplicated per workload. This is the subject of
-[Failover reservations for KVM HA](03-failover-reservations.md).
+[Failover reservations for KVM HA](03-failover-reservations.md) — now **deprecated** in favour of the
+whole-host approach in [Failover host reservations](04-failover-host-reservations.md), which keeps entire
+hosts empty as gang-evacuation targets.
 
 ### In-flight reservations
 
@@ -59,7 +61,7 @@ in-flight reservation blocks that capacity from "decided" until "running", then
 > The reasoning here is pessimistic by necessity: because concurrent requests race, Cortex assumes a slot
 > it just recommended is already taken until the placement is realized or abandoned. In-flight
 > reservations are the persistence layer behind the scheduler's concurrency handling —
-> [Concurrency and in-flight reservations](04-concurrency-and-in-flight-reservations.md) explains the
+> [Concurrency and in-flight reservations](05-concurrency-and-in-flight-reservations.md) explains the
 > races they defend against and how the scheduler serializes around them.
 
 ## Inspecting reservations

docs/03-reservations-and-inventory/03-failover-reservations.md

@@ -6,6 +6,16 @@
 
 # Failover reservations for KVM HA
 
+> [!WARNING]
+> **Deprecated.** The slot-based failover reservation system described on this page (the
+> `failover-reservations-controller`, reserving a shared `FailoverReservation` pool) is being **replaced by the
+> failover *host* system** — see [Failover host reservations](04-failover-host-reservations.md). Rather than
+> reserving scattered headroom slots, the new system keeps whole hosts empty so a failed host's entire workload
+> gang-evacuates onto one reserved host. The `FailoverReservation` type is marked deprecated in
+> `api/v1alpha1/reservation_types.go` and will be removed; new deployments should enable the
+> `failover-host-reservations-controller` instead. This page is retained to document the existing system during
+> migration.
+
 Where committed resources reserve space a customer paid for, **failover reservations** reserve space
 *nobody is using yet* — headroom kept clear so that when a host fails, the workloads on it have somewhere
 to be evacuated. This page explains the two design choices that make failover headroom affordable: it is
@@ -94,4 +104,4 @@ same slot.
 
 ## Next
 
-[Prev: Committed-resource reservations](02-committed-resource-reservations.md) · [Next: Concurrency and in-flight reservations »](04-concurrency-and-in-flight-reservations.md)
+[Prev: Committed-resource reservations](02-committed-resource-reservations.md) · [Next: Failover host reservations »](04-failover-host-reservations.md)