[HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.3 and move the Hopsworks Kueue models to v1beta2

docs/setup_installation/admin/upgrade.md

@@ -239,6 +239,22 @@ From 5.3 the default is 2048, up from 1024, because at 1 GiB the sidecar was OOM
   While the sidecar restarts, the pod reports `Ready=False` and leaves the endpoints of its service, so the agent, healthy and still running, receives no requests.
   Looking for agent restarts finds none.
 
+### Kueue outside the Hopsworks namespace { #upgrade-kueue-external }
+
+Required for a cluster that runs the Kueue bundled with Hopsworks (`global._hopsworks.kueue.enabled=true`); a cluster without Kueue needs nothing.
+From the 5.2.x patch release that uses Kueue 0.20, and in 5.3, Hopsworks uses a Kueue installed as its own release, `kueue` in `kueue-system`, in place of the Kueue 0.12.2 bundled in the Hopsworks namespace: the chart ships no Kueue.
+The new chart refuses to upgrade a release that still runs the bundled Kueue 0.12, and a cluster that runs it cannot be switched with `helm upgrade` alone: Kueue 0.12 stores Cohorts and Topologies as `v1alpha1`, which no later Kueue reads.
+The 5.1 chart's `migrate-to-external.sh` moves it to Kueue 0.18.1 in `kueue-system` and keeps every queue, on 5.1, or on 5.2.0 and 5.2.1 together with the Hopsworks upgrade.
+After the Hopsworks upgrade, the new chart's `upgrade-kueue.sh` upgrades Kueue to 0.20.3.
+Follow [Kueue outside the Hopsworks namespace](kueue_external.md), and stop submitting jobs while the move runs.
+
+What changes for users and operators ([details](kueue_external.md#what-changes)):
+
+- Spark jobs are queued, with static executors: a job saved with dynamic allocation runs its dynamic-allocation minimum, and `spark.dynamicAllocation.enabled=true` in its Spark properties is refused.
+- A workload whose pods are not all ready 30 minutes after admission is evicted and queued again (Kueue 0.19 `waitForPodsReady`).
+- The Ray autoscaler and job-submitter sidecars count against quota (Kueue 0.19).
+- A workload without a priority class gets the WorkloadPriorityClass named `default`, when one exists (Kueue 0.20).
+
 ## Changes from 5.2.2 { #upgrade-changes-from-522 }
 
 These changes ship in 5.2.2 and 5.3.0.

docs/setup_installation/common/helm_chart_values/hw-kueue.md

@@ -1,6 +1,7 @@
 # Kueue values { #helm-values-hw-kueue }
 
 Values under `hw-kueue` configure Kueue, the job queueing controller, and the queues Hopsworks schedules jobs with.
+With `kueue.enabled=false` Hopsworks uses a Kueue installed outside its release; [Kueue outside the Hopsworks namespace](../../admin/kueue_external.md) covers moving an existing cluster there.
 
 <!-- BEGIN GENERATED VALUES -->
 

docs/user_guides/projects/jobs/pyspark_job.md

@@ -108,6 +108,7 @@ Resource allocation for the Spark driver and executors can be configured, also t
   Raise it when executors are OOMKilled.
 
 - `Dynamic/Static`: Run the Spark application in static or dynamic allocation mode (see [spark docs](https://spark.apache.org/docs/latest/configuration.html#dynamic-allocation) for details).
+  On a cluster that schedules jobs with Kueue, only static is allowed: a job with dynamic allocation is refused, and new jobs default to static ([Spark jobs and Kueue](../scheduling/kueue_details.md#spark-jobs)).
 
 <p align="center">
   <figure>
@@ -246,7 +247,7 @@ The following table describes the job configuration parameters for a PYSPARK job
 | <nobr>`conf['spark.executor.memory']`</nobr> | int | Memory allocated per executor (in MB) | `4096` |
 | <nobr>`conf['spark.driver.memoryOverheadFactor']`</nobr> | float | Fraction of the driver memory added to the driver pod as non-heap overhead. `null` keeps Spark's default: 0.10 for Spark jobs, 0.40 for PySpark | `null` |
 | <nobr>`conf['spark.executor.memoryOverheadFactor']`</nobr> | float | Fraction of the executor memory added to each executor pod as non-heap overhead. `null` keeps Spark's default: 0.10 for Spark jobs, 0.40 for PySpark | `null` |
-| <nobr>`conf['spark.dynamicAllocation.enabled']`</nobr> | boolean | Enable dynamic allocation of executors | `true` |
+| <nobr>`conf['spark.dynamicAllocation.enabled']`</nobr> | boolean | Enable dynamic allocation of executors. Must be `false` on a cluster with Kueue | `true`, `false` with Kueue |
 | <nobr>`conf['spark.dynamicAllocation.minExecutors']`</nobr> | int | Minimum number of executors with dynamic allocation | `1` |
 | <nobr>`conf['spark.dynamicAllocation.maxExecutors']`</nobr> | int | Maximum number of executors with dynamic allocation | `2` |
 | <nobr>`conf['spark.dynamicAllocation.initialExecutors']`</nobr> | int | Initial number of executors with dynamic allocation | `1` |

docs/user_guides/projects/jobs/spark_job.md

@@ -114,6 +114,7 @@ Resource allocation for the Spark driver and executors can be configured, also t
   Raise it when executors are OOMKilled.
 
 - `Dynamic/Static`: Run the Spark application in static or dynamic allocation mode (see [spark docs](https://spark.apache.org/docs/latest/configuration.html#dynamic-allocation) for details).
+  On a cluster that schedules jobs with Kueue, only static is allowed: a job with dynamic allocation is refused, and new jobs default to static ([Spark jobs and Kueue](../scheduling/kueue_details.md#spark-jobs)).
 
 <p align="center">
   <figure>
@@ -249,7 +250,7 @@ The following table describes the job configuration parameters for a SPARK job.
 | <nobr>`conf['spark.executor.memory']`</nobr> | int | Memory allocated per executor (in MB) | `4096` |
 | <nobr>`conf['spark.driver.memoryOverheadFactor']`</nobr> | float | Fraction of the driver memory added to the driver pod as non-heap overhead. `null` keeps Spark's default: 0.10 for Spark jobs, 0.40 for PySpark | `null` |
 | <nobr>`conf['spark.executor.memoryOverheadFactor']`</nobr> | float | Fraction of the executor memory added to each executor pod as non-heap overhead. `null` keeps Spark's default: 0.10 for Spark jobs, 0.40 for PySpark | `null` |
-| <nobr>`conf['spark.dynamicAllocation.enabled']`</nobr> | boolean | Enable dynamic allocation of executors | `true` |
+| <nobr>`conf['spark.dynamicAllocation.enabled']`</nobr> | boolean | Enable dynamic allocation of executors. Must be `false` on a cluster with Kueue | `true`, `false` with Kueue |
 | <nobr>`conf['spark.dynamicAllocation.minExecutors']`</nobr> | int | Minimum number of executors with dynamic allocation | `1` |
 | <nobr>`conf['spark.dynamicAllocation.maxExecutors']`</nobr> | int | Maximum number of executors with dynamic allocation | `2` |
 | <nobr>`conf['spark.dynamicAllocation.initialExecutors']`</nobr> | int | Initial number of executors with dynamic allocation | `1` |

docs/user_guides/projects/scheduling/kube_scheduler.md

@@ -10,7 +10,7 @@ Hopsworks allows users to configure some Kubernetes scheduler abstractions, such
 Hopsworks also supports additional scheduling abstractions backed by Kueue.
 This includes [Queues](https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/), [Cohorts](https://kueue.sigs.k8s.io/docs/concepts/cohort/) and [Topologies](https://kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling/).
 All these scheduling abstractions are supported in jobs, jupyter notebooks and model deployments.
-Kueue abstractions however, are currently not supported for Spark jobs.
+Spark jobs run in Kueue queues with static executors only: see [Spark jobs](./kueue_details.md#spark-jobs).
 
 Hopsworks Admins can control which labels and priority classes can be used in the cluster (see [Admin configuration](#admin-configuration) section) and by which project (see [Project Configuration](#project-configuration) section)
 

docs/user_guides/projects/scheduling/kueue_details.md

@@ -10,9 +10,21 @@ Hopsworks provides the integration with Kueue to provide the additional scheduli
 Hopsworks currently acts only as a "reader" to the Kueue abstractions and currently does not manage the lifecycle of Kueue abstraction with the exception of the default localqueue for each namespace.
 All the other abstractions are expected to be managed by the administrators of Hopsworks, directly on the Kubernetes cluster.
 
-However Hopsworks and Kueue integration currently only supports frameworks python and ray for jobs, notebooks and model deployments.
+Kueue schedules Python and Ray jobs, Spark jobs, notebooks and model deployments.
 The same queues are also used for Hopsworks internal jobs (zipping, git operations, python library installation).
-Spark is currently not supported, and thus will not be managed by Kueue for scheduling, and instead it will bypass the queues setup (important to note when thinking about queue quotas) and instead are managed directly by the Kubernetes Scheduler.
+
+### Spark jobs
+
+Spark jobs run as Spark Operator `SparkApplication`s, which Kueue admits through its SparkApplication integration.
+They go to the project's default queue unless the job selects another; a Spark job saved without a queue, as all were before Kueue managed Spark, gets the default queue when it starts.
+Kueue reserves quota for the driver and a fixed number of executors, so on a cluster with Kueue a Spark job runs with static executors:
+
+- `spark.dynamicAllocation.enabled` must be `false`. A Spark job with dynamic allocation is refused when it is saved and when it is started (error code 130082).
+- `spark.executor.instances` sets the number of executors Kueue reserves quota for.
+- New Spark jobs default to static executors, as many as the dynamic-allocation maximum.
+- Spark jobs created before Kueue was enabled have dynamic allocation on, the earlier default. Turn it off and set the number of executors before running them.
+
+Spark in Jupyter notebooks runs in client mode and is not managed by Kueue, so it keeps dynamic allocation.
 
 ### Resource flavors
 
@@ -21,30 +33,30 @@ The resource flavor defines the resources that a queue will later manage.
 Hopsworks helm chart installs and uses a default ResourceFlavor
 
 ```yaml
-apiVersion: kueue.x-k8s.io/v1beta1
+apiVersion: kueue.x-k8s.io/v1beta2
 kind: ResourceFlavor
 metadata:
   name: default-flavor
 spec:
   nodeLabels:
     cloud.provider.com/region: europe
-  topologyName: default
 ```
 
-Node labels filter the available nodes to this resource flavor and is required for [topologies](#topologies)
+Node labels filter the available nodes to this resource flavor.
+A flavor that names a topology with `topologyName` makes Kueue place pods by that [topology](#topologies).
 
 ### Cluster Queues
 
 [Cluster Queues](https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/) are the actual queues for submitting jobs and model deployments to.
 The default hopsworks queue looks like:
 
 ```yaml
-apiVersion: kueue.x-k8s.io/v1beta1
+apiVersion: kueue.x-k8s.io/v1beta2
 kind: ClusterQueue
 metadata:
   name: other
 spec:
-  cohort: cluster
+  cohortName: cluster
   namespaceSelector: {}
   preemption:
     borrowWithinCohort:
@@ -90,7 +102,7 @@ An administrator can define the default queue for projects user jobs and system
 Hopsworks defines a default `cluster` cohort
 
 ```yaml
-apiVersion: kueue.x-k8s.io/v1alpha1
+apiVersion: kueue.x-k8s.io/v1beta2
 kind: Cohort
 metadata:
   name: cluster
@@ -114,7 +126,7 @@ spec:
         nominalQuota: 50
 ```
 
-Cohorts can contain other cohorts and thus you can create a hierarchy of cohorts.
+Cohorts can contain other cohorts, through `parentName`, and thus you can create a hierarchy of cohorts.
 Cohorts can set [fair sharing weight](https://kueue.sigs.k8s.io/docs/concepts/admission_fair_sharing/) where using
 
 ```yaml
@@ -130,7 +142,7 @@ in the definition of a cohort, the user can control a priority towards borrowing
 Hopsworks defines a default topology:
 
 ```yaml
-apiVersion: kueue.x-k8s.io/v1alpha1
+apiVersion: kueue.x-k8s.io/v1beta2
 kind: Topology
 metadata:
   name: default
@@ -141,7 +153,7 @@ spec:
   - nodeLabel: kubernetes.io/hostname
 ```
 
-The topology is defined in the Resource Flavor used by a Cluster Queue.
+A Cluster Queue uses a topology through the `topologyName` of its Resource Flavor; the default flavor names none.
 
 When creating a new job, the user can select a topology unit for the job to run in and thus decide if all pods of a job should run on the same hostname, in the same zone or in the same region.
 The user can select the topology for jobs, notebooks and deployments from `Advanced options`, in the `Scheduler` section of the full configuration page.

mkdocs.yml

@@ -315,6 +315,7 @@ nav:
               - Upgrading with Helm 4: setup_installation/admin/helm4.md
               - Upgrading OpenSearch: setup_installation/admin/upgrade_opensearch.md
               - Kafka 4 and KRaft upgrade notes: setup_installation/admin/kafka4.md
+              - Kueue outside the Hopsworks namespace: setup_installation/admin/kueue_external.md
           - Monitoring:
               - Services Dashboards: setup_installation/admin/monitoring/grafana.md
               - Export metrics: setup_installation/admin/monitoring/export-metrics.md