From 9b10ae7a7db8433ebba9245833937009ae1b556d Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Wed, 21 Oct 2026 21:25:34 +0200 Subject: [PATCH 01/17] [HWORKS-3565] Upgrade Kueue from the bundled 0.12.2 to 0.18.1 installed outside the Hopsworks namespace, migrating existing queues and CRDs https://hopsworks.atlassian.net/browse/HWORKS-3565 Every release from 4.3 bundles Kueue 0.12.2 in the Hopsworks release namespace. From 5.3 Hopsworks can use a Kueue installed as its own release in kueue-system instead (hw-kueue.kueue.enabled=false), and the chart ships files/migrate-to-external.sh to move a running cluster to Kueue 0.18.1 while keeping every Kueue object. The docs add an operator page for that move: why helm upgrade alone cannot do it, the script's phases, what to prepare (snapshot, maintenance window, tools, airgapped images, admin API key), the command, what changes for queues and Hopsworks afterwards (no topologyName on flavors, parentName on Cohorts, queues managed with kubectl, Hopsworks-namespace jobs queued, the queue utilisation view failing until the backend reads Cohorts at v1beta1) and how to verify. The hw-kueue values page links to it. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- .../admin/kueue_external.md | 102 ++++++++++++++++++ .../common/helm_chart_values/hw-kueue.md | 1 + mkdocs.yml | 1 + 3 files changed, 104 insertions(+) create mode 100644 docs/setup_installation/admin/kueue_external.md diff --git a/docs/setup_installation/admin/kueue_external.md b/docs/setup_installation/admin/kueue_external.md new file mode 100644 index 0000000000..e01aa874c0 --- /dev/null +++ b/docs/setup_installation/admin/kueue_external.md @@ -0,0 +1,102 @@ +# Operator Notes: Kueue Outside the Hopsworks Namespace + +Administrative reference for cluster operators who run Kueue with Hopsworks and want to move it out of the Hopsworks release. + +With `global._hopsworks.kueue.enabled=true`, every Hopsworks release from 4.3 bundles Kueue 0.12.2 in the Hopsworks namespace. +From 5.3, `hw-kueue.kueue.enabled=false` makes Hopsworks use a Kueue installed outside its release instead: release `kueue` in namespace `kueue-system`, installed with `fullnameOverride: kueue`. +Hopsworks keeps its RBAC bindings to the Kueue roles and keeps queueing jobs through Kueue. + +The move is optional. +A cluster that keeps the bundled Kueue needs nothing. +A fresh install that wants an external Kueue installs it first and sets `hw-kueue.kueue.enabled=false`; this page is for clusters that already run the bundled one. + +## Why `helm upgrade` alone cannot do it + +- **Kueue 0.12 stores Cohorts and Topologies as `v1alpha1`**, which no Kueue from 0.13 on can read. + Their CRDs have to be replaced and the objects restored. +- **The Kueue CRDs belong to no Helm release.** + The bundled chart ships them in its `crds/` directory, which Helm installs once and never upgrades. +- **The new release creates cluster-wide objects under the names the Hopsworks release owns**: the `kueue-*` ClusterRoles, the webhook configurations and the visibility APIService. + The Hopsworks release has to drop them first. + +## What the migration does + +The chart ships `charts/hw-kueue/files/migrate-to-external.sh`, which moves the cluster to Kueue 0.18.1 and keeps every Kueue object: ClusterQueues, Cohorts, ResourceFlavors, Topologies, LocalQueues, priority classes and Workloads. +It runs in phases, and each one is safe to re-run: + +| Phase | What it does | +| --- | --- | +| `preflight` | Checks the tools and the release. Refuses admitted, unfinished Workloads unless given `--allow-admitted`. | +| `backup` | Saves every Kueue object, the Kueue CRDs and the release values to `--backup-dir`. | +| `remove-bundled` | Upgrades Hopsworks with Kueue off, which removes the bundled controller, its webhooks and its ClusterRoles. CRDs and queue objects stay. | +| `replace-alpha-crds` | Snapshots the Cohorts, Topologies and ResourceFlavors, then replaces the `v1alpha1` Cohort and Topology CRDs. | +| `install-kueue` | Applies the Kueue 0.18.1 CRDs, installs release `kueue` in `kueue-system` and restores the replaced objects. | +| `migrate-storage` | Rewrites every object at `v1beta2` and prunes the CRDs' stored versions. | +| `enable-hopsworks` | Upgrades Hopsworks with Kueue on and `hw-kueue.kueue.enabled=false`. | +| `verify` | Checks the controller, the stored versions, every backed-up queue object and the release values. | + +A failed run resumes by running the same phase again, or `all`. +`--dry-run` prints the commands that change the cluster; with `all` it stops after `remove-bundled`, whose result the later phases depend on. + +## Before you start + +- **Take a snapshot** and check that it completed: from `replace-alpha-crds` on, rollback is the snapshot. +- **Plan a maintenance window and stop submitting jobs.** + From `remove-bundled` to `install-kueue` no Kueue runs, so a job created then runs outside quota. + Workloads already admitted keep running, and Kueue 0.18.1 takes them over. +- **Tools**: `kubectl`, `helm` 3.20 or later, `jq` and `yq` v4. +- **Airgapped clusters** mirror `registry.k8s.io/kueue/kueue:v0.18.1` (the default image is `docker.hops.works/registry.k8s.io/kueue/kueue:v0.18.1`) and the `kueue` 0.18.1 chart; `--kueue-image` and `--kueue-repo` point the script at the mirror. +- **An admin API key** lets the script refresh Hopsworks' settings after each upgrade. + Without it Hopsworks picks up the change within 10 minutes, and a project created in those minutes gets its default queue at its first job. + +## Run the migration + +```sh +helm pull --untar /hopsworks --version <5.3 version> +HELM_EXTRA_ARGS="" \ +HOPSWORKS_URL=https:// \ +HOPSWORKS_API_KEY= \ + hopsworks/charts/hw-kueue/files/migrate-to-external.sh \ + --hopsworks-chart /hopsworks --hopsworks-version <5.3 version> all +``` + +Both Hopsworks upgrades run `helm upgrade --reset-then-reuse-values` with that chart, so the Hopsworks upgrade to 5.3 happens inside the window. +Pass `--force-conflicts` in `HELM_EXTRA_ARGS` if your upgrades need it. +`--namespace` and `--release` name a Hopsworks release other than `hopsworks` in `hopsworks`; `--help` lists every option. + +## What changes + +- **ResourceFlavors lose `topologyName`.** + Topology-aware scheduling is on by default from Kueue 0.14, and no bundled release used it. + A cluster that enabled it on purpose needs its own handling before the move. +- **Cohorts get `parentName`** in place of `parent`. + ClusterQueues keep their quotas, fair sharing, preemption and borrowing settings; Kueue converts `cohort`, `admissionChecks` and the `flavorFungibility` values `Borrow` and `Preempt` to their `v1beta2` forms. +- **Queues are managed with `kubectl`.** + With `hw-kueue.kueue.enabled=false` the chart no longer applies `hw-kueue.clusterQueues`, `cohorts`, `resourceFlavours` or `topologies`. + Queues defined there before the move are kept, but changing those values has no effect. +- **Jobs in the Hopsworks namespace are queued.** + Kueue 0.18 ignores the namespace it runs in, which is now `kueue-system`, so Kueue-labelled jobs in the Hopsworks namespace are queued. +- **The project queue utilisation view fails** until the Hopsworks backend reads Cohorts at `v1beta1`: `GET /project/{id}/kube/queueutilization` returns an error after the move. + Job submission, project queues and quotas are not affected. + +## Verify + +The `verify` phase checks the end state; run it on its own at any time: + +```sh +hopsworks/charts/hw-kueue/files/migrate-to-external.sh verify +``` + +To check by hand: + +```sh +kubectl -n kueue-system get deployment kueue-controller-manager +kubectl get crd -l app.kubernetes.io/name=kueue -o custom-columns=NAME:.metadata.name,STORED:.status.storedVersions +kubectl get clusterqueues,cohorts,resourceflavors +kubectl get localqueues -A +``` + +The controller is ready in `kueue-system`, none runs in the Hopsworks namespace, every CRD stores `["v1beta2"]`, and every queue from before the move is listed. + +With `hw-kueue.kueue.enabled=false` the chart refuses to render while the bundled controller still runs, while Cohorts are stored as `v1alpha1`, or when no Kueue serving `v1beta2` is installed. +`hw-kueue.enforcePrerequisiteCheck=false` turns the check off, for `helm template`. diff --git a/docs/setup_installation/common/helm_chart_values/hw-kueue.md b/docs/setup_installation/common/helm_chart_values/hw-kueue.md index 2c309e58de..b834695790 100644 --- a/docs/setup_installation/common/helm_chart_values/hw-kueue.md +++ b/docs/setup_installation/common/helm_chart_values/hw-kueue.md @@ -1,6 +1,7 @@ # Kueue values { #helm-values-hw-kueue } Values under `hw-kueue` configure Kueue, the job queueing controller, and the queues Hopsworks schedules jobs with. +With `kueue.enabled=false` Hopsworks uses a Kueue installed outside its release; [Kueue outside the Hopsworks namespace](../../admin/kueue_external.md) covers moving an existing cluster there. diff --git a/mkdocs.yml b/mkdocs.yml index 9c9f99cb99..e0dfe3f546 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -303,6 +303,7 @@ nav: - Configure Project Mapping: setup_installation/admin/configure-project-mapping.md - Airflow 3 operator notes: setup_installation/admin/airflow3.md - Kafka 4 and KRaft upgrade notes: setup_installation/admin/kafka4.md + - Kueue outside the Hopsworks namespace: setup_installation/admin/kueue_external.md - Search Index: setup_installation/admin/search_index.md - Upgrading OpenSearch: setup_installation/admin/upgrade_opensearch.md - Monitoring: From 2368688b10b72760c134bb16b109300dc0ee9d66 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Thu, 22 Oct 2026 13:20:59 +0200 Subject: [PATCH 02/17] [HWORKS-3565] Upgrade Kueue from the bundled 0.12.2 to 0.18.1 installed outside the Hopsworks namespace, migrating existing queues and CRDs https://hopsworks.atlassian.net/browse/HWORKS-3565 Clusters whose releases Flux manages. Flux reconciles a HelmRelease back to Git, so the helm upgrades in remove-bundled, install-kueue and enable-hopsworks would be reverted on a Flux-managed cluster. logicalclocks.github.io: the Kueue operator page gains a section on clusters managed by Flux (what to commit at each phase, what the script waits for, the queue export) and lists export-queues. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- .../admin/kueue_external.md | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/setup_installation/admin/kueue_external.md b/docs/setup_installation/admin/kueue_external.md index e01aa874c0..6543e9113b 100644 --- a/docs/setup_installation/admin/kueue_external.md +++ b/docs/setup_installation/admin/kueue_external.md @@ -34,6 +34,7 @@ It runs in phases, and each one is safe to re-run: | `migrate-storage` | Rewrites every object at `v1beta2` and prunes the CRDs' stored versions. | | `enable-hopsworks` | Upgrades Hopsworks with Kueue on and `hw-kueue.kueue.enabled=false`. | | `verify` | Checks the controller, the stored versions, every backed-up queue object and the release values. | +| `export-queues` | Writes the cluster-scoped queue objects as `v1beta2` manifests to `/gitops/kueue-queues.yaml`, for clusters that keep their queues in Git. | A failed run resumes by running the same phase again, or `all`. `--dry-run` prints the commands that change the cluster; with `all` it stops after `remove-bundled`, whose result the later phases depend on. @@ -64,6 +65,29 @@ Both Hopsworks upgrades run `helm upgrade --reset-then-reuse-values` with that c Pass `--force-conflicts` in `HELM_EXTRA_ARGS` if your upgrades need it. `--namespace` and `--release` name a Hopsworks release other than `hopsworks` in `hopsworks`; `--help` lists every option. +## Clusters managed by Flux + +When Flux manages the Hopsworks release, run the script with `--gitops`. +Without it, `preflight` refuses: Flux would undo the script's `helm upgrade` at its next reconcile. +With `--gitops` the script runs no `helm upgrade` and stops at each release change until Flux has applied it: + +| Phase | You commit | The script waits for | +| --- | --- | --- | +| `remove-bundled` | The chart version you upgrade to, and `global._hopsworks.kueue.enabled: false` in the Hopsworks HelmRelease values | The bundled Kueue controller to be gone | +| `install-kueue` | `/gitops/kueue.yaml`: the `kueue-system` namespace, a HelmRepository and HelmRelease `kueue`, with CRDs skipped on install and upgrade | The Kueue controller in `kueue-system` to be ready | +| `enable-hopsworks` | `global._hopsworks.kueue.enabled: true` and `hw-kueue.kueue.enabled: false` | Both values in the Hopsworks release | +| `export-queues` | `/gitops/kueue-queues.yaml` under a Flux Kustomization, after `verify` passes | Nothing | + +The script applies the Kueue CRDs itself and the HelmRelease skips them, so deleting the HelmRelease leaves every queue in place. +`kueue-queues.yaml` holds the ResourceFlavors, Topologies, Cohorts, ClusterQueues, AdmissionChecks and WorkloadPriorityClasses as `kueue.x-k8s.io/v1beta2`; once it is committed, remove `hw-kueue.clusterQueues`, `cohorts`, `resourceFlavours` and `topologies` from the Hopsworks values. +LocalQueues are not exported: Hopsworks creates and deletes them with projects. +Each wait lasts up to `--timeout` (default 60 minutes); running the phase again resumes it. +The manifests use `helm.toolkit.fluxcd.io/v2` and `source.toolkit.fluxcd.io/v1`, which need Flux 2.3 or later. + +```sh +hopsworks/charts/hw-kueue/files/migrate-to-external.sh --gitops all +``` + ## What changes - **ResourceFlavors lose `topologyName`.** From cb82f4e6ff452fea99dca0bccafceb320a69b588 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Thu, 22 Oct 2026 13:39:18 +0200 Subject: [PATCH 03/17] [HWORKS-3565] Upgrade Kueue from the bundled 0.12.2 to 0.18.1 installed outside the Hopsworks namespace, migrating existing queues and CRDs https://hopsworks.atlassian.net/browse/HWORKS-3565 Step-by-step instructions for clusters managed by Flux. logicalclocks.github.io: the Flux section of the Kueue operator page becomes a runbook for --gitops: what to have before starting (Flux 2.3 or later, Git access, the Helm release name and namespace, where the HelmRelease values live, a terminal that survives a dropped connection), then one step per change to commit, with the command, the HelmRelease YAML, the flux reconcile command that applies it at once and what the script waits for, then where the queue manifests go (a Kustomization depending on the Kueue one) and how to resume after a timeout. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- .../admin/kueue_external.md | 134 +++++++++++++++--- 1 file changed, 118 insertions(+), 16 deletions(-) diff --git a/docs/setup_installation/admin/kueue_external.md b/docs/setup_installation/admin/kueue_external.md index 6543e9113b..f387fbce98 100644 --- a/docs/setup_installation/admin/kueue_external.md +++ b/docs/setup_installation/admin/kueue_external.md @@ -68,26 +68,128 @@ Pass `--force-conflicts` in `HELM_EXTRA_ARGS` if your upgrades need it. ## Clusters managed by Flux When Flux manages the Hopsworks release, run the script with `--gitops`. -Without it, `preflight` refuses: Flux would undo the script's `helm upgrade` at its next reconcile. -With `--gitops` the script runs no `helm upgrade` and stops at each release change until Flux has applied it: - -| Phase | You commit | The script waits for | -| --- | --- | --- | -| `remove-bundled` | The chart version you upgrade to, and `global._hopsworks.kueue.enabled: false` in the Hopsworks HelmRelease values | The bundled Kueue controller to be gone | -| `install-kueue` | `/gitops/kueue.yaml`: the `kueue-system` namespace, a HelmRepository and HelmRelease `kueue`, with CRDs skipped on install and upgrade | The Kueue controller in `kueue-system` to be ready | -| `enable-hopsworks` | `global._hopsworks.kueue.enabled: true` and `hw-kueue.kueue.enabled: false` | Both values in the Hopsworks release | -| `export-queues` | `/gitops/kueue-queues.yaml` under a Flux Kustomization, after `verify` passes | Nothing | - -The script applies the Kueue CRDs itself and the HelmRelease skips them, so deleting the HelmRelease leaves every queue in place. -`kueue-queues.yaml` holds the ResourceFlavors, Topologies, Cohorts, ClusterQueues, AdmissionChecks and WorkloadPriorityClasses as `kueue.x-k8s.io/v1beta2`; once it is committed, remove `hw-kueue.clusterQueues`, `cohorts`, `resourceFlavours` and `topologies` from the Hopsworks values. -LocalQueues are not exported: Hopsworks creates and deletes them with projects. -Each wait lasts up to `--timeout` (default 60 minutes); running the phase again resumes it. -The manifests use `helm.toolkit.fluxcd.io/v2` and `source.toolkit.fluxcd.io/v1`, which need Flux 2.3 or later. +Flux keeps running throughout: do not suspend any HelmRelease or Kustomization. +The script never runs `helm upgrade`. +At each step that changes a Helm release it prints the change, you commit it to the Git repository Flux reconciles, and the script waits until Flux has applied it. +The kubectl work (backup, CRDs, restores, the storage rewrite, verification) the script does itself. + +Without `--gitops`, `preflight` and every step that would run `helm upgrade` refuse on a Flux-managed release, because Flux would undo the upgrade at its next reconcile. + +### Before you start with Flux + +Everything under [Before you start](#before-you-start) applies, and: + +- **Flux 2.3 or later.** + The files the script writes use `helm.toolkit.fluxcd.io/v2` and `source.toolkit.fluxcd.io/v1`; `flux version` shows the controllers' versions. +- **Write access to the Git repository** that holds the Hopsworks HelmRelease, and a path in it that a Flux Kustomization applies, for the new Kueue release and, at the end, the queues. +- **The Helm release name and namespace of Hopsworks.** + `flux get helmreleases -A` lists the HelmReleases; the Helm release name is the HelmRelease's `spec.releaseName`, or `-` when that is unset, and `helm list -A` shows it. + Pass both with `--release` and `--namespace` unless they are `hopsworks` and `hopsworks`. +- **Where the HelmRelease takes its values from**: `spec.values`, or a ConfigMap or Secret named in `spec.valuesFrom`. + The value changes below go there. +- **A terminal that survives a dropped connection**, such as `tmux` or `screen`. + The script waits for each commit for up to `--timeout`, 60 minutes by default; pass a longer one, such as `--timeout 4h`, when commits go through review. + +### Step 1: Start the script ```sh -hopsworks/charts/hw-kueue/files/migrate-to-external.sh --gitops all +helm pull --untar /hopsworks --version +HOPSWORKS_URL=https:// \ +HOPSWORKS_API_KEY= \ + hopsworks/charts/hw-kueue/files/migrate-to-external.sh --gitops \ + --release --namespace --backup-dir ./kueue-backup all ``` +`--hopsworks-chart` is not needed: Flux upgrades Hopsworks. +The script runs `preflight` and `backup`, then stops at the first change to commit. + +### Step 2: Turn off the bundled Kueue + +The script prints the change for `remove-bundled`. +In the Hopsworks HelmRelease, set the chart version you upgrade to, and turn Kueue off in its values: + +```yaml +spec: + chart: + spec: + version: "" + values: + global: + _hopsworks: + kueue: + enabled: false +``` + +Commit and push. +To apply it without waiting for the next sync interval, run `flux reconcile helmrelease -n --with-source`. +The script continues once the bundled Kueue controller is gone and the release shows the value. +From here until Step 3 completes no Kueue runs: a job created in that window runs outside quota. + +### Step 3: Install Kueue + +The script replaces the `v1alpha1` Cohort and Topology CRDs, applies the new Kueue CRDs, and writes `/gitops/kueue.yaml`: the `kueue-system` namespace, a HelmRepository and a HelmRelease `kueue`. +Copy that file into a path of the Git repository that a Flux Kustomization applies, such as `clusters//kueue/kueue.yaml`, then commit and push. +`flux reconcile kustomization --with-source` applies it at once. + +Commit the file only when the script asks for it: the HelmRelease skips the CRDs, which the script applies just before. +The script then waits for the Kueue controller, restores the Cohorts and Topologies, and rewrites every Kueue object at `v1beta2`. + +### Step 4: Point Hopsworks at the new Kueue + +The script prints the change for `enable-hopsworks`. +In the Hopsworks HelmRelease values: + +```yaml +spec: + values: + global: + _hopsworks: + kueue: + enabled: true + hw-kueue: + kueue: + enabled: false +``` + +Commit and push; the script continues once the release shows both values. + +### Step 5: Verify and move the queues into Git + +`verify` checks the end state, then `export-queues` writes `/gitops/kueue-queues.yaml`: the ResourceFlavors, Topologies, Cohorts, ClusterQueues, AdmissionChecks and WorkloadPriorityClasses as `kueue.x-k8s.io/v1beta2`, with the server-set fields removed. +Commit it under a Flux Kustomization of its own that depends on the one applying `kueue.yaml`: + +```yaml +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: kueue-queues + namespace: flux-system +spec: + interval: 10m + path: ./clusters//kueue-queues + prune: true + sourceRef: + kind: GitRepository + name: + dependsOn: + - name: +``` + +From then on the queues are changed in Git. +With `prune: true`, a queue removed from Git is deleted from the cluster. +Remove `hw-kueue.clusterQueues`, `cohorts`, `resourceFlavours` and `topologies` from the Hopsworks values: with `hw-kueue.kueue.enabled=false` they have no effect. +LocalQueues are not exported, because Hopsworks creates and deletes them with projects. + +### If a step times out or the script stops + +Each step polls every 15 seconds for up to `--timeout`, then stops with `timed out waiting for ...`. +Run the same command again: every phase checks what is already done and continues from there, and the restores are applied only once. +A single phase can also be run on its own, such as `migrate-to-external.sh --gitops enable-hopsworks`. + +### What the script writes + +The script applies the Kueue CRDs itself and the HelmRelease skips them (`crds: Skip` on install and upgrade), so deleting the HelmRelease leaves every queue in place. + ## What changes - **ResourceFlavors lose `topologyName`.** From b13e655deba5d400bb150601dfe33b98c107db50 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Thu, 22 Oct 2026 21:24:57 +0200 Subject: [PATCH 04/17] [HWORKS-3565] Upgrade Kueue from the bundled 0.12.2 to 0.18.1 installed outside the Hopsworks namespace, migrating existing queues and CRDs https://hopsworks.atlassian.net/browse/HWORKS-3565 Review comments on logicalclocks/hopsworks-helm#2599. logicalclocks.github.io: the Kueue operator page runs the migration script with bash in every command, because helm package stores it without the execute bit and a direct call fails with permission denied. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- docs/setup_installation/admin/kueue_external.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/setup_installation/admin/kueue_external.md b/docs/setup_installation/admin/kueue_external.md index f387fbce98..61ab8ca8c5 100644 --- a/docs/setup_installation/admin/kueue_external.md +++ b/docs/setup_installation/admin/kueue_external.md @@ -21,7 +21,7 @@ A fresh install that wants an external Kueue installs it first and sets `hw-kueu ## What the migration does -The chart ships `charts/hw-kueue/files/migrate-to-external.sh`, which moves the cluster to Kueue 0.18.1 and keeps every Kueue object: ClusterQueues, Cohorts, ResourceFlavors, Topologies, LocalQueues, priority classes and Workloads. +The chart ships `charts/hw-kueue/files/migrate-to-external.sh`; run it with `bash`, because `helm package` stores files without the execute bit. The script moves the cluster to Kueue 0.18.1 and keeps every Kueue object: ClusterQueues, Cohorts, ResourceFlavors, Topologies, LocalQueues, priority classes and Workloads. It runs in phases, and each one is safe to re-run: | Phase | What it does | @@ -57,7 +57,7 @@ helm pull --untar /hopsworks --version <5.3 version> HELM_EXTRA_ARGS="" \ HOPSWORKS_URL=https:// \ HOPSWORKS_API_KEY= \ - hopsworks/charts/hw-kueue/files/migrate-to-external.sh \ + bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh \ --hopsworks-chart /hopsworks --hopsworks-version <5.3 version> all ``` @@ -96,7 +96,7 @@ Everything under [Before you start](#before-you-start) applies, and: helm pull --untar /hopsworks --version HOPSWORKS_URL=https:// \ HOPSWORKS_API_KEY= \ - hopsworks/charts/hw-kueue/files/migrate-to-external.sh --gitops \ + bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh --gitops \ --release --namespace --backup-dir ./kueue-backup all ``` @@ -184,7 +184,7 @@ LocalQueues are not exported, because Hopsworks creates and deletes them with pr Each step polls every 15 seconds for up to `--timeout`, then stops with `timed out waiting for ...`. Run the same command again: every phase checks what is already done and continues from there, and the restores are applied only once. -A single phase can also be run on its own, such as `migrate-to-external.sh --gitops enable-hopsworks`. +A single phase can also be run on its own, such as `bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh --gitops enable-hopsworks`. ### What the script writes @@ -210,7 +210,7 @@ The script applies the Kueue CRDs itself and the HelmRelease skips them (`crds: The `verify` phase checks the end state; run it on its own at any time: ```sh -hopsworks/charts/hw-kueue/files/migrate-to-external.sh verify +bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh verify ``` To check by hand: From cbdf3b464a8b9652f979956b3ec091238e25ce54 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Thu, 22 Oct 2026 08:38:07 +0200 Subject: [PATCH 05/17] [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 Kueue 0.20.2 is the latest release and stops serving kueue.x-k8s.io/v1beta1, which every Hopsworks Kueue model still used. Kueue 0.18.1 serves both v1beta1 and v1beta2, so the backend moves to v1beta2 first and runs on 0.18.1 and 0.20.2, then the external Kueue installed by HWORKS-3565 is upgraded to 0.20.2 and the bundled one in hw-kueue moves to 0.20.2 as well. Tested on the Proxmox cluster jim-kueue-ui (snurran, 4 GPUs), migrated to 0.18.1 by HWORKS-3565: the upgrade kept all 106 queue objects and verify passed, and the loadtests gave 31 passed, 1 skipped on 0.20.2 (one Ray MNIST run failed in its training script and passed when rerun). logicalclocks.github.io: the Kueue operator page says 5.3 bundles Kueue 0.20.2 and that a cluster on the bundled 0.12 moves to the external Kueue when it upgrades. It adds the upgrade-kueue phase, the managedFields reset, the path for a cluster already on 0.18.1, the mirrors an airgapped cluster needs, the 0.19 and 0.20 default changes, and the prerequisite check for both modes. The queue utilisation gap is gone. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- .../admin/kueue_external.md | 50 ++++++++++++++----- 1 file changed, 37 insertions(+), 13 deletions(-) diff --git a/docs/setup_installation/admin/kueue_external.md b/docs/setup_installation/admin/kueue_external.md index 61ab8ca8c5..1ef14d8ffb 100644 --- a/docs/setup_installation/admin/kueue_external.md +++ b/docs/setup_installation/admin/kueue_external.md @@ -1,14 +1,14 @@ # Operator Notes: Kueue Outside the Hopsworks Namespace -Administrative reference for cluster operators who run Kueue with Hopsworks and want to move it out of the Hopsworks release. +Administrative reference for cluster operators who run Kueue with Hopsworks and upgrade to Hopsworks 5.3. -With `global._hopsworks.kueue.enabled=true`, every Hopsworks release from 4.3 bundles Kueue 0.12.2 in the Hopsworks namespace. -From 5.3, `hw-kueue.kueue.enabled=false` makes Hopsworks use a Kueue installed outside its release instead: release `kueue` in namespace `kueue-system`, installed with `fullnameOverride: kueue`. +With `global._hopsworks.kueue.enabled=true`, Hopsworks releases 4.3 to 5.2 bundle Kueue 0.12.2 in the Hopsworks namespace. +Hopsworks 5.3 bundles Kueue 0.20.2, and its backend reads Kueue objects as `kueue.x-k8s.io/v1beta2`, which Kueue 0.12 does not serve. +A cluster that runs the bundled Kueue 0.12 therefore moves to a Kueue installed outside the Hopsworks release when it upgrades to 5.3: release `kueue` in namespace `kueue-system`, installed with `fullnameOverride: kueue`, which Hopsworks uses with `hw-kueue.kueue.enabled=false`. Hopsworks keeps its RBAC bindings to the Kueue roles and keeps queueing jobs through Kueue. -The move is optional. -A cluster that keeps the bundled Kueue needs nothing. -A fresh install that wants an external Kueue installs it first and sets `hw-kueue.kueue.enabled=false`; this page is for clusters that already run the bundled one. +A cluster without Kueue needs nothing. +A fresh 5.3 install gets Kueue 0.20.2 in the Hopsworks namespace, or uses an external Kueue installed first with `hw-kueue.kueue.enabled=false`. ## Why `helm upgrade` alone cannot do it @@ -21,7 +21,8 @@ A fresh install that wants an external Kueue installs it first and sets `hw-kueu ## What the migration does -The chart ships `charts/hw-kueue/files/migrate-to-external.sh`; run it with `bash`, because `helm package` stores files without the execute bit. The script moves the cluster to Kueue 0.18.1 and keeps every Kueue object: ClusterQueues, Cohorts, ResourceFlavors, Topologies, LocalQueues, priority classes and Workloads. +The chart ships `charts/hw-kueue/files/migrate-to-external.sh`; run it with `bash`, because `helm package` stores files without the execute bit. The script keeps every Kueue object: ClusterQueues, Cohorts, ResourceFlavors, Topologies, LocalQueues, priority classes and Workloads. +It moves the cluster to Kueue 0.18.1, the last release that serves both `v1beta1` and `v1beta2`, rewrites every object as `v1beta2`, then upgrades to Kueue 0.20.2. It runs in phases, and each one is safe to re-run: | Phase | What it does | @@ -31,11 +32,14 @@ It runs in phases, and each one is safe to re-run: | `remove-bundled` | Upgrades Hopsworks with Kueue off, which removes the bundled controller, its webhooks and its ClusterRoles. CRDs and queue objects stay. | | `replace-alpha-crds` | Snapshots the Cohorts, Topologies and ResourceFlavors, then replaces the `v1alpha1` Cohort and Topology CRDs. | | `install-kueue` | Applies the Kueue 0.18.1 CRDs, installs release `kueue` in `kueue-system` and restores the replaced objects. | -| `migrate-storage` | Rewrites every object at `v1beta2` and prunes the CRDs' stored versions. | +| `migrate-storage` | Rewrites every object at `v1beta2`, prunes the CRDs' stored versions and resets `managedFields` entries recorded at older versions. | | `enable-hopsworks` | Upgrades Hopsworks with Kueue on and `hw-kueue.kueue.enabled=false`. | -| `verify` | Checks the controller, the stored versions, every backed-up queue object and the release values. | +| `upgrade-kueue` | Upgrades release `kueue` to Kueue 0.20.2 (`--kueue-target`). Refuses while any Kueue CRD stores a version other than `v1beta2`, and refuses a downgrade. Resets the `managedFields` recorded at `v1beta1`, which Kueue 0.20 can no longer convert. | +| `verify` | Checks the controller and its version, the stored versions, every backed-up queue object and the release values. | | `export-queues` | Writes the cluster-scoped queue objects as `v1beta2` manifests to `/gitops/kueue-queues.yaml`, for clusters that keep their queues in Git. | +The script applies the Kueue CRDs itself and installs release `kueue` without them, so `helm uninstall kueue` leaves the CRDs, and every queue, in place. + A failed run resumes by running the same phase again, or `all`. `--dry-run` prints the commands that change the cluster; with `all` it stops after `remove-bundled`, whose result the later phases depend on. @@ -46,7 +50,8 @@ A failed run resumes by running the same phase again, or `all`. From `remove-bundled` to `install-kueue` no Kueue runs, so a job created then runs outside quota. Workloads already admitted keep running, and Kueue 0.18.1 takes them over. - **Tools**: `kubectl`, `helm` 3.20 or later, `jq` and `yq` v4. -- **Airgapped clusters** mirror `registry.k8s.io/kueue/kueue:v0.18.1` (the default image is `docker.hops.works/registry.k8s.io/kueue/kueue:v0.18.1`) and the `kueue` 0.18.1 chart; `--kueue-image` and `--kueue-repo` point the script at the mirror. +- **Airgapped clusters** mirror the images `registry.k8s.io/kueue/kueue:v0.18.1` and `v0.20.2` (the default repository is `docker.hops.works/registry.k8s.io/kueue/kueue`) and the `kueue` 0.18.1 and 0.20.2 charts; `--kueue-image` and `--kueue-repo` point the script at the mirror. + `--kueue-repo` takes a chart repository URL or an `oci://` path, such as `oci://registry.k8s.io/kueue/charts`. - **An admin API key** lets the script refresh Hopsworks' settings after each upgrade. Without it Hopsworks picks up the change within 10 minutes, and a project created in those minutes gets its default queue at its first job. @@ -63,8 +68,18 @@ HOPSWORKS_API_KEY= \ Both Hopsworks upgrades run `helm upgrade --reset-then-reuse-values` with that chart, so the Hopsworks upgrade to 5.3 happens inside the window. Pass `--force-conflicts` in `HELM_EXTRA_ARGS` if your upgrades need it. +`upgrade-kueue` runs after `enable-hopsworks`, because a Hopsworks backend older than 5.3 reads Kueue as `v1beta1`, which Kueue 0.20 no longer serves. `--namespace` and `--release` name a Hopsworks release other than `hopsworks` in `hopsworks`; `--help` lists every option. +### A cluster already on Kueue 0.18.1 + +A cluster moved to Kueue 0.18.1 in `kueue-system` by an earlier version of the script runs only the last two phases, after Hopsworks is on 5.3: + +```sh +bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh upgrade-kueue +bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh verify +``` + ## Clusters managed by Flux When Flux manages the Hopsworks release, run the script with `--gitops`. @@ -200,10 +215,15 @@ The script applies the Kueue CRDs itself and the HelmRelease skips them (`crds: - **Queues are managed with `kubectl`.** With `hw-kueue.kueue.enabled=false` the chart no longer applies `hw-kueue.clusterQueues`, `cohorts`, `resourceFlavours` or `topologies`. Queues defined there before the move are kept, but changing those values has no effect. + Write queue objects as `kueue.x-k8s.io/v1beta2`: Kueue 0.20 serves no other version. - **Jobs in the Hopsworks namespace are queued.** Kueue 0.18 ignores the namespace it runs in, which is now `kueue-system`, so Kueue-labelled jobs in the Hopsworks namespace are queued. -- **The project queue utilisation view fails** until the Hopsworks backend reads Cohorts at `v1beta1`: `GET /project/{id}/kube/queueutilization` returns an error after the move. - Job submission, project queues and quotas are not affected. +- **Workloads must be ready within 30 minutes.** + Kueue 0.19 turns `waitForPodsReady` on by default: a workload whose pods are not all ready 30 minutes after admission is evicted and queued again. +- **Ray sidecars count against quota.** + From Kueue 0.19 the Ray autoscaler and job-submitter sidecars are charged to the head, so a queue sized exactly for its Ray heads may need more CPU and memory. +- **Workloads without a priority class get `default`.** + From Kueue 0.20 such a workload gets the WorkloadPriorityClass named `default`, when one exists. ## Verify @@ -224,5 +244,9 @@ kubectl get localqueues -A The controller is ready in `kueue-system`, none runs in the Hopsworks namespace, every CRD stores `["v1beta2"]`, and every queue from before the move is listed. -With `hw-kueue.kueue.enabled=false` the chart refuses to render while the bundled controller still runs, while Cohorts are stored as `v1alpha1`, or when no Kueue serving `v1beta2` is installed. +The chart checks the cluster before it renders: + +- With `hw-kueue.kueue.enabled=false`, it refuses while the bundled controller still runs, while Cohorts are stored as `v1alpha1`, or when no Kueue serving `v1beta2` is installed. +- With `hw-kueue.kueue.enabled=true`, it refuses when the Kueue CRDs exist and the Hopsworks release does not own them: those of the bundled Kueue 0.12, or those of an external Kueue. + `hw-kueue.enforcePrerequisiteCheck=false` turns the check off, for `helm template`. From 937115bb656a849f3a247583ee69271375c38f94 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Thu, 22 Oct 2026 09:08:39 +0200 Subject: [PATCH 06/17] [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 Spark jobs on Kueue. Kueue never managed Hopsworks Spark jobs: they run as Spark Operator SparkApplications, which Kueue admits only through its SparkApplication integration, alpha since 0.17 and off by default (SparkApplicationIntegration gate, sparkoperator.k8s.io/sparkapplication commented out of the default frameworks). The integration reserves quota for the driver plus spec.executor.instances and refuses dynamicAllocation.enabled unless the job is elastic, and even an elastic job leaves the executors its driver adds uncharged. So the integration is turned on and, while Kueue is on, Spark jobs run with static executors only. logicalclocks.github.io: the Kueue details page gains a Spark jobs section (static executors only, error 130082, defaults, jobs saved before Kueue, notebooks unaffected) and its examples move to kueue.x-k8s.io/v1beta2 (cohortName, no topologyName on the default flavor). The scheduler page, the Spark and PySpark job guides and the Kueue operator page say Spark jobs are queued with static executors. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- .../admin/kueue_external.md | 5 +++ docs/user_guides/projects/jobs/pyspark_job.md | 3 +- docs/user_guides/projects/jobs/spark_job.md | 3 +- .../projects/scheduling/kube_scheduler.md | 2 +- .../projects/scheduling/kueue_details.md | 33 ++++++++++++------- 5 files changed, 32 insertions(+), 14 deletions(-) diff --git a/docs/setup_installation/admin/kueue_external.md b/docs/setup_installation/admin/kueue_external.md index 1ef14d8ffb..16a6c3a87e 100644 --- a/docs/setup_installation/admin/kueue_external.md +++ b/docs/setup_installation/admin/kueue_external.md @@ -218,6 +218,11 @@ The script applies the Kueue CRDs itself and the HelmRelease skips them (`crds: Write queue objects as `kueue.x-k8s.io/v1beta2`: Kueue 0.20 serves no other version. - **Jobs in the Hopsworks namespace are queued.** Kueue 0.18 ignores the namespace it runs in, which is now `kueue-system`, so Kueue-labelled jobs in the Hopsworks namespace are queued. +- **Spark jobs are queued, with static executors only.** + The script turns on Kueue's SparkApplication integration, so Spark jobs get quota and queueing. + Kueue reserves quota for a fixed number of executors, so Hopsworks refuses a Spark job with dynamic allocation on a Kueue cluster. + Spark jobs saved before the upgrade have dynamic allocation on: turn it off and set the number of executors before running them. + Spark in notebooks is unchanged. - **Workloads must be ready within 30 minutes.** Kueue 0.19 turns `waitForPodsReady` on by default: a workload whose pods are not all ready 30 minutes after admission is evicted and queued again. - **Ray sidecars count against quota.** diff --git a/docs/user_guides/projects/jobs/pyspark_job.md b/docs/user_guides/projects/jobs/pyspark_job.md index fcac3821fa..85d1514e61 100644 --- a/docs/user_guides/projects/jobs/pyspark_job.md +++ b/docs/user_guides/projects/jobs/pyspark_job.md @@ -108,6 +108,7 @@ Resource allocation for the Spark driver and executors can be configured, also t Raise it when executors are OOMKilled. - `Dynamic/Static`: Run the Spark application in static or dynamic allocation mode (see [spark docs](https://spark.apache.org/docs/latest/configuration.html#dynamic-allocation) for details). + On a cluster that schedules jobs with Kueue, only static is allowed: a job with dynamic allocation is refused, and new jobs default to static ([Spark jobs and Kueue](../scheduling/kueue_details.md#spark-jobs)).

@@ -246,7 +247,7 @@ The following table describes the job configuration parameters for a PYSPARK job | `conf['spark.executor.memory']` | int | Memory allocated per executor (in MB) | `4096` | | `conf['spark.driver.memoryOverheadFactor']` | float | Fraction of the driver memory added to the driver pod as non-heap overhead. `null` keeps Spark's default: 0.10 for Spark jobs, 0.40 for PySpark | `null` | | `conf['spark.executor.memoryOverheadFactor']` | float | Fraction of the executor memory added to each executor pod as non-heap overhead. `null` keeps Spark's default: 0.10 for Spark jobs, 0.40 for PySpark | `null` | -| `conf['spark.dynamicAllocation.enabled']` | boolean | Enable dynamic allocation of executors | `true` | +| `conf['spark.dynamicAllocation.enabled']` | boolean | Enable dynamic allocation of executors. Must be `false` on a cluster with Kueue | `true`, `false` with Kueue | | `conf['spark.dynamicAllocation.minExecutors']` | int | Minimum number of executors with dynamic allocation | `1` | | `conf['spark.dynamicAllocation.maxExecutors']` | int | Maximum number of executors with dynamic allocation | `2` | | `conf['spark.dynamicAllocation.initialExecutors']` | int | Initial number of executors with dynamic allocation | `1` | diff --git a/docs/user_guides/projects/jobs/spark_job.md b/docs/user_guides/projects/jobs/spark_job.md index 342ef4c113..83f26a04dd 100644 --- a/docs/user_guides/projects/jobs/spark_job.md +++ b/docs/user_guides/projects/jobs/spark_job.md @@ -114,6 +114,7 @@ Resource allocation for the Spark driver and executors can be configured, also t Raise it when executors are OOMKilled. - `Dynamic/Static`: Run the Spark application in static or dynamic allocation mode (see [spark docs](https://spark.apache.org/docs/latest/configuration.html#dynamic-allocation) for details). + On a cluster that schedules jobs with Kueue, only static is allowed: a job with dynamic allocation is refused, and new jobs default to static ([Spark jobs and Kueue](../scheduling/kueue_details.md#spark-jobs)).

@@ -249,7 +250,7 @@ The following table describes the job configuration parameters for a SPARK job. | `conf['spark.executor.memory']` | int | Memory allocated per executor (in MB) | `4096` | | `conf['spark.driver.memoryOverheadFactor']` | float | Fraction of the driver memory added to the driver pod as non-heap overhead. `null` keeps Spark's default: 0.10 for Spark jobs, 0.40 for PySpark | `null` | | `conf['spark.executor.memoryOverheadFactor']` | float | Fraction of the executor memory added to each executor pod as non-heap overhead. `null` keeps Spark's default: 0.10 for Spark jobs, 0.40 for PySpark | `null` | -| `conf['spark.dynamicAllocation.enabled']` | boolean | Enable dynamic allocation of executors | `true` | +| `conf['spark.dynamicAllocation.enabled']` | boolean | Enable dynamic allocation of executors. Must be `false` on a cluster with Kueue | `true`, `false` with Kueue | | `conf['spark.dynamicAllocation.minExecutors']` | int | Minimum number of executors with dynamic allocation | `1` | | `conf['spark.dynamicAllocation.maxExecutors']` | int | Maximum number of executors with dynamic allocation | `2` | | `conf['spark.dynamicAllocation.initialExecutors']` | int | Initial number of executors with dynamic allocation | `1` | diff --git a/docs/user_guides/projects/scheduling/kube_scheduler.md b/docs/user_guides/projects/scheduling/kube_scheduler.md index 59817b64e0..5f59ebe6bc 100644 --- a/docs/user_guides/projects/scheduling/kube_scheduler.md +++ b/docs/user_guides/projects/scheduling/kube_scheduler.md @@ -10,7 +10,7 @@ Hopsworks allows users to configure some Kubernetes scheduler abstractions, such Hopsworks also supports additional scheduling abstractions backed by Kueue. This includes [Queues](https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/), [Cohorts](https://kueue.sigs.k8s.io/docs/concepts/cohort/) and [Topologies](https://kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling/). All these scheduling abstractions are supported in jobs, jupyter notebooks and model deployments. -Kueue abstractions however, are currently not supported for Spark jobs. +Spark jobs run in Kueue queues with static executors only: see [Spark jobs](./kueue_details.md#spark-jobs). Hopsworks Admins can control which labels and priority classes can be used in the cluster (see [Admin configuration](#admin-configuration) section) and by which project (see [Project Configuration](#project-configuration) section) diff --git a/docs/user_guides/projects/scheduling/kueue_details.md b/docs/user_guides/projects/scheduling/kueue_details.md index a922cb7ad5..e772666611 100644 --- a/docs/user_guides/projects/scheduling/kueue_details.md +++ b/docs/user_guides/projects/scheduling/kueue_details.md @@ -10,9 +10,20 @@ Hopsworks provides the integration with Kueue to provide the additional scheduli Hopsworks currently acts only as a "reader" to the Kueue abstractions and currently does not manage the lifecycle of Kueue abstraction with the exception of the default localqueue for each namespace. All the other abstractions are expected to be managed by the administrators of Hopsworks, directly on the Kubernetes cluster. -However Hopsworks and Kueue integration currently only supports frameworks python and ray for jobs, notebooks and model deployments. +Kueue schedules Python and Ray jobs, Spark jobs, notebooks and model deployments. The same queues are also used for Hopsworks internal jobs (zipping, git operations, python library installation). -Spark is currently not supported, and thus will not be managed by Kueue for scheduling, and instead it will bypass the queues setup (important to note when thinking about queue quotas) and instead are managed directly by the Kubernetes Scheduler. + +### Spark jobs + +Spark jobs run as Spark Operator `SparkApplication`s, which Kueue admits through its SparkApplication integration. +Kueue reserves quota for the driver and a fixed number of executors, so on a cluster with Kueue a Spark job runs with static executors: + +- `spark.dynamicAllocation.enabled` must be `false`. A Spark job with dynamic allocation is refused when it is saved and when it is started (error code 130082). +- `spark.executor.instances` sets the number of executors Kueue reserves quota for. +- New Spark jobs default to static executors, as many as the dynamic-allocation maximum. +- Spark jobs created before Kueue was enabled have dynamic allocation on, the earlier default. Turn it off and set the number of executors before running them. + +Spark in Jupyter notebooks runs in client mode and is not managed by Kueue, so it keeps dynamic allocation. ### Resource flavors @@ -21,17 +32,17 @@ The resource flavor defines the resources that a queue will later manage. Hopsworks helm chart installs and uses a default ResourceFlavor ```yaml -apiVersion: kueue.x-k8s.io/v1beta1 +apiVersion: kueue.x-k8s.io/v1beta2 kind: ResourceFlavor metadata: name: default-flavor spec: nodeLabels: cloud.provider.com/region: europe - topologyName: default ``` -Node labels filter the available nodes to this resource flavor and is required for [topologies](#topologies) +Node labels filter the available nodes to this resource flavor. +A flavor that names a topology with `topologyName` makes Kueue place pods by that [topology](#topologies). ### Cluster Queues @@ -39,12 +50,12 @@ Node labels filter the available nodes to this resource flavor and is required f The default hopsworks queue looks like: ```yaml -apiVersion: kueue.x-k8s.io/v1beta1 +apiVersion: kueue.x-k8s.io/v1beta2 kind: ClusterQueue metadata: name: other spec: - cohort: cluster + cohortName: cluster namespaceSelector: {} preemption: borrowWithinCohort: @@ -90,7 +101,7 @@ An administrator can define the default queue for projects user jobs and system Hopsworks defines a default `cluster` cohort ```yaml -apiVersion: kueue.x-k8s.io/v1alpha1 +apiVersion: kueue.x-k8s.io/v1beta2 kind: Cohort metadata: name: cluster @@ -114,7 +125,7 @@ spec: nominalQuota: 50 ``` -Cohorts can contain other cohorts and thus you can create a hierarchy of cohorts. +Cohorts can contain other cohorts, through `parentName`, and thus you can create a hierarchy of cohorts. Cohorts can set [fair sharing weight](https://kueue.sigs.k8s.io/docs/concepts/admission_fair_sharing/) where using ```yaml @@ -130,7 +141,7 @@ in the definition of a cohort, the user can control a priority towards borrowing Hopsworks defines a default topology: ```yaml -apiVersion: kueue.x-k8s.io/v1alpha1 +apiVersion: kueue.x-k8s.io/v1beta2 kind: Topology metadata: name: default @@ -141,7 +152,7 @@ spec: - nodeLabel: kubernetes.io/hostname ``` -The topology is defined in the Resource Flavor used by a Cluster Queue. +A Cluster Queue uses a topology through the `topologyName` of its Resource Flavor; the default flavor names none. When creating a new job, the user can select a topology unit for the job to run in and thus decide if all pods of a job should run on the same hostname, in the same zone or in the same region. The user can select the topology for jobs, notebooks and deployments from `Advanced options`, in the `Scheduler` section of the full configuration page. From 2eb70b759f7641dba81fce9941fccf0f5c41e790 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Thu, 22 Oct 2026 11:10:08 +0200 Subject: [PATCH 07/17] [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 Spark jobs reach Kueue. With the SparkApplication integration on, a test run still showed no Spark Workload: the backend gave Spark jobs no queue, so their SparkApplications carried no queue-name label and Kueue ignored them. With a queue, Kueue admitted the job and then could not start it: unsuspending sets the driver and executor pod templates, and the Spark Operator 2.5.1 webhook refused that with "pod template feature requires Spark version 3.0.0 or higher", because it compares spec.sparkVersion as semver and ranks an invalid one below every valid one, and Hopsworks writes 4.1.3.0. On jim-kueue-ui, Kueue 0.20.2 with the integration on, the Kueue, Spark job, default job configuration, rerun and job failure loadtests gave 14 passed, and Kueue admitted 9 SparkApplication Workloads into ClusterQueue other. logicalclocks.github.io: the Kueue details page says Spark jobs go to the project's default queue unless they select another, and that a Spark job saved without a queue gets the default when it starts. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- docs/user_guides/projects/scheduling/kueue_details.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/user_guides/projects/scheduling/kueue_details.md b/docs/user_guides/projects/scheduling/kueue_details.md index e772666611..e14d7f7856 100644 --- a/docs/user_guides/projects/scheduling/kueue_details.md +++ b/docs/user_guides/projects/scheduling/kueue_details.md @@ -16,6 +16,7 @@ The same queues are also used for Hopsworks internal jobs (zipping, git operatio ### Spark jobs Spark jobs run as Spark Operator `SparkApplication`s, which Kueue admits through its SparkApplication integration. +They go to the project's default queue unless the job selects another; a Spark job saved without a queue, as all were before Kueue managed Spark, gets the default queue when it starts. Kueue reserves quota for the driver and a fixed number of executors, so on a cluster with Kueue a Spark job runs with static executors: - `spark.dynamicAllocation.enabled` must be `false`. A Spark job with dynamic allocation is refused when it is saved and when it is started (error code 130082). From 384150e0fc3298541d9aaf242ddd66ebe453e294 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Thu, 22 Oct 2026 13:20:51 +0200 Subject: [PATCH 08/17] [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 Fixes from a Codex review of the change set (nine findings and one suspicion, each checked against the code before it was fixed). logicalclocks.github.io: the Kueue operator page says the managedFields cleanup drops only old entries, that the 0.20.2 chart comes from the upstream OCI registry (--kueue-target-repo), and which flags point an airgapped cluster at its mirrors. Reviewed-by: OpenAI Codex (codex-plugin-cc 1.0.6) Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- docs/setup_installation/admin/kueue_external.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/setup_installation/admin/kueue_external.md b/docs/setup_installation/admin/kueue_external.md index 16a6c3a87e..612f7a59de 100644 --- a/docs/setup_installation/admin/kueue_external.md +++ b/docs/setup_installation/admin/kueue_external.md @@ -32,9 +32,9 @@ It runs in phases, and each one is safe to re-run: | `remove-bundled` | Upgrades Hopsworks with Kueue off, which removes the bundled controller, its webhooks and its ClusterRoles. CRDs and queue objects stay. | | `replace-alpha-crds` | Snapshots the Cohorts, Topologies and ResourceFlavors, then replaces the `v1alpha1` Cohort and Topology CRDs. | | `install-kueue` | Applies the Kueue 0.18.1 CRDs, installs release `kueue` in `kueue-system` and restores the replaced objects. | -| `migrate-storage` | Rewrites every object at `v1beta2`, prunes the CRDs' stored versions and resets `managedFields` entries recorded at older versions. | +| `migrate-storage` | Rewrites every object at `v1beta2`, prunes the CRDs' stored versions and drops the `managedFields` entries recorded at older versions. | | `enable-hopsworks` | Upgrades Hopsworks with Kueue on and `hw-kueue.kueue.enabled=false`. | -| `upgrade-kueue` | Upgrades release `kueue` to Kueue 0.20.2 (`--kueue-target`). Refuses while any Kueue CRD stores a version other than `v1beta2`, and refuses a downgrade. Resets the `managedFields` recorded at `v1beta1`, which Kueue 0.20 can no longer convert. | +| `upgrade-kueue` | Upgrades release `kueue` to Kueue 0.20.2 (`--kueue-target`). Refuses while any Kueue CRD stores a version other than `v1beta2`, and refuses a downgrade. Drops the `managedFields` entries recorded at `v1beta1`, which Kueue 0.20 can no longer convert. Takes the chart from `oci://registry.k8s.io/kueue/charts` (`--kueue-target-repo`). | | `verify` | Checks the controller and its version, the stored versions, every backed-up queue object and the release values. | | `export-queues` | Writes the cluster-scoped queue objects as `v1beta2` manifests to `/gitops/kueue-queues.yaml`, for clusters that keep their queues in Git. | @@ -50,8 +50,8 @@ A failed run resumes by running the same phase again, or `all`. From `remove-bundled` to `install-kueue` no Kueue runs, so a job created then runs outside quota. Workloads already admitted keep running, and Kueue 0.18.1 takes them over. - **Tools**: `kubectl`, `helm` 3.20 or later, `jq` and `yq` v4. -- **Airgapped clusters** mirror the images `registry.k8s.io/kueue/kueue:v0.18.1` and `v0.20.2` (the default repository is `docker.hops.works/registry.k8s.io/kueue/kueue`) and the `kueue` 0.18.1 and 0.20.2 charts; `--kueue-image` and `--kueue-repo` point the script at the mirror. - `--kueue-repo` takes a chart repository URL or an `oci://` path, such as `oci://registry.k8s.io/kueue/charts`. +- **Airgapped clusters** mirror the images `registry.k8s.io/kueue/kueue:v0.18.1` and `v0.20.2` (the default repository is `docker.hops.works/registry.k8s.io/kueue/kueue`) and the `kueue` 0.18.1 and 0.20.2 charts; `--kueue-image`, `--kueue-repo` and `--kueue-target-repo` point the script at the mirror. + `--kueue-repo` (the 0.18.1 chart) and `--kueue-target-repo` (the 0.20.2 chart) take a chart repository URL or an `oci://` path. - **An admin API key** lets the script refresh Hopsworks' settings after each upgrade. Without it Hopsworks picks up the change within 10 minutes, and a project created in those minutes gets its default queue at its first job. From 9cdd58bc4532f04a3770601ef3d5041ed90b2ba7 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Thu, 22 Oct 2026 13:27:49 +0200 Subject: [PATCH 09/17] [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 Rebased on HWORKS-3565 with its Flux runbook. logicalclocks.github.io: the Flux runbook on the Kueue operator page gains Step 5, upgrade-kueue: the script applies the 0.20.2 CRDs and rewrites the Kueue HelmRelease for 0.20.2 (chart from the upstream OCI registry, Spark settings, a post-render patch that keeps the chart's templated CRDs out of the release), the operator replaces the file committed in Step 3, and the script waits for the controller to run 0.20.2. Verification and the queue export become Step 6. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- docs/setup_installation/admin/kueue_external.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/setup_installation/admin/kueue_external.md b/docs/setup_installation/admin/kueue_external.md index 612f7a59de..c7fd07e2fd 100644 --- a/docs/setup_installation/admin/kueue_external.md +++ b/docs/setup_installation/admin/kueue_external.md @@ -168,7 +168,13 @@ spec: Commit and push; the script continues once the release shows both values. -### Step 5: Verify and move the queues into Git +### Step 5: Upgrade Kueue to 0.20.2 + +The script applies the 0.20.2 CRDs and rewrites `/gitops/kueue.yaml` for 0.20.2: the chart from `oci://registry.k8s.io/kueue/charts`, the Spark manager configuration and feature gate, and a post-render patch that keeps the chart's CRDs out of the release. +Replace the file you committed in Step 3 with it, then commit and push. +The script continues once the Kueue controller runs 0.20.2. + +### Step 6: Verify and move the queues into Git `verify` checks the end state, then `export-queues` writes `/gitops/kueue-queues.yaml`: the ResourceFlavors, Topologies, Cohorts, ClusterQueues, AdmissionChecks and WorkloadPriorityClasses as `kueue.x-k8s.io/v1beta2`, with the server-set fields removed. Commit it under a Flux Kustomization of its own that depends on the one applying `kueue.yaml`: @@ -204,6 +210,7 @@ A single phase can also be run on its own, such as `bash hopsworks/charts/hw-kue ### What the script writes The script applies the Kueue CRDs itself and the HelmRelease skips them (`crds: Skip` on install and upgrade), so deleting the HelmRelease leaves every queue in place. +The 0.20.2 chart ships its CRDs as templates, which `crds: Skip` does not cover, so its HelmRelease also carries a kustomize post-render patch that removes them from what Flux installs. ## What changes From 5c629bc7a68aeec284042fae45f0f482e0936c0a Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Fri, 23 Oct 2026 10:16:20 +0200 Subject: [PATCH 10/17] [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 Kueue 0.20.3. The upgrade-kueue entry states that it waits for the Hopsworks release to finish deploying, keeps release kueue's own values and refuses a manager configuration that is not v1beta2. Reviewed-by: OpenAI Codex (codex-plugin-cc 1.0.6) Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- .../admin/kueue_external.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/setup_installation/admin/kueue_external.md b/docs/setup_installation/admin/kueue_external.md index c7fd07e2fd..9e028747d8 100644 --- a/docs/setup_installation/admin/kueue_external.md +++ b/docs/setup_installation/admin/kueue_external.md @@ -3,12 +3,12 @@ Administrative reference for cluster operators who run Kueue with Hopsworks and upgrade to Hopsworks 5.3. With `global._hopsworks.kueue.enabled=true`, Hopsworks releases 4.3 to 5.2 bundle Kueue 0.12.2 in the Hopsworks namespace. -Hopsworks 5.3 bundles Kueue 0.20.2, and its backend reads Kueue objects as `kueue.x-k8s.io/v1beta2`, which Kueue 0.12 does not serve. +Hopsworks 5.3 bundles Kueue 0.20.3, and its backend reads Kueue objects as `kueue.x-k8s.io/v1beta2`, which Kueue 0.12 does not serve. A cluster that runs the bundled Kueue 0.12 therefore moves to a Kueue installed outside the Hopsworks release when it upgrades to 5.3: release `kueue` in namespace `kueue-system`, installed with `fullnameOverride: kueue`, which Hopsworks uses with `hw-kueue.kueue.enabled=false`. Hopsworks keeps its RBAC bindings to the Kueue roles and keeps queueing jobs through Kueue. A cluster without Kueue needs nothing. -A fresh 5.3 install gets Kueue 0.20.2 in the Hopsworks namespace, or uses an external Kueue installed first with `hw-kueue.kueue.enabled=false`. +A fresh 5.3 install gets Kueue 0.20.3 in the Hopsworks namespace, or uses an external Kueue installed first with `hw-kueue.kueue.enabled=false`. ## Why `helm upgrade` alone cannot do it @@ -22,7 +22,7 @@ A fresh 5.3 install gets Kueue 0.20.2 in the Hopsworks namespace, or uses an ext ## What the migration does The chart ships `charts/hw-kueue/files/migrate-to-external.sh`; run it with `bash`, because `helm package` stores files without the execute bit. The script keeps every Kueue object: ClusterQueues, Cohorts, ResourceFlavors, Topologies, LocalQueues, priority classes and Workloads. -It moves the cluster to Kueue 0.18.1, the last release that serves both `v1beta1` and `v1beta2`, rewrites every object as `v1beta2`, then upgrades to Kueue 0.20.2. +It moves the cluster to Kueue 0.18.1, the last release that serves both `v1beta1` and `v1beta2`, rewrites every object as `v1beta2`, then upgrades to Kueue 0.20.3. It runs in phases, and each one is safe to re-run: | Phase | What it does | @@ -34,7 +34,7 @@ It runs in phases, and each one is safe to re-run: | `install-kueue` | Applies the Kueue 0.18.1 CRDs, installs release `kueue` in `kueue-system` and restores the replaced objects. | | `migrate-storage` | Rewrites every object at `v1beta2`, prunes the CRDs' stored versions and drops the `managedFields` entries recorded at older versions. | | `enable-hopsworks` | Upgrades Hopsworks with Kueue on and `hw-kueue.kueue.enabled=false`. | -| `upgrade-kueue` | Upgrades release `kueue` to Kueue 0.20.2 (`--kueue-target`). Refuses while any Kueue CRD stores a version other than `v1beta2`, and refuses a downgrade. Drops the `managedFields` entries recorded at `v1beta1`, which Kueue 0.20 can no longer convert. Takes the chart from `oci://registry.k8s.io/kueue/charts` (`--kueue-target-repo`). | +| `upgrade-kueue` | Upgrades release `kueue` to Kueue 0.20.3 (`--kueue-target`). Refuses while any Kueue CRD stores a version other than `v1beta2`, while the Hopsworks release does not use the external Kueue or has not finished deploying, and refuses a downgrade. Keeps the `kueue` release's own values, and refuses a manager configuration that is not `config.kueue.x-k8s.io/v1beta2` before it changes anything. Drops the `managedFields` entries recorded at `v1beta1`, which Kueue 0.20 can no longer convert. Takes the chart from `oci://registry.k8s.io/kueue/charts` (`--kueue-target-repo`). | | `verify` | Checks the controller and its version, the stored versions, every backed-up queue object and the release values. | | `export-queues` | Writes the cluster-scoped queue objects as `v1beta2` manifests to `/gitops/kueue-queues.yaml`, for clusters that keep their queues in Git. | @@ -50,8 +50,8 @@ A failed run resumes by running the same phase again, or `all`. From `remove-bundled` to `install-kueue` no Kueue runs, so a job created then runs outside quota. Workloads already admitted keep running, and Kueue 0.18.1 takes them over. - **Tools**: `kubectl`, `helm` 3.20 or later, `jq` and `yq` v4. -- **Airgapped clusters** mirror the images `registry.k8s.io/kueue/kueue:v0.18.1` and `v0.20.2` (the default repository is `docker.hops.works/registry.k8s.io/kueue/kueue`) and the `kueue` 0.18.1 and 0.20.2 charts; `--kueue-image`, `--kueue-repo` and `--kueue-target-repo` point the script at the mirror. - `--kueue-repo` (the 0.18.1 chart) and `--kueue-target-repo` (the 0.20.2 chart) take a chart repository URL or an `oci://` path. +- **Airgapped clusters** mirror the images `registry.k8s.io/kueue/kueue:v0.18.1` and `v0.20.3` (the default repository is `docker.hops.works/registry.k8s.io/kueue/kueue`) and the `kueue` 0.18.1 and 0.20.3 charts; `--kueue-image`, `--kueue-repo` and `--kueue-target-repo` point the script at the mirror. + `--kueue-repo` (the 0.18.1 chart) and `--kueue-target-repo` (the 0.20.3 chart) take a chart repository URL or an `oci://` path. - **An admin API key** lets the script refresh Hopsworks' settings after each upgrade. Without it Hopsworks picks up the change within 10 minutes, and a project created in those minutes gets its default queue at its first job. @@ -168,11 +168,11 @@ spec: Commit and push; the script continues once the release shows both values. -### Step 5: Upgrade Kueue to 0.20.2 +### Step 5: Upgrade Kueue to 0.20.3 -The script applies the 0.20.2 CRDs and rewrites `/gitops/kueue.yaml` for 0.20.2: the chart from `oci://registry.k8s.io/kueue/charts`, the Spark manager configuration and feature gate, and a post-render patch that keeps the chart's CRDs out of the release. +The script applies the 0.20.3 CRDs and rewrites `/gitops/kueue.yaml` for 0.20.3: the chart from `oci://registry.k8s.io/kueue/charts`, the Spark manager configuration and feature gate, and a post-render patch that keeps the chart's CRDs out of the release. Replace the file you committed in Step 3 with it, then commit and push. -The script continues once the Kueue controller runs 0.20.2. +The script continues once the Kueue controller runs 0.20.3. ### Step 6: Verify and move the queues into Git @@ -210,7 +210,7 @@ A single phase can also be run on its own, such as `bash hopsworks/charts/hw-kue ### What the script writes The script applies the Kueue CRDs itself and the HelmRelease skips them (`crds: Skip` on install and upgrade), so deleting the HelmRelease leaves every queue in place. -The 0.20.2 chart ships its CRDs as templates, which `crds: Skip` does not cover, so its HelmRelease also carries a kustomize post-render patch that removes them from what Flux installs. +The 0.20.3 chart ships its CRDs as templates, which `crds: Skip` does not cover, so its HelmRelease also carries a kustomize post-render patch that removes them from what Flux installs. ## What changes From bea67f589755d9cc0f60c9e3439171f3a2e16149 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Fri, 23 Oct 2026 16:58:36 +0200 Subject: [PATCH 11/17] [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 The 5.3 chart no longer ships migrate-to-external.sh: a cluster on the bundled Kueue 0.12 moves to the external Kueue 0.18.1 on Hopsworks 5.1, upgrades Hopsworks to 5.3, then upgrades Kueue to 0.20.3 with the 5.1 script's upgrade-kueue phase. docs: kueue_external.md runs the move in those three steps, with the latest 5.1 chart, and the Flux steps follow the same order. all no longer includes upgrade-kueue, which refuses a Hopsworks release older than 5.3. The --kueue-image option for the upstream images is listed. The upgrade guide's Kueue entry says where the script comes from. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- .../admin/kueue_external.md | 78 ++++++++++++------- docs/setup_installation/admin/upgrade.md | 4 +- 2 files changed, 51 insertions(+), 31 deletions(-) diff --git a/docs/setup_installation/admin/kueue_external.md b/docs/setup_installation/admin/kueue_external.md index 9e028747d8..857d951cfe 100644 --- a/docs/setup_installation/admin/kueue_external.md +++ b/docs/setup_installation/admin/kueue_external.md @@ -4,7 +4,7 @@ Administrative reference for cluster operators who run Kueue with Hopsworks and With `global._hopsworks.kueue.enabled=true`, Hopsworks releases 4.3 to 5.2 bundle Kueue 0.12.2 in the Hopsworks namespace. Hopsworks 5.3 bundles Kueue 0.20.3, and its backend reads Kueue objects as `kueue.x-k8s.io/v1beta2`, which Kueue 0.12 does not serve. -A cluster that runs the bundled Kueue 0.12 therefore moves to a Kueue installed outside the Hopsworks release when it upgrades to 5.3: release `kueue` in namespace `kueue-system`, installed with `fullnameOverride: kueue`, which Hopsworks uses with `hw-kueue.kueue.enabled=false`. +A cluster that runs the bundled Kueue 0.12 therefore moves to a Kueue installed outside the Hopsworks release before it upgrades to 5.3, while it runs Hopsworks 5.1: release `kueue` in namespace `kueue-system`, installed with `fullnameOverride: kueue`, which Hopsworks uses with `hw-kueue.kueue.enabled=false`. Hopsworks keeps its RBAC bindings to the Kueue roles and keeps queueing jobs through Kueue. A cluster without Kueue needs nothing. @@ -21,9 +21,10 @@ A fresh 5.3 install gets Kueue 0.20.3 in the Hopsworks namespace, or uses an ext ## What the migration does -The chart ships `charts/hw-kueue/files/migrate-to-external.sh`; run it with `bash`, because `helm package` stores files without the execute bit. The script keeps every Kueue object: ClusterQueues, Cohorts, ResourceFlavors, Topologies, LocalQueues, priority classes and Workloads. -It moves the cluster to Kueue 0.18.1, the last release that serves both `v1beta1` and `v1beta2`, rewrites every object as `v1beta2`, then upgrades to Kueue 0.20.3. -It runs in phases, and each one is safe to re-run: +The 5.1 chart ships `charts/hw-kueue/files/migrate-to-external.sh`; the 5.3 chart does not. Use the latest 5.1 chart, whose `--help` lists the `upgrade-kueue` phase. Run the script with `bash`, because `helm package` stores files without the execute bit. The script keeps every Kueue object: ClusterQueues, Cohorts, ResourceFlavors, Topologies, LocalQueues, priority classes and Workloads. +On 5.1 it moves the cluster to Kueue 0.18.1, the last release that serves both `v1beta1` and `v1beta2`, and rewrites every object as `v1beta2`. +After the Hopsworks upgrade to 5.3, its `upgrade-kueue` phase upgrades Kueue to 0.20.3. +It runs in phases, and each one is safe to re-run; `all` runs every phase except `upgrade-kueue`: | Phase | What it does | | --- | --- | @@ -34,8 +35,8 @@ It runs in phases, and each one is safe to re-run: | `install-kueue` | Applies the Kueue 0.18.1 CRDs, installs release `kueue` in `kueue-system` and restores the replaced objects. | | `migrate-storage` | Rewrites every object at `v1beta2`, prunes the CRDs' stored versions and drops the `managedFields` entries recorded at older versions. | | `enable-hopsworks` | Upgrades Hopsworks with Kueue on and `hw-kueue.kueue.enabled=false`. | -| `upgrade-kueue` | Upgrades release `kueue` to Kueue 0.20.3 (`--kueue-target`). Refuses while any Kueue CRD stores a version other than `v1beta2`, while the Hopsworks release does not use the external Kueue or has not finished deploying, and refuses a downgrade. Keeps the `kueue` release's own values, and refuses a manager configuration that is not `config.kueue.x-k8s.io/v1beta2` before it changes anything. Drops the `managedFields` entries recorded at `v1beta1`, which Kueue 0.20 can no longer convert. Takes the chart from `oci://registry.k8s.io/kueue/charts` (`--kueue-target-repo`). | -| `verify` | Checks the controller and its version, the stored versions, every backed-up queue object and the release values. | +| `upgrade-kueue` | Upgrades release `kueue` to Kueue 0.20.3 (`--kueue-target`). Refuses while the Hopsworks release is older than 5.3, while any Kueue CRD stores a version other than `v1beta2`, while the Hopsworks release does not use the external Kueue or has not finished deploying, and refuses a downgrade. Keeps the `kueue` release's own values, and refuses a manager configuration that is not `config.kueue.x-k8s.io/v1beta2` before it changes anything. Drops the `managedFields` entries recorded at `v1beta1`, which Kueue 0.20 can no longer convert. Takes the chart from `oci://registry.k8s.io/kueue/charts` (`--kueue-target-repo`). | +| `verify` | Checks the controller and its version (0.18.1 before Hopsworks 5.3, 0.20.3 after), the stored versions, every backed-up queue object and the release values. | | `export-queues` | Writes the cluster-scoped queue objects as `v1beta2` manifests to `/gitops/kueue-queues.yaml`, for clusters that keep their queues in Git. | The script applies the Kueue CRDs itself and installs release `kueue` without them, so `helm uninstall kueue` leaves the CRDs, and every queue, in place. @@ -50,6 +51,7 @@ A failed run resumes by running the same phase again, or `all`. From `remove-bundled` to `install-kueue` no Kueue runs, so a job created then runs outside quota. Workloads already admitted keep running, and Kueue 0.18.1 takes them over. - **Tools**: `kubectl`, `helm` 3.20 or later, `jq` and `yq` v4. +- **Kueue images**: the default repository is `docker.hops.works/registry.k8s.io/kueue/kueue`. `--kueue-image registry.k8s.io/kueue/kueue` takes the upstream images. - **Airgapped clusters** mirror the images `registry.k8s.io/kueue/kueue:v0.18.1` and `v0.20.3` (the default repository is `docker.hops.works/registry.k8s.io/kueue/kueue`) and the `kueue` 0.18.1 and 0.20.3 charts; `--kueue-image`, `--kueue-repo` and `--kueue-target-repo` point the script at the mirror. `--kueue-repo` (the 0.18.1 chart) and `--kueue-target-repo` (the 0.20.3 chart) take a chart repository URL or an `oci://` path. - **An admin API key** lets the script refresh Hopsworks' settings after each upgrade. @@ -57,28 +59,37 @@ A failed run resumes by running the same phase again, or `all`. ## Run the migration -```sh -helm pull --untar /hopsworks --version <5.3 version> -HELM_EXTRA_ARGS="" \ -HOPSWORKS_URL=https:// \ -HOPSWORKS_API_KEY= \ - bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh \ - --hopsworks-chart /hopsworks --hopsworks-version <5.3 version> all -``` +The move runs in three steps. -Both Hopsworks upgrades run `helm upgrade --reset-then-reuse-values` with that chart, so the Hopsworks upgrade to 5.3 happens inside the window. -Pass `--force-conflicts` in `HELM_EXTRA_ARGS` if your upgrades need it. -`upgrade-kueue` runs after `enable-hopsworks`, because a Hopsworks backend older than 5.3 reads Kueue as `v1beta1`, which Kueue 0.20 no longer serves. -`--namespace` and `--release` name a Hopsworks release other than `hopsworks` in `hopsworks`; `--help` lists every option. +1. On Hopsworks 5.1, move the cluster to Kueue 0.18.1: -### A cluster already on Kueue 0.18.1 + ```sh + helm pull --untar /hopsworks --version + HELM_EXTRA_ARGS="" \ + HOPSWORKS_URL=https:// \ + HOPSWORKS_API_KEY= \ + bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh \ + --hopsworks-chart /hopsworks --hopsworks-version all + ``` -A cluster moved to Kueue 0.18.1 in `kueue-system` by an earlier version of the script runs only the last two phases, after Hopsworks is on 5.3: + Both Hopsworks upgrades in `all` run `helm upgrade --reset-then-reuse-values` with that chart. + Pass `--force-conflicts` in `HELM_EXTRA_ARGS` if your upgrades need it. + `verify` at the end of `all` expects Kueue 0.18.1. + Jobs can be submitted again once it passes. -```sh -bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh upgrade-kueue -bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh verify -``` +2. Upgrade Hopsworks to 5.3, keeping `global._hopsworks.kueue.enabled=true` and `hw-kueue.kueue.enabled=false`. + Kueue 0.18.1 serves `v1beta2`, which the 5.3 backend reads. + +3. Upgrade Kueue to 0.20.3, with the script from the 5.1 chart: + + ```sh + bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh upgrade-kueue + bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh verify + ``` + + Until then Kueue 0.18.1 queues jobs, and Spark jobs run without queueing. + +`--namespace` and `--release` name a Hopsworks release other than `hopsworks` in `hopsworks`; `--help` lists every option. ## Clusters managed by Flux @@ -108,7 +119,7 @@ Everything under [Before you start](#before-you-start) applies, and: ### Step 1: Start the script ```sh -helm pull --untar /hopsworks --version +helm pull --untar /hopsworks --version HOPSWORKS_URL=https:// \ HOPSWORKS_API_KEY= \ bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh --gitops \ @@ -121,13 +132,13 @@ The script runs `preflight` and `backup`, then stops at the first change to comm ### Step 2: Turn off the bundled Kueue The script prints the change for `remove-bundled`. -In the Hopsworks HelmRelease, set the chart version you upgrade to, and turn Kueue off in its values: +In the Hopsworks HelmRelease, set the latest 5.1 chart version, and turn Kueue off in its values: ```yaml spec: chart: spec: - version: "" + version: "" values: global: _hopsworks: @@ -167,12 +178,21 @@ spec: ``` Commit and push; the script continues once the release shows both values. +It then runs `verify` and `export-queues` (Step 6) against Kueue 0.18.1 and exits. -### Step 5: Upgrade Kueue to 0.20.3 +### Step 5: Upgrade Hopsworks and Kueue + +Set the Hopsworks HelmRelease's chart version to 5.3, keeping the values from Step 4, then commit and push. +Once Flux has upgraded Hopsworks, run the script's last phase: + +```sh +bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh --gitops \ + --release --namespace --backup-dir ./kueue-backup upgrade-kueue +``` The script applies the 0.20.3 CRDs and rewrites `/gitops/kueue.yaml` for 0.20.3: the chart from `oci://registry.k8s.io/kueue/charts`, the Spark manager configuration and feature gate, and a post-render patch that keeps the chart's CRDs out of the release. Replace the file you committed in Step 3 with it, then commit and push. -The script continues once the Kueue controller runs 0.20.3. +The script continues once the Kueue controller runs 0.20.3; `verify` then checks the end state. ### Step 6: Verify and move the queues into Git diff --git a/docs/setup_installation/admin/upgrade.md b/docs/setup_installation/admin/upgrade.md index 139ec0e4dc..493c447bd5 100644 --- a/docs/setup_installation/admin/upgrade.md +++ b/docs/setup_installation/admin/upgrade.md @@ -244,8 +244,8 @@ From 5.3 the default is 2048, up from 1024, because at 1 GiB the sidecar was OOM Optional. From 5.3, `hw-kueue.kueue.enabled=false` makes Hopsworks use a Kueue installed as its own release, `kueue` in `kueue-system`, in place of the Kueue 0.12.2 bundled in the Hopsworks namespace. A cluster that runs the bundled Kueue cannot be switched with `helm upgrade` alone: Kueue 0.12 stores Cohorts and Topologies as `v1alpha1`, which no later Kueue reads. -The chart's `migrate-to-external.sh` moves it across and keeps every queue. -Follow [Kueue outside the Hopsworks namespace](kueue_external.md), and stop submitting jobs while it runs. +The 5.1 chart's `migrate-to-external.sh` moves it across on 5.1 and keeps every queue; after the upgrade to 5.3 it upgrades Kueue to 0.20.3. +Follow [Kueue outside the Hopsworks namespace](kueue_external.md), and stop submitting jobs while the move runs. ## Changes from 5.2.2 { #upgrade-changes-from-522 } From 8d2d55348ac580c220091f7d61f65a237f1cc5c0 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Fri, 23 Oct 2026 17:16:26 +0200 Subject: [PATCH 12/17] [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 The Kueue 0.20 changes ship in a 5.2.x patch release as well as in 5.3. docs: kueue_external.md and the upgrade guide's Kueue entry name the target as a Hopsworks release that bundles Kueue 0.20, a 5.2.x patch release or 5.3, in place of 5.3, and upgrade-kueue's refusal as a chart that bundles an older Kueue. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- .../admin/kueue_external.md | 22 +++++++++---------- docs/setup_installation/admin/upgrade.md | 4 ++-- 2 files changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/setup_installation/admin/kueue_external.md b/docs/setup_installation/admin/kueue_external.md index 857d951cfe..178d8b9bb4 100644 --- a/docs/setup_installation/admin/kueue_external.md +++ b/docs/setup_installation/admin/kueue_external.md @@ -1,14 +1,14 @@ # Operator Notes: Kueue Outside the Hopsworks Namespace -Administrative reference for cluster operators who run Kueue with Hopsworks and upgrade to Hopsworks 5.3. +Administrative reference for cluster operators who run Kueue with Hopsworks and upgrade to a Hopsworks release that bundles Kueue 0.20: a 5.2.x patch release, or 5.3. With `global._hopsworks.kueue.enabled=true`, Hopsworks releases 4.3 to 5.2 bundle Kueue 0.12.2 in the Hopsworks namespace. -Hopsworks 5.3 bundles Kueue 0.20.3, and its backend reads Kueue objects as `kueue.x-k8s.io/v1beta2`, which Kueue 0.12 does not serve. -A cluster that runs the bundled Kueue 0.12 therefore moves to a Kueue installed outside the Hopsworks release before it upgrades to 5.3, while it runs Hopsworks 5.1: release `kueue` in namespace `kueue-system`, installed with `fullnameOverride: kueue`, which Hopsworks uses with `hw-kueue.kueue.enabled=false`. +Those releases bundle Kueue 0.20.3, and their backend reads Kueue objects as `kueue.x-k8s.io/v1beta2`, which Kueue 0.12 does not serve. +A cluster that runs the bundled Kueue 0.12 therefore moves to a Kueue installed outside the Hopsworks release before that upgrade, while it runs Hopsworks 5.1: release `kueue` in namespace `kueue-system`, installed with `fullnameOverride: kueue`, which Hopsworks uses with `hw-kueue.kueue.enabled=false`. Hopsworks keeps its RBAC bindings to the Kueue roles and keeps queueing jobs through Kueue. A cluster without Kueue needs nothing. -A fresh 5.3 install gets Kueue 0.20.3 in the Hopsworks namespace, or uses an external Kueue installed first with `hw-kueue.kueue.enabled=false`. +A fresh install of such a release gets Kueue 0.20.3 in the Hopsworks namespace, or uses an external Kueue installed first with `hw-kueue.kueue.enabled=false`. ## Why `helm upgrade` alone cannot do it @@ -21,9 +21,9 @@ A fresh 5.3 install gets Kueue 0.20.3 in the Hopsworks namespace, or uses an ext ## What the migration does -The 5.1 chart ships `charts/hw-kueue/files/migrate-to-external.sh`; the 5.3 chart does not. Use the latest 5.1 chart, whose `--help` lists the `upgrade-kueue` phase. Run the script with `bash`, because `helm package` stores files without the execute bit. The script keeps every Kueue object: ClusterQueues, Cohorts, ResourceFlavors, Topologies, LocalQueues, priority classes and Workloads. +The 5.1 chart ships `charts/hw-kueue/files/migrate-to-external.sh`; the charts with Kueue 0.20 do not. Use the latest 5.1 chart, whose `--help` lists the `upgrade-kueue` phase. Run the script with `bash`, because `helm package` stores files without the execute bit. The script keeps every Kueue object: ClusterQueues, Cohorts, ResourceFlavors, Topologies, LocalQueues, priority classes and Workloads. On 5.1 it moves the cluster to Kueue 0.18.1, the last release that serves both `v1beta1` and `v1beta2`, and rewrites every object as `v1beta2`. -After the Hopsworks upgrade to 5.3, its `upgrade-kueue` phase upgrades Kueue to 0.20.3. +After the Hopsworks upgrade to a release with Kueue 0.20, its `upgrade-kueue` phase upgrades Kueue to 0.20.3. It runs in phases, and each one is safe to re-run; `all` runs every phase except `upgrade-kueue`: | Phase | What it does | @@ -35,8 +35,8 @@ It runs in phases, and each one is safe to re-run; `all` runs every phase except | `install-kueue` | Applies the Kueue 0.18.1 CRDs, installs release `kueue` in `kueue-system` and restores the replaced objects. | | `migrate-storage` | Rewrites every object at `v1beta2`, prunes the CRDs' stored versions and drops the `managedFields` entries recorded at older versions. | | `enable-hopsworks` | Upgrades Hopsworks with Kueue on and `hw-kueue.kueue.enabled=false`. | -| `upgrade-kueue` | Upgrades release `kueue` to Kueue 0.20.3 (`--kueue-target`). Refuses while the Hopsworks release is older than 5.3, while any Kueue CRD stores a version other than `v1beta2`, while the Hopsworks release does not use the external Kueue or has not finished deploying, and refuses a downgrade. Keeps the `kueue` release's own values, and refuses a manager configuration that is not `config.kueue.x-k8s.io/v1beta2` before it changes anything. Drops the `managedFields` entries recorded at `v1beta1`, which Kueue 0.20 can no longer convert. Takes the chart from `oci://registry.k8s.io/kueue/charts` (`--kueue-target-repo`). | -| `verify` | Checks the controller and its version (0.18.1 before Hopsworks 5.3, 0.20.3 after), the stored versions, every backed-up queue object and the release values. | +| `upgrade-kueue` | Upgrades release `kueue` to Kueue 0.20.3 (`--kueue-target`). Refuses while the Hopsworks release's chart bundles a Kueue older than 0.20, while any Kueue CRD stores a version other than `v1beta2`, while the Hopsworks release does not use the external Kueue or has not finished deploying, and refuses a downgrade. Keeps the `kueue` release's own values, and refuses a manager configuration that is not `config.kueue.x-k8s.io/v1beta2` before it changes anything. Drops the `managedFields` entries recorded at `v1beta1`, which Kueue 0.20 can no longer convert. Takes the chart from `oci://registry.k8s.io/kueue/charts` (`--kueue-target-repo`). | +| `verify` | Checks the controller and its version (0.18.1 before the Hopsworks upgrade, 0.20.3 after), the stored versions, every backed-up queue object and the release values. | | `export-queues` | Writes the cluster-scoped queue objects as `v1beta2` manifests to `/gitops/kueue-queues.yaml`, for clusters that keep their queues in Git. | The script applies the Kueue CRDs itself and installs release `kueue` without them, so `helm uninstall kueue` leaves the CRDs, and every queue, in place. @@ -77,8 +77,8 @@ The move runs in three steps. `verify` at the end of `all` expects Kueue 0.18.1. Jobs can be submitted again once it passes. -2. Upgrade Hopsworks to 5.3, keeping `global._hopsworks.kueue.enabled=true` and `hw-kueue.kueue.enabled=false`. - Kueue 0.18.1 serves `v1beta2`, which the 5.3 backend reads. +2. Upgrade Hopsworks to a release with Kueue 0.20, keeping `global._hopsworks.kueue.enabled=true` and `hw-kueue.kueue.enabled=false`. + Kueue 0.18.1 serves `v1beta2`, which its backend reads. 3. Upgrade Kueue to 0.20.3, with the script from the 5.1 chart: @@ -182,7 +182,7 @@ It then runs `verify` and `export-queues` (Step 6) against Kueue 0.18.1 and exit ### Step 5: Upgrade Hopsworks and Kueue -Set the Hopsworks HelmRelease's chart version to 5.3, keeping the values from Step 4, then commit and push. +Set the Hopsworks HelmRelease's chart version to a release with Kueue 0.20, keeping the values from Step 4, then commit and push. Once Flux has upgraded Hopsworks, run the script's last phase: ```sh diff --git a/docs/setup_installation/admin/upgrade.md b/docs/setup_installation/admin/upgrade.md index 493c447bd5..c0d2e6d008 100644 --- a/docs/setup_installation/admin/upgrade.md +++ b/docs/setup_installation/admin/upgrade.md @@ -242,9 +242,9 @@ From 5.3 the default is 2048, up from 1024, because at 1 GiB the sidecar was OOM ### Kueue outside the Hopsworks namespace { #upgrade-kueue-external } Optional. -From 5.3, `hw-kueue.kueue.enabled=false` makes Hopsworks use a Kueue installed as its own release, `kueue` in `kueue-system`, in place of the Kueue 0.12.2 bundled in the Hopsworks namespace. +From the 5.2.x patch release that bundles Kueue 0.20, and in 5.3, `hw-kueue.kueue.enabled=false` makes Hopsworks use a Kueue installed as its own release, `kueue` in `kueue-system`, in place of the Kueue 0.12.2 bundled in the Hopsworks namespace. A cluster that runs the bundled Kueue cannot be switched with `helm upgrade` alone: Kueue 0.12 stores Cohorts and Topologies as `v1alpha1`, which no later Kueue reads. -The 5.1 chart's `migrate-to-external.sh` moves it across on 5.1 and keeps every queue; after the upgrade to 5.3 it upgrades Kueue to 0.20.3. +The 5.1 chart's `migrate-to-external.sh` moves it across on 5.1 and keeps every queue; after the Hopsworks upgrade it upgrades Kueue to 0.20.3. Follow [Kueue outside the Hopsworks namespace](kueue_external.md), and stop submitting jobs while the move runs. ## Changes from 5.2.2 { #upgrade-changes-from-522 } From 2d01b9e1c4c9e9ab64196f0c0df61c4e099c822c Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Fri, 23 Oct 2026 17:30:48 +0200 Subject: [PATCH 13/17] [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 A 5.1 cluster stays on Kueue 0.18; the step to Kueue 0.20.3 ships with the Hopsworks release that bundles it. docs: kueue_external.md names two scripts. The 5.1 chart's migrate-to-external.sh moves a cluster to Kueue 0.18.1 on 5.1, and the new chart's upgrade-kueue.sh, run after the Hopsworks upgrade, takes it to 0.20.3. The upgrade-kueue phase row gives way to a description of upgrade-kueue.sh and its refusals; the Flux Step 5 runs it with --gitops and commits the HelmRelease it writes to ./kueue-upgrade. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- .../admin/kueue_external.md | 57 +++++++++++-------- 1 file changed, 34 insertions(+), 23 deletions(-) diff --git a/docs/setup_installation/admin/kueue_external.md b/docs/setup_installation/admin/kueue_external.md index 178d8b9bb4..aada554a54 100644 --- a/docs/setup_installation/admin/kueue_external.md +++ b/docs/setup_installation/admin/kueue_external.md @@ -21,10 +21,13 @@ A fresh install of such a release gets Kueue 0.20.3 in the Hopsworks namespace, ## What the migration does -The 5.1 chart ships `charts/hw-kueue/files/migrate-to-external.sh`; the charts with Kueue 0.20 do not. Use the latest 5.1 chart, whose `--help` lists the `upgrade-kueue` phase. Run the script with `bash`, because `helm package` stores files without the execute bit. The script keeps every Kueue object: ClusterQueues, Cohorts, ResourceFlavors, Topologies, LocalQueues, priority classes and Workloads. -On 5.1 it moves the cluster to Kueue 0.18.1, the last release that serves both `v1beta1` and `v1beta2`, and rewrites every object as `v1beta2`. -After the Hopsworks upgrade to a release with Kueue 0.20, its `upgrade-kueue` phase upgrades Kueue to 0.20.3. -It runs in phases, and each one is safe to re-run; `all` runs every phase except `upgrade-kueue`: +Two scripts do the move, each shipped with the release it runs on; run them with `bash`, because `helm package` stores files without the execute bit: + +- On Hopsworks 5.1, the 5.1 chart's `charts/hw-kueue/files/migrate-to-external.sh` moves the cluster to Kueue 0.18.1, the last release that serves both `v1beta1` and `v1beta2`, and rewrites every object as `v1beta2`. A 5.1 cluster stays on Kueue 0.18. +- After the Hopsworks upgrade to a release with Kueue 0.20, that chart's `charts/hw-kueue/files/upgrade-kueue.sh` upgrades Kueue to 0.20.3. + +Every Kueue object is kept: ClusterQueues, Cohorts, ResourceFlavors, Topologies, LocalQueues, priority classes and Workloads. +`migrate-to-external.sh` runs in phases, and each one is safe to re-run; `all` runs them in order: | Phase | What it does | | --- | --- | @@ -33,10 +36,9 @@ It runs in phases, and each one is safe to re-run; `all` runs every phase except | `remove-bundled` | Upgrades Hopsworks with Kueue off, which removes the bundled controller, its webhooks and its ClusterRoles. CRDs and queue objects stay. | | `replace-alpha-crds` | Snapshots the Cohorts, Topologies and ResourceFlavors, then replaces the `v1alpha1` Cohort and Topology CRDs. | | `install-kueue` | Applies the Kueue 0.18.1 CRDs, installs release `kueue` in `kueue-system` and restores the replaced objects. | -| `migrate-storage` | Rewrites every object at `v1beta2`, prunes the CRDs' stored versions and drops the `managedFields` entries recorded at older versions. | +| `migrate-storage` | Rewrites every object at `v1beta2` and prunes the CRDs' stored versions. | | `enable-hopsworks` | Upgrades Hopsworks with Kueue on and `hw-kueue.kueue.enabled=false`. | -| `upgrade-kueue` | Upgrades release `kueue` to Kueue 0.20.3 (`--kueue-target`). Refuses while the Hopsworks release's chart bundles a Kueue older than 0.20, while any Kueue CRD stores a version other than `v1beta2`, while the Hopsworks release does not use the external Kueue or has not finished deploying, and refuses a downgrade. Keeps the `kueue` release's own values, and refuses a manager configuration that is not `config.kueue.x-k8s.io/v1beta2` before it changes anything. Drops the `managedFields` entries recorded at `v1beta1`, which Kueue 0.20 can no longer convert. Takes the chart from `oci://registry.k8s.io/kueue/charts` (`--kueue-target-repo`). | -| `verify` | Checks the controller and its version (0.18.1 before the Hopsworks upgrade, 0.20.3 after), the stored versions, every backed-up queue object and the release values. | +| `verify` | Checks the controller, the stored versions, every backed-up queue object and the release values. | | `export-queues` | Writes the cluster-scoped queue objects as `v1beta2` manifests to `/gitops/kueue-queues.yaml`, for clusters that keep their queues in Git. | The script applies the Kueue CRDs itself and installs release `kueue` without them, so `helm uninstall kueue` leaves the CRDs, and every queue, in place. @@ -51,9 +53,9 @@ A failed run resumes by running the same phase again, or `all`. From `remove-bundled` to `install-kueue` no Kueue runs, so a job created then runs outside quota. Workloads already admitted keep running, and Kueue 0.18.1 takes them over. - **Tools**: `kubectl`, `helm` 3.20 or later, `jq` and `yq` v4. -- **Kueue images**: the default repository is `docker.hops.works/registry.k8s.io/kueue/kueue`. `--kueue-image registry.k8s.io/kueue/kueue` takes the upstream images. -- **Airgapped clusters** mirror the images `registry.k8s.io/kueue/kueue:v0.18.1` and `v0.20.3` (the default repository is `docker.hops.works/registry.k8s.io/kueue/kueue`) and the `kueue` 0.18.1 and 0.20.3 charts; `--kueue-image`, `--kueue-repo` and `--kueue-target-repo` point the script at the mirror. - `--kueue-repo` (the 0.18.1 chart) and `--kueue-target-repo` (the 0.20.3 chart) take a chart repository URL or an `oci://` path. +- **Kueue images**: both scripts default to the repository `docker.hops.works/registry.k8s.io/kueue/kueue`; `--kueue-image registry.k8s.io/kueue/kueue` takes the upstream images. +- **Airgapped clusters** mirror the images `registry.k8s.io/kueue/kueue:v0.18.1` and `v0.20.3` and the `kueue` 0.18.1 and 0.20.3 charts. + `--kueue-image` and `--kueue-repo` point each script at the mirror; `--kueue-repo` takes a chart repository URL or an `oci://` path. - **An admin API key** lets the script refresh Hopsworks' settings after each upgrade. Without it Hopsworks picks up the change within 10 minutes, and a project created in those minutes gets its default queue at its first job. @@ -80,15 +82,21 @@ The move runs in three steps. 2. Upgrade Hopsworks to a release with Kueue 0.20, keeping `global._hopsworks.kueue.enabled=true` and `hw-kueue.kueue.enabled=false`. Kueue 0.18.1 serves `v1beta2`, which its backend reads. -3. Upgrade Kueue to 0.20.3, with the script from the 5.1 chart: +3. Upgrade Kueue to 0.20.3, with the script from the new chart: ```sh - bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh upgrade-kueue - bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh verify + helm pull --untar --untardir new /hopsworks --version + bash new/hopsworks/charts/hw-kueue/files/upgrade-kueue.sh ``` + It upgrades release `kueue`, then checks the end state; `upgrade-kueue.sh verify` runs the check on its own. Until then Kueue 0.18.1 queues jobs, and Spark jobs run without queueing. +`upgrade-kueue.sh` refuses to run while the Hopsworks release's chart bundles a Kueue older than 0.20, while the release does not use the external Kueue or has not finished deploying, while any Kueue CRD stores a version other than `v1beta2`, and refuses a downgrade. +It applies the Kueue CRDs itself and keeps them out of the release, keeps the `kueue` release's own values, and refuses a manager configuration that is not `config.kueue.x-k8s.io/v1beta2` before it changes anything. +It turns on Kueue's SparkApplication integration and drops the `managedFields` entries recorded at `v1beta1`, which Kueue 0.20 can no longer convert. +It takes the chart from `oci://registry.k8s.io/kueue/charts` (`--kueue-repo`), version 0.20.3 (`--kueue-version`). + `--namespace` and `--release` name a Hopsworks release other than `hopsworks` in `hopsworks`; `--help` lists every option. ## Clusters managed by Flux @@ -178,21 +186,23 @@ spec: ``` Commit and push; the script continues once the release shows both values. -It then runs `verify` and `export-queues` (Step 6) against Kueue 0.18.1 and exits. +It then runs `verify` and `export-queues` (Step 6) and exits. ### Step 5: Upgrade Hopsworks and Kueue Set the Hopsworks HelmRelease's chart version to a release with Kueue 0.20, keeping the values from Step 4, then commit and push. -Once Flux has upgraded Hopsworks, run the script's last phase: +Once Flux has upgraded Hopsworks, run the new chart's `upgrade-kueue.sh`: ```sh -bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh --gitops \ - --release --namespace --backup-dir ./kueue-backup upgrade-kueue +helm pull --untar --untardir new /hopsworks --version +bash new/hopsworks/charts/hw-kueue/files/upgrade-kueue.sh --gitops \ + --release --namespace ``` -The script applies the 0.20.3 CRDs and rewrites `/gitops/kueue.yaml` for 0.20.3: the chart from `oci://registry.k8s.io/kueue/charts`, the Spark manager configuration and feature gate, and a post-render patch that keeps the chart's CRDs out of the release. +With `--gitops` it runs no `helm upgrade`. +It applies the 0.20.3 CRDs and writes `./kueue-upgrade/kueue.yaml` (`--out-dir`): the HelmRelease `kueue` for 0.20.3 with the release's values, the chart from `oci://registry.k8s.io/kueue/charts`, the Spark manager configuration and feature gate, and a post-render patch that keeps the chart's CRDs out of the release. Replace the file you committed in Step 3 with it, then commit and push. -The script continues once the Kueue controller runs 0.20.3; `verify` then checks the end state. +The script continues once Flux runs the 0.20.3 controller, then checks the end state. ### Step 6: Verify and move the queues into Git @@ -230,7 +240,7 @@ A single phase can also be run on its own, such as `bash hopsworks/charts/hw-kue ### What the script writes The script applies the Kueue CRDs itself and the HelmRelease skips them (`crds: Skip` on install and upgrade), so deleting the HelmRelease leaves every queue in place. -The 0.20.3 chart ships its CRDs as templates, which `crds: Skip` does not cover, so its HelmRelease also carries a kustomize post-render patch that removes them from what Flux installs. +The 0.20.3 chart ships its CRDs as templates, which `crds: Skip` does not cover, so the HelmRelease `upgrade-kueue.sh` writes also carries a kustomize post-render patch that removes them from what Flux installs. ## What changes @@ -246,7 +256,7 @@ The 0.20.3 chart ships its CRDs as templates, which `crds: Skip` does not cover, - **Jobs in the Hopsworks namespace are queued.** Kueue 0.18 ignores the namespace it runs in, which is now `kueue-system`, so Kueue-labelled jobs in the Hopsworks namespace are queued. - **Spark jobs are queued, with static executors only.** - The script turns on Kueue's SparkApplication integration, so Spark jobs get quota and queueing. + `upgrade-kueue.sh` turns on Kueue's SparkApplication integration, so Spark jobs get quota and queueing. Kueue reserves quota for a fixed number of executors, so Hopsworks refuses a Spark job with dynamic allocation on a Kueue cluster. Spark jobs saved before the upgrade have dynamic allocation on: turn it off and set the number of executors before running them. Spark in notebooks is unchanged. @@ -259,10 +269,11 @@ The 0.20.3 chart ships its CRDs as templates, which `crds: Skip` does not cover, ## Verify -The `verify` phase checks the end state; run it on its own at any time: +Each script checks the end state, and runs the check on its own at any time: ```sh -bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh verify +bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh verify # on 5.1 +bash new/hopsworks/charts/hw-kueue/files/upgrade-kueue.sh verify # after the upgrade to Kueue 0.20 ``` To check by hand: From bc107f8eabb2cea115cf375d5ec4b941f04935d1 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Sat, 24 Oct 2026 12:57:58 +0200 Subject: [PATCH 14/17] [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 docs: a 5.2.0 or 5.2.1 cluster on the bundled Kueue 0.12 moves with the 5.1 chart's migrate-to-external.sh run phase by phase: the phases up to migrate-storage on its own 5.2 chart, then enable-hopsworks with the new release, so Hopsworks is never downgraded. The upgrade note says the move is required for a cluster on the bundled Kueue, since the new chart refuses it, and that upgrade-kueue.sh, not the 5.1 script, takes Kueue to 0.20.3. The guide also covers what changed in the scripts and chart: Spark jobs saved with dynamic allocation run their dynamic-allocation minimum instead of being refused; upgrade-kueue.sh keeps TopologyAwareScheduling off when a release turns it off, and with --gitops writes kueue-crds.yaml and waits for the SparkApplication integration; the bundled CRDs carry an Argo CD no-prune, no-delete option; the chart tells its own CRDs by label and refuses an upgrade that turns Kueue on while they are missing. Neither script supports a release managed by Argo CD. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011tczJCVzi8QNsr26x1rDNh --- .../admin/kueue_external.md | 51 ++++++++++++++++--- docs/setup_installation/admin/upgrade.md | 7 +-- 2 files changed, 49 insertions(+), 9 deletions(-) diff --git a/docs/setup_installation/admin/kueue_external.md b/docs/setup_installation/admin/kueue_external.md index aada554a54..40854b0701 100644 --- a/docs/setup_installation/admin/kueue_external.md +++ b/docs/setup_installation/admin/kueue_external.md @@ -4,11 +4,13 @@ Administrative reference for cluster operators who run Kueue with Hopsworks and With `global._hopsworks.kueue.enabled=true`, Hopsworks releases 4.3 to 5.2 bundle Kueue 0.12.2 in the Hopsworks namespace. Those releases bundle Kueue 0.20.3, and their backend reads Kueue objects as `kueue.x-k8s.io/v1beta2`, which Kueue 0.12 does not serve. -A cluster that runs the bundled Kueue 0.12 therefore moves to a Kueue installed outside the Hopsworks release before that upgrade, while it runs Hopsworks 5.1: release `kueue` in namespace `kueue-system`, installed with `fullnameOverride: kueue`, which Hopsworks uses with `hw-kueue.kueue.enabled=false`. +A cluster that runs the bundled Kueue 0.12 therefore moves to a Kueue installed outside the Hopsworks release: release `kueue` in namespace `kueue-system`, installed with `fullnameOverride: kueue`, which Hopsworks uses with `hw-kueue.kueue.enabled=false`. +On Hopsworks 5.1 it moves before the upgrade; on 5.2.0 and 5.2.1, which ship no migration script, it moves as part of the upgrade ([From Hopsworks 5.2.0 or 5.2.1](#from-hopsworks-520-or-521)). Hopsworks keeps its RBAC bindings to the Kueue roles and keeps queueing jobs through Kueue. A cluster without Kueue needs nothing. A fresh install of such a release gets Kueue 0.20.3 in the Hopsworks namespace, or uses an external Kueue installed first with `hw-kueue.kueue.enabled=false`. +The bundled Kueue's CRDs ship in the chart's `crds/` directory with `argocd.argoproj.io/sync-options: Prune=false,Delete=false`, so Helm and Argo CD install them and neither deletes them, or the queues with them. ## Why `helm upgrade` alone cannot do it @@ -94,11 +96,43 @@ The move runs in three steps. `upgrade-kueue.sh` refuses to run while the Hopsworks release's chart bundles a Kueue older than 0.20, while the release does not use the external Kueue or has not finished deploying, while any Kueue CRD stores a version other than `v1beta2`, and refuses a downgrade. It applies the Kueue CRDs itself and keeps them out of the release, keeps the `kueue` release's own values, and refuses a manager configuration that is not `config.kueue.x-k8s.io/v1beta2` before it changes anything. +A release that turns `TopologyAwareScheduling` off keeps it off: Kueue 0.20 adds six topology-aware scheduling gates, on by default, that depend on it, and the script turns them off too, since Kueue refuses to start with a gate on whose dependency is off. It turns on Kueue's SparkApplication integration and drops the `managedFields` entries recorded at `v1beta1`, which Kueue 0.20 can no longer convert. It takes the chart from `oci://registry.k8s.io/kueue/charts` (`--kueue-repo`), version 0.20.3 (`--kueue-version`). `--namespace` and `--release` name a Hopsworks release other than `hopsworks` in `hopsworks`; `--help` lists every option. +Both scripts read and upgrade Helm release records, directly or through Flux's helm-controller, which keeps them. +A Hopsworks or Kueue release managed by Argo CD, which renders charts and stores no Helm release, is not supported by either script. + +## From Hopsworks 5.2.0 or 5.2.1 + +Hopsworks 5.2.0 and 5.2.1 bundle Kueue 0.12.2 and ship no migration script. +The 5.1 chart's `migrate-to-external.sh` does the move on them too, run phase by phase: the phases up to `migrate-storage` keep Hopsworks on its current chart, and `enable-hopsworks` upgrades it to the new release. +Hopsworks is never downgraded. + +```sh +helm pull --untar --untardir v51 /hopsworks --version +M="bash v51/hopsworks/charts/hw-kueue/files/migrate-to-external.sh" +export HELM_EXTRA_ARGS="" +export HOPSWORKS_URL=https:// HOPSWORKS_API_KEY= + +# Kueue off, Hopsworks stays on its current version +for phase in preflight backup remove-bundled replace-alpha-crds install-kueue migrate-storage; do + $M --hopsworks-chart /hopsworks --hopsworks-version $phase || break +done +# Hopsworks upgraded to the release with Kueue 0.20, on the external Kueue +$M --hopsworks-chart /hopsworks --hopsworks-version enable-hopsworks +$M verify + +helm pull --untar --untardir new /hopsworks --version +bash new/hopsworks/charts/hw-kueue/files/upgrade-kueue.sh +``` + +`enable-hopsworks` upgrades Hopsworks with `--reset-then-reuse-values`, Kueue on and `hw-kueue.kueue.enabled=false`; Kueue 0.18.1 serves `v1beta2`, which the new backend reads. +From `remove-bundled` until `install-kueue` no Kueue runs, as on 5.1. +With Flux, follow [Clusters managed by Flux](#clusters-managed-by-flux) with the same split: keep the current 5.2 chart version in Step 2, and set the new version in Step 4. + ## Clusters managed by Flux When Flux manages the Hopsworks release, run the script with `--gitops`. @@ -201,8 +235,8 @@ bash new/hopsworks/charts/hw-kueue/files/upgrade-kueue.sh --gitops \ With `--gitops` it runs no `helm upgrade`. It applies the 0.20.3 CRDs and writes `./kueue-upgrade/kueue.yaml` (`--out-dir`): the HelmRelease `kueue` for 0.20.3 with the release's values, the chart from `oci://registry.k8s.io/kueue/charts`, the Spark manager configuration and feature gate, and a post-render patch that keeps the chart's CRDs out of the release. -Replace the file you committed in Step 3 with it, then commit and push. -The script continues once Flux runs the 0.20.3 controller, then checks the end state. +Replace the file you committed in Step 3 with it, add `./kueue-upgrade/kueue-crds.yaml` next to it, then commit and push. +The script continues once Flux runs the 0.20.3 controller with the SparkApplication integration on, then checks the end state. ### Step 6: Verify and move the queues into Git @@ -241,6 +275,8 @@ A single phase can also be run on its own, such as `bash hopsworks/charts/hw-kue The script applies the Kueue CRDs itself and the HelmRelease skips them (`crds: Skip` on install and upgrade), so deleting the HelmRelease leaves every queue in place. The 0.20.3 chart ships its CRDs as templates, which `crds: Skip` does not cover, so the HelmRelease `upgrade-kueue.sh` writes also carries a kustomize post-render patch that removes them from what Flux installs. +`upgrade-kueue.sh` also writes the CRDs it applied to `./kueue-upgrade/kueue-crds.yaml`, annotated `kustomize.toolkit.fluxcd.io/prune: disabled`. +Commit it next to `kueue.yaml`, in place of any Kueue CRDs there: Flux applies CRDs before the objects that use them, so Git can restore Kueue on an empty cluster, and never prunes them, so removing the file does not delete the queues. ## What changes @@ -257,8 +293,10 @@ The 0.20.3 chart ships its CRDs as templates, which `crds: Skip` does not cover, Kueue 0.18 ignores the namespace it runs in, which is now `kueue-system`, so Kueue-labelled jobs in the Hopsworks namespace are queued. - **Spark jobs are queued, with static executors only.** `upgrade-kueue.sh` turns on Kueue's SparkApplication integration, so Spark jobs get quota and queueing. - Kueue reserves quota for a fixed number of executors, so Hopsworks refuses a Spark job with dynamic allocation on a Kueue cluster. - Spark jobs saved before the upgrade have dynamic allocation on: turn it off and set the number of executors before running them. + Kueue reserves quota for a fixed number of executors, so on a Kueue cluster Spark jobs run static executors. + A job saved with dynamic allocation, as every Spark job was by default, runs its dynamic-allocation minimum, at least one executor: it is converted and saved at its next start, save, or read of the project default. + A job that needs more executors sets `spark.executor.instances`. + `spark.dynamicAllocation.enabled=true` in a job's Spark properties is refused. Spark in notebooks is unchanged. - **Workloads must be ready within 30 minutes.** Kueue 0.19 turns `waitForPodsReady` on by default: a workload whose pods are not all ready 30 minutes after admission is evicted and queued again. @@ -290,6 +328,7 @@ The controller is ready in `kueue-system`, none runs in the Hopsworks namespace, The chart checks the cluster before it renders: - With `hw-kueue.kueue.enabled=false`, it refuses while the bundled controller still runs, while Cohorts are stored as `v1alpha1`, or when no Kueue serving `v1beta2` is installed. -- With `hw-kueue.kueue.enabled=true`, it refuses when the Kueue CRDs exist and the Hopsworks release does not own them: those of the bundled Kueue 0.12, or those of an external Kueue. +- With `hw-kueue.kueue.enabled=true`, it refuses when Kueue CRDs exist that the chart did not install, those without the label `hw-kueue.hops.works/bundled-crds`: those of the bundled Kueue 0.12, or those of an external Kueue. + It also refuses an upgrade that turns the bundled Kueue on while its CRDs are missing, since `helm upgrade` does not install a chart's `crds/`. `hw-kueue.enforcePrerequisiteCheck=false` turns the check off, for `helm template`. diff --git a/docs/setup_installation/admin/upgrade.md b/docs/setup_installation/admin/upgrade.md index c0d2e6d008..28ed92dd95 100644 --- a/docs/setup_installation/admin/upgrade.md +++ b/docs/setup_installation/admin/upgrade.md @@ -241,10 +241,11 @@ From 5.3 the default is 2048, up from 1024, because at 1 GiB the sidecar was OOM ### Kueue outside the Hopsworks namespace { #upgrade-kueue-external } -Optional. +Required for a cluster that runs the Kueue bundled with Hopsworks (`global._hopsworks.kueue.enabled=true`); a cluster without Kueue needs nothing. From the 5.2.x patch release that bundles Kueue 0.20, and in 5.3, `hw-kueue.kueue.enabled=false` makes Hopsworks use a Kueue installed as its own release, `kueue` in `kueue-system`, in place of the Kueue 0.12.2 bundled in the Hopsworks namespace. -A cluster that runs the bundled Kueue cannot be switched with `helm upgrade` alone: Kueue 0.12 stores Cohorts and Topologies as `v1alpha1`, which no later Kueue reads. -The 5.1 chart's `migrate-to-external.sh` moves it across on 5.1 and keeps every queue; after the Hopsworks upgrade it upgrades Kueue to 0.20.3. +The new chart refuses to upgrade a release that still runs the bundled Kueue 0.12, and a cluster that runs it cannot be switched with `helm upgrade` alone: Kueue 0.12 stores Cohorts and Topologies as `v1alpha1`, which no later Kueue reads. +The 5.1 chart's `migrate-to-external.sh` moves it to Kueue 0.18.1 in `kueue-system` and keeps every queue, on 5.1, or on 5.2.0 and 5.2.1 together with the Hopsworks upgrade. +After the Hopsworks upgrade, the new chart's `upgrade-kueue.sh` upgrades Kueue to 0.20.3. Follow [Kueue outside the Hopsworks namespace](kueue_external.md), and stop submitting jobs while the move runs. ## Changes from 5.2.2 { #upgrade-changes-from-522 } From 1ba52fee6459cdf3b91263dd59f95b544a5459cf Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Sat, 24 Oct 2026 13:24:49 +0200 Subject: [PATCH 15/17] [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 docs: the "Jobs in the Hopsworks namespace are queued" change is removed. Hopsworks creates its Kueue-queued system jobs in the project namespaces, so Kueue 0.18 excluding its own namespace changes nothing for Hopsworks. The Kueue entry in upgrade.md lists what changes for users and operators: static Spark executors, the 30-minute waitForPodsReady, Ray sidecars charged to quota and the default priority class, with a link to the details. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011tczJCVzi8QNsr26x1rDNh --- docs/setup_installation/admin/kueue_external.md | 2 -- docs/setup_installation/admin/upgrade.md | 7 +++++++ 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/setup_installation/admin/kueue_external.md b/docs/setup_installation/admin/kueue_external.md index 40854b0701..c95b6e5ed4 100644 --- a/docs/setup_installation/admin/kueue_external.md +++ b/docs/setup_installation/admin/kueue_external.md @@ -289,8 +289,6 @@ Commit it next to `kueue.yaml`, in place of any Kueue CRDs there: Flux applies C With `hw-kueue.kueue.enabled=false` the chart no longer applies `hw-kueue.clusterQueues`, `cohorts`, `resourceFlavours` or `topologies`. Queues defined there before the move are kept, but changing those values has no effect. Write queue objects as `kueue.x-k8s.io/v1beta2`: Kueue 0.20 serves no other version. -- **Jobs in the Hopsworks namespace are queued.** - Kueue 0.18 ignores the namespace it runs in, which is now `kueue-system`, so Kueue-labelled jobs in the Hopsworks namespace are queued. - **Spark jobs are queued, with static executors only.** `upgrade-kueue.sh` turns on Kueue's SparkApplication integration, so Spark jobs get quota and queueing. Kueue reserves quota for a fixed number of executors, so on a Kueue cluster Spark jobs run static executors. diff --git a/docs/setup_installation/admin/upgrade.md b/docs/setup_installation/admin/upgrade.md index 28ed92dd95..61540a3f59 100644 --- a/docs/setup_installation/admin/upgrade.md +++ b/docs/setup_installation/admin/upgrade.md @@ -248,6 +248,13 @@ The 5.1 chart's `migrate-to-external.sh` moves it to Kueue 0.18.1 in `kueue-syst After the Hopsworks upgrade, the new chart's `upgrade-kueue.sh` upgrades Kueue to 0.20.3. Follow [Kueue outside the Hopsworks namespace](kueue_external.md), and stop submitting jobs while the move runs. +What changes for users and operators ([details](kueue_external.md#what-changes)): + +- Spark jobs are queued, with static executors: a job saved with dynamic allocation runs its dynamic-allocation minimum, and `spark.dynamicAllocation.enabled=true` in its Spark properties is refused. +- A workload whose pods are not all ready 30 minutes after admission is evicted and queued again (Kueue 0.19 `waitForPodsReady`). +- The Ray autoscaler and job-submitter sidecars count against quota (Kueue 0.19). +- A workload without a priority class gets the WorkloadPriorityClass named `default`, when one exists (Kueue 0.20). + ## Changes from 5.2.2 { #upgrade-changes-from-522 } These changes ship in 5.2.2 and 5.3.0. From 4b456a9dee89d2d51e45f9f19232e1d6770c43ae Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Sat, 24 Oct 2026 14:07:45 +0200 Subject: [PATCH 16/17] [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.3 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 docs: the intro names the releases that bundle Kueue 0.20.3; "Those releases" pointed back at 4.3 to 5.2, which bundle 0.12.2. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011tczJCVzi8QNsr26x1rDNh --- docs/setup_installation/admin/kueue_external.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/setup_installation/admin/kueue_external.md b/docs/setup_installation/admin/kueue_external.md index c95b6e5ed4..28aa323196 100644 --- a/docs/setup_installation/admin/kueue_external.md +++ b/docs/setup_installation/admin/kueue_external.md @@ -3,7 +3,7 @@ Administrative reference for cluster operators who run Kueue with Hopsworks and upgrade to a Hopsworks release that bundles Kueue 0.20: a 5.2.x patch release, or 5.3. With `global._hopsworks.kueue.enabled=true`, Hopsworks releases 4.3 to 5.2 bundle Kueue 0.12.2 in the Hopsworks namespace. -Those releases bundle Kueue 0.20.3, and their backend reads Kueue objects as `kueue.x-k8s.io/v1beta2`, which Kueue 0.12 does not serve. +A 5.2.x patch release and 5.3 bundle Kueue 0.20.3, and their backend reads Kueue objects as `kueue.x-k8s.io/v1beta2`, which Kueue 0.12 does not serve. A cluster that runs the bundled Kueue 0.12 therefore moves to a Kueue installed outside the Hopsworks release: release `kueue` in namespace `kueue-system`, installed with `fullnameOverride: kueue`, which Hopsworks uses with `hw-kueue.kueue.enabled=false`. On Hopsworks 5.1 it moves before the upgrade; on 5.2.0 and 5.2.1, which ship no migration script, it moves as part of the upgrade ([From Hopsworks 5.2.0 or 5.2.1](#from-hopsworks-520-or-521)). Hopsworks keeps its RBAC bindings to the Kueue roles and keeps queueing jobs through Kueue. From 9d79baae27aa21ae8c5f6d9c508970458b6eac06 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Sat, 24 Oct 2026 17:28:47 +0200 Subject: [PATCH 17/17] [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.3 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 docs: the Hopsworks chart ships no Kueue, as it ships no Kyverno. A new Fresh install section installs Kueue 0.20.3 first as release kueue in kueue-system with the chart's upgrade-kueue.sh install, and covers Argo CD: an Application of its own for the upstream chart with the SparkApplication settings, synced first and without the resources finalizer. The chart creates the default queues only when missing, and its prerequisite check refuses kueue.enabled=true. The bundled 0.20.3 and its crds/ are gone from the guide and the upgrade note. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011tczJCVzi8QNsr26x1rDNh --- .../admin/kueue_external.md | 55 +++++++++++++++---- docs/setup_installation/admin/upgrade.md | 2 +- 2 files changed, 44 insertions(+), 13 deletions(-) diff --git a/docs/setup_installation/admin/kueue_external.md b/docs/setup_installation/admin/kueue_external.md index 28aa323196..37aa30ae28 100644 --- a/docs/setup_installation/admin/kueue_external.md +++ b/docs/setup_installation/admin/kueue_external.md @@ -1,16 +1,48 @@ # Operator Notes: Kueue Outside the Hopsworks Namespace -Administrative reference for cluster operators who run Kueue with Hopsworks and upgrade to a Hopsworks release that bundles Kueue 0.20: a 5.2.x patch release, or 5.3. +Administrative reference for cluster operators who run Kueue with Hopsworks, install a Hopsworks release that uses Kueue 0.20 (a 5.2.x patch release, or 5.3), or upgrade to one. With `global._hopsworks.kueue.enabled=true`, Hopsworks releases 4.3 to 5.2 bundle Kueue 0.12.2 in the Hopsworks namespace. -A 5.2.x patch release and 5.3 bundle Kueue 0.20.3, and their backend reads Kueue objects as `kueue.x-k8s.io/v1beta2`, which Kueue 0.12 does not serve. +A 5.2.x patch release and 5.3 ship no Kueue: they use Kueue 0.20.3 installed outside the Hopsworks release, and their backend reads Kueue objects as `kueue.x-k8s.io/v1beta2`, which Kueue 0.12 does not serve. A cluster that runs the bundled Kueue 0.12 therefore moves to a Kueue installed outside the Hopsworks release: release `kueue` in namespace `kueue-system`, installed with `fullnameOverride: kueue`, which Hopsworks uses with `hw-kueue.kueue.enabled=false`. On Hopsworks 5.1 it moves before the upgrade; on 5.2.0 and 5.2.1, which ship no migration script, it moves as part of the upgrade ([From Hopsworks 5.2.0 or 5.2.1](#from-hopsworks-520-or-521)). Hopsworks keeps its RBAC bindings to the Kueue roles and keeps queueing jobs through Kueue. A cluster without Kueue needs nothing. -A fresh install of such a release gets Kueue 0.20.3 in the Hopsworks namespace, or uses an external Kueue installed first with `hw-kueue.kueue.enabled=false`. -The bundled Kueue's CRDs ship in the chart's `crds/` directory with `argocd.argoproj.io/sync-options: Prune=false,Delete=false`, so Helm and Argo CD install them and neither deletes them, or the queues with them. +A fresh install of such a release with Kueue installs Kueue first ([Fresh install](#fresh-install)). + +## Fresh install + +The Hopsworks chart ships no Kueue, as it ships no Kyverno: install Kueue 0.20.3 first, as release `kueue` in `kueue-system`, then Hopsworks with `global._hopsworks.kueue.enabled=true`. +The chart's `upgrade-kueue.sh` installs it with the SparkApplication integration Hopsworks queues Spark jobs through: + +```sh +helm pull --untar /hopsworks --version +bash hopsworks/charts/hw-kueue/files/upgrade-kueue.sh install +helm install hopsworks hopsworks -n hopsworks --set global._hopsworks.kueue.enabled=true ... +``` + +It applies the Kueue CRDs itself and installs release `kueue` without them, so uninstalling it never deletes the queues. +With `--gitops` it writes the HelmRelease and the CRDs to commit, as in [Clusters managed by Flux](#clusters-managed-by-flux). +The Hopsworks chart then creates the default queues, ResourceFlavor `default-flavor`, Topology `default`, Cohort `cluster` and ClusterQueue `other`, when they are missing. +The default ResourceFlavor selects nodes labelled `cloud.provider.com/region=europe`; label the nodes, or change `hw-kueue.resourceFlavours`. + +With Argo CD, install Kueue as an Application of its own, synced before the Hopsworks one: the chart `kueue` version 0.20.3 from `oci://registry.k8s.io/kueue/charts`, release name `kueue` in `kueue-system`, with these values: + +```yaml +fullnameOverride: kueue +controllerManager: + featureGates: + - name: SparkApplicationIntegration + enabled: true +managerConfig: + controllerManagerConfigYaml: | + # the chart's default, with sparkoperator.k8s.io/sparkapplication + # added to integrations.frameworks +``` + +That chart keeps its CRDs among its templates, so Argo CD upgrades them with a version bump. +Leave the `resources-finalizer.argocd.argoproj.io` finalizer off that Application: deleting it then leaves the CRDs, and every queue, in place. ## Why `helm upgrade` alone cannot do it @@ -39,7 +71,7 @@ Every Kueue object is kept: ClusterQueues, Cohorts, ResourceFlavors, Topologies, | `replace-alpha-crds` | Snapshots the Cohorts, Topologies and ResourceFlavors, then replaces the `v1alpha1` Cohort and Topology CRDs. | | `install-kueue` | Applies the Kueue 0.18.1 CRDs, installs release `kueue` in `kueue-system` and restores the replaced objects. | | `migrate-storage` | Rewrites every object at `v1beta2` and prunes the CRDs' stored versions. | -| `enable-hopsworks` | Upgrades Hopsworks with Kueue on and `hw-kueue.kueue.enabled=false`. | +| `enable-hopsworks` | Upgrades Hopsworks with Kueue on and `hw-kueue.kueue.enabled=false`, which a 5.2.x patch release or 5.3 keeps by default. | | `verify` | Checks the controller, the stored versions, every backed-up queue object and the release values. | | `export-queues` | Writes the cluster-scoped queue objects as `v1beta2` manifests to `/gitops/kueue-queues.yaml`, for clusters that keep their queues in Git. | @@ -81,7 +113,7 @@ The move runs in three steps. `verify` at the end of `all` expects Kueue 0.18.1. Jobs can be submitted again once it passes. -2. Upgrade Hopsworks to a release with Kueue 0.20, keeping `global._hopsworks.kueue.enabled=true` and `hw-kueue.kueue.enabled=false`. +2. Upgrade Hopsworks to a release with Kueue 0.20, keeping `global._hopsworks.kueue.enabled=true`. Kueue 0.18.1 serves `v1beta2`, which its backend reads. 3. Upgrade Kueue to 0.20.3, with the script from the new chart: @@ -262,7 +294,7 @@ spec: From then on the queues are changed in Git. With `prune: true`, a queue removed from Git is deleted from the cluster. -Remove `hw-kueue.clusterQueues`, `cohorts`, `resourceFlavours` and `topologies` from the Hopsworks values: with `hw-kueue.kueue.enabled=false` they have no effect. +Remove `hw-kueue.clusterQueues`, `cohorts`, `resourceFlavours` and `topologies` from the Hopsworks values: the chart creates those queues only when they are missing, and Git now owns them. LocalQueues are not exported, because Hopsworks creates and deletes them with projects. ### If a step times out or the script stops @@ -286,8 +318,8 @@ Commit it next to `kueue.yaml`, in place of any Kueue CRDs there: Flux applies C - **Cohorts get `parentName`** in place of `parent`. ClusterQueues keep their quotas, fair sharing, preemption and borrowing settings; Kueue converts `cohort`, `admissionChecks` and the `flavorFungibility` values `Borrow` and `Preempt` to their `v1beta2` forms. - **Queues are managed with `kubectl`.** - With `hw-kueue.kueue.enabled=false` the chart no longer applies `hw-kueue.clusterQueues`, `cohorts`, `resourceFlavours` or `topologies`. - Queues defined there before the move are kept, but changing those values has no effect. + The chart creates the queues in `hw-kueue.clusterQueues`, `cohorts`, `resourceFlavours` and `topologies` only when they are missing. + Existing queues are kept as they are, and changing those values has no effect on them. Write queue objects as `kueue.x-k8s.io/v1beta2`: Kueue 0.20 serves no other version. - **Spark jobs are queued, with static executors only.** `upgrade-kueue.sh` turns on Kueue's SparkApplication integration, so Spark jobs get quota and queueing. @@ -325,8 +357,7 @@ The controller is ready in `kueue-system`, none runs in the Hopsworks namespace, The chart checks the cluster before it renders: -- With `hw-kueue.kueue.enabled=false`, it refuses while the bundled controller still runs, while Cohorts are stored as `v1alpha1`, or when no Kueue serving `v1beta2` is installed. -- With `hw-kueue.kueue.enabled=true`, it refuses when Kueue CRDs exist that the chart did not install, those without the label `hw-kueue.hops.works/bundled-crds`: those of the bundled Kueue 0.12, or those of an external Kueue. - It also refuses an upgrade that turns the bundled Kueue on while its CRDs are missing, since `helm upgrade` does not install a chart's `crds/`. +- It refuses while the bundled Kueue's controller still runs, while Cohorts are stored as `v1alpha1`, or when no Kueue serving `v1beta2` is installed. +- It refuses `hw-kueue.kueue.enabled=true`: the chart ships no Kueue. `hw-kueue.enforcePrerequisiteCheck=false` turns the check off, for `helm template`. diff --git a/docs/setup_installation/admin/upgrade.md b/docs/setup_installation/admin/upgrade.md index 61540a3f59..8c349ccda4 100644 --- a/docs/setup_installation/admin/upgrade.md +++ b/docs/setup_installation/admin/upgrade.md @@ -242,7 +242,7 @@ From 5.3 the default is 2048, up from 1024, because at 1 GiB the sidecar was OOM ### Kueue outside the Hopsworks namespace { #upgrade-kueue-external } Required for a cluster that runs the Kueue bundled with Hopsworks (`global._hopsworks.kueue.enabled=true`); a cluster without Kueue needs nothing. -From the 5.2.x patch release that bundles Kueue 0.20, and in 5.3, `hw-kueue.kueue.enabled=false` makes Hopsworks use a Kueue installed as its own release, `kueue` in `kueue-system`, in place of the Kueue 0.12.2 bundled in the Hopsworks namespace. +From the 5.2.x patch release that uses Kueue 0.20, and in 5.3, Hopsworks uses a Kueue installed as its own release, `kueue` in `kueue-system`, in place of the Kueue 0.12.2 bundled in the Hopsworks namespace: the chart ships no Kueue. The new chart refuses to upgrade a release that still runs the bundled Kueue 0.12, and a cluster that runs it cannot be switched with `helm upgrade` alone: Kueue 0.12 stores Cohorts and Topologies as `v1alpha1`, which no later Kueue reads. The 5.1 chart's `migrate-to-external.sh` moves it to Kueue 0.18.1 in `kueue-system` and keeps every queue, on 5.1, or on 5.2.0 and 5.2.1 together with the Hopsworks upgrade. After the Hopsworks upgrade, the new chart's `upgrade-kueue.sh` upgrades Kueue to 0.20.3.