From 9b10ae7a7db8433ebba9245833937009ae1b556d Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Wed, 21 Oct 2026 21:25:34 +0200 Subject: [PATCH 1/4] [HWORKS-3565] Upgrade Kueue from the bundled 0.12.2 to 0.18.1 installed outside the Hopsworks namespace, migrating existing queues and CRDs https://hopsworks.atlassian.net/browse/HWORKS-3565 Every release from 4.3 bundles Kueue 0.12.2 in the Hopsworks release namespace. From 5.3 Hopsworks can use a Kueue installed as its own release in kueue-system instead (hw-kueue.kueue.enabled=false), and the chart ships files/migrate-to-external.sh to move a running cluster to Kueue 0.18.1 while keeping every Kueue object. The docs add an operator page for that move: why helm upgrade alone cannot do it, the script's phases, what to prepare (snapshot, maintenance window, tools, airgapped images, admin API key), the command, what changes for queues and Hopsworks afterwards (no topologyName on flavors, parentName on Cohorts, queues managed with kubectl, Hopsworks-namespace jobs queued, the queue utilisation view failing until the backend reads Cohorts at v1beta1) and how to verify. The hw-kueue values page links to it. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- .../admin/kueue_external.md | 102 ++++++++++++++++++ .../common/helm_chart_values/hw-kueue.md | 1 + mkdocs.yml | 1 + 3 files changed, 104 insertions(+) create mode 100644 docs/setup_installation/admin/kueue_external.md diff --git a/docs/setup_installation/admin/kueue_external.md b/docs/setup_installation/admin/kueue_external.md new file mode 100644 index 0000000000..e01aa874c0 --- /dev/null +++ b/docs/setup_installation/admin/kueue_external.md @@ -0,0 +1,102 @@ +# Operator Notes: Kueue Outside the Hopsworks Namespace + +Administrative reference for cluster operators who run Kueue with Hopsworks and want to move it out of the Hopsworks release. + +With `global._hopsworks.kueue.enabled=true`, every Hopsworks release from 4.3 bundles Kueue 0.12.2 in the Hopsworks namespace. +From 5.3, `hw-kueue.kueue.enabled=false` makes Hopsworks use a Kueue installed outside its release instead: release `kueue` in namespace `kueue-system`, installed with `fullnameOverride: kueue`. +Hopsworks keeps its RBAC bindings to the Kueue roles and keeps queueing jobs through Kueue. + +The move is optional. +A cluster that keeps the bundled Kueue needs nothing. +A fresh install that wants an external Kueue installs it first and sets `hw-kueue.kueue.enabled=false`; this page is for clusters that already run the bundled one. + +## Why `helm upgrade` alone cannot do it + +- **Kueue 0.12 stores Cohorts and Topologies as `v1alpha1`**, which no Kueue from 0.13 on can read. + Their CRDs have to be replaced and the objects restored. +- **The Kueue CRDs belong to no Helm release.** + The bundled chart ships them in its `crds/` directory, which Helm installs once and never upgrades. +- **The new release creates cluster-wide objects under the names the Hopsworks release owns**: the `kueue-*` ClusterRoles, the webhook configurations and the visibility APIService. + The Hopsworks release has to drop them first. + +## What the migration does + +The chart ships `charts/hw-kueue/files/migrate-to-external.sh`, which moves the cluster to Kueue 0.18.1 and keeps every Kueue object: ClusterQueues, Cohorts, ResourceFlavors, Topologies, LocalQueues, priority classes and Workloads. +It runs in phases, and each one is safe to re-run: + +| Phase | What it does | +| --- | --- | +| `preflight` | Checks the tools and the release. Refuses admitted, unfinished Workloads unless given `--allow-admitted`. | +| `backup` | Saves every Kueue object, the Kueue CRDs and the release values to `--backup-dir`. | +| `remove-bundled` | Upgrades Hopsworks with Kueue off, which removes the bundled controller, its webhooks and its ClusterRoles. CRDs and queue objects stay. | +| `replace-alpha-crds` | Snapshots the Cohorts, Topologies and ResourceFlavors, then replaces the `v1alpha1` Cohort and Topology CRDs. | +| `install-kueue` | Applies the Kueue 0.18.1 CRDs, installs release `kueue` in `kueue-system` and restores the replaced objects. | +| `migrate-storage` | Rewrites every object at `v1beta2` and prunes the CRDs' stored versions. | +| `enable-hopsworks` | Upgrades Hopsworks with Kueue on and `hw-kueue.kueue.enabled=false`. | +| `verify` | Checks the controller, the stored versions, every backed-up queue object and the release values. | + +A failed run resumes by running the same phase again, or `all`. +`--dry-run` prints the commands that change the cluster; with `all` it stops after `remove-bundled`, whose result the later phases depend on. + +## Before you start + +- **Take a snapshot** and check that it completed: from `replace-alpha-crds` on, rollback is the snapshot. +- **Plan a maintenance window and stop submitting jobs.** + From `remove-bundled` to `install-kueue` no Kueue runs, so a job created then runs outside quota. + Workloads already admitted keep running, and Kueue 0.18.1 takes them over. +- **Tools**: `kubectl`, `helm` 3.20 or later, `jq` and `yq` v4. +- **Airgapped clusters** mirror `registry.k8s.io/kueue/kueue:v0.18.1` (the default image is `docker.hops.works/registry.k8s.io/kueue/kueue:v0.18.1`) and the `kueue` 0.18.1 chart; `--kueue-image` and `--kueue-repo` point the script at the mirror. +- **An admin API key** lets the script refresh Hopsworks' settings after each upgrade. + Without it Hopsworks picks up the change within 10 minutes, and a project created in those minutes gets its default queue at its first job. + +## Run the migration + +```sh +helm pull --untar /hopsworks --version <5.3 version> +HELM_EXTRA_ARGS="" \ +HOPSWORKS_URL=https:// \ +HOPSWORKS_API_KEY= \ + hopsworks/charts/hw-kueue/files/migrate-to-external.sh \ + --hopsworks-chart /hopsworks --hopsworks-version <5.3 version> all +``` + +Both Hopsworks upgrades run `helm upgrade --reset-then-reuse-values` with that chart, so the Hopsworks upgrade to 5.3 happens inside the window. +Pass `--force-conflicts` in `HELM_EXTRA_ARGS` if your upgrades need it. +`--namespace` and `--release` name a Hopsworks release other than `hopsworks` in `hopsworks`; `--help` lists every option. + +## What changes + +- **ResourceFlavors lose `topologyName`.** + Topology-aware scheduling is on by default from Kueue 0.14, and no bundled release used it. + A cluster that enabled it on purpose needs its own handling before the move. +- **Cohorts get `parentName`** in place of `parent`. + ClusterQueues keep their quotas, fair sharing, preemption and borrowing settings; Kueue converts `cohort`, `admissionChecks` and the `flavorFungibility` values `Borrow` and `Preempt` to their `v1beta2` forms. +- **Queues are managed with `kubectl`.** + With `hw-kueue.kueue.enabled=false` the chart no longer applies `hw-kueue.clusterQueues`, `cohorts`, `resourceFlavours` or `topologies`. + Queues defined there before the move are kept, but changing those values has no effect. +- **Jobs in the Hopsworks namespace are queued.** + Kueue 0.18 ignores the namespace it runs in, which is now `kueue-system`, so Kueue-labelled jobs in the Hopsworks namespace are queued. +- **The project queue utilisation view fails** until the Hopsworks backend reads Cohorts at `v1beta1`: `GET /project/{id}/kube/queueutilization` returns an error after the move. + Job submission, project queues and quotas are not affected. + +## Verify + +The `verify` phase checks the end state; run it on its own at any time: + +```sh +hopsworks/charts/hw-kueue/files/migrate-to-external.sh verify +``` + +To check by hand: + +```sh +kubectl -n kueue-system get deployment kueue-controller-manager +kubectl get crd -l app.kubernetes.io/name=kueue -o custom-columns=NAME:.metadata.name,STORED:.status.storedVersions +kubectl get clusterqueues,cohorts,resourceflavors +kubectl get localqueues -A +``` + +The controller is ready in `kueue-system`, none runs in the Hopsworks namespace, every CRD stores `["v1beta2"]`, and every queue from before the move is listed. + +With `hw-kueue.kueue.enabled=false` the chart refuses to render while the bundled controller still runs, while Cohorts are stored as `v1alpha1`, or when no Kueue serving `v1beta2` is installed. +`hw-kueue.enforcePrerequisiteCheck=false` turns the check off, for `helm template`. diff --git a/docs/setup_installation/common/helm_chart_values/hw-kueue.md b/docs/setup_installation/common/helm_chart_values/hw-kueue.md index 2c309e58de..b834695790 100644 --- a/docs/setup_installation/common/helm_chart_values/hw-kueue.md +++ b/docs/setup_installation/common/helm_chart_values/hw-kueue.md @@ -1,6 +1,7 @@ # Kueue values { #helm-values-hw-kueue } Values under `hw-kueue` configure Kueue, the job queueing controller, and the queues Hopsworks schedules jobs with. +With `kueue.enabled=false` Hopsworks uses a Kueue installed outside its release; [Kueue outside the Hopsworks namespace](../../admin/kueue_external.md) covers moving an existing cluster there. diff --git a/mkdocs.yml b/mkdocs.yml index 9c9f99cb99..e0dfe3f546 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -303,6 +303,7 @@ nav: - Configure Project Mapping: setup_installation/admin/configure-project-mapping.md - Airflow 3 operator notes: setup_installation/admin/airflow3.md - Kafka 4 and KRaft upgrade notes: setup_installation/admin/kafka4.md + - Kueue outside the Hopsworks namespace: setup_installation/admin/kueue_external.md - Search Index: setup_installation/admin/search_index.md - Upgrading OpenSearch: setup_installation/admin/upgrade_opensearch.md - Monitoring: From 2368688b10b72760c134bb16b109300dc0ee9d66 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Thu, 22 Oct 2026 13:20:59 +0200 Subject: [PATCH 2/4] [HWORKS-3565] Upgrade Kueue from the bundled 0.12.2 to 0.18.1 installed outside the Hopsworks namespace, migrating existing queues and CRDs https://hopsworks.atlassian.net/browse/HWORKS-3565 Clusters whose releases Flux manages. Flux reconciles a HelmRelease back to Git, so the helm upgrades in remove-bundled, install-kueue and enable-hopsworks would be reverted on a Flux-managed cluster. logicalclocks.github.io: the Kueue operator page gains a section on clusters managed by Flux (what to commit at each phase, what the script waits for, the queue export) and lists export-queues. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- .../admin/kueue_external.md | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/setup_installation/admin/kueue_external.md b/docs/setup_installation/admin/kueue_external.md index e01aa874c0..6543e9113b 100644 --- a/docs/setup_installation/admin/kueue_external.md +++ b/docs/setup_installation/admin/kueue_external.md @@ -34,6 +34,7 @@ It runs in phases, and each one is safe to re-run: | `migrate-storage` | Rewrites every object at `v1beta2` and prunes the CRDs' stored versions. | | `enable-hopsworks` | Upgrades Hopsworks with Kueue on and `hw-kueue.kueue.enabled=false`. | | `verify` | Checks the controller, the stored versions, every backed-up queue object and the release values. | +| `export-queues` | Writes the cluster-scoped queue objects as `v1beta2` manifests to `/gitops/kueue-queues.yaml`, for clusters that keep their queues in Git. | A failed run resumes by running the same phase again, or `all`. `--dry-run` prints the commands that change the cluster; with `all` it stops after `remove-bundled`, whose result the later phases depend on. @@ -64,6 +65,29 @@ Both Hopsworks upgrades run `helm upgrade --reset-then-reuse-values` with that c Pass `--force-conflicts` in `HELM_EXTRA_ARGS` if your upgrades need it. `--namespace` and `--release` name a Hopsworks release other than `hopsworks` in `hopsworks`; `--help` lists every option. +## Clusters managed by Flux + +When Flux manages the Hopsworks release, run the script with `--gitops`. +Without it, `preflight` refuses: Flux would undo the script's `helm upgrade` at its next reconcile. +With `--gitops` the script runs no `helm upgrade` and stops at each release change until Flux has applied it: + +| Phase | You commit | The script waits for | +| --- | --- | --- | +| `remove-bundled` | The chart version you upgrade to, and `global._hopsworks.kueue.enabled: false` in the Hopsworks HelmRelease values | The bundled Kueue controller to be gone | +| `install-kueue` | `/gitops/kueue.yaml`: the `kueue-system` namespace, a HelmRepository and HelmRelease `kueue`, with CRDs skipped on install and upgrade | The Kueue controller in `kueue-system` to be ready | +| `enable-hopsworks` | `global._hopsworks.kueue.enabled: true` and `hw-kueue.kueue.enabled: false` | Both values in the Hopsworks release | +| `export-queues` | `/gitops/kueue-queues.yaml` under a Flux Kustomization, after `verify` passes | Nothing | + +The script applies the Kueue CRDs itself and the HelmRelease skips them, so deleting the HelmRelease leaves every queue in place. +`kueue-queues.yaml` holds the ResourceFlavors, Topologies, Cohorts, ClusterQueues, AdmissionChecks and WorkloadPriorityClasses as `kueue.x-k8s.io/v1beta2`; once it is committed, remove `hw-kueue.clusterQueues`, `cohorts`, `resourceFlavours` and `topologies` from the Hopsworks values. +LocalQueues are not exported: Hopsworks creates and deletes them with projects. +Each wait lasts up to `--timeout` (default 60 minutes); running the phase again resumes it. +The manifests use `helm.toolkit.fluxcd.io/v2` and `source.toolkit.fluxcd.io/v1`, which need Flux 2.3 or later. + +```sh +hopsworks/charts/hw-kueue/files/migrate-to-external.sh --gitops all +``` + ## What changes - **ResourceFlavors lose `topologyName`.** From cb82f4e6ff452fea99dca0bccafceb320a69b588 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Thu, 22 Oct 2026 13:39:18 +0200 Subject: [PATCH 3/4] [HWORKS-3565] Upgrade Kueue from the bundled 0.12.2 to 0.18.1 installed outside the Hopsworks namespace, migrating existing queues and CRDs https://hopsworks.atlassian.net/browse/HWORKS-3565 Step-by-step instructions for clusters managed by Flux. logicalclocks.github.io: the Flux section of the Kueue operator page becomes a runbook for --gitops: what to have before starting (Flux 2.3 or later, Git access, the Helm release name and namespace, where the HelmRelease values live, a terminal that survives a dropped connection), then one step per change to commit, with the command, the HelmRelease YAML, the flux reconcile command that applies it at once and what the script waits for, then where the queue manifests go (a Kustomization depending on the Kueue one) and how to resume after a timeout. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- .../admin/kueue_external.md | 134 +++++++++++++++--- 1 file changed, 118 insertions(+), 16 deletions(-) diff --git a/docs/setup_installation/admin/kueue_external.md b/docs/setup_installation/admin/kueue_external.md index 6543e9113b..f387fbce98 100644 --- a/docs/setup_installation/admin/kueue_external.md +++ b/docs/setup_installation/admin/kueue_external.md @@ -68,26 +68,128 @@ Pass `--force-conflicts` in `HELM_EXTRA_ARGS` if your upgrades need it. ## Clusters managed by Flux When Flux manages the Hopsworks release, run the script with `--gitops`. -Without it, `preflight` refuses: Flux would undo the script's `helm upgrade` at its next reconcile. -With `--gitops` the script runs no `helm upgrade` and stops at each release change until Flux has applied it: - -| Phase | You commit | The script waits for | -| --- | --- | --- | -| `remove-bundled` | The chart version you upgrade to, and `global._hopsworks.kueue.enabled: false` in the Hopsworks HelmRelease values | The bundled Kueue controller to be gone | -| `install-kueue` | `/gitops/kueue.yaml`: the `kueue-system` namespace, a HelmRepository and HelmRelease `kueue`, with CRDs skipped on install and upgrade | The Kueue controller in `kueue-system` to be ready | -| `enable-hopsworks` | `global._hopsworks.kueue.enabled: true` and `hw-kueue.kueue.enabled: false` | Both values in the Hopsworks release | -| `export-queues` | `/gitops/kueue-queues.yaml` under a Flux Kustomization, after `verify` passes | Nothing | - -The script applies the Kueue CRDs itself and the HelmRelease skips them, so deleting the HelmRelease leaves every queue in place. -`kueue-queues.yaml` holds the ResourceFlavors, Topologies, Cohorts, ClusterQueues, AdmissionChecks and WorkloadPriorityClasses as `kueue.x-k8s.io/v1beta2`; once it is committed, remove `hw-kueue.clusterQueues`, `cohorts`, `resourceFlavours` and `topologies` from the Hopsworks values. -LocalQueues are not exported: Hopsworks creates and deletes them with projects. -Each wait lasts up to `--timeout` (default 60 minutes); running the phase again resumes it. -The manifests use `helm.toolkit.fluxcd.io/v2` and `source.toolkit.fluxcd.io/v1`, which need Flux 2.3 or later. +Flux keeps running throughout: do not suspend any HelmRelease or Kustomization. +The script never runs `helm upgrade`. +At each step that changes a Helm release it prints the change, you commit it to the Git repository Flux reconciles, and the script waits until Flux has applied it. +The kubectl work (backup, CRDs, restores, the storage rewrite, verification) the script does itself. + +Without `--gitops`, `preflight` and every step that would run `helm upgrade` refuse on a Flux-managed release, because Flux would undo the upgrade at its next reconcile. + +### Before you start with Flux + +Everything under [Before you start](#before-you-start) applies, and: + +- **Flux 2.3 or later.** + The files the script writes use `helm.toolkit.fluxcd.io/v2` and `source.toolkit.fluxcd.io/v1`; `flux version` shows the controllers' versions. +- **Write access to the Git repository** that holds the Hopsworks HelmRelease, and a path in it that a Flux Kustomization applies, for the new Kueue release and, at the end, the queues. +- **The Helm release name and namespace of Hopsworks.** + `flux get helmreleases -A` lists the HelmReleases; the Helm release name is the HelmRelease's `spec.releaseName`, or `-` when that is unset, and `helm list -A` shows it. + Pass both with `--release` and `--namespace` unless they are `hopsworks` and `hopsworks`. +- **Where the HelmRelease takes its values from**: `spec.values`, or a ConfigMap or Secret named in `spec.valuesFrom`. + The value changes below go there. +- **A terminal that survives a dropped connection**, such as `tmux` or `screen`. + The script waits for each commit for up to `--timeout`, 60 minutes by default; pass a longer one, such as `--timeout 4h`, when commits go through review. + +### Step 1: Start the script ```sh -hopsworks/charts/hw-kueue/files/migrate-to-external.sh --gitops all +helm pull --untar /hopsworks --version +HOPSWORKS_URL=https:// \ +HOPSWORKS_API_KEY= \ + hopsworks/charts/hw-kueue/files/migrate-to-external.sh --gitops \ + --release --namespace --backup-dir ./kueue-backup all ``` +`--hopsworks-chart` is not needed: Flux upgrades Hopsworks. +The script runs `preflight` and `backup`, then stops at the first change to commit. + +### Step 2: Turn off the bundled Kueue + +The script prints the change for `remove-bundled`. +In the Hopsworks HelmRelease, set the chart version you upgrade to, and turn Kueue off in its values: + +```yaml +spec: + chart: + spec: + version: "" + values: + global: + _hopsworks: + kueue: + enabled: false +``` + +Commit and push. +To apply it without waiting for the next sync interval, run `flux reconcile helmrelease -n --with-source`. +The script continues once the bundled Kueue controller is gone and the release shows the value. +From here until Step 3 completes no Kueue runs: a job created in that window runs outside quota. + +### Step 3: Install Kueue + +The script replaces the `v1alpha1` Cohort and Topology CRDs, applies the new Kueue CRDs, and writes `/gitops/kueue.yaml`: the `kueue-system` namespace, a HelmRepository and a HelmRelease `kueue`. +Copy that file into a path of the Git repository that a Flux Kustomization applies, such as `clusters//kueue/kueue.yaml`, then commit and push. +`flux reconcile kustomization --with-source` applies it at once. + +Commit the file only when the script asks for it: the HelmRelease skips the CRDs, which the script applies just before. +The script then waits for the Kueue controller, restores the Cohorts and Topologies, and rewrites every Kueue object at `v1beta2`. + +### Step 4: Point Hopsworks at the new Kueue + +The script prints the change for `enable-hopsworks`. +In the Hopsworks HelmRelease values: + +```yaml +spec: + values: + global: + _hopsworks: + kueue: + enabled: true + hw-kueue: + kueue: + enabled: false +``` + +Commit and push; the script continues once the release shows both values. + +### Step 5: Verify and move the queues into Git + +`verify` checks the end state, then `export-queues` writes `/gitops/kueue-queues.yaml`: the ResourceFlavors, Topologies, Cohorts, ClusterQueues, AdmissionChecks and WorkloadPriorityClasses as `kueue.x-k8s.io/v1beta2`, with the server-set fields removed. +Commit it under a Flux Kustomization of its own that depends on the one applying `kueue.yaml`: + +```yaml +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: kueue-queues + namespace: flux-system +spec: + interval: 10m + path: ./clusters//kueue-queues + prune: true + sourceRef: + kind: GitRepository + name: + dependsOn: + - name: +``` + +From then on the queues are changed in Git. +With `prune: true`, a queue removed from Git is deleted from the cluster. +Remove `hw-kueue.clusterQueues`, `cohorts`, `resourceFlavours` and `topologies` from the Hopsworks values: with `hw-kueue.kueue.enabled=false` they have no effect. +LocalQueues are not exported, because Hopsworks creates and deletes them with projects. + +### If a step times out or the script stops + +Each step polls every 15 seconds for up to `--timeout`, then stops with `timed out waiting for ...`. +Run the same command again: every phase checks what is already done and continues from there, and the restores are applied only once. +A single phase can also be run on its own, such as `migrate-to-external.sh --gitops enable-hopsworks`. + +### What the script writes + +The script applies the Kueue CRDs itself and the HelmRelease skips them (`crds: Skip` on install and upgrade), so deleting the HelmRelease leaves every queue in place. + ## What changes - **ResourceFlavors lose `topologyName`.** From b13e655deba5d400bb150601dfe33b98c107db50 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Thu, 22 Oct 2026 21:24:57 +0200 Subject: [PATCH 4/4] [HWORKS-3565] Upgrade Kueue from the bundled 0.12.2 to 0.18.1 installed outside the Hopsworks namespace, migrating existing queues and CRDs https://hopsworks.atlassian.net/browse/HWORKS-3565 Review comments on logicalclocks/hopsworks-helm#2599. logicalclocks.github.io: the Kueue operator page runs the migration script with bash in every command, because helm package stores it without the execute bit and a direct call fails with permission denied. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- docs/setup_installation/admin/kueue_external.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/setup_installation/admin/kueue_external.md b/docs/setup_installation/admin/kueue_external.md index f387fbce98..61ab8ca8c5 100644 --- a/docs/setup_installation/admin/kueue_external.md +++ b/docs/setup_installation/admin/kueue_external.md @@ -21,7 +21,7 @@ A fresh install that wants an external Kueue installs it first and sets `hw-kueu ## What the migration does -The chart ships `charts/hw-kueue/files/migrate-to-external.sh`, which moves the cluster to Kueue 0.18.1 and keeps every Kueue object: ClusterQueues, Cohorts, ResourceFlavors, Topologies, LocalQueues, priority classes and Workloads. +The chart ships `charts/hw-kueue/files/migrate-to-external.sh`; run it with `bash`, because `helm package` stores files without the execute bit. The script moves the cluster to Kueue 0.18.1 and keeps every Kueue object: ClusterQueues, Cohorts, ResourceFlavors, Topologies, LocalQueues, priority classes and Workloads. It runs in phases, and each one is safe to re-run: | Phase | What it does | @@ -57,7 +57,7 @@ helm pull --untar /hopsworks --version <5.3 version> HELM_EXTRA_ARGS="" \ HOPSWORKS_URL=https:// \ HOPSWORKS_API_KEY= \ - hopsworks/charts/hw-kueue/files/migrate-to-external.sh \ + bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh \ --hopsworks-chart /hopsworks --hopsworks-version <5.3 version> all ``` @@ -96,7 +96,7 @@ Everything under [Before you start](#before-you-start) applies, and: helm pull --untar /hopsworks --version HOPSWORKS_URL=https:// \ HOPSWORKS_API_KEY= \ - hopsworks/charts/hw-kueue/files/migrate-to-external.sh --gitops \ + bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh --gitops \ --release --namespace --backup-dir ./kueue-backup all ``` @@ -184,7 +184,7 @@ LocalQueues are not exported, because Hopsworks creates and deletes them with pr Each step polls every 15 seconds for up to `--timeout`, then stops with `timed out waiting for ...`. Run the same command again: every phase checks what is already done and continues from there, and the restores are applied only once. -A single phase can also be run on its own, such as `migrate-to-external.sh --gitops enable-hopsworks`. +A single phase can also be run on its own, such as `bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh --gitops enable-hopsworks`. ### What the script writes @@ -210,7 +210,7 @@ The script applies the Kueue CRDs itself and the HelmRelease skips them (`crds: The `verify` phase checks the end state; run it on its own at any time: ```sh -hopsworks/charts/hw-kueue/files/migrate-to-external.sh verify +bash hopsworks/charts/hw-kueue/files/migrate-to-external.sh verify ``` To check by hand: