[HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.3 and move the Hopsworks Kueue models to v1beta2 by jimdowling · Pull Request #709 · logicalclocks/logicalclocks.github.io

jimdowling · 2026-10-22T06:40:50Z

https://hopsworks.atlassian.net/browse/HWORKS-3579

Summary

docs/setup_installation/admin/kueue_external.md and the Kueue entry in upgrade.md:

A Hopsworks release with Kueue 0.20.3 (a 5.2.x patch release, 5.3) ships no Kueue and reads v1beta2. Fresh install: Kueue first, with the chart's upgrade-kueue.sh install, terraform install_kueue, or under Argo CD an Application for the upstream chart (values for the Spark integration, no resources finalizer). A cluster on the bundled Kueue 0.12 moves to the external Kueue: required, since the new chart refuses the bundled 0.12.
Two scripts: on 5.1 the 5.1 chart's migrate-to-external.sh moves the cluster to Kueue 0.18.1; after the Hopsworks upgrade the new chart's upgrade-kueue.sh takes it to 0.20.3, with Flux (--gitops, which also writes kueue-crds.yaml).
From Hopsworks 5.2.0 or 5.2.1, which ship no migration script: the 5.1 script's phases up to migrate-storage on the cluster's own 5.2 chart, then enable-hopsworks with the new release, so Hopsworks is never downgraded. Tested on jim-kueue-51 (logicalclocks/hopsworks-helm#2638).
upgrade-kueue.sh keeps TopologyAwareScheduling off when the release turns it off, including the six 0.20 gates that depend on it.
Neither script supports a release managed by Argo CD.
The chart creates the default queues only when missing; changing hw-kueue queue values does not change existing queues.
Spark jobs saved with dynamic allocation run their dynamic-allocation minimum.
Airgapped mirrors for 0.18.1 and 0.20.3; --kueue-repo takes oci://. The 0.19 and 0.20 default changes, and the prerequisite check in both modes.

markdownlint-cli2 passes on both files.

Spark jobs on Kueue

Kueue never managed Hopsworks Spark jobs: they run as Spark Operator SparkApplications, which Kueue admits only through its SparkApplication integration (alpha since 0.17, off by default). That integration reserves quota for the driver plus spec.executor.instances and refuses dynamicAllocation.enabled unless the job is elastic, and even then the executors the driver adds are not charged. This PR set therefore:

turns the integration on (SparkApplicationIntegration gate plus the sparkoperator.k8s.io/sparkapplication framework) in hopsworks-helm's upgrade-kueue.sh, the bundled hw-kueue values and terraform's install_kueue.sh (logicalclocks/terraform#178);
runs Spark jobs with static executors while Kueue is on: a configuration with dynamic allocation on is converted at start, save and project-default read to its dynamic-allocation minimum, at least one executor, and written back; only an explicit spark.dynamicAllocation.enabled=true in the Spark properties is refused (130082). Notebook Spark (client mode through Livy) is unchanged;
puts Spark jobs in the project's default queue, including jobs saved before (at save and at start);
writes spec.sparkVersion as semver (4.1.3, not 4.1.3.0): the Spark Operator 2.5.1 webhook ranks an invalid version below 3.0.0 and refused the pod templates Kueue sets when it resumes an admitted job, which left every Spark job suspended.

Spark jobs saved before this release have dynamic allocation on (the old default); they run their dynamic-allocation minimum from their next start, and the docs say so.

Superseded, on Kueue 0.20.2: Spark testing on jim-kueue-ui (integration on, backend from this branch): test_kueue_queues.py, test_spark_job_execution.py, test_default_job_configuration.py, test_execution_rerun.py and test_job_failure_handling.py gave 14 passed. A watcher on the cluster recorded 9 Workloads owned by SparkApplications, all admitted to ClusterQueue other.

Related PRs

logicalclocks/hopsworks-ee#3548 (backend models on v1beta2, Spark under Kueue)
logicalclocks/hopsworks-helm#2638 (bundled 0.20.3, upgrade-kueue.sh, prerequisite check)
logicalclocks/loadtest#1181 (test_kueue_queues.py)
logicalclocks/docker-images#1130 (Kueue image mirrors)

🤖 Generated with Claude Code

…ed outside the Hopsworks namespace, migrating existing queues and CRDs https://hopsworks.atlassian.net/browse/HWORKS-3565 Every release from 4.3 bundles Kueue 0.12.2 in the Hopsworks release namespace. From 5.3 Hopsworks can use a Kueue installed as its own release in kueue-system instead (hw-kueue.kueue.enabled=false), and the chart ships files/migrate-to-external.sh to move a running cluster to Kueue 0.18.1 while keeping every Kueue object. The docs add an operator page for that move: why helm upgrade alone cannot do it, the script's phases, what to prepare (snapshot, maintenance window, tools, airgapped images, admin API key), the command, what changes for queues and Hopsworks afterwards (no topologyName on flavors, parentName on Cohorts, queues managed with kubectl, Hopsworks-namespace jobs queued, the queue utilisation view failing until the backend reads Cohorts at v1beta1) and how to verify. The hw-kueue values page links to it. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ed outside the Hopsworks namespace, migrating existing queues and CRDs https://hopsworks.atlassian.net/browse/HWORKS-3565 Clusters whose releases Flux manages. Flux reconciles a HelmRelease back to Git, so the helm upgrades in remove-bundled, install-kueue and enable-hopsworks would be reverted on a Flux-managed cluster. logicalclocks.github.io: the Kueue operator page gains a section on clusters managed by Flux (what to commit at each phase, what the script waits for, the queue export) and lists export-queues. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ed outside the Hopsworks namespace, migrating existing queues and CRDs https://hopsworks.atlassian.net/browse/HWORKS-3565 Step-by-step instructions for clusters managed by Flux. logicalclocks.github.io: the Flux section of the Kueue operator page becomes a runbook for --gitops: what to have before starting (Flux 2.3 or later, Git access, the Helm release name and namespace, where the HelmRelease values live, a terminal that survives a dropped connection), then one step per change to commit, with the command, the HelmRelease YAML, the flux reconcile command that applies it at once and what the script waits for, then where the queue manifests go (a Kustomization depending on the Kueue one) and how to resume after a timeout. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ed outside the Hopsworks namespace, migrating existing queues and CRDs https://hopsworks.atlassian.net/browse/HWORKS-3565 Review comments on logicalclocks/hopsworks-helm#2599. logicalclocks.github.io: the Kueue operator page runs the migration script with bash in every command, because helm package stores it without the execute bit and a direct call fails with permission denied. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 Kueue 0.20.2 is the latest release and stops serving kueue.x-k8s.io/v1beta1, which every Hopsworks Kueue model still used. Kueue 0.18.1 serves both v1beta1 and v1beta2, so the backend moves to v1beta2 first and runs on 0.18.1 and 0.20.2, then the external Kueue installed by HWORKS-3565 is upgraded to 0.20.2 and the bundled one in hw-kueue moves to 0.20.2 as well. Tested on the Proxmox cluster jim-kueue-ui (snurran, 4 GPUs), migrated to 0.18.1 by HWORKS-3565: the upgrade kept all 106 queue objects and verify passed, and the loadtests gave 31 passed, 1 skipped on 0.20.2 (one Ray MNIST run failed in its training script and passed when rerun). logicalclocks.github.io: the Kueue operator page says 5.3 bundles Kueue 0.20.2 and that a cluster on the bundled 0.12 moves to the external Kueue when it upgrades. It adds the upgrade-kueue phase, the managedFields reset, the path for a cluster already on 0.18.1, the mirrors an airgapped cluster needs, the 0.19 and 0.20 default changes, and the prerequisite check for both modes. The queue utilisation gap is gone. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 Spark jobs on Kueue. Kueue never managed Hopsworks Spark jobs: they run as Spark Operator SparkApplications, which Kueue admits only through its SparkApplication integration, alpha since 0.17 and off by default (SparkApplicationIntegration gate, sparkoperator.k8s.io/sparkapplication commented out of the default frameworks). The integration reserves quota for the driver plus spec.executor.instances and refuses dynamicAllocation.enabled unless the job is elastic, and even an elastic job leaves the executors its driver adds uncharged. So the integration is turned on and, while Kueue is on, Spark jobs run with static executors only. logicalclocks.github.io: the Kueue details page gains a Spark jobs section (static executors only, error 130082, defaults, jobs saved before Kueue, notebooks unaffected) and its examples move to kueue.x-k8s.io/v1beta2 (cohortName, no topologyName on the default flavor). The scheduler page, the Spark and PySpark job guides and the Kueue operator page say Spark jobs are queued with static executors. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 Spark jobs reach Kueue. With the SparkApplication integration on, a test run still showed no Spark Workload: the backend gave Spark jobs no queue, so their SparkApplications carried no queue-name label and Kueue ignored them. With a queue, Kueue admitted the job and then could not start it: unsuspending sets the driver and executor pod templates, and the Spark Operator 2.5.1 webhook refused that with "pod template feature requires Spark version 3.0.0 or higher", because it compares spec.sparkVersion as semver and ranks an invalid one below every valid one, and Hopsworks writes 4.1.3.0. On jim-kueue-ui, Kueue 0.20.2 with the integration on, the Kueue, Spark job, default job configuration, rerun and job failure loadtests gave 14 passed, and Kueue admitted 9 SparkApplication Workloads into ClusterQueue other. logicalclocks.github.io: the Kueue details page says Spark jobs go to the project's default queue unless they select another, and that a Spark job saved without a queue gets the default when it starts. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 Fixes from a Codex review of the change set (nine findings and one suspicion, each checked against the code before it was fixed). logicalclocks.github.io: the Kueue operator page says the managedFields cleanup drops only old entries, that the 0.20.2 chart comes from the upstream OCI registry (--kueue-target-repo), and which flags point an airgapped cluster at its mirrors. Reviewed-by: OpenAI Codex (codex-plugin-cc 1.0.6) <codex@openai.com> Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 Rebased on HWORKS-3565 with its Flux runbook. logicalclocks.github.io: the Flux runbook on the Kueue operator page gains Step 5, upgrade-kueue: the script applies the 0.20.2 CRDs and rewrites the Kueue HelmRelease for 0.20.2 (chart from the upstream OCI registry, Spark settings, a post-render patch that keeps the chart's templated CRDs out of the release), the operator replaces the file committed in Step 3, and the script waits for the controller to run 0.20.2. Verification and the queue export become Step 6. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 Kueue 0.20.3. The upgrade-kueue entry states that it waits for the Hopsworks release to finish deploying, keeps release kueue's own values and refuses a manager configuration that is not v1beta2. Reviewed-by: OpenAI Codex (codex-plugin-cc 1.0.6) <codex@openai.com> Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

o-alex · 2026-10-23T09:26:02Z

Move the upgrade note into upgrade.md. logicalclocks.github.io#711 has merged: docs/setup_installation/admin/upgrade.md is now the upgrade guide, with a ## 5.2.x to 5.3.x section (#upgrade-52-to-53). hopsworks-helm#2609 deletes UPGRADE.md, and maismail asked on hopsworks-helm#2602 for that PR's upgrade note to move to the docs site and merge no later than the chart PR.

Please add a ### Kueue 0.20 subsection under 5.2.x to 5.3.x and drop the 150 UPGRADE.md lines from hopsworks-helm#2602. Given the forced hop through 5.1/5.2 (see the comment on hopsworks-helm#2602), it only needs:

the prerequisite: a cluster with Kueue must already be on external Kueue 0.18.1, moved there by the 5.1 or 5.2 migrate-to-external.sh. A cluster still on the bundled 0.12 upgrades to 5.1 or 5.2 first.
the step: upgrade Hopsworks to 5.3, then run upgrade-kueue and verify.
the behaviour changes: Spark jobs need static executors on Kueue, 0.19/0.20 waitForPodsReady 30m, Ray sidecars charged to quota, the default priority class.
a link to kueue_external.md for the details.

kueue_external.md itself can then drop the bundled 0.12 migration and its Flux runbook (:19-78, :89-215), which apply to 5.1/5.2, not 5.3.

Upstream published the Helm upgrade guide here (HWORKS-3580) and moved the Kafka 4 notes under a new Upgrades section of the admin nav. Kueue outside the Hopsworks namespace moves into that section, and the upgrade guide's 5.2.x to 5.3.x section gets an entry for it that links the runbook, replacing the UPGRADE.md section the chart no longer ships. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 The 5.3 chart no longer ships migrate-to-external.sh: a cluster on the bundled Kueue 0.12 moves to the external Kueue 0.18.1 on Hopsworks 5.1, upgrades Hopsworks to 5.3, then upgrades Kueue to 0.20.3 with the 5.1 script's upgrade-kueue phase. docs: kueue_external.md runs the move in those three steps, with the latest 5.1 chart, and the Flux steps follow the same order. all no longer includes upgrade-kueue, which refuses a Hopsworks release older than 5.3. The --kueue-image option for the upstream images is listed. The upgrade guide's Kueue entry says where the script comes from. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 The Kueue 0.20 changes ship in a 5.2.x patch release as well as in 5.3. docs: kueue_external.md and the upgrade guide's Kueue entry name the target as a Hopsworks release that bundles Kueue 0.20, a 5.2.x patch release or 5.3, in place of 5.3, and upgrade-kueue's refusal as a chart that bundles an older Kueue. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 A 5.1 cluster stays on Kueue 0.18; the step to Kueue 0.20.3 ships with the Hopsworks release that bundles it. docs: kueue_external.md names two scripts. The 5.1 chart's migrate-to-external.sh moves a cluster to Kueue 0.18.1 on 5.1, and the new chart's upgrade-kueue.sh, run after the Hopsworks upgrade, takes it to 0.20.3. The upgrade-kueue phase row gives way to a description of upgrade-kueue.sh and its refusals; the Flux Step 5 runs it with --gitops and commits the HelmRelease it writes to ./kueue-upgrade. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 docs: a 5.2.0 or 5.2.1 cluster on the bundled Kueue 0.12 moves with the 5.1 chart's migrate-to-external.sh run phase by phase: the phases up to migrate-storage on its own 5.2 chart, then enable-hopsworks with the new release, so Hopsworks is never downgraded. The upgrade note says the move is required for a cluster on the bundled Kueue, since the new chart refuses it, and that upgrade-kueue.sh, not the 5.1 script, takes Kueue to 0.20.3. The guide also covers what changed in the scripts and chart: Spark jobs saved with dynamic allocation run their dynamic-allocation minimum instead of being refused; upgrade-kueue.sh keeps TopologyAwareScheduling off when a release turns it off, and with --gitops writes kueue-crds.yaml and waits for the SparkApplication integration; the bundled CRDs carry an Argo CD no-prune, no-delete option; the chart tells its own CRDs by label and refuses an upgrade that turns Kueue on while they are missing. Neither script supports a release managed by Argo CD. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011tczJCVzi8QNsr26x1rDNh

…rks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 docs: the "Jobs in the Hopsworks namespace are queued" change is removed. Hopsworks creates its Kueue-queued system jobs in the project namespaces, so Kueue 0.18 excluding its own namespace changes nothing for Hopsworks. The Kueue entry in upgrade.md lists what changes for users and operators: static Spark executors, the 30-minute waitForPodsReady, Ray sidecars charged to quota and the default priority class, with a link to the details. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011tczJCVzi8QNsr26x1rDNh

jimdowling · 2026-10-24T11:41:22Z

@o-alex The upgrade note is in upgrade.md, under 5.2.x to 5.3.x (#upgrade-kueue-external). Since 1ba52fe it lists the behaviour changes you named (static Spark executors, the 30-minute waitForPodsReady, Ray sidecars charged to quota, the default priority class) and links kueue_external.md#what-changes. hopsworks-helm#2602 was closed; its replacement, hopsworks-helm#2638, carries no UPGRADE.md.

I kept the 0.12 migration and its Flux runbook in kueue_external.md. The change now ships in a 5.2.x patch release as well as 5.3, so the forced hop no longer holds. A cluster on 5.2.0 or 5.2.1 with the bundled Kueue 0.12 upgrades straight to the patch: the 5.1 chart's migrate-to-external.sh runs its phases up to migrate-storage on the cluster's own 5.2 chart, then enable-hopsworks with the new release, and Hopsworks is never downgraded. The guide's "From Hopsworks 5.2.0 or 5.2.1" section describes it, and it was tested from 5.2.1 with seeded queues and saved Spark jobs (hopsworks-helm#2638). Operators on those releases need those sections.

…rks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 docs: the intro names the releases that bundle Kueue 0.20.3; "Those releases" pointed back at 4.3 to 5.2, which bundle 0.12.2. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011tczJCVzi8QNsr26x1rDNh

…rks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 docs: the Hopsworks chart ships no Kueue, as it ships no Kyverno. A new Fresh install section installs Kueue 0.20.3 first as release kueue in kueue-system with the chart's upgrade-kueue.sh install, and covers Argo CD: an Application of its own for the upstream chart with the SparkApplication settings, synced first and without the resources finalizer. The chart creates the default queues only when missing, and its prerequisite check refuses kueue.enabled=true. The bundled 0.20.3 and its crds/ are gone from the guide and the upgrade note. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011tczJCVzi8QNsr26x1rDNh

jimdowling force-pushed the HWORKS-3579 branch from 66fd21e to 52482e1 Compare October 22, 2026 07:19

jimdowling marked this pull request as ready for review October 22, 2026 09:11

jimdowling force-pushed the HWORKS-3579 branch 2 times, most recently from 2aa0b81 to 22ee8dc Compare October 22, 2026 11:39

jimdowling and others added 3 commits October 22, 2026 19:04

jimdowling force-pushed the HWORKS-3579 branch from 22ee8dc to 16c9c30 Compare October 22, 2026 17:05

jimdowling and others added 6 commits October 22, 2026 21:24

jimdowling force-pushed the HWORKS-3579 branch from 16c9c30 to 9cdd58b Compare October 22, 2026 19:25

o-alex reviewed Oct 23, 2026

View reviewed changes

Comment thread docs/setup_installation/admin/kueue_external.md Outdated

jimdowling mentioned this pull request Oct 23, 2026

[HWORKS-3565] Upgrade Kueue from the bundled 0.12.2 to 0.18.1 installed outside the Hopsworks namespace, migrating existing queues and CRDs #708

Closed

jimdowling and others added 7 commits October 23, 2026 16:57

Merge remote-tracking branch 'origin/HWORKS-3565' into HWORKS-3579

fc60510

Merge remote-tracking branch 'upstream/main' into HWORKS-3579

bf5b590

o-alex reviewed Oct 24, 2026

View reviewed changes

Comment thread docs/setup_installation/admin/kueue_external.md Outdated

jimdowling changed the title ~~[HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2~~ [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.3 and move the Hopsworks Kueue models to v1beta2 Oct 24, 2026

jimdowling and others added 2 commits October 24, 2026 14:07

o-alex approved these changes Oct 25, 2026

View reviewed changes

jimdowling merged commit f195a5d into logicalclocks:main Oct 25, 2026
1 check passed

Navigation Menu

Repository navigation

[HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.3 and move the Hopsworks Kueue models to v1beta2 - #709

[HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.3 and move the Hopsworks Kueue models to v1beta2#709
jimdowling merged 20 commits into
logicalclocks:mainfrom
jimdowling:HWORKS-3579

jimdowling commented Oct 22, 2026 •

edited

Loading

Uh oh!

o-alex commented Oct 23, 2026

Uh oh!

Uh oh!

Uh oh!

jimdowling commented Oct 24, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jimdowling commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Spark jobs on Kueue

Related PRs

Uh oh!

o-alex commented Oct 23, 2026

Uh oh!

Uh oh!

Uh oh!

jimdowling commented Oct 24, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jimdowling commented Oct 22, 2026 •

edited

Loading