From 5d1a35055bed22a642d67cedf95e1fed8cfe05f6 Mon Sep 17 00:00:00 2001 From: MagicLex <64143547+MagicLex@users.noreply.github.com> Date: Fri, 23 Oct 2026 15:01:16 +0300 Subject: [PATCH 1/2] [HWORKS-3555] Upgrade notes: sidecar resource values Co-Authored-By: Claude Opus 5.5 --- docs/setup_installation/admin/upgrade.md | 56 +++++++++++++++++++++++- 1 file changed, 55 insertions(+), 1 deletion(-) diff --git a/docs/setup_installation/admin/upgrade.md b/docs/setup_installation/admin/upgrade.md index c91d0c31c9..c7c18589d3 100644 --- a/docs/setup_installation/admin/upgrade.md +++ b/docs/setup_installation/admin/upgrade.md @@ -15,7 +15,7 @@ The commands assume Hopsworks is installed in the `hopsworks` namespace. | ------- | ---------- | ----------------- | | [5.0.x to 5.1.x][upgrade-50-to-51] | 1.3.x to 2.19.x | Kubernetes 1.34 or later before the upgrade, a first Helm 4 upgrade, and from 5.0.13 or earlier to 5.1.2 or later [the Trino query history][upgrade-trino-query-history]. | | [5.1.x to 5.2.x][upgrade-51-to-52] | 2.19.x to 3.8.0: upgrade to 5.2.1 or later | The index pass, Kafka moves to KRaft, KEDA, Kubernetes 1.37 only afterwards, Trino, and from 5.2.2 [OnlineFS and the schema migrations][upgrade-changes-from-522]. | -| [5.2.x to 5.3.x][upgrade-52-to-53] | stays on 3.8.0 | ICEBERG feature groups in Trino, the Filebeat registry, the memory of the OTLP sidecar of agent deployments, from 5.2.0 [the subset shares in Trino][upgrade-trino-subset-shares] and the OpenSearch security configuration, and from 5.2.0 or 5.2.1 [OnlineFS and the schema migrations][upgrade-changes-from-522]. | +| [5.2.x to 5.3.x][upgrade-52-to-53] | stays on 3.8.0 | ICEBERG feature groups in Trino, the Filebeat registry, the memory of the OTLP sidecar of agent deployments, [new variables for the resources of the hopsfs-fuse and envoy sidecars][upgrade-sidecar-resources], from 5.2.0 [the subset shares in Trino][upgrade-trino-subset-shares] and the OpenSearch security configuration, and from 5.2.0 or 5.2.1 [OnlineFS and the schema migrations][upgrade-changes-from-522]. | ## Before every upgrade { #upgrade-before-every-upgrade } @@ -218,6 +218,60 @@ On the upgrade to 5.3: - Leave the existing contents of `/var/lib/filebeat-data` in place. Nothing in the chart reads them, and they may belong to another Filebeat on the node. +### hopsfs-fuse and envoy sidecar resources { #upgrade-sidecar-resources } + +5.3 adds three variables, each empty by default, which keeps the pod resources of 5.2: + +- `kube_scheduling_hopsfsmount_memory_requests_mb` is the memory request of the hopsfs-fuse sidecar, in MiB. + Empty means it is not set, so Kubernetes uses `kube_scheduling_hopsfsmount_memory_limits_mb`. +- `python_app_envoy_cpu_limit` and `python_app_envoy_memory_limit_mb` are the limits of the envoy sidecar of a Python app. + Empty means they equal `python_app_envoy_cpu` and `python_app_envoy_memory_mb`. + +5.3 checks these values when it builds a pod. +An invalid value fails the start with an error that names the key. +On a default installation all values are valid. + +- `kube_scheduling_hopsfsmount_cpu_requests`, `kube_scheduling_hopsfsmount_memory_requests_mb`, `kube_scheduling_jobinit_cpu_requests` and `kube_scheduling_jobinit_memory_requests_mb`: a negative value fails. + Before, a negative request was dropped. +- `kube_scheduling_hopsfsmount_memory_limits_mb` and `kube_scheduling_jobinit_memory_limits_mb`: 0 or below fails. + Before, the limit was dropped. +- `kube_scheduling_hopsfsmount_cpu_limits` and `kube_scheduling_jobinit_cpu_limits`: 0 fails. + Before, 0 meant no limit. + A negative value still means no limit, as described below. +- A hopsfs-fuse or job-init request above its limit fails. + Before, Kubernetes rejected the pod. +- A bad hopsfs-fuse key fails every start of a pod that runs the hopsfs-fuse sidecar. + A bad job-init key fails every Spark and Ray job start, and every Jupyter start whose Spark environment replaces the Uniffle client. + For example, `kube_scheduling_jobinit_cpu_requests=-1` fails every Spark and Ray job start. +- `python_app_envoy_cpu` and `python_app_envoy_memory_mb`: a negative value fails. + `python_app_envoy_cpu_limit` and `python_app_envoy_memory_limit_mb`: 0 or below fails. + A request above its limit fails. + Every start of a Python app with monitoring fails. + Before, the values went into the pod unchecked. +- `git_watcher_cpu` and `git_watcher_memory_mb`: 0 or below fails. + Every start of a Python app or agent deployment with git auto-redeploy fails. + Before, the values went into the pod unchecked. +- `agent_deployment_otel_cpu` and `agent_deployment_otel_memory_mb`: 0 or below fails. + Every start of a traced agent deployment fails, for example with `agent_deployment_otel_memory_mb=0`. + Before, the values went into the pod unchecked. +- Every key above: a value that is not a number, or not finite (`NaN`, `Infinity`), fails. + `git_watcher_memory_mb`, `agent_deployment_otel_memory_mb`, `python_app_envoy_memory_mb` and `python_app_envoy_memory_limit_mb` are whole numbers, so a decimal there fails too. + The `kube_scheduling_*_mb` keys accept a decimal. +- `docker_operations_buildkit_storage`, `docker_operations_buildkit_request_cpu`, `docker_operations_buildkit_request_memory`, `docker_operations_buildkit_limit_cpu` and `docker_operations_buildkit_limit_memory`: a value that is not a Kubernetes quantity, or a negative one, fails. + Every Python environment image build fails. + Before, Kubernetes rejected the build pod without naming the key. + +Before upgrading, check the live value of each key above in the values file and in the admin UI. + +`kube_scheduling_hopsfsmount_cpu_limits` and `kube_scheduling_jobinit_cpu_limits` move from -1 to empty, which means no CPU limit. +Deploy a Hopsworks version that reads empty as no limit before this chart version. +The chart rejects -1 or 0 for these two keys, so remove any such override from the values file before upgrading. +Hopsworks still reads a negative value set outside the chart as no limit. +This is deprecated and a later release removes it. + +Chart-owned: set `kube_scheduling_hopsfsmount_memory_requests_mb`, `python_app_envoy_cpu_limit` and `python_app_envoy_memory_limit_mb` in values. +An admin-UI change is reset on the next upgrade. + ### Agent deployments reserve 2 GiB for their OTLP sidecar { #upgrade-agent-otlp-sidecar } An agent deployment with tracing enabled runs a `hops-otlp` sidecar next to the agent. From fdb9d3c44f27f08ee5fca582738c657249efb47f Mon Sep 17 00:00:00 2001 From: MagicLex <64143547+MagicLex@users.noreply.github.com> Date: Fri, 23 Oct 2026 15:18:33 +0300 Subject: [PATCH 2/2] [HWORKS-3555] Upgrade notes: name the Hopsworks version that reads an empty CPU limit Co-Authored-By: Claude Opus 5.5 --- docs/setup_installation/admin/upgrade.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/setup_installation/admin/upgrade.md b/docs/setup_installation/admin/upgrade.md index c7c18589d3..8b4eb773bc 100644 --- a/docs/setup_installation/admin/upgrade.md +++ b/docs/setup_installation/admin/upgrade.md @@ -264,7 +264,8 @@ On a default installation all values are valid. Before upgrading, check the live value of each key above in the values file and in the admin UI. `kube_scheduling_hopsfsmount_cpu_limits` and `kube_scheduling_jobinit_cpu_limits` move from -1 to empty, which means no CPU limit. -Deploy a Hopsworks version that reads empty as no limit before this chart version. +Hopsworks 5.3 reads empty as no limit. +Do not run this chart version with an earlier Hopsworks image. The chart rejects -1 or 0 for these two keys, so remove any such override from the values file before upgrading. Hopsworks still reads a negative value set outside the chart as no limit. This is deprecated and a later release removes it.