From 30e1158ab767f4aae739a0a743e21e808d574785 Mon Sep 17 00:00:00 2001 From: MagicLex <64143547+MagicLex@users.noreply.github.com> Date: Sat, 24 Oct 2026 12:04:59 +0300 Subject: [PATCH 1/5] [HABLE-142] Upgrade notes: MinIO priority class, registry probes, node filesystem alert Co-Authored-By: Claude Opus 5.5 --- docs/setup_installation/admin/upgrade.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/setup_installation/admin/upgrade.md b/docs/setup_installation/admin/upgrade.md index c91d0c31c9..fb0d3b3055 100644 --- a/docs/setup_installation/admin/upgrade.md +++ b/docs/setup_installation/admin/upgrade.md @@ -239,6 +239,23 @@ From 5.3 the default is 2048, up from 1024, because at 1 GiB the sidecar was OOM While the sidecar restarts, the pod reports `Ready=False` and leaves the endpoints of its service, so the agent, healthy and still running, receives no requests. Looking for agent restarts finds none. +### MinIO priority class and registry probes { #upgrade-minio-priority } + +When a node runs short of disk, the kubelet evicts its lowest-priority pods first. +Before 5.3 MinIO had no priority class, so it could be evicted before ordinary workloads, and HopsFS had no block store until it came back. + +- The chart creates a cluster-scoped PriorityClass `-minio-priority` (value 1000000) and sets it on the MinIO StatefulSet. + The account that runs `helm upgrade` needs permission to create PriorityClasses. + The value cannot be changed after the class is created. + To use an existing class, set `minio.priorityClassName`; to keep MinIO without a priority class, set `minio.priorityClass.create: false`. +- The MinIO pod is recreated by the upgrade, so HopsFS reads and writes through MinIO pause until it is ready again. +- The registry's liveness probe checks that the process answers, and its readiness probe keeps `/v2/`. + While MinIO is unavailable the registry pods leave the service instead of restarting, and image pulls fail until MinIO is back. + The registry pods are recreated by the upgrade. +- The registry no longer logs its S3 requests (`REGISTRY_STORAGE_S3_LOGLEVEL` is `off`). +- A new Prometheus alert, `NodeRootFilesystemLow` (warning), fires when a node's root filesystem has less than 15% free for 10 minutes. + It is only rendered when `prometheus-node-exporter.enabled` is true. + ## Changes from 5.2.2 { #upgrade-changes-from-522 } These changes ship in 5.2.2 and 5.3.0. From e68980e3db5ddf0c9e2a52ef2a64cc4c87fbcf6d Mon Sep 17 00:00:00 2001 From: MagicLex <64143547+MagicLex@users.noreply.github.com> Date: Sat, 24 Oct 2026 12:12:16 +0300 Subject: [PATCH 2/5] [HABLE-142] Upgrade notes: priority orders eviction, it does not prevent it Co-Authored-By: Claude Opus 5.5 --- docs/setup_installation/admin/upgrade.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/setup_installation/admin/upgrade.md b/docs/setup_installation/admin/upgrade.md index fb0d3b3055..5b482d183f 100644 --- a/docs/setup_installation/admin/upgrade.md +++ b/docs/setup_installation/admin/upgrade.md @@ -241,8 +241,9 @@ From 5.3 the default is 2048, up from 1024, because at 1 GiB the sidecar was OOM ### MinIO priority class and registry probes { #upgrade-minio-priority } -When a node runs short of disk, the kubelet evicts its lowest-priority pods first. -Before 5.3 MinIO had no priority class, so it could be evicted before ordinary workloads, and HopsFS had no block store until it came back. +When a node runs short of disk, the kubelet evicts pods ranked by their usage against their requests, then by priority. +Before 5.3 MinIO had no priority class, so it could be evicted ahead of ordinary workloads, and HopsFS had no block store until it came back. +A priority class moves MinIO later in that order; it does not prevent its eviction. - The chart creates a cluster-scoped PriorityClass `-minio-priority` (value 1000000) and sets it on the MinIO StatefulSet. The account that runs `helm upgrade` needs permission to create PriorityClasses. From 01f0d2d1e8e8bfe10de1c44bd06a3a0d0f07050a Mon Sep 17 00:00:00 2001 From: MagicLex <64143547+MagicLex@users.noreply.github.com> Date: Tue, 27 Oct 2026 11:41:18 +0200 Subject: [PATCH 3/5] [HABLE-142] Upgrade notes: MinIO runs as a critical pod Co-Authored-By: Claude Opus 5.5 --- docs/setup_installation/admin/upgrade.md | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/docs/setup_installation/admin/upgrade.md b/docs/setup_installation/admin/upgrade.md index 5b482d183f..901d330b86 100644 --- a/docs/setup_installation/admin/upgrade.md +++ b/docs/setup_installation/admin/upgrade.md @@ -239,16 +239,14 @@ From 5.3 the default is 2048, up from 1024, because at 1 GiB the sidecar was OOM While the sidecar restarts, the pod reports `Ready=False` and leaves the endpoints of its service, so the agent, healthy and still running, receives no requests. Looking for agent restarts finds none. -### MinIO priority class and registry probes { #upgrade-minio-priority } +### MinIO as a critical pod and registry probes { #upgrade-minio-priority } -When a node runs short of disk, the kubelet evicts pods ranked by their usage against their requests, then by priority. -Before 5.3 MinIO had no priority class, so it could be evicted ahead of ordinary workloads, and HopsFS had no block store until it came back. -A priority class moves MinIO later in that order; it does not prevent its eviction. +When a node runs short of disk, the kubelet evicts the pods that use more ephemeral storage than they request first, then by priority. +It never evicts a critical pod (priority class `system-cluster-critical` or `system-node-critical`), and while the pressure lasts it admits only critical pods. +Before 5.3 MinIO was not critical, so it could be evicted ahead of ordinary workloads and could not start again until the pressure cleared; HopsFS had no block store in that time. +Evicting MinIO frees no disk: its data is a hostPath volume. -- The chart creates a cluster-scoped PriorityClass `-minio-priority` (value 1000000) and sets it on the MinIO StatefulSet. - The account that runs `helm upgrade` needs permission to create PriorityClasses. - The value cannot be changed after the class is created. - To use an existing class, set `minio.priorityClassName`; to keep MinIO without a priority class, set `minio.priorityClass.create: false`. +- The MinIO StatefulSet runs with priority class `system-cluster-critical` (`minio.priorityClassName`). - The MinIO pod is recreated by the upgrade, so HopsFS reads and writes through MinIO pause until it is ready again. - The registry's liveness probe checks that the process answers, and its readiness probe keeps `/v2/`. While MinIO is unavailable the registry pods leave the service instead of restarting, and image pulls fail until MinIO is back. From 2f5f27d13ec5ea075f7307fced7ef739d32f6e91 Mon Sep 17 00:00:00 2001 From: MagicLex <64143547+MagicLex@users.noreply.github.com> Date: Tue, 27 Oct 2026 11:55:55 +0200 Subject: [PATCH 4/5] [HABLE-142] Upgrade notes name the kubelet's eviction manager Co-Authored-By: Claude Opus 5.5 --- docs/setup_installation/admin/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/setup_installation/admin/upgrade.md b/docs/setup_installation/admin/upgrade.md index 901d330b86..7268fb7959 100644 --- a/docs/setup_installation/admin/upgrade.md +++ b/docs/setup_installation/admin/upgrade.md @@ -242,7 +242,7 @@ From 5.3 the default is 2048, up from 1024, because at 1 GiB the sidecar was OOM ### MinIO as a critical pod and registry probes { #upgrade-minio-priority } When a node runs short of disk, the kubelet evicts the pods that use more ephemeral storage than they request first, then by priority. -It never evicts a critical pod (priority class `system-cluster-critical` or `system-node-critical`), and while the pressure lasts it admits only critical pods. +Under resource pressure its eviction manager does not evict a critical pod (priority class `system-cluster-critical` or `system-node-critical`) and admits only critical pods. Before 5.3 MinIO was not critical, so it could be evicted ahead of ordinary workloads and could not start again until the pressure cleared; HopsFS had no block store in that time. Evicting MinIO frees no disk: its data is a hostPath volume. From 26d925321fa7c59e6921a9667a01f30e9175bcf2 Mon Sep 17 00:00:00 2001 From: MagicLex <64143547+MagicLex@users.noreply.github.com> Date: Tue, 27 Oct 2026 14:17:58 +0200 Subject: [PATCH 5/5] [HABLE-142] Upgrade notes: MinIO's priority class is an optional value Co-Authored-By: Claude Opus 5.5 --- docs/setup_installation/admin/upgrade.md | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/docs/setup_installation/admin/upgrade.md b/docs/setup_installation/admin/upgrade.md index 7268fb7959..1498aac349 100644 --- a/docs/setup_installation/admin/upgrade.md +++ b/docs/setup_installation/admin/upgrade.md @@ -239,15 +239,10 @@ From 5.3 the default is 2048, up from 1024, because at 1 GiB the sidecar was OOM While the sidecar restarts, the pod reports `Ready=False` and leaves the endpoints of its service, so the agent, healthy and still running, receives no requests. Looking for agent restarts finds none. -### MinIO as a critical pod and registry probes { #upgrade-minio-priority } +### MinIO priority class and registry probes { #upgrade-minio-priority } -When a node runs short of disk, the kubelet evicts the pods that use more ephemeral storage than they request first, then by priority. -Under resource pressure its eviction manager does not evict a critical pod (priority class `system-cluster-critical` or `system-node-critical`) and admits only critical pods. -Before 5.3 MinIO was not critical, so it could be evicted ahead of ordinary workloads and could not start again until the pressure cleared; HopsFS had no block store in that time. -Evicting MinIO frees no disk: its data is a hostPath volume. - -- The MinIO StatefulSet runs with priority class `system-cluster-critical` (`minio.priorityClassName`). -- The MinIO pod is recreated by the upgrade, so HopsFS reads and writes through MinIO pause until it is ready again. +- `minio.priorityClassName` sets a priority class on the MinIO StatefulSet; it is empty by default. + On a cluster where MinIO keeps its data on a node-local volume, `system-cluster-critical` keeps the kubelet's eviction manager from evicting it under disk pressure, where evicting it frees no disk. - The registry's liveness probe checks that the process answers, and its readiness probe keeps `/v2/`. While MinIO is unavailable the registry pods leave the service instead of restarting, and image pulls fail until MinIO is back. The registry pods are recreated by the upgrade.