From 3025284499c78d6f7d8cddc8eb472befdeb76b4a Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 14 Oct 2026 11:59:16 +0000 Subject: [PATCH 1/6] docs(networking): bring the Gateway API page up to 1.7 The page still described pre-1.7 behaviour as fact. TLS-passthrough hostnames are now reserved by the TenantGateway spec alone (cozystack/cozystack#3342, #4638): under HTTP-01 such a name gets no HTTPS listener and no certificate, route or no route, and a hostname only a TLSRoute claims earns neither. The three platform passthrough listeners render only on the publishing tenant, while gateway.enabled is on, for the names publishing.exposedServices lists, so a removed service loses its listener as well. The listener budget, the HTTP-01 listener layout and the rate-limit note are corrected to match. The route hostname policy now requires a non-empty spec.hostnames (cozystack/cozystack#3475) and has a third policy for GRPCRoute (cozystack/cozystack#3872). The Layer 5 text, the diagram label and the troubleshooting entry, whose quoted message changed, follow. Adds a section for TenantGateway tlsPassthroughListeners, the native-port passthrough form from #3342. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- .../en/docs/next/networking/gateway-api.md | 82 +++++++++++++++---- 1 file changed, 68 insertions(+), 14 deletions(-) diff --git a/content/en/docs/next/networking/gateway-api.md b/content/en/docs/next/networking/gateway-api.md index f1b7d56a..9eb5d0b3 100644 --- a/content/en/docs/next/networking/gateway-api.md +++ b/content/en/docs/next/networking/gateway-api.md @@ -48,7 +48,7 @@ flowchart TD The controller: - Materialises the `Gateway`, the redirect HTTPRoute, and — on the ACME cert modes — the per-tenant `Issuer` and the Certificate(s) from `TenantGateway.spec`. In `existingSecret` mode it mints neither, and points the listeners at the operator-supplied Secret instead. In `edge` mode it renders no redirect HTTPRoute either, and deletes one it previously owned. -- Watches `HTTPRoute` and `TLSRoute` resources cluster-wide. For each route attached to its Gateway, it picks up the hostnames and (in HTTP-01 mode) appends a per-app HTTPS listener + a per-app `Certificate`. +- Watches `HTTPRoute` and `TLSRoute` resources cluster-wide. For each hostname an `HTTPRoute` attached to its Gateway claims, it (in HTTP-01 mode) appends a per-app HTTPS listener + a per-app `Certificate`, unless a TLS-passthrough listener answers that hostname. A hostname that only a `TLSRoute` claims earns neither: its backend presents its own certificate. - In DNS-01 mode, extends the wildcard `Certificate` with `` + `*.` SANs for every tenant inheriting through this Gateway (discovered by listing namespaces with `namespace.cozystack.io/gateway = ` and reading their `namespace.cozystack.io/host`), and adds one `*.` HTTPS listener per inheriting child. - Patches `namespace.cozystack.io/gateway = ` onto every namespace in `TenantGateway.spec.attachedNamespaces` (the cozy-* system namespaces published through the Gateway). The patch is annotated with `cozystack.io/gateway-attached-by` so the controller knows which labels it wrote and which are owned by the `apps/tenant` chart — labels written by the chart are never touched. Labels written by the controller are garbage-collected when the namespace is removed from `attachedNamespaces`. - In HTTP-01 mode, resolves cross-namespace hostname conflicts: `cozy-*` namespaces (cluster-admin-managed platform services) win over tenant namespaces; the loser receives a `HostnameConflict` condition under the controller's name in `Status.Parents`. The other modes gather no claims, so nothing is resolved and no route condition is written. @@ -93,19 +93,19 @@ On every cert mode except [edge](#edge-tls-terminated-by-the-class-provider), a Plus HTTPS listeners that depend on cert mode: -- **HTTP-01 mode (default):** one HTTPS listener per attached route hostname — `HTTPRoute` and `TLSRoute` alike, because the controller collects hostnames from both kinds without distinguishing them — named `https--<8-hex>`. Every claimed hostname gets exactly one listener and one certificate, owned by the route that wins it; a route that loses a cross-namespace race is marked `Accepted=False` by [conflict resolution](#hostnameconflict-resolution) rather than rejected, and the hostname itself is still served. The hex suffix is the first 32 bits of `sha256` over the **lower-cased** hostname, so two different hostnames sharing the same first label (`harbor.foo.example.com` vs `harbor.alice.example.com`) get distinct listener names. Each listener's `tls.certificateRefs` points at a per-listener `Certificate` named `--<8-hex>-tls`, also auto-issued. +- **HTTP-01 mode (default):** one HTTPS listener per hostname an attached `HTTPRoute` claims, named `https--<8-hex>`. A hostname a TLS-passthrough listener answers is reserved to that listener and gets no HTTPS listener (see [TLS passthrough](#tlsroute-tls-passthrough)), and a hostname that only a `TLSRoute` claims gets none either. Every other claimed hostname gets exactly one listener and one certificate, owned by the route that wins it; a route that loses a cross-namespace race is marked `Accepted=False` by [conflict resolution](#hostnameconflict-resolution) rather than rejected, and the hostname itself is still served. The hex suffix is the first 32 bits of `sha256` over the **lower-cased** hostname, so two different hostnames sharing the same first label (`harbor.foo.example.com` vs `harbor.alice.example.com`) get distinct listener names. Each listener's `tls.certificateRefs` points at a per-listener `Certificate` named `--<8-hex>-tls`, also auto-issued. - **DNS-01 mode (opt-in):** `https` (`*.`) and `https-apex` (``) listeners consuming a single wildcard Certificate, plus one `https-child--<8-hex>` listener per inheriting child apex (referencing the same wildcard cert, whose dnsNames are extended with `` + `*.` SANs). - **existingSecret mode (operator-provided wildcard):** the same listener set as DNS-01 — `https` (`*.`), `https-apex` (``), and one `https-child--<8-hex>` per inheriting child apex — except that every `tls.certificateRefs` points at the operator-supplied Secret named by `publishing.certificates.wildcardSecretName`, and no `Certificate` is issued for any of them. - **[edge mode](#edge-tls-terminated-by-the-class-provider):** no HTTPS listeners at all. The Gateway serves port 80 only, with `edge` (`*.`), `edge-apex` (``) and one `edge-child--<8-hex>` per inheriting child apex, and neither the hostname-less `http` listener above nor any passthrough listener is rendered. -Plus one extra listener per TLS-passthrough service (see [TLS passthrough](#tlsroute-tls-passthrough)), on every mode except edge. +Plus, on every mode except edge, one extra listener per TLS-passthrough service on the publishing tenant (see [TLS passthrough](#tlsroute-tls-passthrough)) and one per entry in the `TenantGateway`'s `tlsPassthroughListeners` (see [TLS passthrough on native ports](#tls-passthrough-on-native-ports)). Listener `allowedRoutes.namespaces` uses two different selectors by listener role: - **HTTPS and TLS-passthrough listeners — and, in [edge](#edge-tls-terminated-by-the-class-provider) mode, the port-80 `edge*` listeners that stand in for them** — match the `namespace.cozystack.io/gateway` label and admit routes from any namespace whose label equals the owner tenant's namespace name (e.g. `tenant-root`, `tenant-alice` — the namespace name, not the bare tenant name). This is the inheritance hinge — every inheriting child's namespace carries the same label value (written by the `apps/tenant` chart), and cozy-* system namespaces in `attachedNamespaces` get the same label patched on by the controller. - **The plain-HTTP listener (port 80)** uses a strictly narrower whitelist on the built-in `kubernetes.io/metadata.name` label — only the owner tenant's namespace itself (where the controller-owned redirect HTTPRoute lives) and `cozy-cert-manager` (HTTP-01 ACME challenge HTTPRoutes). App HTTPRoutes attaching to the Gateway by hostname therefore cannot bind to port 80 and serve plaintext. This listener, and with it that guarantee, exists only on the non-edge modes: an [edge](#edge-tls-terminated-by-the-class-provider) Gateway has no hostname-less port-80 listener, and the port-80 listeners it does render carry the wide selector, so app routes bind them by design. -Port-443 listeners pin `allowedRoutes.kinds` to `HTTPRoute` and `TLSRoute` (the pair is listed on each of them so that Cilium keeps the listeners apart), preventing GRPCRoute / TCPRoute / UDPRoute from attaching outside the route-hostname VAP's coverage. +Port-443 listeners pin `allowedRoutes.kinds` to `HTTPRoute` and `TLSRoute` (the pair is listed on each of them so that Cilium keeps the listeners apart), so `GRPCRoute`, `TCPRoute` and `UDPRoute` cannot attach to them. The port-80 listener leaves `allowedRoutes.kinds` unset, which for an HTTP listener admits `GRPCRoute` as well, so the route-hostname VAP covers `GRPCRoute` too (Layer 5 under [Security](#security)). ## Enabling Gateway API @@ -287,7 +287,7 @@ Edge therefore wins over an operator-supplied wildcard Secret, which wins over t Out of the box, no extra config required. The controller: - Renders an ACME `Issuer` in the tenant namespace with an `http01.gatewayHTTPRoute` solver pointing at the tenant's own Gateway / `http` listener. -- Watches HTTPRoutes / TLSRoutes attached to the Gateway (parentRefs pointing at it), collecting hostnames from both kinds without distinguishing them. For each unique hostname claimed it adds one per-app HTTPS listener and one per-app `Certificate` (dnsNames containing exactly that hostname), owned by the route that wins that hostname. +- Watches HTTPRoutes / TLSRoutes attached to the Gateway (parentRefs pointing at it). For each unique hostname an `HTTPRoute` claims it adds one per-app HTTPS listener and one per-app `Certificate` (dnsNames containing exactly that hostname), owned by the route that wins that hostname. A hostname a TLS-passthrough listener answers earns neither, whether or not a `TLSRoute` is attached there, and neither does a hostname that only a `TLSRoute` claims. - Per-app listener naming: `https--<8-hex>` (e.g. `https-harbor-deadbeef`). - Per-app cert naming: `--<8-hex>-tls`. @@ -310,7 +310,7 @@ DNS-01 mode renders a single wildcard `Certificate` covering `` and The platform chart writes the provider config into `_cluster.dns01-*` keys consumed by both the per-tenant gateway chart (rendering the TenantGateway CR) and the cluster-wide `letsencrypt-prod` / `letsencrypt-stage` ClusterIssuers used by the legacy ingress flow. Both paths agree on which provider is active. -Pick DNS-01 when you specifically want a wildcard cert — a long-lived cluster with many apps under one apex, deep inheritance trees, or tight Let's Encrypt rate limits. Gateway API caps `Gateway.spec.listeners` at 64, and HTTP-01 spends that budget fastest: one HTTPS listener per published hostname, plus the mandatory `http` listener, plus one passthrough listener per TLS-passthrough service. Budget those passthrough services at **two** slots each, not one — the controller collects hostnames from `TLSRoute` as well as `HTTPRoute`, so each passthrough FQDN also claims an HTTPS listener of its own. With the three default passthrough services that is `1 + N + 6`, and a single-tenant deployment approaching 60 published apps on HTTP-01 will hit the cap, at which point the rendered `Gateway` fails admission. DNS-01 collapses every hostname under the apex into a small fixed number of listeners. +Pick DNS-01 when you specifically want a wildcard cert — a long-lived cluster with many apps under one apex, deep inheritance trees, or tight Let's Encrypt rate limits. Gateway API caps `Gateway.spec.listeners` at 64, and HTTP-01 spends that budget fastest: one HTTPS listener per published hostname, plus the mandatory `http` listener, plus one passthrough listener per TLS-passthrough service and one per `tlsPassthroughListeners` entry. A passthrough hostname takes one slot, not two: it is reserved to its passthrough listener and earns no HTTPS listener of its own. On the publishing tenant with the three default passthrough services that is `1 + N + 3`, so a single-tenant deployment approaching 60 published apps on HTTP-01 will hit the cap. The controller counts the assembled listeners before it writes the `Gateway` and reports a total over 64 as `Ready=False` on the `TenantGateway`, with a message naming the budget. DNS-01 collapses every hostname under the apex into a small fixed number of listeners. ### existingSecret (operator-provided wildcard) @@ -387,10 +387,64 @@ Services that need SNI-based passthrough (clients present certificates, backend | KubeVirt VM export | `cozy-kubevirt` | `vm-exportproxy` | `vm-exportproxy:443` | `tls-vm-exportproxy` | `gateway.enabled` AND `vm-exportproxy` in `publishing.exposedServices` | | KubeVirt CDI upload | `cozy-kubevirt-cdi` | `cdi-uploadproxy` | `cdi-uploadproxy:443` | `tls-cdi-uploadproxy` | `gateway.enabled` AND `cdi-uploadproxy` in `publishing.exposedServices` | -On every cert mode except [edge](#edge-tls-terminated-by-the-class-provider), all three Passthrough listeners (`tls-api`, `tls-vm-exportproxy`, `tls-cdi-uploadproxy`) are rendered on the Gateway regardless of `publishing.exposedServices` — the controller materialises one per entry in the chart's `tlsPassthroughServices` value (defaults: `[api, vm-exportproxy, cdi-uploadproxy]`). In edge mode that value is not written at all and none of the three exists, so the paragraph below does not apply there. What `publishing.exposedServices` actually gates is the matching `TLSRoute` template in each upstream chart: if a service is removed from `publishing.exposedServices`, its listener still exists but nothing attaches, so no traffic is admitted. +The controller materialises one Passthrough listener per entry in the gateway chart's `tlsPassthroughServices` value (defaults: `[api, vm-exportproxy, cdi-uploadproxy]`), and the chart writes that value only where the platform's own `TLSRoute`s attach: on the publishing tenant, the namespace `publishing.ingressName` names (`tenant-root` by default), while `gateway.enabled` is on, and only for the names `publishing.exposedServices` lists. No other tenant's Gateway carries these listeners, and a service removed from `publishing.exposedServices` loses its listener together with its `TLSRoute`. In [edge](#edge-tls-terminated-by-the-class-provider) mode the value is not written at all and none of the three exists. + +Under HTTP-01, a hostname a passthrough listener answers is reserved to that listener by the `TenantGateway` spec alone: no HTTPS listener and no certificate is rendered for it on 443, whether or not a `TLSRoute` is attached and wherever that route's backends point. An `HTTPRoute` whose every hostname is reserved this way reads `Accepted=False` with reason `NoMatchingListenerHostname`; one that also claims a free hostname stays `Accepted` and is served on that one. For `api.` this means clients always reach the management kube-apiserver's own serving certificate, so that certificate has to carry `api.` in its SANs. `TLSRoute` comes from the Gateway API experimental channel, which is the bundle Cozystack vendors. That bundle serves `tlsroutes` at `v1` — the storage version — alongside `v1alpha2` and `v1alpha3`, both still served and both marked deprecated. The platform's own TLSRoute templates (`cozystack-api`, `kubevirt`, `kubevirt-cdi`) write `v1alpha2`, so moving them onto `v1` is a change on the Cozystack side rather than something to wait for upstream. +### TLS passthrough on native ports + +A `TenantGateway` also accepts `spec.tlsPassthroughListeners`, a list of layer-4 TLS-passthrough listeners on ports other than 443. Each entry renders one listener named `tls-` on its own port, in `Passthrough` mode, matching one SNI hostname, next to the tenant's other listeners. The listener never terminates TLS: the SNI in the `ClientHello` selects the route and the raw TLS stream goes to the backend, so a database engine can present its own certificate to clients on the port they already expect while sharing the tenant Gateway's address. It is independent of `tlsPassthroughServices`, the port-443 form described above, and neither replaces the other. + +No chart or platform value sets the field yet, and no managed application attaches a route to such a listener, so an entry is written onto the `TenantGateway` directly. A merge patch replaces the whole list, so include every entry you want to keep: + +```bash +kubectl -n tenant-root patch tenantgateway cozystack --type=merge \ + -p '{"spec":{"tlsPassthroughListeners":[{"name":"redis","port":6379,"hostname":"redis.example.org"}]}}' +``` + +Each entry has three required fields: + +| Field | Rule | +| --- | --- | +| `name` | A DNS-1123 label of at most 63 characters, unique in the list. The rendered listener is `tls-`. It must not repeat an entry of `tlsPassthroughServices`, which renders a listener of the same name. | +| `port` | From 1 to 65535, distinct from every other entry, and neither 80 nor 443, which belong to the Gateway's own listeners. | +| `hostname` | The SNI the listener matches, either an exact hostname or a wildcard in the left-most label such as `*.db.example.org`. It must equal the tenant apex or be a subdomain of it. | + +The port, distinct-port, apex and name-collision rules are CEL rules on the CRD, so a bad entry is refused when it is written. The controller checks two more rules and reports a violation as `Ready=False` on the `TenantGateway`: no two passthrough listeners may match the same SNI, counting the port-443 `tls-` listeners and counting wildcards (so `*.` is refused next to the shipped `api.` listener), and the assembled listener count must stay within the Gateway API cap of 64. + +The listener is all an entry creates. Nothing is served until a `TLSRoute` attaches to it with `sectionName: tls-` and names a backend `Service`, and that backend, not the Gateway, has to present a certificate for the listener's hostname: + +```yaml +apiVersion: gateway.networking.k8s.io/v1alpha2 +kind: TLSRoute +metadata: + name: redis + namespace: tenant-root +spec: + parentRefs: + - name: cozystack + sectionName: tls-redis + hostnames: + - redis.example.org + rules: + - backendRefs: + - name: + port: 6379 +``` + +Only routes from the namespace of the tenant that owns the Gateway can attach. These listeners select on the `kubernetes.io/metadata.name` label rather than on `namespace.cozystack.io/gateway`, so neither inheriting child tenants nor `attachedNamespaces` entries reach a database port. Under HTTP-01 the hostname is reserved from the moment the entry is declared: no HTTPS listener or certificate is rendered for it on 443, route or no route. + +The engine has to open the connection with a TLS handshake. A protocol whose server speaks first in plaintext, MySQL and MariaDB on 3306 among them, never sends the `ClientHello` the listener waits for, so it cannot be published this way. PostgreSQL on 5432 works only where both the server and the client negotiate TLS directly. + +Current limitations: + +- The field is refused while the tenant is in `edge` cert mode, and a stored entry blocks a later move of that tenant to `edge`: its gateway release keeps failing until the field is removed. `http01`, `dns01` and `existingSecret` accept it. +- Each entry needs its own port, so two instances of the same engine cannot share the engine's native port and be told apart by SNI alone. +- The declared port is not an access boundary. Depending on the rest of the Gateway, Cilium may serve every port of the Gateway from one proxy listener matched by SNI alone, so a `ClientHello` carrying the hostname can reach the backend through another port the load balancer forwards. Treat the hostname as public and give the backend its own authentication. +- The gateway release neither reverts nor removes a hand-added entry, so a published port stays published until someone deletes the entry. Record what you added. + ## Security Tenants in Cozystack interact with the platform exclusively through `apps.cozystack.io/*` resources (Tenant, Bucket, Kubernetes, …) served by `cozystack-api`. Tenant RBAC (`cozy:tenant:*` aggregated to a RoleBinding in the tenant's own namespace) does not grant write access to `gateway.networking.k8s.io/*`, core `Namespaces`, or `cozystack.io/Package`. The protections below split into two groups by who they defend against — most of the five layers do not protect against tenant-user input (that RBAC isn't granted in the first place); they guard against bugs in cozystack-controller / Flux, supply-chain compromise of an app chart, and confused-deputy mistakes by a cluster admin. Every admission-time check is fail-closed **on policy error** (`failurePolicy: Fail`, `validationActions: [Deny]`) — a policy that cannot be evaluated denies rather than admits. That is a separate axis from what each CEL expression does when the namespace host label is missing, where the layers differ: Layer 5 denies, Layer 2 allows. See each layer for its own behaviour. @@ -414,7 +468,7 @@ flowchart TD L1["L1: Listener allowedRoutes selector
(HTTPS: namespace.cozystack.io/gateway;
HTTP-80: kubernetes.io/metadata.name;
edge mode: no HTTP-80 whitelist)"] L2["L2 VAP: Gateway listener hostname
matches namespace.cozystack.io/host"] L4["L4 VAP: namespace.cozystack.io/host label
writes restricted to trusted callers"] - L5["L5 VAP: HTTPRoute/TLSRoute hostnames
match namespace label (tenant-* only)"] + L5["L5 VAP: HTTPRoute/TLSRoute/GRPCRoute
hostnames required, match namespace label
(tenant-* only)"] GW["Cross-tenant hostname hijack
BLOCKED"] @@ -440,7 +494,7 @@ Every listener on a tenant Gateway pins `allowedRoutes.namespaces.from: Selector - **HTTPS and TLS-passthrough listeners** use `matchLabels: { namespace.cozystack.io/gateway: }`. In [edge](#edge-tls-terminated-by-the-class-provider) mode there are none of either, and the port-80 `edge*` listeners that replace them carry this same wide selector. The label value is the namespace of the TenantGateway — for `tenant-root` that resolves to `tenant-root`, for `tenant-alice` to `tenant-alice` (i.e. the namespace name, not the bare tenant name). The label is written by the `apps/tenant` chart on every tenant namespace (own namespace name when owning a Gateway, inherited ancestor's namespace name otherwise) and patched by the controller onto every namespace in `attachedNamespaces`. Namespaces without the matching value cannot attach any HTTPRoute / TLSRoute to those listeners. - **The plain-HTTP listener (port 80)** uses a strictly narrower `matchExpressions` whitelist on the built-in `kubernetes.io/metadata.name` label — only the owner tenant's own namespace (where the controller-owned redirect HTTPRoute lives) and `cozy-cert-manager` (HTTP-01 ACME challenge HTTPRoutes). App HTTPRoutes attaching by hostname therefore cannot bind to port 80 and silently serve plaintext. **This layer does not apply in [`edge` cert mode](#edge-tls-terminated-by-the-class-provider)**, which renders no hostname-less port-80 listener at all and gives its port-80 listeners the same wide `namespace.cozystack.io/gateway` selector the HTTPS listeners carry elsewhere. There, app routes bind port 80 and are served in cleartext deliberately, because TLS ended at the class provider's edge — see [What this does NOT defend](#what-this-does-not-defend). -Port-443 listeners pin `allowedRoutes.kinds` to `HTTPRoute` and `TLSRoute` — the pair is listed on each so Cilium keeps them apart — preventing `GRPCRoute` / `TCPRoute` / `UDPRoute` from attaching outside the Layer 5 VAP's coverage. +Port-443 listeners pin `allowedRoutes.kinds` to `HTTPRoute` and `TLSRoute` (the pair is listed on each so Cilium keeps them apart), so `GRPCRoute`, `TCPRoute` and `UDPRoute` cannot attach to them. The port-80 listener leaves the kinds unset, which for an HTTP listener admits `GRPCRoute` too, and Layer 5 covers `GRPCRoute` for that reason. ### Layer 2 — `cozystack-gateway-hostname-policy` @@ -464,9 +518,9 @@ This closes the path where a tenant user creates a Tenant with `spec.host=dashbo Combined with Layer 3, a tenant user cannot establish or change their host through either the Tenant CR or the namespace label. -### Layer 5 — `cozystack-route-hostname-policy` (HTTPRoute) and `cozystack-route-hostname-policy-tls` (TLSRoute) +### Layer 5 — `cozystack-route-hostname-policy` (HTTPRoute), `cozystack-route-hostname-policy-tls` (TLSRoute) and `cozystack-route-hostname-policy-grpc` (GRPCRoute) -A pair of `ValidatingAdmissionPolicy` objects sharing the same CEL expression. `cozystack-route-hostname-policy` targets `gateway.networking.k8s.io` `HTTPRoute` CREATE/UPDATE, naming `v1` and `v1beta1`; `cozystack-route-hostname-policy-tls` targets `TLSRoute`. Whichever versions a rule names, coverage is not limited to them: `matchConstraints.matchPolicy` is unset on both policies and so defaults to `Equivalent`, which has the apiserver convert a request submitted under another served version of the same resource into one the rule does name, before the policy evaluates it. It does not reach a *different* resource, which is why `GRPCRoute` — a CRD of its own, not a version of `httproutes` — is outside both policies rather than converted into one. Both are scoped to `tenant-*` namespaces (cozy-* are cluster-admin-managed and trusted to publish under any apex) and reject any `spec.hostnames` entry that is not equal to the namespace's `namespace.cozystack.io/host` label or a subdomain of it. **Fail-closed when the label is absent** — a `tenant-*` namespace without `namespace.cozystack.io/host` is rejected, not silently allowed. It does, however, admit a route that declares no `spec.hostnames` at all: the CEL short-circuits before the suffix test, so a hostname-less route is allowed in a labelled namespace. That is the hole the Cloudflare Tunnel package closes with its own ownership layer, because on a shared tunnel a hostname-less route would match every host. Operators querying `kubectl get validatingadmissionpolicy` will see both objects. +Three `ValidatingAdmissionPolicy` objects sharing one CEL expression: `cozystack-route-hostname-policy` targets `gateway.networking.k8s.io` `HTTPRoute` CREATE/UPDATE, `cozystack-route-hostname-policy-tls` targets `TLSRoute`, and `cozystack-route-hostname-policy-grpc` targets `GRPCRoute`. Each rule names every version the vendored CRD bundle serves for its resource: `v1` and `v1beta1` for `HTTPRoute`, `v1`, `v1alpha2` and `v1alpha3` for `TLSRoute`, and `v1` for `GRPCRoute`. `GRPCRoute` is covered because the port-80 listener leaves `allowedRoutes.kinds` unset, and for an HTTP listener an unset list admits `GRPCRoute`, which carries `spec.hostnames` of its own. All three are scoped to `tenant-*` namespaces (cozy-* are cluster-admin-managed and trusted to publish under any apex). A route must declare a non-empty `spec.hostnames`, and every entry must equal the namespace's `namespace.cozystack.io/host` label or be a subdomain of it. A route that omits `spec.hostnames`, or sets it to an empty list, is rejected: under Gateway API such a route does not match nothing, it inherits the hostnames of every listener it attaches to, and a policy cannot resolve the parent Gateway to check them. **Fail-closed when the label is absent:** a `tenant-*` namespace without `namespace.cozystack.io/host` is rejected, not silently allowed. Operators querying `kubectl get validatingadmissionpolicy` will see all three objects. Defense-in-depth against an app chart bug or supply-chain compromise that emits Gateway API resources outside the tenant's apex — tenants in Cozystack do not hold `gateway.networking.k8s.io/*` RBAC by design, so this is not a tenant-user defense. The within-apex cross-namespace case (a tenant chart claiming a hostname owned by a `cozy-*` app) is handled by the controller at reconcile time **in HTTP-01 mode only** — see [HostnameConflict resolution](#hostnameconflict-resolution) below for which modes resolve it and which leave it to the class controller. @@ -538,7 +592,7 @@ Let's Encrypt enforces per-account and per-registered-domain quotas: - 5 duplicate certificates per week for the same hostname set - 300 new orders per account per 3 hours -Setting `publishing.certificates.wildcardSecretName` sidesteps the quotas entirely — that mode issues no ACME certificates at all — at the cost of the caveats in [Certificates](#certificates). A cluster where many tenants share the same apex domain can exhaust these quickly, especially in HTTP-01 mode where each published app contributes one certificate — and so does each TLS-passthrough service, since its hostname is claimed the same way even though the backend, not the Gateway, terminates its TLS. Mitigations: +Setting `publishing.certificates.wildcardSecretName` sidesteps the quotas entirely — that mode issues no ACME certificates at all — at the cost of the caveats in [Certificates](#certificates). A cluster where many tenants share the same apex domain can exhaust these quickly, especially in HTTP-01 mode where each published app contributes one certificate. A TLS-passthrough hostname contributes none, because its backend presents its own certificate. Mitigations: - `publishing.certificates.issuerName: letsencrypt-stage` for non-production clusters (staging quotas do not affect prod). - `tenant.spec.resourceQuotas.count/certificates.cert-manager.io` to cap per-tenant certificate creations. @@ -734,9 +788,9 @@ Look for an entry under `Status.Parents` with `controllerName: gateway.cozystack Layer 2 (`cozystack-gateway-hostname-policy`) rejected the Gateway because a listener hostname does not match `namespace.cozystack.io/host` on the Gateway's namespace. Fix the listener hostname, or (if the namespace label is wrong) update the tenant's `spec.host` via a trusted caller. -### Admission denied: "HTTPRoute hostnames must equal..." +### Admission denied: "HTTPRoute must declare spec.hostnames, and each hostname must equal..." -Layer 5 (`cozystack-route-hostname-policy`) rejected the HTTPRoute or TLSRoute because a hostname falls outside the apex of the namespace's `namespace.cozystack.io/host` label. Either change the hostname to live under the apex, or move the route to a namespace whose label covers the desired hostname. +Layer 5 rejected the route. The message starts with the route kind, `HTTPRoute`, `TLSRoute` or `GRPCRoute`, and the cause is one of two: the route declares no `spec.hostnames` (or an empty list), or a hostname falls outside the apex of the namespace's `namespace.cozystack.io/host` label. Name the route's hostnames explicitly under the apex instead of relying on the listener's, or move the route to a namespace whose label covers the desired hostname. ### Admission denied: "tenant.spec.host can only be set..." From aeab68906f105058ba7987b13ce39aba7b00283a Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 14 Oct 2026 12:01:39 +0000 Subject: [PATCH 2/6] docs(operations): document the platform values added in 1.7 Four platform values that 1.7 adds had no row on the Platform Package reference: - authentication.oidc.realmName (cozystack/cozystack#4237), with the character rule, the refusal of `master` and what a rename on a live cluster does. The keycloakInternalUrl row is corrected next to it: its default is now unset and derived from the realm name. - networking.encryption.enabled (cozystack/cozystack#2984), with the variants that apply, refuse or ignore it. - backupStrategyController, the pass-through that sizes the Redis backup strategy Pod. - kubernetesWorkerImage (cozystack/cozystack#4171), the values of the opt-in golden worker image catalog. cozystackOperator.systemNamespaceMemoryLimit is an installer value, not a platform one, so it is not part of this page. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- .../configuration/platform-package.md | 48 ++++++++++++++++++- 1 file changed, 47 insertions(+), 1 deletion(-) diff --git a/content/en/docs/next/operations/configuration/platform-package.md b/content/en/docs/next/operations/configuration/platform-package.md index bf3dfc66..2b17aeb2 100644 --- a/content/en/docs/next/operations/configuration/platform-package.md +++ b/content/en/docs/next/operations/configuration/platform-package.md @@ -99,6 +99,7 @@ spec: | `networking.serviceCIDR` | `"10.96.0.0/16"` | The service subnet used by Services to assign IPs. Kube-OVN only. | | `networking.joinCIDR` | `"100.64.0.0/16"` | The `join` subnet for network communication between the Node and Pod. Follow the [kube-ovn] documentation to learn more. Kube-OVN only. | | `networking.kubeovn.MASTER_NODES` | `""` | Comma-separated list of KubeOVN master node IPs. By default, KubeOVN uses `lookup` to find control-plane nodes by label `node-role.kubernetes.io/control-plane`. On fresh clusters, lookup may return empty results. Set this to override. | +| `networking.encryption.enabled` | `false` | Opt-in encryption of pod-to-pod traffic that crosses nodes, at the network layer and whether or not the workload speaks TLS. On `isp-full` and `isp-full-generic`, where Kube-OVN owns the datapath, it turns on Kube-OVN IPsec for the Geneve overlay; Kube-OVN provisions the IPsec CA and the per-node certificates itself through the Kubernetes CSR API, so no cert-manager setup is needed. `isp-slim` and `isp-slim-generic` refuse it and fail the platform render, because their Cilium-only datapath has no encryption wiring yet. On `isp-hosted` and `isp-hosted-slim`, where Cozystack does not manage the CNI, it has no effect. Traffic between pods on the same node is not tunneled and stays unencrypted. The ESP overhead lowers the effective MTU, so east-west throughput drops and latency rises, most for high-bandwidth flows. Turning it on for an existing cluster rolls out the CNI components. It complements TLS on exposed endpoints and does not replace it: no external client and no tenant certificate is involved. | #### Bundles @@ -119,7 +120,8 @@ spec: | `authentication.oidc.enabled` | `false` | Enable [OIDC][oidc] feature in Cozystack. | | `authentication.oidc.insecureSkipVerify` | `false` | Skip TLS certificate verification for the OIDC provider. | | `authentication.oidc.keycloakExtraRedirectUri` | `""` | Additional redirect URI for Keycloak OIDC client. | -| `authentication.oidc.keycloakInternalUrl` | `""` | Internal URL for backend-to-backend requests to Keycloak. When set, the dashboard's oauth2-proxy skips OIDC discovery and routes token, JWKS, userinfo, and logout requests through this URL while keeping browser redirects on the external URL. Example: `http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy`. | +| `authentication.oidc.realmName` | `"cozy"` | Name of the Keycloak realm that holds the Cozystack clients and the cluster-admin roles. `keycloak-configure` creates the realm under this name, and the dashboard, linstor-gui, the generated kubeconfig and the derived `keycloakInternalUrl` point at `/realms/`. The OIDC issuer of the management cluster's API server is configured outside this Package, through the talm `oidcIssuerUrl`, and must name the same realm, or the tokens issued here are rejected. The value must be a quoted string of letters, digits, `.`, `_` and `-` that starts with a letter or digit, and `master` is refused because that realm holds the Keycloak super-admin; any other value fails the platform render. Set it at install time. Changing it on a running cluster does not rename the realm: a new, empty realm is created under the new name and every account, session and client in the old one is left behind. | +| `authentication.oidc.keycloakInternalUrl` | `null` | Internal URL for backend-to-backend requests to Keycloak. The dashboard's oauth2-proxy skips OIDC discovery and sends token, JWKS, userinfo and logout requests to this URL, while browser redirects keep the external URL. Left unset, it is derived from `realmName` as `http://keycloak-http.cozy-keycloak.svc:8080/realms/`. An empty string `""` restores full OIDC discovery against the external issuer. Any other value is used as is, so an explicit URL has to name the same realm as `realmName`. | #### Gateway @@ -170,6 +172,20 @@ gateway: | Value | Default | Description | | --- | --- | --- | | `backupStorage` | `{}` | S3 coordinates for the platform-managed `cozy-default` BackupClass. The whole block is forwarded into the `backupstrategy-controller` component and deep-merged over its chart defaults; keys include `provisionBucket`, `bucketName`, `endpoint`, `region`, `forcePathStyle`, `systemSecretName`, and `systemNamespaces`. See [Backup Classes]({{% ref "/docs/next/operations/services/backup-classes" %}}) for the knob-by-knob reference. | +| `backupStrategyController` | `{}` | Overrides for the backup strategy objects the platform ships, forwarded as is into the `backupstrategy-controller` component as its `backupStrategyController` values. The knob that needs it is `redisBackupResources`, the requests and limits of the Pod the Redis strategy runs. The restore side holds the whole dataset in memory and writes it to ephemeral storage, so a large Redis needs more memory and `ephemeral-storage` than the defaults, which suit small and medium instances. The `cozy-default-redis` strategy is managed by Helm and a direct edit is reverted on upgrade, so raise the values here. Empty forwards nothing. | + +Example under `spec.components.platform.values`: + +```yaml +backupStrategyController: + redisBackupResources: + requests: + memory: 2Gi + ephemeral-storage: 8Gi + limits: + memory: 4Gi + ephemeral-storage: 16Gi +``` #### Branding @@ -230,6 +246,36 @@ kubevirt: `bandwidthPerMigration` is a quantity in bytes per second: `625M` is 625,000,000 bytes/s (5 Gbit/s) per migration. This example allows up to two migrations across the cluster and one outbound migration per node. +#### Kubernetes worker images + +| Value | Default | Description | +| --- | --- | --- | +| `kubernetesWorkerImage` | `{}` | Values for the optional `cozystack.kubernetes-worker-image` package, forwarded as is into that component. The package keeps a catalog of golden Talos worker OS images as DataVolumes in the `cozy-public` namespace, and a `KubernetesNodes` pool that sets `osImage.builtin` boots its workers from a storage-level clone of one instead of each worker importing the image from the Talos Image Factory. The package is opt-in: add `cozystack.kubernetes-worker-image` to `bundles.enabledPackages`, since this key alone does not enable it. Empty forwards nothing and keeps the chart defaults. | + +The keys the package accepts: + +| Key | Default | Description | +| --- | --- | --- | +| `imageFactoryURL` | `"https://factory.talos.dev"` | Base URL of the Talos Image Factory the golden images are imported from, without a trailing slash. Repoint it on an air-gapped or rate-limited cluster. | +| `storageClass` | `"replicated"` | StorageClass of the golden images. It must match the StorageClass of the worker disks that clone them, because CDI cannot clone across StorageClasses; a pool whose class differs fails its render. Use a class whose volumes are reachable from any node, in practice a replicated DRBD class: cloning on a node-pinned class is unsupported. | +| `storage` | `"6Gi"` | Size of each golden image. It must be at least the imported image size and at most the smallest worker disk that clones it. | +| `images` | one entry | The catalog, a list of entries with `schematicID` and `version` and optionally their own `imageFactoryURL`, `storage` and `storageClass`. Each entry becomes a golden named `talos-worker--`. | + +A list set here replaces the shipped `images` list rather than extending it. A pool that names a `(schematicID, version)` pair missing from `images`, or one whose golden failed to import, fails its render rather than falling back to the HTTP import, so add an entry before any pool points at it. An entry is effectively immutable once its golden exists: its size, class and source URL cannot be patched in place, so treat a change to any of them as a new entry and move the pools onto it. Removing an entry stops managing its golden but does not delete it. + +Example under `spec.components.platform.values`: + +```yaml +bundles: + enabledPackages: + - cozystack.kubernetes-worker-image +kubernetesWorkerImage: + imageFactoryURL: "https://factory.internal.example.org" + images: + - schematicID: "" + version: "" +``` + #### Internal fields These fields are managed automatically by the Cozystack operator and should not be modified manually. From 3eb62d5036931c8f776376c236da3f00c788aa5b Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 14 Oct 2026 12:04:34 +0000 Subject: [PATCH 3/6] docs(operations): re-sync Backup Classes with the product doc The website copy of the Backup Classes page had drifted from docs/operations/backup-classes.md in cozystack/cozystack. It listed seven applications and none of the drivers 1.7 adds. The body now follows the product page at v1.7.0-rc.1: - MongoDB (cozystack/cozystack#3562, #3967), Kafka topic metadata (#4000), RabbitMQ definitions (#3990) and Redis (#3992) bound by cozy-default, with their endpoint forms, credentials keys, storage flows and restore semantics. - The corrected PostgreSQL point-in-time recovery text from cozystack/cozystack#4443: base-backup selection, the recoverable window bounded by the last archived transaction, the one-second margin after stoppedAt, and archives with several timelines. This covers the corrections listed in cozystack/website#716. - Password handling after a restore, the default-objects gate and its metrics, the backupStrategyController override, concurrent restores and the Barman Cloud plugin upgrade notes. Adapted to the site: front matter and See also kept, relative links to the monorepo turned into GitHub links, `users[].password` written as the map path `users..password` the charts use, a test-run remark and the stale CloudNativePG version in the plugin note dropped. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- .../operations/services/backup-classes.md | 261 ++++++++++++++++-- 1 file changed, 233 insertions(+), 28 deletions(-) diff --git a/content/en/docs/next/operations/services/backup-classes.md b/content/en/docs/next/operations/services/backup-classes.md index da3f88b6..01b73c93 100644 --- a/content/en/docs/next/operations/services/backup-classes.md +++ b/content/en/docs/next/operations/services/backup-classes.md @@ -8,7 +8,6 @@ aliases: - /docs/next/operations/services/velero-backup-configuration --- - Cozystack ships a single platform-managed `BackupClass` named `cozy-default`. It is provisioned automatically when the `backupstrategy-controller` package is installed and references the system-managed bucket provisioned through the `apps.cozystack.io/Bucket` CR `cozy-backups` in the `tenant-root` namespace (the real S3 bucket name is the COSI-assigned one from `BucketClaim.status.bucketName`). Tenants reference `cozy-default` from `BackupJob`, `Plan`, and `RestoreJob` resources — they do **not** supply S3 credentials, endpoints, or paths. The platform projects the system-managed credentials Secret into the tenant namespace per BackupJob (or, for long-lived references like Velero's `BackupStorageLocation`, into a fixed list of system namespaces on a periodic tick), and the default strategy templates encode `/` into every S3 path so two tenants with the same application name never collide. @@ -22,7 +21,11 @@ Tenants reference `cozy-default` from `BackupJob`, `Plan`, and `RestoreJob` reso | `apps.cozystack.io/Postgres` | CloudNativePG (barman) | `strategy.backups.cozystack.io/CNPG` `cozy-default-cnpg` | | `apps.cozystack.io/MariaDB` | mariadb-operator dump | `strategy.backups.cozystack.io/MariaDB` `cozy-default-mariadb` | | `apps.cozystack.io/ClickHouse` | Altinity `clickhouse-backup` sidecar | `strategy.backups.cozystack.io/Altinity` `cozy-default-altinity` | +| `apps.cozystack.io/MongoDB` | Percona psmdb operator (pbm) dump | `strategy.backups.cozystack.io/MongoDB` `cozy-default-mongodb` | +| `apps.cozystack.io/Kafka` | Kafka Admin API (topic metadata) | `strategy.backups.cozystack.io/Kafka` `cozy-default-kafka` | | `apps.cozystack.io/Etcd` | etcd-operator snapshot | `strategy.backups.cozystack.io/Etcd` `cozy-default-etcd` | +| `apps.cozystack.io/RabbitMQ` | RabbitMQ definitions (management API) | `strategy.backups.cozystack.io/Rabbitmq` `cozy-default-rabbitmq` | +| `apps.cozystack.io/Redis` | RDB dump Job (sentinel-discovered master) | `strategy.backups.cozystack.io/Redis` `cozy-default-redis` | | `apps.cozystack.io/VMInstance` | Velero + kubevirt-velero-plugin | `strategy.backups.cozystack.io/Velero` `cozy-default-velero-vminstance` | | `apps.cozystack.io/VMDisk` | Velero | `strategy.backups.cozystack.io/Velero` `cozy-default-velero-vmdisk` | @@ -43,9 +46,12 @@ Different operators expect different endpoint shapes; the strategy templates ren | CNPG (Postgres) | `barmanObjectStore.endpointURL` | full URL (scheme preserved) | | Etcd | `destination.s3.endpoint` | full URL (scheme preserved) | | MariaDB | `storage.s3.endpoint` | bare host:port (scheme stripped); `tls.enabled` derived from the scheme | +| MongoDB | `s3.endpointUrl` | full URL (scheme preserved); on the default flow the app's `backup.endpointURL`, on `useSystemBucket` the strategy coordinate injected by the driver | | FoundationDB | `blobStoreConfiguration.accountName` + `urlParameters.secure_connection` | bare host:port + derived secure flag | | Velero | `BackupStorageLocation.spec.config.s3Url` | full URL (scheme preserved) | | ClickHouse sidecar | `S3_ENDPOINT` env | bare host:port (from projected Secret) | +| Redis (dump Job) | `S3_ENDPOINT` env | bare host:port (from projected Secret); `https://` prepended when unscheme'd | +| Kafka | `S3_ENDPOINT` env on the strategy Job | full URL (scheme preserved); the Job's `curl --aws-sigv4` prepends `https://` if the endpoint carries no scheme, honours `backupStorage.forcePathStyle`, and (for the image's curl 7.76.1) signs a portless URL with `--connect-to` to the real port so a ported endpoint still verifies | The projected `cozy-backups-creds.endpoint` key is **stripped of scheme** so chart-emitted sidecars (ClickHouse) consume it directly. Drivers that need the full URL receive the resolved endpoint described above — derived from the COSI system Secret (forced `https://`) for a provisioned bucket, or the `backupStorage.endpoint` fallback for external S3. @@ -57,6 +63,33 @@ The strategy CR `cozy-default-foundationdb` is shipped, but it is **not** bound **Cleanup gotcha (zombie backup_agent).** Unlike CNPG/MariaDB/Altinity (one-shot operator-side Backup CRs), the FoundationDB driver creates an `apps.foundationdb.org/FoundationDBBackup` CR that drives a **long-lived** `backup_agent` Deployment streaming continuously to S3. Deleting a Cozystack `Backup` (e.g. via retention sweeping) does NOT stop that Deployment — the agent keeps writing until the next BackupJob's `stopOtherFoundationDBBackups` call swaps it out, until an admin invokes `examples/backups/foundationdb/cleanup.sh`, or until the operator-side CR is deleted by hand. If a tenant deletes their last Cozystack Backup and never submits another BackupJob, the agent pods will continue running indefinitely and accumulate S3 PUTs. This is intentional today (the driver has no RBAC verb to stop the operator-side CR on Cozystack-Backup deletion) but admins should be aware of it. +## Postgres and MariaDB: a restored copy converges its application passwords shortly after recovery + +Managed Postgres no longer accepts a plaintext `users..password`; every application-user password is chart-generated into `-credentials` and read only from there. The CNPG backup snapshot therefore carries no passwords — deliberate, since a snapshot that did was the one place a tenant password sat in cleartext. + +**Residue from before the upgrade — this is forward-looking, not retroactive.** The removal scrubs new renders and new snapshots; it does not rewrite copies made earlier, and upgrading preserves the live password (the chart reads the existing `-credentials` through `lookup`), so those older copies still expose the *current* credential. Two sinks retained it: a Postgres `Backup` object created before this release snapshotted `app.spec.users` while that spec still carried a plaintext `password`, so its `status.underlyingResources` held that password in cleartext — a field any tenant with `get backups` in the namespace can read (see the `underlyingResources` caution below); and every pre-upgrade Helm revision of `-init-script` rendered `ALTER ROLE … WITH PASSWORD ''` into its manifest, retained for `MaxHistory` revisions. There is no in-band rotation to invalidate the exposed value (chart-based rotation was intentionally not added — a chart-rendered Secret cannot revoke a leaked credential; rotation is being reworked as a controller, [cozystack/community#72](https://github.com/cozystack/community/pull/72)). The backup strategy controller removes `users.*.password` from every CNPG `Backup` snapshot each time its leader starts, so the first sink is cleaned in-cluster after the upgrade, including a `Backup` written by a pre-upgrade replica during the rollout; the start-up log line `legacy snapshot password scrub finished` reports how many it changed. That does not reach copies made outside the cluster — apiserver audit logs, etcd backups, or object-level exports of the `Backup` resources — and the old init-script Secret persists in Helm history until those revisions age out of `MaxHistory`. So after upgrading, treat any password that was ever set through `users..password` as exposed until it is rotated. + +The consequence for a **restore into a copy** (a differently-named target): the recovered database keeps the source roles with their source password hashes, but the target chart generates fresh random passwords into the target's `-credentials`, and the role-reconciling init-job is skipped while `bootstrap.enabled` is set on the restored app. The restore driver clears `bootstrap.enabled` automatically once recovery is healthy, so the next HelmRelease reconcile runs the init-job, which applies `ALTER ROLE … WITH PASSWORD` for every `spec.users` entry and converges the recovered roles onto the target Secret. Convergence is therefore automatic but not instant: between the RestoreJob reaching `Succeeded` and that init-job completing, the passwords advertised in `-credentials` do not yet match the recovered roles, so a client reading that Secret cannot log in as an application user for that brief window. The chart flags the Secret `postgres.cozystack.io/credentials-pending: "true"` and the NOTES print the same caution while bootstrap is still on, so a reader does not treat the advertised password as usable mid-restore; both clear on their own once the driver clears bootstrap. The data is intact and reachable as the CNPG superuser throughout. After a to-copy restore, wait for the application login to succeed rather than assuming the advertised password works the instant the RestoreJob succeeds. The driver clears `bootstrap.enabled` by writing it onto the Postgres app's own spec, so a GitOps source (Argo CD, Flux, Terraform) that pins `bootstrap.enabled: true` for that app will revert the clear on its next apply and the copy's credentials never converge — exclude the restored app from such a source until convergence completes, or clear `bootstrap.enabled` in the source itself. If the driver cannot clear the flag at all (the app was deleted or renamed mid-restore), the RestoreJob fails with reason `BootstrapDisableFailed` once its deadline elapses rather than sitting `Running` forever with no signal. An in-place restore is unaffected: the release keeps its existing `-credentials`, whose passwords already match the roles the recovery brings back. One-time migration: an app restored to a copy under an earlier release — before this auto-clear existed — may still be sitting with `bootstrap.enabled` set and its init-job permanently skipped; the driver clears the flag only for a restore it drives to completion, never retroactively for one that already finished, so such an app keeps advertising unconverged credentials until you clear `bootstrap.enabled` on it once by hand. + +Databases and users after a Postgres restore, in place or into a copy: the driver sets the app's `spec.databases` and `spec.users` to the snapshot taken when the Backup was created, while recovery replays WAL past that point (to the end of the archive when no `recoveryTime` is set), so the recovered cluster can hold databases and users the snapshot does not list. On the first init-job run after the restore, a database the chart manages but the restored spec does not declare is released instead of dropped: its data stays, and it and its `_admin` and `_readonly` roles lose the `managed by helm` comment, so the chart no longer grants on it or drops it. Add it to `databases` to manage it again. Users are not released: a recovered user the restored spec does not declare, including a source user the target of a copy does not declare, is dropped on that run, so a client still logging in as that user loses the login. Every object it owned is reassigned to `cozystack:orphaned`, a role that cannot log in and that the chart grants nothing. Functions it owns are switched to `SECURITY INVOKER`, and on PostgreSQL 15 and later its views to `security_invoker`. A materialized view, a rule, and a view on PostgreSQL 13 or 14 have no such switch and keep reading with the rights of `cozystack:orphaned`. The user's grants and memberships do not carry over, so a view or function of that user can lose access it had through them. Declare the user in `users` to keep it; the chart then generates a new password for it into `-credentials`. + +**Objects reassigned by earlier releases stay as they were.** Before this change the chart reassigned a dropped user's or role's objects to the `postgres` superuser, and upgrading does not move them. After upgrading, run the query below in each database: it lists the relations and routines `postgres` owns outside the system catalog and extensions, leaving out a sequence that belongs to a column, which moves with its table. Of those, the chart itself owns only its `auto_grant_schema_privileges()` event-trigger function, so anything else came from a tenant role or was created by hand. Hand what a tenant role created to `cozystack:orphaned` with the matching `ALTER TABLE`, `ALTER VIEW`, `ALTER ROUTINE` or other `ALTER … OWNER TO "cozystack:orphaned"`; the next init-job run switches its functions and, on PostgreSQL 15 and later, its views as above. Handle first the rows with `security_definer` set, and views, materialized views and tables with rules, since those run with their owner's rights: + +```sql +SELECT c.oid::regclass::text AS object, NULL::boolean AS security_definer FROM pg_class c +WHERE c.relowner = 'postgres'::regrole AND c.relkind NOT IN ('i', 'I', 't') + AND c.relnamespace NOT IN ('pg_catalog'::regnamespace, 'information_schema'::regnamespace) + AND NOT EXISTS (SELECT FROM pg_depend d WHERE d.classid = 'pg_class'::regclass AND d.objid = c.oid + AND (d.deptype = 'e' OR (c.relkind = 'S' AND d.refclassid = 'pg_class'::regclass AND d.deptype IN ('a', 'i')))) +UNION ALL +SELECT p.oid::regprocedure::text, p.prosecdef FROM pg_proc p +WHERE p.proowner = 'postgres'::regrole + AND p.pronamespace NOT IN ('pg_catalog'::regnamespace, 'information_schema'::regnamespace) + AND NOT EXISTS (SELECT FROM pg_depend d WHERE d.classid = 'pg_proc'::regclass AND d.objid = p.oid AND d.deptype = 'e'); +``` + +MariaDB avoids the same problem at backup time: the driver sets `ignoreGlobalPriv` on every operator `Backup` it creates, so `mysql.global_priv` — the grant table holding every user's password hash, root included — is excluded from the logical dump and a restore into a copy never overwrites the target's chart-managed users. The same exclusion is also a limitation, and it applies to every restore, not only a to-copy one: because the grant table is never in the dump, only accounts Cozystack declares through `User` CRs (the chart's `users` map) are recreated on restore — an account created out of band, directly in MySQL rather than through `users`, is in no backup, so a restore brings back its data and its `mysql.db` grants but not the account that owns them, and that login fails until the account is recreated. Cozystack treats the `User` CRs as the source of truth for accounts precisely so the grant table need not be, so keep managed accounts in `users` rather than creating them by hand. That matters most for `root`, which the operator only applies at datadir bootstrap via `rootPasswordSecretKeyRef` and could never repair afterwards; keeping the source's grants out of the dump is what lets the target's advertised `root` password stay correct through a restore. This is a **backup-side** fix, so it only protects backups captured from this release onward. A restore from a backup taken **before** this change still replays the source's `mysql.global_priv` and overwrites the target's application users and `root` with the source's hashes; after restoring such an older backup, treat the target's advertised credentials as unreliable and reset them — application users converge on the next reconcile through their `User` CRs, but `root` must be reset out of band (`ALTER USER 'root' … IDENTIFIED BY …` against the restored server) since the operator applies `rootPasswordSecretKeyRef` only at datadir bootstrap. That once-only write is also why the Secret itself now matters: `root` lives solely in `-credentials` with no plaintext escape hatch left in values, so if that Secret is lost after bootstrap — a tenant deletes it, or a partial Velero restore drops it — the advertised `root` password and the live server's diverge with no in-band way to reconcile them; back the Secret up alongside the data, and reset `root` out of band if it is ever lost. + ## ClickHouse: opt-in to the system bucket The `clickhouse-backup` sidecar runs inside the ClickHouse Pod itself, so the Helm chart is what wires its S3 credentials. Existing tenants on the legacy `backup.s3*` values continue to work unchanged. To switch a release onto the platform bucket, set: @@ -75,6 +108,73 @@ When `useSystemBucket: true`: `s3Region`, `s3Bucket`, `endpoint`, `s3AccessKey`, `s3SecretKey`, and `s3CredentialsSecret` are ignored in this mode. +## MongoDB: backup storage + +MongoDB supports two storage flows. By default the S3 target lives on the app CR and the tenant supplies its own credentials; alternatively the tenant opts into the platform system bucket and the driver injects the storage onto the live cluster at backup time. + +### Default: the application owns the backup storage + +The Percona psmdb operator runs the percona-backup-mongodb (pbm) agents whenever the `PerconaServerMongoDB` cluster has `spec.backup.enabled: true` (a declared storage is not needed for the agents to start), and services a `PerconaServerMongoDBBackup` only once the storage it references by name is declared on the cluster. On the default flow the MongoDB application **opts into backups** and points at the bucket in its own chart values: + +```yaml +apiVersion: apps.cozystack.io/v1alpha1 +kind: MongoDB +metadata: + name: orders-db +spec: + backup: + enabled: true + destinationPath: "s3:///orders-db/" + endpointURL: "https://seaweedfs-s3.tenant-root:8333" + insecureSkipTLSVerify: true # self-signed in-cluster seaweedfs; psmdb s3 storage has no CA-bundle field + s3AccessKey: "" + s3SecretKey: "" +``` + +The chart declares that storage as `s3-storage`, which the `cozy-default-mongodb` strategy names (`storageName: s3-storage`, `type: logical`). The driver then drives on-demand backups and restores through the unified `BackupJob` / `RestoreJob` / `Plan` interface on top of the operator's storage config — adding restore-to-differently-named instances (via `PerconaServerMongoDBRestore.spec.backupSource`) and point-in-time recovery that the raw scheduled-task / `mongodump` paths do not offer. When a target cluster has backups disabled, the driver surfaces a clear `Ready=False` precondition on the BackupJob/RestoreJob rather than hanging until the deadline. + +For a to-copy restore the operator reads the dump from the **source** backup's bucket/endpoint. The driver re-points the restore at the **target** cluster's own S3 credentials only when the target declares a storage on that same bucket (the common case where both share one bucket), since the source release's Secret dies with it in a DR scenario; a system-bucket source keeps its projected `cozy-backups-creds` reference, which outlives the source app. A target whose storages are all on a different bucket gets no credential swap: the restore keeps the source's Secret reference and is held at `Ready=False RestoreCredentialsMissing` naming that Secret until it exists in the restore namespace, so a cross-bucket restore never runs with a credential for the wrong bucket. Re-create the source release's `-s3-creds` Secret by name to let it proceed. + +A full, scripted example (write a marker document, back up, restore to a copy, assert the round-trip while the source stays untouched) is in the Cozystack repository at [`examples/backups/mongodb/`](https://github.com/cozystack/cozystack/tree/main/examples/backups/mongodb), driven by `run-all.sh`. + +### Opt-in to the system bucket + +Unlike ClickHouse's sidecar, PBM takes the bucket/endpoint/prefix as static fields on the `PerconaServerMongoDB` CR, and the platform bucket name is only known at BackupJob time — so the chart cannot render them. Instead the `cozy-default-mongodb` driver injects the storage onto the live cluster. To back up to the platform bucket without supplying S3 credentials, set: + +```yaml +backup: + enabled: true + useSystemBucket: true +``` + +When `useSystemBucket: true`: + +- The chart-emitted `-s3-creds` Secret is no longer rendered, and the chart leaves `spec.backup.storages`, `spec.backup.tasks` and `spec.backup.pitr` unset on the `PerconaServerMongoDB`. +- On every BackupJob the driver SSA-injects the `s3-storage` entry from the strategy coordinates (bucket/endpoint from the platform system bucket, `credentialsSecret: cozy-backups-creds`, prefix `/`) under its own field manager, so a Flux re-render never reverts it and a later change to the coordinates is picked up on the next backup rather than frozen at the first. The backup itself is not started until the operator reports that it has pushed that `spec.backup` into pbm's own configuration (`status.observedGeneration` caught up with the cluster's generation and the `PBMReady` condition `True`): pbm writes to whatever its config last said, not to the spec, so a backup started before that push would land at the previous coordinates while its `Backup` recorded the new ones. The job waits at `Ready=False PerconaServerMongoDBPBMConfigPending` meanwhile, and fails at the backup deadline if the operator never gets there. +- The injection waits for the release to actually render the flag. The app's `backup.useSystemBucket` is a desired value (it reads `true` as soon as it is written, before helm-controller renders the revision), while the tenant's own storage, scheduled tasks and PITR stream from the previous render stay on the `PerconaServerMongoDB` until it does. As long as the cluster still carries `spec.backup.tasks` or `spec.backup.pitr`, the driver does not inject and holds the BackupJob at `Ready=False` (`PerconaServerMongoDBLegacyRender`), failing it at the backup deadline; a release wedged on an old revision therefore never has its nightly dump and oplog redirected into the shared bucket. `useSystemBucket: true` together with `enabled: false` is refused by the chart at render time; the schema carries the same rule as an `x-kubernetes-validations` marker, which the aggregated apiserver does not evaluate yet ([#2657](https://github.com/cozystack/cozystack/issues/2657)), so until then the render-time refusal is what stops such a release, and a release wedged on it shows up as a failed HelmRelease rather than a redirected backup. + +`destinationPath`, `endpointURL`, `insecureSkipTLSVerify`, `s3AccessKey` and `s3SecretKey` are ignored in this mode. + +Because the chart drops `spec.backup.tasks` and `spec.backup.pitr`, the psmdb operator's own scheduled backups and oplog PITR do **not** run on this flow — `backup.schedule` and `backup.retentionPolicy` stay in the schema for the default flow but have no effect here, and the `recoveryTime` PITR restores described below are not available. Migrate scheduled backups to a `backups.cozystack.io/Plan` against `cozy-default` instead. **Flipping this flag on a running app** removes the storage, tasks and pitr the app had, so a release that was taking scheduled backups + PITR before the flip stops until a `Plan` takes over. + +Dropping `spec.backup.tasks` also drops the psmdb `keep` retention it carried, so on this flow **the driver owns the archive** the way the Rabbitmq/Redis drivers own theirs (an on-demand `PerconaServerMongoDBBackup` is not covered by psmdb task retention). The driver stamps the operator's `percona.com/delete-backup` finalizer onto each backup it mints against a storage that carries the platform coordinates (the strategy's bucket and `credentialsSecret`, as read back from the live cluster after the injection; a strategy without an `s3` block, or a storage pointing anywhere else, holds the job at `Ready=False` instead of minting an unowned dump), and deleting a Cozystack `Backup` (by hand today; by platform retention once it lands) deletes that `PerconaServerMongoDBBackup` CR — whose finalizer removes the pbm object from the shared bucket before the `Backup` is released, so deleting `Backup` objects is what reclaims `cozy-backups`. A legacy backup writes to the tenant's own bucket and keeps the pre-existing hands-off contract (its lifecycle stays the tenant's / operator's; nothing is deleted on `Backup` removal). Note this prunes only when a `Backup` is deleted: unlike CNPG, whose shipped strategy sets a `retentionPolicy` the engine trims on, MongoDB has no engine-side scheduled trim on this flow, so keeping `cozy-backups` bounded depends on a `Plan`'s retention actually deleting old `Backup` objects. **A `Plan` has no retention field yet** (`PlanSpec` carries only `applicationRef`, `backupClassName` and `schedule`), so today a nightly `useSystemBucket` Plan grows the shared bucket unbounded — `schedule` and `retentionPolicy` on the app are inert here (see [Opt-in to the system bucket](#opt-in-to-the-system-bucket)), and platform-level retention is tracked in the [`BackupRetentionPolicy` design (#4192)](https://github.com/cozystack/cozystack/pull/4192). Teardown GC of orphaned `Backup`s / S3 data is [#3966](https://github.com/cozystack/cozystack/issues/3966). Until those land, bound `cozy-backups` with an S3 lifecycle policy on the object store. Two cases release the `Backup` without waiting for the object to be pruned, so a wedged prune never pins a namespace in `Terminating`: a terminating namespace (where the operator's delete-backup step cannot run) and the `backups.cozystack.io/skip-artifact-cleanup: "true"` annotation. In both the driver strips its own finalizer and leaves the object for a bucket lifecycle policy to reclaim. A down operator or unreachable storage in a live namespace is NOT released automatically — the `Backup` stays `Terminating` as a visible signal until the operator recovers or you set that annotation to force the release. + +A dump that has started streaming on this flow is not cancelled on the 30-minute backup deadline (a real dataset routinely outruns it, and deleting the CR mid-stream would prune a partial), but it is not unbounded either: after 24 hours in `running` the `BackupJob` fails with a `BackupAbandoned` warning naming the `PerconaServerMongoDBBackup`, which is left in place. If that dump later completes, its archive is reachable only through that CR; delete the CR to prune it. + +**Opting back out** (`useSystemBucket` true→false) is not automatic in reverse: the driver stops injecting, but the `s3-storage` entry it already applied stays on the live `PerconaServerMongoDB` under the driver's own field manager (`cozystack-psmdb-backup-driver`) — the chart does not own it and so never prunes it. It is inert once `backup.enabled=false`; to fully return a release to the legacy flow, delete that storage entry from the `PerconaServerMongoDB` by hand before re-declaring the tenant's own `backup.*` S3 values. Until it is removed, a new `BackupJob` on that release is refused rather than written into the platform bucket unowned: the driver recognises the injected entry by its `credentialsSecret` and holds the job at `Ready=False` (`PerconaServerMongoDBStorageStale`) naming that manual step, failing it at the backup deadline. A backup already streaming when the flag was flipped is unaffected — its flow is fixed by the finalizer on its own operator CR. + +### Point-in-time recovery (MongoDB) + +psmdb records an oplog stream between logical backups. A MongoDB `RestoreJob` recovers to a timestamp via `spec.options.recoveryTime` — the same option name and RFC3339 format the Postgres/CNPG driver uses, so the two are uniform: + +```yaml +spec: + options: + recoveryTime: "2026-08-05T12:34:56Z" # RFC3339 (UTC), same as the CNPG driver +``` + +The driver converts `recoveryTime` to the psmdb oplog target internally. Unlike CNPG (whose empty `recoveryTime` replays WAL to the latest archived point), an empty `recoveryTime` here restores the backup snapshot as taken — a psmdb logical backup is already a consistent point, so "restore this backup" is the safe default. `spec.options.restoreTimeoutSeconds` caps how long the driver waits for the operator restore before failing (default 30m), matching the CNPG option. Any unrecognised key under `spec.options` (e.g. a `recoverytime` typo) is ignored but surfaced as a `UnknownRestoreOption` Warning event on the RestoreJob rather than silently dropped. + ## Inspecting the defaults ```bash @@ -89,11 +189,15 @@ The bucket lives in `tenant-root` and is provisioned through the `apps.cozystack | Key | Consumer | |-----------------------------------------------|-------------------------------------------| -| `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` | CNPG, MariaDB, Etcd | +| `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` | CNPG, MariaDB, Etcd, RabbitMQ, Redis | | `accessKey` / `secretKey` (plus `bucketName`, `endpoint`, `region`) | ClickHouse sidecar | | `cloud` | Velero (AWS credentials file format) | | `blob_credentials.json` | FoundationDB backup_agent | +The Redis dump Job additionally reads the `endpoint`, `bucketName`, and `region` keys (alongside the `AWS_*` pair) via `secretKeyRef`, the same set the ClickHouse sidecar consumes. + +The dump Job also speaks to the Redis app's own Sentinel and master, which is a separate TLS axis from the S3 endpoint above. When the `Redis` app has `tls.enabled`, the operator moves both to a TLS-only listener and publishes the CA at `redis-.ca-cert`; the Job mounts that Secret optionally and connects with `--tls --cacert` exactly when its `ca.crt` is present, so a plaintext (non-TLS) app is unaffected. This covers the default `tls.authClients: no`. Mutual TLS (`authClients: yes`) is not supported for backup — it requires a client certificate signed by the release CA, which the platform does not issue. + ### Bootstrap window On a fresh-cluster install, the Velero `BackupStorageLocation` `cozy-default` is rendered before the credentials projector has had a chance to copy `cozy-backups-creds` into `cozy-velero`. The BSL reports `Unavailable` until the projector's first synchronous round completes (which runs as soon as the `backupstrategy-controller` acquires leadership — in practice moments after the Pod becomes Ready, typically tens of seconds after `helm install` returns, not minutes). Velero rejects new `Backup` AND `Restore` requests against `storageLocation: cozy-default` during that window. Plan VM backup automation accordingly, or wait for the BSL to become ready before submitting backups: `kubectl -n cozy-velero wait backupstoragelocation cozy-default --for=jsonpath='{.status.phase}'=Available --timeout=5m`. @@ -102,9 +206,52 @@ On a fresh-cluster install, the Velero `BackupStorageLocation` `cozy-default` is ### Cozy-default Bucket bootstrap -`cozy-default` ships an `apps.cozystack.io/Bucket cozy-backups` CR in `tenant-root`, which the bucket-application chart turns into a `BucketClaim`; the COSI driver then assigns the real S3 bucket name and writes it to the BucketClaim's `.status.bucketName`. The strategy templates and the Velero BSL all read that real bucket name (Helm `lookup` against the BucketClaim). On a fresh install the BucketClaim takes a short reconcile cycle to populate its status — until it does, the strategy templates render empty and only the `Bucket` CR + `BackupClass` are present in the cluster. The HelmRelease re-reconciles on its interval (5 minutes by default — set by the cozystack operator's `helmrelease-interval` flag, not a Flux default), at which point the populated BucketClaim status causes the missing strategy templates to materialise. +`cozy-default` ships an `apps.cozystack.io/Bucket cozy-backups` CR in `tenant-root`, which the bucket-application chart turns into a `BucketClaim`; the COSI driver then assigns the real S3 bucket name (`bucket-`, so it cannot be computed in advance) and writes it to the BucketClaim's `.status.bucketName`. The strategy templates and the Velero BSL all read that real bucket name (Helm `lookup` against the BucketClaim). On a fresh install the BucketClaim takes a reconcile cycle to populate its status — until it does, the strategy templates render empty and only the `Bucket` CR + `BackupClass` are present in the cluster. -If you need the BackupClass functional immediately (e.g. an e2e), trigger a Flux reconcile (`flux reconcile helmrelease backupstrategy-controller -n cozy-backup-controller`) once you see `kubectl get bucketclaim -n tenant-root bucket-cozy-backups -o jsonpath='{.status.bucketName}'` non-empty. +**That skip does not repair itself on a reconcile.** helm-controller re-renders a release only when its chart or values change; the interval reconcile is a no-op for a healthy release, and drift detection is off on operator-generated HelmReleases. A cluster that lost the race at install time therefore keeps `BackupClass cozy-default` with **no** `Strategy` CRs and **no** Velero BSL indefinitely — which also fail-closes the pre-adoption snapshot in the v1.6.0 etcd migration. + +The same trap sits one release earlier, on the Secret everything else depends on. `bucket-cozy-backups-system-credentials` is rendered by the `bucket-cozy-backups-system` release behind a `lookup` of the COSI Secret, and is skipped just as permanently when that lookup is empty. Without it the credentials projector has no source at all, so no strategy, no Velero and no v1.6.0 etcd migration can resolve — and the gate cannot even determine the bucket name, which it reads from that Secret. + +Convergence is driven instead by the controller's default-objects gate (`backupStorage.reconcileDefaultObjects`, on by default), which repairs both, in order. While the credentials Secret is absent or carries no bucket name, the gate stamps `reconcile.fluxcd.io/forceAt` + `requestedAt` on the **bucket's** `bucket--system` HelmRelease. Once the bucket name resolves it checks that every object `cozy-default` routes to exists and stamps the same pair on the **`backupstrategy-controller`** HelmRelease, forcing the real Helm upgrade that re-runs the lookups. The two are throttled independently (one force per 5 minutes each), so the second is not delayed by the first. Expect the objects within a minute or two of the bucket becoming ready. + +The gate does not force a suspended HelmRelease — helm-controller ignores both annotations while `spec.suspend` is true, so it logs the skip and leaves the force counter alone. A release left suspended (`cozyhr suspend`) is therefore never repaired until it is resumed. + +Watch it with: + +```bash +kubectl -n cozy-backup-controller logs deploy/backupstrategy-controller | grep default-objects-gate +``` + +#### Manual recovery on an affected cluster + +Only needed on a cluster running a version without the gate (or with `reconcileDefaultObjects: false`). A plain `flux reconcile helmrelease` does **nothing** here — it does not re-render. You need a forced upgrade, and **both** annotations: `forceAt` is what makes helm-controller run a real Helm upgrade, and it is only honoured together with `requestedAt`. + +First confirm the bucket name is actually resolvable — forcing before that just re-runs the same empty lookup: + +```bash +kubectl -n tenant-root get bucketclaim bucket-cozy-backups -o jsonpath='{.status.bucketName}' +``` + +Then force the two releases, **in this order**: + +```bash +# 1. The credentials Secret. It is rendered by the -system release, not by +# bucket-cozy-backups — easy to miss, and the projector (hence every +# strategy and Velero) has no source without it. +ts=$(date -u +%Y-%m-%dT%H:%M:%SZ) +kubectl -n tenant-root annotate helmrelease bucket-cozy-backups-system \ + reconcile.fluxcd.io/forceAt="$ts" reconcile.fluxcd.io/requestedAt="$ts" --overwrite +kubectl -n tenant-root get secret bucket-cozy-backups-system-credentials + +# 2. The Strategy CRs and the Velero BSL. +ts=$(date -u +%Y-%m-%dT%H:%M:%SZ) +kubectl -n cozy-backup-controller annotate helmrelease backupstrategy-controller \ + reconcile.fluxcd.io/forceAt="$ts" reconcile.fluxcd.io/requestedAt="$ts" --overwrite +kubectl get $(kubectl get crd -o name | grep strategy.backups.cozystack.io) 2>/dev/null +kubectl -n cozy-velero get backupstoragelocation cozy-default +``` + +The race is nested: step 2's `lookup` resolves from the BucketClaim status, but the projector — and the v1.6.0 etcd migration — read the Secret from step 1, so a cluster missing both needs both. ### Observability @@ -115,9 +262,15 @@ The credentials projector emits two Prometheus counters labelled by `namespace` Alert on `rate(cozystack_backup_credentials_projection_failures_total[5m]) > 0` or `absent_over_time(cozystack_backup_credentials_projection_successes_total[10m])` to catch a stale BSL credential or a malformed source Secret without log scraping. +The default-objects gate emits three more: + +- `cozystack_backup_default_objects_missing{backupclass="cozy-default"}` — how many of the objects the platform default backups depend on are absent: the credentials Secret, the Strategy CRs `cozy-default` routes to, and the Velero BSL. It is non-zero whatever the HelmRelease's `Ready` condition says, so it catches missing Strategy CRs that the release status does not show. Alert on `min_over_time(cozystack_backup_default_objects_missing[15m]) > 0`. +- `cozystack_backup_default_objects_check_errors_total{backupclass="cozy-default"}` — checks that could not reach a conclusion (an API error reading the source Secret, the BackupClass, or one of the routed objects). The gauge above is deliberately **not** written on those ticks, so that it does not flap on a transient API error — which means that while this counter climbs, the gauge is stale and a `0` on it proves nothing. Pair the two: `min_over_time(cozystack_backup_default_objects_missing[15m]) > 0 or rate(cozystack_backup_default_objects_check_errors_total[15m]) > 0`. +- `cozystack_backup_default_objects_force_reconciles_total{namespace,name}` — forced Helm upgrades issued, labelled by the release forced (this chart's own, or the bucket's `-system` release). A counter that keeps climbing means the forced render is not producing the objects (a missing CRD, for instance), which is a different problem from the install-time race. A suspended release is skipped before the patch and is **not** counted here, so a paused release cannot masquerade as a render that keeps failing — look for the `skipped forcing a suspended HelmRelease` log line instead. + ## Admin overrides for `cozy-default` -`cozy-default` is rendered by the `backupstrategy-controller` chart and owned by Flux's helm-controller. **Direct `kubectl edit backupclass cozy-default` is overwritten on the next helm reconcile** — the same applies to its companion `strategy.backups.cozystack.io/*` CRs (`cozy-default-cnpg`, `cozy-default-etcd`, `cozy-default-mariadb`, `cozy-default-altinity`, `cozy-default-foundationdb`, the two `cozy-default-velero-*`). The supported override path is the `backupStorage` block on the **`platform` component** of the `cozystack.cozystack-platform` Package CR: +`cozy-default` is rendered by the `backupstrategy-controller` chart and owned by Flux's helm-controller. **Direct `kubectl edit backupclass cozy-default` is overwritten on the next helm reconcile** — the same applies to its companion `strategy.backups.cozystack.io/*` CRs (`cozy-default-cnpg`, `cozy-default-etcd`, `cozy-default-mariadb`, `cozy-default-altinity`, `cozy-default-mongodb`, `cozy-default-foundationdb`, `cozy-default-rabbitmq`, `cozy-default-redis`, the two `cozy-default-velero-*`). The supported override path is the `backupStorage` block on the **`platform` component** of the `cozystack.cozystack-platform` Package CR: ```yaml apiVersion: cozystack.io/v1alpha1 @@ -139,7 +292,21 @@ spec: - cozy-velero ``` -The platform chart forwards this block into the child `Package cozystack.backupstrategy-controller` as component values, from where the cozystack operator merges it into the `backupstrategy-controller` HelmRelease over the chart defaults. Two paths that look plausible do **not** work: `spec.components.backupstrategy-controller` on the `cozystack.cozystack-platform` Package is silently ignored (the only component under that PackageSource is `platform`), and patching the child `Package cozystack.backupstrategy-controller` directly is reverted whenever the platform helm-reconcile re-renders it. +The platform chart forwards this block into the child `Package cozystack.backupstrategy-controller` as `components.backupstrategy-controller.values.backupStorage` (`packages/core/platform/templates/bundles/system.yaml`), from where the cozystack operator merges it into the `backupstrategy-controller` HelmRelease over the chart defaults. Two paths that look plausible do **not** work: `spec.components.backupstrategy-controller` on the `cozystack.cozystack-platform` Package is silently ignored (the only component under that PackageSource is `platform`), and patching the child `Package cozystack.backupstrategy-controller` directly is reverted whenever the platform helm-reconcile re-renders it. + +A sibling [`backupStrategyController`]({{% ref "/docs/next/operations/configuration/platform-package" %}}) block on the same `platform` component is forwarded the same way, for the controller's own knobs rather than the bucket. The one an operator reaches for is `redisBackupResources`, which sizes the Redis strategy Pod — restore loads the whole dataset into a throwaway loader, so a large Redis needs more memory / ephemeral-storage than the defaults (the loader-not-ready log line points here): + +```yaml +spec: + components: + platform: + values: + backupStrategyController: + redisBackupResources: + limits: + memory: 8Gi + ephemeral-storage: 32Gi +``` | Knob | Effect | |---|---| @@ -147,7 +314,9 @@ The platform chart forwards this block into the child `Package cozystack.backups | `bucketName` | Two modes. With `provisionBucket: true` (default): K8s name of the Bucket CR + lookup key for the COSI BucketClaim — the actual S3 bucket name is the COSI-assigned UUID, surfaced through `BucketClaim.status.bucketName`. With `provisionBucket: false`: taken **verbatim as the real S3 bucket name** and baked into every strategy CR + the Velero BSL. | | `namespace` | Namespace the Bucket CR (and its system-credentials Secret) lives in — `tenant-root` by default. Must be a tenant namespace (`tenant-*`): the Bucket chart's RBAC helper fails the Helm render for any other prefix. | | `bucketNameOverride` | Escape hatch for offline `helm template` renders — bypasses the live-cluster BucketClaim lookup. Leave empty in production. | -| `endpoint` | **Fallback** S3 endpoint. For a provisioned bucket the strategy CRs + Velero BSL derive the endpoint from the COSI system Secret (external ACME ingress, forced `https://`) instead; this value is used only for external S3 (`provisionBucket: false`) and offline renders. For external S3, switching it to `https://` enables TLS in the MariaDB/FoundationDB strategies — ensure the CA bundle is reachable to the relevant operator/driver Pods first. | +| `endpoint` | **Fallback** S3 endpoint. For a provisioned bucket the strategy CRs + Velero BSL derive the endpoint from the COSI system Secret (external ACME ingress, forced `https://`) instead; this value is used only for external S3 (`provisionBucket: false`) and offline renders. For external S3, switching it to `https://` enables TLS in the MariaDB/FoundationDB strategies, which derive TLS from the scheme. The Redis dump Job is not scheme-driven: the projector delivers a bare host and the script prepends `https://` unconditionally, so the Job always connects over TLS — point `endpointCASecretName` (below) at the private CA rather than relying on the scheme, and ensure that CA is reachable to the relevant Pods. | +| `endpointCASecretName` | Optional Secret (key `ca.crt`) in the app namespace the Redis Job trusts for a self-signed S3 endpoint. Empty by default: the projected endpoint is always `https://` and the platform bucket's ACME cert verifies against the image's system CA store unaided. Set it only for a private CA. The Job mounts it optionally (nothing projects this Secret automatically), so a name typo does not wedge the Pod on `FailedMount`: a missing `ca.crt` falls through to the system CA store and the Job fails fast at the TLS handshake with a legible error — it does not skip verification, which still needs the explicit `insecureSkipTLSVerify` opt-in. | +| `insecureSkipTLSVerify` | Disables S3 certificate verification for the Redis Job (`curl -k`). `false` by default and an explicit opt-in, never a fallback — an untrusted-cert endpoint fails closed unless this is set. Prefer `endpointCASecretName`. | | `region` | Re-projected into `cozy-backups-creds` on the next reconcile. Pod-restart required for chart-emitted clients consuming the region via env (ClickHouse sidecar today). | | `forcePathStyle` | Path-style addressing; SeaweedFS S3 requires it, AWS S3 typically doesn't. | | `systemSecretName` | Name of the human-friendly Secret produced by the Bucket app (or pre-created manually for external S3). The projector also accepts the raw COSI Secret format. | @@ -157,29 +326,42 @@ When the override needs to go beyond storage coordinates — different retention ## Tuning via a custom BackupClass -The defaults aim at a reasonable middle (30-day retention, gzip compression where applicable). To override for a specific tenant or workload, create your own `BackupClass` pointing at the same strategy CRs but with tweaked `parameters`, or a fresh strategy CR. Common knobs: +The defaults aim at a reasonable middle (gzip compression where applicable; engine-side retention only where the strategy carries it, which today is the CNPG `retentionPolicy: "30d"` and the two Velero `ttl: 720h`; every other shipped strategy, MongoDB included, sets none, so its archives on the shared bucket are bounded only by an S3 lifecycle policy or a manual `Backup` delete: a `Plan` only creates BackupJobs and never deletes a `Backup`). To override for a specific tenant or workload, create your own `BackupClass` pointing at the same strategy CRs but with tweaked `parameters`, or a fresh strategy CR. Common knobs: - **CNPG strategy**: `barmanObjectStore.retentionPolicy`, `data.compression`, `wal.compression`. - **MariaDB strategy**: `compression`, `maxRetention`, `databases[]`. -- **Altinity strategy**: tune the `clickhouse-backup` sidecar via `backup.*` values on the ClickHouse release; the strategy Pod is a thin HTTP client. +- **MongoDB strategy**: `storageName` (which `spec.backup.storages` entry on the psmdb cluster to use), `type` (`logical`), `compressionType` / `compressionLevel`, and — for the `useSystemBucket` flow — the `s3` coordinates the driver injects. On the default flow the S3 target is tuned via `backup.*` values on the MongoDB release instead (see [MongoDB: backup storage](#mongodb-backup-storage)). +- **Altinity strategy**: tune the `clickhouse-backup` sidecar via `backup.*` values on the ClickHouse release; the strategy Pod is a thin HTTP client. When the S3 endpoint's certificate is signed by a private CA rather than a publicly-trusted one — SeaweedFS's in-cluster `:8333` being the case in point — point `backup.endpointCA` at a Secret holding that CA bundle; the chart mounts it into the sidecar and adds it to the trust store via `SSL_CERT_DIR`, which supplements the system CA set rather than replacing it. - **FoundationDB strategy**: `snapshotPeriodSeconds`, `agentCount`, `urlParameters[]`. +- **Rabbitmq strategy**: `artifactURITemplate` (the `///definitions.json` object-key layout) and the pod `template` (image, resources). The backup is a definitions export over the management API — vhosts, users, permissions, queues, exchanges, bindings, policies, parameters — so message payloads are out of scope and there is no point-in-time recovery; a full message-data backup is a Velero volume snapshot instead. **Restore is a merge, not a reset**: importing definitions (`POST /api/definitions`) creates or updates what the export contains and never deletes, so anything created after the backup survives the restore and the broker is not returned to its exact backup-time state. A to-copy restore imports the source's users (with password hashes) into the target, so those source credentials become valid logins there. Unlike the operator-backed strategies (whose engine owns archive retention), the Rabbitmq driver owns its object outright and deletes it from the bucket when its `Backup` is deleted — via a one-shot Job the Backup's removal waits on — so a retention-pruned `Plan` does not normally accumulate objects. The delete is **best-effort**: when it cannot run from here the Backup is released without deleting the object (with a `Warning` Event naming what was left behind), so the Backup — and any namespace being torn down — never wedges. The named give-up conditions are: the namespace is terminating (`CREATE` is forbidden there), the strategy CR or the projected credentials are unavailable, the rendered Job name is invalid, or the skip annotation below is set. Otherwise a genuinely failing delete (object storage unreachable) keeps the `Backup` in `Terminating` (a visible signal to act on) and retries. To release such a stuck `Backup`, annotate it `backups.cozystack.io/skip-artifact-cleanup: "true"` — cleanup then skips the delete and lets the `Backup` go, leaving the object in the bucket to be reclaimed manually or by a bucket lifecycle policy (`kubectl annotate backup -n backups.cozystack.io/skip-artifact-cleanup=true`). - **Velero strategy (VMInstance / VMDisk)**: `ttl`, `includedResources[]`, `excludedResources[]`. -- **Etcd strategy**: today the strategy is path-only; combine with `Plan.spec.retentionPolicy` for trim cadence. +- **Etcd strategy**: today the strategy is path-only, and a `Plan` carries no retention field yet (see [#4192](https://github.com/cozystack/cozystack/pull/4192)), so nothing trims etcd snapshots on the shared bucket automatically either. The system-managed credentials Secret is the **only** way for in-cluster strategies to reach `cozy-backups`. Do not embed access keys in `BackupClass.parameters` — the security model relies on Secret references, and `parameters` end up in `Backup.status.underlyingResources`, which tenants can read. ## Disabling the platform-managed bucket -If a deployment runs against an external S3 (no SeaweedFS), set `backupStorage.provisionBucket: false` through the same platform Package path as above (`spec.components.platform.values.backupStorage`) and create the source credentials Secret in `tenant-root` manually (flat-key format: `accessKey` / `secretKey` / `endpoint` / `bucketName`; or the raw COSI `BucketInfo` JSON). In the same `backupStorage` block, update `endpoint`, `region`, **and `bucketName`**: with `provisionBucket: false` the strategies and the Velero BSL take `bucketName` verbatim as the real S3 bucket name (no COSI lookup), so it must name the actual bucket on the external S3 — the `bucketName` key inside the Secret alone is not enough. The Velero `BackupStorageLocation` picks the same values up automatically (the chart renders it from the same `backupStorage` block), so no separate BSL configuration is needed. Note that disabling the cluster-default BSL itself (the chart's `velero.bslEnabled` value) is **not** carried by the `backupStorage` override path — the platform Package forwards only the `backupStorage` block. +If a deployment runs against an external S3 (no SeaweedFS), set `backupStorage.provisionBucket: false` via the `platform` component override described above and create the source credentials Secret in `tenant-root` manually (flat-key format: `accessKey` / `secretKey` / `endpoint` / `bucketName`; or the raw COSI `BucketInfo` JSON). In the same `backupStorage` block, update `endpoint`, `region`, **and `bucketName`**: with `provisionBucket: false` the strategies and the Velero BSL take `bucketName` verbatim as the real S3 bucket name (no COSI lookup), so it must name the actual bucket on the external S3 — the `bucketName` key inside the Secret alone is not enough. The Velero `BackupStorageLocation` picks the same values up automatically (the chart renders it from the same `backupStorage` block), so no separate BSL configuration is needed. Note that disabling the cluster-default BSL itself (the chart's `velero.bslEnabled` value) is **not** carried by the `backupStorage` override path — the platform Package forwards only the `backupStorage` block. ## Upgrade notes from chart-managed backups -> **Postgres `backup.enabled: true` with placeholder credentials no longer renders `barmanObjectStore` on upgrade.** +> **Postgres backups now use the CloudNativePG Barman Cloud plugin, not native `spec.backup.barmanObjectStore`.** +> +> Postgres backups were migrated off the deprecated native `barmanObjectStore` (removed in CNPG 1.29) onto the [Barman Cloud plugin](https://github.com/cloudnative-pg/plugin-barman-cloud). The `barman-cloud-*` binaries the native path needs are absent from the `standard` image variant `keycloak-db` pins; `apps/postgres` pins bare (system-flavor) tags that still ship them — its breakage was the operator/CRD skew below, not missing binaries — but the system flavor is deprecated upstream, so both move to the plugin. The chart now renders `spec.plugins` on the `cnpg.io/Cluster` plus a `barmancloud.cnpg.io/ObjectStore` CR that carries the S3 configuration, and the CNPG backup driver SSA-applies an `ObjectStore` and patches `spec.plugins` (instead of `spec.backup.barmanObjectStore`) for the platform flow; `Backup`/`ScheduledBackup` use `method: plugin`. On `helm upgrade` a legacy chart-managed Cluster's `spec.backup.barmanObjectStore` is replaced by `spec.plugins` + an `ObjectStore`. This only works once the `plugin-barman-cloud` operator is installed and co-located with the CNPG operator in `cozy-postgres-operator` (shipped by the `postgres-operator` chart) and its cert-manager dependency is present — otherwise the Cluster references a plugin that is not registered and backups / WAL archiving stall. +> +> **Mixed-version window: BackupJobs fail until the app re-renders with the new chart.** A `BackupJob` that fires while a `Cluster` still carries the pre-upgrade chart's native `spec.backup.barmanObjectStore` is rejected by the CNPG admission webhook (`Cannot enable a WAL archiver plugin when barmanObjectStore is configured`) — the driver cannot attach `spec.plugins` alongside the native config. This resolves itself once Flux re-renders the Postgres app with the new chart (which replaces `barmanObjectStore` with `spec.plugins`); re-run the BackupJob afterward. Upgrading only the operators/driver without the app charts leaves every `backup.enabled: true` Postgres in this state permanently. +> +> **Postgres `backup.enabled: true` with placeholder credentials no longer wires backups on upgrade.** +> +> The pre-v1.5 defaults for `backup.s3AccessKey` / `backup.s3SecretKey` in `packages/apps/postgres/values.yaml` were the literal `""` / `""` placeholders, so the Postgres chart still rendered a backup block on the `cnpg.io/Cluster` (with junk credentials, WAL archiving failing at runtime). Starting with v1.5 those defaults are empty strings and the chart NO LONGER renders any backup wiring when the placeholders are unmodified. Tenants on the legacy chart-managed flow who relied on those placeholders see `spec.plugins` and the `ObjectStore` disappear from the live `Cluster` on `helm upgrade`. Action — pick one: +> +> - **Move to the platform flow (recommended).** Set `backup.useSystemBucket: true`; the chart leaves `spec.plugins` unset and the CNPG backup driver SSA-applies the `ObjectStore` and patches `spec.plugins` onto the live `Cluster` at first BackupJob time. No tenant-side keys required. +> - **Stay on the legacy chart-managed flow.** Supply real `backup.s3AccessKey` / `backup.s3SecretKey` (or a pre-existing `backup.s3CredentialsSecret.name`); the chart renders `spec.plugins` + the `ObjectStore` from those coordinates. > -> The pre-v1.5 defaults for `backup.s3AccessKey` / `backup.s3SecretKey` in `packages/apps/postgres/values.yaml` were the literal `""` / `""` placeholders, so the Postgres chart still rendered `spec.backup.barmanObjectStore` on the `cnpg.io/Cluster` (with junk credentials, `archive_command` failing at runtime). Starting with v1.5 those defaults are empty strings and the chart NO LONGER renders the backup block at all when the placeholders are unmodified. Tenants on the legacy chart-managed flow who relied on those placeholders see their `barmanObjectStore` disappear from the live `Cluster` on `helm upgrade`. Action — pick one: +> **Two platform-level changes ride along with this migration (they are required for the plugin to run):** > -> - **Move to the platform flow (recommended).** Set `backup.useSystemBucket: true`; the chart leaves `barmanObjectStore` unset and the CNPG backup driver SSA-patches it onto the live `Cluster` at first BackupJob time. No tenant-side keys required. -> - **Stay on the legacy chart-managed flow.** Supply real `backup.s3AccessKey` / `backup.s3SecretKey` (or a pre-existing `backup.s3CredentialsSecret.name`); the chart renders `barmanObjectStore` exactly as before. +> - **CloudNativePG operator upgraded** (`postgres-operator` chart). A minor operator upgrade for the whole fleet; the CNPG CRDs move with it. This also fixes a latent bug where the operator image was newer than its bundled CRDs (`status.instanceID.sessionID` pruned), which made the operator believe the instance manager restarted and fail **every** backup cluster-wide with `instance manager was restarted during backup`. The vendored chart ships the CNPG CRDs as ordinary release manifests (gated by `crds.create`), so a normal `helm upgrade` updates them together with the operator — the `helm.sh/resource-policy: keep` annotation only prevents their deletion, not updates. The historical skew came from the `image.tag` pin outrunning the vendored chart's CRDs, which this PR removes; no out-of-band CRD apply is needed unless the HelmRelease itself is held back. +> - **LINSTOR scheduler admission webhook must not strip native-sidecar fields.** Older `linstor-scheduler` extender builds (≤ v0.3.5's predecessors) decoded every pod through a stale vendored Kubernetes API and re-encoded it, stripping `initContainers[].restartPolicy` cluster-wide — which breaks the plugin's restartable barman sidecar ("startupProbe forbidden without restartPolicy=Always"). The platform already ships the fixed webhook (`linstor-scheduler` re-vendored to upstream chart 0.3.1 / extender v0.3.6, which stopped stripping unknown pod fields), so no action is needed on an up-to-date platform; just do not hold the `linstor-scheduler` package back on an older version when rolling out the plugin. > > The same `useSystemBucket` opt-in applies to ClickHouse — see [ClickHouse: opt-in to the system bucket](#clickhouse-opt-in-to-the-system-bucket). When `useSystemBucket: true` is set on ClickHouse, the legacy `-backup` CronJob, credential Secret, and backup script are no longer rendered (they are mutually exclusive with the platform flow); migrate scheduled backups to a `backups.cozystack.io/Plan` against `cozy-default`. @@ -201,9 +383,13 @@ spec: name: orders-db ``` +An application takes one restore at a time. A RestoreJob that targets an application another unfinished RestoreJob is already restoring fails with reason `ConcurrentRestore` before it changes anything, and its message names the job that goes first. It is not queued: once that job has finished, create a new RestoreJob if the restore is still wanted. A restore that has started always goes first. Between two that have not, the older one runs; creation times have one-second precision, so of two created in the same second, the one whose name sorts first runs. + ## Point-in-time recovery (PostgreSQL) -A `RestoreJob` restores a `Postgres` application from a `Backup`. Omit `spec.options.recoveryTime` to recover to the latest point in the WAL archive; set it (RFC3339) to recover the database to an exact instant — a point-in-time recovery (PITR). Under the hood the CNPG barman-cloud plugin restores the newest base backup taken at/before that instant and replays archived WAL up to it, so the restored cluster reflects the database exactly as of `recoveryTime`; later writes are absent. +A `RestoreJob` restores a `Postgres` application from a `Backup`. Omit `spec.options.recoveryTime` to recover to the latest point in the WAL archive; set it (RFC3339) to recover the database to an exact instant — a point-in-time recovery (PITR). The CNPG barman-cloud plugin restores a base backup and replays archived WAL from it: up to `recoveryTime` when one is set, so the restored cluster reflects the database as of that instant and later writes are absent, and to the end of the archive otherwise. + +The base backup a restore starts from is the one the `Backup` refers to: the driver passes its barman backup ID to CNPG as `bootstrap.recovery.recoveryTarget.backupID`. There are two exceptions, and for both the choice is left to the plugin. The first is a `recoveryTime` earlier than one second after that backup ended, which the backup cannot serve (recovery would have to stop before the backup is consistent); the plugin then takes the newest base backup that ended at or before `recoveryTime`. The second is a `Backup` whose `cnpg.io/Backup` no longer exists: retention (`barmanObjectStore.retentionPolicy`) deletes it together with the base backup it describes, so the archive can no longer start from that backup, and the `RestoreJob` records a `BaseBackupGone` warning event. A base backup removed from the bucket by anything else — a lifecycle rule, a manual delete — is still pinned for as long as its `cnpg.io/Backup` stays. The plugin deletes a `cnpg.io/Backup` whose base backup has left the catalog on its next maintenance pass (every `instanceSidecarConfiguration.retentionPolicyIntervalSeconds` of the `ObjectStore`, 30 minutes by default, retention policy or not), but only from the primary of the cluster that took it, running under the same UID. Until then a restore from it fails after the target has been purged, with `no backup found with ID` in the recovery log, and a retry fails the same way; deleting that `cnpg.io/Backup` by hand, which leaves the object store untouched, lets the next `RestoreJob` go through the plugin's choice. Left to itself for every restore, the plugin would start from the newest base backup in the archive whenever `recoveryTime` is empty, and from the newest one ending by `recoveryTime` otherwise, whichever `Backup` the `RestoreJob` names and on any timeline — see [Archives with several timelines](#archives-with-several-timelines) for why that matters. ```yaml apiVersion: backups.cozystack.io/v1alpha1 @@ -222,41 +408,60 @@ spec: recoveryTime: "2026-07-21T09:30:00Z" ``` -A full, scripted example (write a marker, capture the timestamp, restore to it, assert what survived) lives in the monorepo at [`examples/backups/postgres/`](https://github.com/cozystack/cozystack/tree/main/examples/backups/postgres) — `45-restorejob-pitr.yaml`, driven by `run-all.sh`. +A full, scripted example (write a marker, capture the timestamp, restore to it, assert what survived) is in the Cozystack repository at [`examples/backups/postgres/`](https://github.com/cozystack/cozystack/tree/main/examples/backups/postgres): `45-restorejob-pitr.yaml`, driven by `run-all.sh`. ### The recoverable window `recoveryTime` must fall inside the window the archive can reconstruct: -- **Earliest** — the completion of the oldest base backup still in the archive. You cannot recover to an instant before the first base backup: WAL replay always starts from a base backup, and retention (`barmanObjectStore.retentionPolicy`) eventually trims the oldest ones together with the WAL that predates them. -- **Latest** — the timestamp of the most recent WAL segment shipped to object storage. While the source runs with `backup.enabled: true` it archives continuously, so the latest restorable time trails "now" only by the archive lag (seconds for a healthy cluster). Once the source is deleted, the latest restorable time is frozen at whatever WAL made it to S3 before it went away. +- **Earliest** — the completion of the oldest base backup still in the archive. You cannot recover to an instant before the first base backup: WAL replay always starts from a base backup, and retention (`barmanObjectStore.retentionPolicy`) eventually trims the oldest ones together with the WAL that predates them. Aim at least one second after a backup's `status.stoppedAt`: `stoppedAt` is truncated to the second, while the plugin compares `recoveryTime` with the end time barman records in `backup.info`, to the microsecond, and keeps only a backup that ended at or before it. A `recoveryTime` equal to `stoppedAt` therefore falls before that backup's actual end: an older backup is used instead, and when there is none the restore fails with `no target backup found`. +- **Latest** — the last transaction, commit or abort, contained in the WAL shipped to object storage. It is not the newest WAL segment: a time-targeted recovery ends just before the first transaction recorded after `recoveryTime`, and a segment switch, a checkpoint or the end of a base backup is not one. A `recoveryTime` at or after the last archived transaction leaves PostgreSQL nothing to stop on. On a busy database the last transaction trails "now" by the archive lag; on a database at rest it is its last write, however recent the WAL archive and however recent the latest base backup. Once the source is deleted, the window is frozen at the last transaction that made it to S3 before it went away. -A `recoveryTime` **past the latest archived WAL cannot converge**: PostgreSQL replays every available WAL, never reaches the target, exits with `FATAL: recovery ended before configured recovery target was reached`, and CNPG re-creates the recovery instance in a loop. Such a wedged restore is failed by the restore deadline (`spec.options.restoreTimeoutSeconds`, default 30m) — and the driver reads the recovery pod's log at that point, so instead of a generic timeout the RestoreJob ends `status.phase: Failed` with conditions `RecoveryConverged=False` and `Ready=False`, both reason `RecoveryTargetUnreachable`, and a message naming the target. The driver deliberately does **not** fail earlier off that FATAL: a *reachable* near-now target hits the identical FATAL transiently — often repeatedly, on a slow archiver — before it converges, so an earlier trip would reject a recoverable restore. To reject an unreachable target quickly, set a short `restoreTimeoutSeconds`; otherwise widen the window (take a fresh backup, or restore closer to now) or pick a time inside it. +A `recoveryTime` **at or after the last archived transaction cannot converge**: PostgreSQL replays every available WAL, never finds a transaction after the target, exits with `FATAL: recovery ended before configured recovery target was reached`, and CNPG re-creates the recovery instance in a loop. Such a wedged restore is failed by the restore deadline (`spec.options.restoreTimeoutSeconds`, default 30m) — and the driver reads the recovery pod's log at that point, so instead of a generic timeout the RestoreJob ends `status.phase: Failed` with conditions `RecoveryConverged=False` and `Ready=False`, both reason `RecoveryTargetUnreachable`, and a message naming the target. The driver deliberately does **not** fail earlier off that FATAL: a *reachable* near-now target hits the identical FATAL transiently — often repeatedly, on a slow archiver — before it converges, so an earlier trip would reject a recoverable restore. To reject an unreachable target quickly, set a short `restoreTimeoutSeconds`; otherwise pick a time before the last archived transaction. The state as of the last write needs no `recoveryTime` at all: omitting it replays the whole archive, and that is the state it recovers. -Restoring to a **very recent** instant is safe as long as WAL archiving is current: the segment covering the target may not be in object storage yet and the same FATAL fires transiently, but because the driver only fails at the deadline (not on that FATAL), the recovery has the whole window to catch up — the next attempt promotes once the WAL ships and the cluster goes healthy, which completes the restore. Only a target that never becomes reachable within the deadline is failed. If archiving is badly behind (a stalled `archive_command`, an overloaded cluster) a near-now target can exceed even the default 30m window; restore to a point you can confirm is archived (e.g. at/before the most recent completed backup, per the discovery query below). +Restoring to a **very recent** instant is safe as long as the database keeps writing and WAL archiving is current: the segment covering the target, or the transaction after it, may not be in object storage yet and the same FATAL fires transiently, but because the driver only fails at the deadline (not on that FATAL), the recovery has the whole window to catch up — the next attempt promotes once the WAL ships and the cluster goes healthy, which completes the restore. Only a target that never becomes reachable within the deadline is failed: on a database that has stopped writing, that is every target after its last transaction. If archiving is badly behind (a stalled `archive_command`, an overloaded cluster) a near-now target can exceed even the default 30m window; restore to a point you can confirm is followed by an archived transaction (see the discovery section below). -A `recoveryTime` **before the earliest base backup** fails differently: PostgreSQL cannot begin replay before the base backup it restored, so recovery never reaches a consistent state and never emits the "recovery ended before …" FATAL the driver classifies on. That case also surfaces at the deadline, but with the generic reason `RestoreFailed` rather than `RecoveryTargetUnreachable`. Choosing a `recoveryTime` at/after the oldest base backup's completion (below) avoids it. +A `recoveryTime` **before the earliest base backup** fails differently: PostgreSQL cannot begin replay before the base backup it restored, so recovery never reaches a consistent state and never emits the "recovery ended before …" FATAL the driver classifies on. That case also surfaces at the deadline, but with the generic reason `RestoreFailed` rather than `RecoveryTargetUnreachable`. Choosing a `recoveryTime` at least one second after the oldest base backup's `stoppedAt` (below) avoids it. ### Discovering the earliest / latest restorable time CNPG's `Cluster.status.firstRecoverabilityPoint` exists in the status schema but is not reliably populated under the barman-cloud plugin, so read the window from the backup catalog instead. Each completed base backup records its WAL range and timestamps on the underlying `cnpg.io/Backup`: ```bash -# Completed base backups, oldest first: STOP is the earliest instant that -# backup alone can restore to; the oldest STOP is the window's lower bound. +# Completed base backups, oldest first: STOP plus one second is the earliest +# instant that backup alone can restore to; the oldest one bounds the window. kubectl -n get backups.postgresql.cnpg.io \ --sort-by=.status.stoppedAt \ - -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,START:.status.startedAt,STOP:.status.stoppedAt,BEGINWAL:.status.beginWal,ENDWAL:.status.endWal + -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,ID:.status.backupId,START:.status.startedAt,STOP:.status.stoppedAt,BEGINWAL:.status.beginWal,ENDWAL:.status.endWal -# The Cozystack Backup points at its underlying cnpg.io/Backup and S3 prefix. +# The cozystack Backup points at its underlying cnpg.io/Backup and S3 prefix. kubectl -n get backup.backups.cozystack.io -o jsonpath='{.spec.driverMetadata}' ``` -The upper bound — the latest archived WAL — is whatever the source has shipped so far; with archiving healthy, any time up to a few seconds ago is safe. To confirm the archive is current, check that the source cluster's WAL is flowing to S3 (the `barman-cloud` sidecar logs an upload per segment) before restoring to a near-now target. +The upper bound is the timestamp of the last `COMMIT` or `ABORT` record in the archived WAL, which nothing in the cluster status reports. `pg_waldump` prints it from the newest segments under `//wals/`, working back from the newest segment until one holds such a record; on a database at rest that can be several segments back, because every segment switch and every base backup adds one with no transaction in it. Any `recoveryTime` before that timestamp is reachable; for the state as of that last transaction, omit `recoveryTime`. + +### Archives with several timelines + +A WAL archive can hold several PostgreSQL timelines under one `serverName`. Every failover starts one, and so did every in-place restore made before restored clusters archived to a fresh prefix (`bootstrap.newServerName`): those wrote the new timeline back into the prefix they had recovered from, often without its `.history` file, so the archive ends up with branches that are siblings rather than a single line of descent. + +Recovery follows `recovery_target_timeline: latest`, which CNPG leaves as PostgreSQL's default: starting from the base backup's timeline, PostgreSQL looks for the history file of the next timeline number, then the next, stops at the first one missing, and targets the last one it found, replaying only the segments on that line of descent. Without any history file that is the base backup's own timeline alone. When the timeline it targets does not descend from the base backup — a sibling branch whose history file happens to carry the next number — recovery does not fall back to another timeline: it stops with `FATAL: requested timeline N is not a child of this server's history` (or `... is not in this server's history`). Two consequences: + +- the last archived transaction that bounds a restore is the last one on the timeline that restore follows, not the newest one in the archive; +- a restore must start from a base backup on the branch it means to bring back. The plugin's own choice ignores timelines — CNPG passes no `targetTLI` — so the newest backup, or the newest one ending by `recoveryTime`, can lie on another branch and restore that branch's data instead. The driver pins the `Backup`'s own base backup with `recoveryTarget.backupID` for that reason; a `recoveryTime` earlier than that backup's end still goes through the plugin's time-based choice, and on such an archive can land on another branch. ### Idempotency under GitOps -An in-progress restore is safe to reconcile. The driver purges the target `Cluster` + PVCs exactly once per RestoreJob (guarded by the `TargetPurged` condition and a freshly-recovered check), suspends the target's HelmRelease across the purge so Flux cannot race the bootstrap swap, and resumes it once the recovery cluster is rendered. A Flux reconcile (or a controller restart) mid-restore therefore re-attaches to the recovering cluster rather than deleting it and starting over. +An in-progress restore is safe to reconcile. The driver purges the target `Cluster` + PVCs exactly once per RestoreJob (guarded by the `TargetPurged` condition and by the `postgres.cozystack.io/restored-server-name` annotation, which identifies the RestoreJob whose purge produced the live recovery `Cluster`), suspends the target's HelmRelease across the purge so Flux cannot race the bootstrap swap, and resumes it once the recovery cluster is rendered. A Flux reconcile (or a controller restart) mid-restore therefore re-attaches to the recovering cluster rather than deleting it and starting over. + +## Kafka: topic metadata only + +The `cozy-default-kafka` strategy backs up **topic metadata**, not message data. Its Job talks to the Kafka Admin API (`kafka-topics --describe` on backup, `kafka-topics --create` + `kafka-configs --alter` on restore) and stores one small object per run at `s3://////kafka-metadata.txt`. It captures every non-internal topic's partition count, replication factor and non-default configs; message payloads, consumer-group offsets, ACLs, quotas and `KafkaUser`s are out of scope, and there is no point-in-time recovery. The driver gates each run on the Strimzi `Kafka` cluster reporting `Ready` and never mutates it. + +This is the "reconstruct a cluster's topic topology" flow: an in-place restore recreates dropped topics into the source, and a to-copy restore applies the source's topics onto a freshly-bootstrapped empty `Kafka` (partitions and configs preserved). Restore is additive at the topic-set level — it never deletes topics that exist live but not in the backup — but it does not silently accept a divergent existing topic: it grows the partition count when the backup asks for more, and fails loudly (naming the topic and both values) when the backup asks for fewer partitions or a different replication factor, since Kafka cannot shrink partitions or change RF in place and a restore that cannot reach the recorded state must not report `Succeeded`. Restored topics are created directly through the Admin API, so on the target they are **unmanaged** (no `KafkaTopic` CR); that is the accepted tradeoff of an Admin-API-only driver. Restoring a topic whose replication factor exceeds the target cluster's broker count fails at `--create`. Topic configs are restored **additively** (`kafka-configs --add-config`): a config the backup recorded is re-applied, but a config added to a live topic after the backup is not removed, so an in-place restore does not return a topic's config set to its exact backup-time state (partition count and replication factor are the only shape the restore reconciles or fails loudly on). + +One consequence of the Admin-API approach: if a topic is declared in the Kafka app's `spec.topics`, the Strimzi Topic Operator owns it and reconciles its config from the `KafkaTopic` CR, reverting any dynamic config an in-place restore sets that the CR does not list. So this strategy is durable for **out-of-band** topics (created directly, no CR); for CR-declared topics the `KafkaTopic` CR / GitOps is the source of truth. A to-copy restore onto a fresh empty cluster is unaffected. For message-data durability, use a volume-snapshot strategy instead. See `examples/backups/kafka-metadata/` for the end-to-end flow. + +Restore is not transactional: it applies topics one at a time as it reads the object, so a failure part-way through (an RF mismatch, an unreachable broker) leaves the topics already created behind — re-running the restore is safe (existing topics are reconciled, not recreated). Like the Rabbitmq driver, this one owns its object outright and deletes it from the bucket when its `Backup` is deleted — via a one-shot Job the Backup's removal waits on — so a retention-pruned `Plan` does not accumulate objects; the delete is **best-effort** with the same give-up conditions and `backups.cozystack.io/skip-artifact-cleanup` escape hatch documented for Rabbitmq above. One caveat specific to this driver: its image ships an older `curl` (7.76.1) whose `--aws-sigv4` omits a non-default port from the SigV4 canonical host, which every S3 backend then rejects with `403 SignatureDoesNotMatch` on a ported endpoint (fixed in curl 7.86.0). The strategy script works around it with `--connect-to` — it signs and sends a portless URL and redirects the connection to the real port — so a ported endpoint (the `backupStorage.endpoint` fallback `…:8333`, `provisionBucket: false`, or SeaweedFS with `s3.ingress.enabled: false`) works too; the sibling drivers run curl 8.x and need no such workaround. Export time scales with topic count — the backup spends two Kafka CLI invocations (each a fresh JVM, ~1.5–3s) per topic — so a cluster with many hundreds of topics takes tens of minutes; the run is bounded at the Job layer by `activeDeadlineSeconds` (default 30 minutes, from the same wait the readiness precondition uses), so a wedged or unschedulable run is failed by Kubernetes as `DeadlineExceeded` — the pod is killed before the script's final upload, so no object is left orphaned — rather than requeuing forever. A cluster whose export legitimately runs longer raises the bound with the `backupTimeout` BackupClass parameter (a Go duration, e.g. `2h`), which round-trips through the `Backup` so a restore is bounded by the same value. ## See also From c1d3060f2fdd4c66b0a1264096f4a071604c230d Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 14 Oct 2026 12:17:25 +0000 Subject: [PATCH 4/6] docs(operations): add upgrade notes for v1.6 to v1.7 The docs had no page for moving between v1.x releases; the actions a v1.7 upgrade needs lived only in the release notes. Adds "Upgrading from v1.6 to v1.7" next to the generic upgrade procedure, which now points to it. The page names 1.7 in its title and path, so it stays correct once the trunk is promoted to v1.7 and copied again later. It covers the platform migrations 54 to 59 and the default variant, which runs none of them; the drbd_transport_tcp module Talos v1.14 needs before a node is upgraded, and the etcd metrics port; worker pools as KubernetesNodes resources, the FluxCD addon removal and the Kamaji update with its extraArgs refusals; the Cilium address block, Multus CNI plugin staging and the Gateway API route and passthrough changes; KubeVirt eviction and declarative hotplug; Redis, PostgreSQL and OpenSearch exposure rules, generated database passwords, FoundationDB 7.1, the Kafka KRaft migration and the SeaweedFS volume size limit; and the smaller breaking changes. Every statement was checked against cozystack/cozystack at v1.7.0-rc.1. The listing commands were run against local fixtures. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- .../operations/cluster/upgrade-to-v1.7.md | 303 ++++++++++++++++++ .../docs/next/operations/cluster/upgrade.md | 2 + 2 files changed, 305 insertions(+) create mode 100644 content/en/docs/next/operations/cluster/upgrade-to-v1.7.md diff --git a/content/en/docs/next/operations/cluster/upgrade-to-v1.7.md b/content/en/docs/next/operations/cluster/upgrade-to-v1.7.md new file mode 100644 index 00000000..977c0842 --- /dev/null +++ b/content/en/docs/next/operations/cluster/upgrade-to-v1.7.md @@ -0,0 +1,303 @@ +--- +title: "Upgrading from v1.6 to v1.7" +linkTitle: "Upgrading to v1.7" +description: "What changes when a Cozystack v1.6 cluster is upgraded to v1.7, and what has to be done before and after the upgrade." +weight: 11 +--- + +Cozystack v1.7 changes the behaviour of running clusters in several places. This page lists those changes for a cluster that runs v1.6.x, and the action each one needs. The upgrade itself follows the usual procedure in [Upgrading Cozystack]({{% ref "/docs/next/operations/cluster/upgrade" %}}). Read this page first: several actions have to be taken before the upgrade starts, because once it runs it is too late to act on them. + +{{% alert title="Platform migrations do not run on the default variant" color="warning" %}} + +On every other variant the v1.7 platform release runs platform migrations 54 to 59 as a pre-upgrade hook, before any package moves to its new chart. The [`default`]({{% ref "/docs/next/operations/configuration/variants#default" %}}) variant runs none, so on such a cluster the steps those migrations would take for SeaweedFS, VM volume hotplug and HTTPCache are yours, before the upgrade. + +{{% /alert %}} + +## Before you upgrade + +- Every node: add `drbd_transport_tcp` to the Talos kernel modules before the node moves to Talos v1.14. See [Talos v1.14](#talos-v114). +- `isp-full` and `isp-full-generic`: check that `networking.podCIDR` has a prefix of /20 or shorter. See [Cilium address block](#cilium-address-block). +- Tenant Kubernetes clusters: remove `controlPlane.apiServer.extraArgs` entries that name a flag the tenant control plane sets itself. See [Kamaji and tenant control planes](#kamaji-and-tenant-control-planes). +- Multus: decide whether the reference CNI plugins may be written to `/opt/cni/bin` on every node. See [Multus stages CNI plugins](#multus-stages-cni-plugins). +- FoundationDB: upgrade every cluster that runs 7.1.x to 7.3. See [FoundationDB 7.1](#foundationdb-71). +- PostgreSQL with `external: true`: find clients that disable TLS. See [PostgreSQL](#postgresql). +- OpenSearch with `external: true` or Dashboards: see [OpenSearch](#opensearch). +- Gateway API: check the certificate served on `api.` and the hostnames of routes in tenant namespaces. See [Gateway API](#gateway-api). +- VM instances: fix `resources` values that cannot size a VM. See [Smaller changes](#smaller-changes). +- Kafka: plan a maintenance window for every instance. See [Kafka moves to KRaft](#kafka-moves-to-kraft). +- `default` variant only: pin the SeaweedFS volume size limit and look for volumes hotplugged on a VM instance only. See [SeaweedFS volume size limit](#seaweedfs-volume-size-limit) and [Declarative volume hotplug](#declarative-volume-hotplug). +- While the platform upgrade runs, do not edit `Kubernetes` resources. See [Worker pools become KubernetesNodes resources](#worker-pools-become-kubernetesnodes-resources). + +## Platform migrations + +The platform migration version moves from 54, where v1.6 left it, to 60. On every variant except `default`, the migrations below run once each, in order, as a pre-upgrade hook of the platform release: + +| Migration | What it does | +| --- | --- | +| 54 | Labels every existing `RedisFailover` for the new Redis operator (`redis-failover.freshworks.com/operator-group: cozystack`) and sets `redis:6.2.6-alpine` wherever its Redis or Sentinel image is empty, so the new operator does not move the engine to another version when it adopts the instance. | +| 55 | Detaches the HelmReleases of the removed FluxCD addon of tenant Kubernetes clusters without uninstalling Flux inside the tenant cluster. See [FluxCD addon removed](#fluxcd-addon-removed). | +| 56 | Moves the worker pools of every tenant Kubernetes cluster into one `kubernetes-nodes--` release per pool. See [Worker pools become KubernetesNodes resources](#worker-pools-become-kubernetesnodes-resources). | +| 57 | Writes `master.volumeSizeLimitMB: 1000` into every SeaweedFS instance that has no valid value of its own. See [SeaweedFS volume size limit](#seaweedfs-volume-size-limit). | +| 58 | Freezes HTTPCache on the packages artifact the cluster already runs, or removes it from the platform where no instance exists. See [Smaller changes](#smaller-changes). | +| 59 | Copies volumes that are hotplugged on a running VM instance but missing from its `VirtualMachine` spec into that spec. See [Declarative volume hotplug](#declarative-volume-hotplug). | + +After the upgrade, the `cozystack-version` ConfigMap shows the version the migrations reached: + +```bash +kubectl -n cozy-system get configmap cozystack-version -o jsonpath='{.data.version}{"\n"}' +``` + +A migration that fails stops the platform upgrade and is retried on the next attempt; one that succeeded does not run again. + +## Talos v1.14 + +Cozystack v1.7.0 ships Talos v1.14.2, with DRBD 9.3.4 and ZFS 2.4.4; v1.6 shipped Talos v1.13.6. The platform upgrade does not touch the Talos version of the nodes: each node moves when you upgrade it, and the steps below belong to that point. + +**Add `drbd_transport_tcp` before a node moves to Talos v1.14.** Talos v1.14 kernels no longer load kernel modules on demand, so DRBD cannot load its TCP transport when it connects to a peer, and every DRBD resource on the node stays in `Connecting`. List the module explicitly, next to `drbd`, in the node's machine config: + +```yaml +machine: + kernel: + modules: + - name: drbd + parameters: + - usermode_helper=disabled + - name: drbd_transport_tcp +``` + +The module ships in the DRBD extension of Talos v1.13 as well, so the line can be applied before the node is upgraded. The `cozystack` preset of talm v0.35.0 and earlier does not list it; with such a preset add it through `extraKernelModules` in the project's `values.yaml`, and apply the configuration before upgrading the node: + +```yaml +extraKernelModules: + - name: drbd_transport_tcp +``` + +Talos v1.14 accepts a direct upgrade from v1.12.0 or later, and supports Kubernetes 1.32 to 1.37. See [talm]({{% ref "/docs/next/install/kubernetes/talm" %}}) for how nodes are configured and upgraded. + +**etcd metrics move to port 2383.** Talos v1.14 serves etcd's HTTP endpoints, such as `/metrics` and `/health`, on a separate listener on port 2383, and port 2379 serves gRPC only. On the variants where Cozystack manages Cilium, the host network policy `restrict-system-components` denies port 2383 from outside the cluster, as it does 2379 and 2380. Cozystack's own monitoring reads neither port. A scraper of your own inside the cluster that reads etcd's HTTP endpoints from port 2379 has to move to 2383. + +Tenant Kubernetes clusters are not part of this: their workers keep the Talos version their `KubernetesNodes` pool sets. + +## Tenant Kubernetes clusters + +### Worker pools become KubernetesNodes resources + +The `Kubernetes` resource no longer describes worker nodes. `spec.nodeGroups` and `spec.nodeHealthCheck` are removed, and each worker pool is a `KubernetesNodes` resource (`apps.cozystack.io/v1alpha1`) in the same namespace, named `-`, whose `spec.cluster` names the parent cluster. Pool size and machine settings, the health-check settings (`maxUnhealthy`, `nodeStartupTimeout`, `readyUnknownTimeout`, `readyFalseTimeout`) and the autoscaler budget `maxNodeProvisionTime` are set per pool. `cluster` and `storageClass` are immutable once the pool exists. + +Platform migration 56 moves the pools of existing clusters. For every pool in `spec.nodeGroups` it creates a `kubernetes-nodes--` release, which appears as a `KubernetesNodes` resource, and hands the pool's existing worker objects over to it. A cluster that had no `nodeGroups` gets a pool named `md0` with the defaults the old chart used for it: `u1.medium` workers, 0 to 10 replicas, a 20Gi disk and the `ingress-nginx` role. The adoption changes the bootstrap configuration template each pool's `MachineDeployment` refers to, so plan the upgrade with room for the workers of a pool to be replaced. A pool that holds `nvidia.com/*` GPUs and does not set `kernelModules` rolls in any case, because v1.7 adds the NVIDIA kernel modules to its machine configuration. + +While the platform upgrade runs, do not edit `Kubernetes` resources: a change in that window renders the old chart, and its apply fails until the new chart arrives. + +A pool the migration cannot adopt keeps running but is left unmanaged. That happens, among other cases the migration's log names, when `kubernetes-nodes--` is longer than the 53 characters a Helm release name allows, so `-` must fit in 36 characters, when the pool name holds characters a release name cannot carry, or when the pool's objects belong to another release. The migration pins such a pool's objects with `helm.sh/resource-policy: keep` so nothing deletes them, and records each one in a ConfigMap: + +```bash +kubectl -n cozy-system get configmap cozystack-migration-56-unadopted -o yaml +``` + +The ConfigMap exists only when at least one pool was skipped. Migration 56 does not run again, so such a pool has to be recreated or adopted by hand. + +The removed fields stay stored on upgraded `Kubernetes` resources and have no effect. Every write to a resource that still carries one returns a warning such as `spec.nodeGroups is ignored on the Kubernetes resource since Phase 2: worker pools are managed as separate KubernetesNodes resources`. Change a pool through its `KubernetesNodes` resource instead. + +Rolling the platform back to v1.6 after migration 56 is not automatic: the old chart cannot render worker objects that now belong to the per-pool releases. + +A `Kubernetes` resource created on v1.7 has a name of at most 32 characters; existing clusters with longer names keep working. A cluster created on v1.7 has no workers until a `KubernetesNodes` pool exists for it. With the ingress-nginx addon in its default `Proxied` mode, at least one pool needs `roles: [ingress-nginx]`, because the Service that publishes the tenant's ingress selects the workers carrying that role: + +```yaml +apiVersion: apps.cozystack.io/v1alpha1 +kind: KubernetesNodes +metadata: + name: mycluster-md0 + namespace: tenant-example +spec: + cluster: mycluster + roles: + - ingress-nginx + minReplicas: 1 + maxReplicas: 3 + instanceType: u1.medium +``` + +### FluxCD addon removed + +The `addons.fluxcd` addon of the `Kubernetes` application is removed. Platform migration 55 detaches its HelmReleases from the management cluster without uninstalling anything, so a tenant cluster that had the addon keeps its Flux running, unmanaged. To remove it, uninstall its two Helm releases from inside the tenant cluster, the instance first and the operator second, using the tenant cluster's kubeconfig: + +```bash +helm uninstall fluxcd --namespace cozy-fluxcd +helm uninstall fluxcd-operator --namespace cozy-fluxcd +``` + +An `addons.fluxcd` key left in a `Kubernetes` resource is accepted and has no effect. + +### Kamaji and tenant control planes + +Kamaji moves from 26.3.6-edge to 26.9.4-edge. Kamaji now tracks the kube-apiserver arguments of a tenant control plane by a hash of its `extraArgs` and rebuilds the list from scratch whenever the hash differs, so on the first reconcile after the upgrade every tenant control plane rebuilds its arguments and its kube-apiserver rolls once. A tenant with OIDC enabled can roll a second time when the chart's own flags land. Plan the upgrade window for that. + +Kamaji now sets a list of kube-apiserver flags itself and drops any `controlPlane.apiServer.extraArgs` entry that names one, and the `Kubernetes` application refuses to render such an entry. A tenant cluster that carries one stops reconciling after the upgrade until the entry is fixed. Two of these flags are moved instead of refused: an `--enable-admission-plugins=` entry goes into the new `controlPlane.apiServer.admissionControllers`, and a `--kubelet-preferred-address-types=` entry into the new `controlPlane.kubelet.preferredAddressTypes`, as long as the field is empty and the entry is a single unquoted `--flag=value` string. List the entries to look at before upgrading: + +```bash +kubectl get kuberneteses.apps.cozystack.io --all-namespaces --output json | jq --raw-output '["--advertise-address","--client-ca-file","--enable-admission-plugins","--service-cluster-ip-range","--kubelet-client-certificate","--kubelet-client-key","--kubelet-preferred-address-types","--proxy-client-cert-file","--proxy-client-key-file","--requestheader-allowed-names","--requestheader-client-ca-file","--secure-port","--service-account-key-file","--service-account-signing-key-file","--tls-cert-file","--tls-private-key-file","--etcd-servers","--etcd-cafile","--etcd-certfile","--etcd-keyfile","--etcd-prefix","--etcd-compaction-interval","--egress-selector-config-file"] as $owned | .items[] | .metadata as $m | (.spec.controlPlane.apiServer.extraArgs // [])[] | select((split("=")[0]) as $f | $owned | index($f)) | "\($m.namespace)/\($m.name): \(.)"' +``` + +Remove every entry it lists, together with its value if the value sits in a separate entry. The exceptions are the two moved flags written as one unquoted `--flag=value` entry next to an empty field, which need no change. + +Do not roll Kamaji back to an earlier build after the upgrade. The older build expects a number where the new one stores the hash, and from then on it never removes a flag you take out of `extraArgs`, on any tenant control plane, until the annotation `kube-apiserver.kamaji.clastix.io/args` is set back to a number by hand. + +## Networking + +### Cilium address block + +On `isp-full` and `isp-full-generic`, Cilium now takes the addresses of its router and of its Gateway API ingress from the last /21 of `networking.podCIDR`, and Kube-OVN no longer hands out addresses from that block. `networking.podCIDR` therefore needs a prefix of /20 or shorter; a narrower one fails the platform render with `networking.podCIDR is too small`. The default `10.244.0.0/16` qualifies. Each node takes a /29 of the block, which caps these variants at 256 nodes. + +An existing cluster moves to the block in a post-upgrade hook of the Cilium release, the Job `cilium-ipam-migration` in `cozy-cilium`. It excludes the block from the Kube-OVN default subnet, then moves the nodes one at a time, so each Cilium agent restarts twice: once for the configuration change and once in the hook. The hook stops, and the Cilium release upgrade fails, if pods still hold addresses inside the block; its log names them, and once they are recreated the next upgrade attempt continues. A failed Job is kept for its log. + +### Multus stages CNI plugins + +On the variants that ship Multus, it now installs the upstream reference CNI plugins into `/opt/cni/bin` on every node: bridge, host-local, static, macvlan, ipvlan, vlan, tuning, portmap, firewall, bandwidth, host-device, ptp, sbr and vrf. They are written every time a Multus pod is recreated and replace files of the same name. On Talos four of them, bridge, firewall, host-local and portmap, are rewritten with the same upstream version Talos carries. On a generic Linux node a plugin of your own under one of those names does not survive. + +To decline, set `networking.stageCniPlugins: false` on the `platform` component of the Platform Package before upgrading. Setting it later does not restore files already replaced. The value accepts `true`, `false` and their usual YAML spellings; anything else fails the whole platform render. + +### East-west encryption + +`networking.encryption.enabled` is a new opt-in that encrypts pod traffic between nodes with Kube-OVN IPsec on `isp-full` and `isp-full-generic`. It is off by default, so the upgrade changes nothing. See the [Platform Package reference]({{% ref "/docs/next/operations/configuration/platform-package" %}}). + +### Gateway API + +These apply where Gateway API resources are in use. + +- An `HTTPRoute`, `TLSRoute` or `GRPCRoute` in a `tenant-*` namespace must declare a non-empty `spec.hostnames`. A route that relied on inheriting its listener's hostnames is refused on its next create or update and has to name them. +- Under HTTP-01, a hostname a TLS-passthrough listener answers is reserved to that listener and gets no HTTPS listener and no certificate on 443. With `gateway.enabled` on, the publishing tenant therefore passes `api.`, `vm-exportproxy.` and `cdi-uploadproxy.` through after the upgrade, for the names `publishing.exposedServices` lists, and the only certificate on `api.` is the management kube-apiserver's own serving certificate. It has to carry `api.` in its SANs before the upgrade. A cluster that came to the Gateway from ingress-nginx already meets this. +- The three platform passthrough listeners are rendered only on the publishing tenant. A child tenant that rendered them before loses them, together with any `TLSRoute` attached to them by hand. +- The certificates of the shed HTTPS listeners are deleted, but their TLS Secrets stay in the namespace and can be removed by hand. + +See [Gateway API]({{% ref "/docs/next/networking/gateway-api" %}}) for the details, including `tlsPassthroughListeners`, the new native-port passthrough form. + +## Virtual machines + +### Eviction of non-migratable VMs + +The cluster-wide KubeVirt eviction strategy changes from `LiveMigrate` to `LiveMigrateIfPossible`. A VM that can live-migrate still migrates when its pod is evicted. A VM that cannot, for example one on a ReadWriteOnce disk or with a GPU or host device passed through, is shut down instead of blocking the drain, and started again according to its `runStrategy`: a VM with `runStrategy: Always` comes back, while one with `RerunOnFailure`, `Manual` or `Once` that shuts down cleanly stays off until it is started. A VM whose disk is bound to the drained node stays down until the node returns. This applies to any eviction through the API, not only a drain. + +A running VM instance keeps the strategy it was created with, so a non-migratable VM started before the upgrade still blocks a drain until its VM is restarted. Restart such VMs once after the upgrade if their nodes have to drain. From then on, a KubeVirt upgrade shuts down non-migratable VMs to move them to the new version, ten every minute. + +### Declarative volume hotplug + +KubeVirt v1.9.0 runs with `DeclarativeHotplugVolumes` instead of the deprecated `HotplugVolumes` feature gate, and the kubevirt-csi driver of tenant Kubernetes clusters no longer hot-plugs volumes through the VM instance. Hotplugged volumes now follow the `VirtualMachine` spec: on its first reconcile, KubeVirt removes every hotplugged volume of a running VM instance that the VM spec does not list. + +Platform migration 59 copies such volumes into the VM spec before that happens, so they stay attached. This also makes a volume that was added to the VM instance only, without being persisted, permanent: it attaches again after a VM restart, and from then on it can be removed only through the VM. + +On the `default` variant this migration does not run. Before upgrading such a cluster, list the affected volumes: + +```bash +kubectl get vm,vmi --all-namespaces --output json | jq --raw-output ' + ([.items[] | select(.kind == "VirtualMachine") + | {key: "\(.metadata.namespace)/\(.metadata.name)", value: [.spec.template.spec.volumes[]?.name]}] + | from_entries) as $vm + | .items[] | select(.kind == "VirtualMachineInstance") + | ([.metadata.ownerReferences[]? | select(.controller == true and .kind == "VirtualMachine" and (.apiVersion | startswith("kubevirt.io/"))) | .name][0]) as $owner + | "\(.metadata.namespace)/\($owner)" as $key + | select(.status.phase == "Running" and $owner != null and $vm[$key] != null) + | .spec.volumes[]? | select(.dataVolume.hotpluggable == true or .persistentVolumeClaim.hotpluggable == true) + | .name as $n | select($vm[$key] | index($n) | not) + | "\($key) \($n)"' +``` + +Empty output means nothing is affected. Each line names a VM and a volume the upgrade would detach. Recreate each tenant Kubernetes node it lists before the upgrade, so that its volumes are attached the new way. + +## Managed applications + +### Redis + +A Redis instance can now be exposed outside the cluster only when it is created with both `external: true` and `tls.enabled: true`. The `cozystack-database-external-tls-policy` admission policy refuses to create an external instance without TLS, to turn `external` on for an existing instance, whether or not it has TLS, and to change `tls.enabled` while `external` is on. TLS is fixed when an instance is created, so an existing instance cannot gain it: to expose one with TLS, create a new instance with both values set and move the clients to it. + +Instances that were already external without TLS keep working and stay editable, apart from `tls.enabled`. Setting `external: false` in the same edit lifts that freeze, but external access can then be turned on again only by creating a new instance. + +On upgrade the Sentinel pods of every instance move to the Redis image of the data pods and roll once, and the data pods roll once for a new metrics exporter image. + +### PostgreSQL + +An external PostgreSQL instance now refuses sessions that are not encrypted. The new `tls.required` value has three states: + +- unset (the default): follows `external`, so an instance with `external: true` accepts only TLS sessions; +- `true`: TLS sessions only, whether or not the instance is external; +- `false`: no extra rule, so plaintext sessions are accepted as before. + +The rule covers every client of such an instance, in-cluster ones included. libpq's default `sslmode=prefer` already negotiates TLS, so only clients that disable TLS explicitly, with `sslmode=disable`, are refused. An instance that was external before the upgrade refuses those clients afterwards; update them, or set `tls.required: false` to keep accepting them. + +### PostgreSQL and MariaDB passwords + +User passwords can no longer be set from values: `users..password` is removed from both applications, and every password, MariaDB's `root` included, is generated into the `-credentials` Secret. On upgrade the charts keep the passwords already stored in that Secret, so existing logins keep working. A `password` left in values is ignored with an admission warning, but the value it set stays the live password until it is rotated. Read a user's password from the Secret: + +```bash +kubectl get secret -credentials -o json | jq -r '.data["user1"]' | base64 -d +``` + +An upgraded MariaDB release refuses to render if its `-credentials` Secret is missing, rather than generating new passwords that would not match the server. + +### OpenSearch + +- An external release that never set `tls.issuer` moves from the operator's certificate authority to a per-release cert-manager CA the first time it reconciles after the upgrade. That rolls the cluster and re-anchors it on the new CA, published to the tenant as `opensearch-.tenant-ca`, so anything pinned to the old CA has to move. Set `tls.issuer: operator` before the upgrade to stay on the operator's CA. +- The external-dns names of an external release move from the in-cluster domain to `.` and `-dashboards.`, whatever `tls.issuer` says. The old records are not deleted and have to be removed from the zone by hand. +- With `dashboards.enabled`, an application name is limited to 41 characters, and to 32 when `external` is also on. A longer existing release stops rendering after the upgrade; set `dashboards.enabled: false`, or recreate the application under a shorter name to keep Dashboards. +- Every OpenSearch cluster rolls once on upgrade. + +### FoundationDB 7.1 + +The FoundationDB operator no longer ships the 7.1 client, so it cannot manage or upgrade a cluster that runs FoundationDB 7.1.x. Before upgrading the platform, move every such cluster to 7.3 through its `cluster.version` value. + +### Kafka moves to KRaft + +The Kafka application now runs Kafka in KRaft mode, and existing ZooKeeper-based instances are migrated in place, automatically, on their first upgrade to the new chart. The migration runs as the Job `-kafka-migration` in the instance's namespace; follow it with: + +```bash +kubectl logs -n job/-kafka-migration +``` + +Plan a maintenance window of up to a few hours per instance: the migration rolls the brokers, and when several ZooKeeper-based instances share a namespace they migrate one after another. Once a cluster passes the `PreKRaft` state the migration cannot be reversed, so do not downgrade the Kafka application or the platform afterwards. + +This release is a mandatory stop on the upgrade path: make sure every Kafka instance has reached KRaft on v1.7 before upgrading past it, because later Strimzi releases no longer migrate ZooKeeper clusters. The application also gains a `version` value, `v3.9` by default. + +### SeaweedFS volume size limit + +New SeaweedFS instances start with `master.volumeSizeLimitMB: 400` instead of 1000, and a bucket now grows three volumes at a time instead of six at `replicationFactor: 2`. Lowering the limit on an instance that already holds data stops writes to buckets whose volumes have all crossed the new limit until they can grow, so existing instances keep 1000: + +- On every variant except `default`, platform migration 57 writes `master.volumeSizeLimitMB: 1000` into each instance that has no valid value of its own. +- An instance whose value was already set to a whole number from 1 to 30000 keeps it. The previous chart ignored that value, so it takes effect with the upgrade. +- On the `default` variant nothing writes the value, and every instance without one would drop to 400. Before upgrading, apply the migration's change yourself with the commands below. + +```sh +kubectl get helmreleases.helm.toolkit.fluxcd.io -A -o json \ + -l apps.cozystack.io/application.kind=SeaweedFS,apps.cozystack.io/application.group=apps.cozystack.io \ + | jq -r '"instances \(.items | length)", + (.items[] | "\(.metadata.namespace) \(.metadata.name)" as $id | .spec.values as $s + | [(.spec.valuesFrom // [])[] | select(. != {kind: "Secret", name: "cozystack-values"})] as $extra + | (if $extra != [] then "valuesfrom \($id) \($extra | tojson)" else empty end), + (if $s != null and ($s | type) != "object" then "unreadable \($id) \($s | tojson)", "pin \($id)" + elif $s.master != null and ($s.master | type) != "object" then "unrenderable \($id) \($s.master | tojson)" + else $s.master.volumeSizeLimitMB as $v + | select(($v | type) != "number" or $v != ($v | floor) or $v < 1 or $v > 30000) + | "pin \($id)" end)), + "complete \(.items | length)"' \ + | while read -r what ns name detail; do + case "$what" in + pin) ;; + instances|complete) echo "$what $ns"; continue ;; + *) echo "$what $ns $name $detail"; continue ;; + esac + echo "pinning $ns/$name" + kubectl -n "$ns" patch helmreleases.helm.toolkit.fluxcd.io "$name" --type=merge \ + -p '{"spec":{"values":{"master":{"volumeSizeLimitMB":1000}}}}' Date: Fri, 16 Oct 2026 09:39:21 +0000 Subject: [PATCH 5/6] docs(operations): describe the worker roll as expected behavior The upgrade notes said the pool adoption changes the bootstrap template of each MachineDeployment, without saying why or that this is intended. v1.7 names each pool's TalosConfigTemplate after a hash of its content (cozystack/cozystack#3523), so every pool's MachineDeployment refers to a new template and Cluster API rolls the Talos workers once. The page now states the roll as expected, one-time behaviour with that cause and the rollout strategy, asks to plan the window for it, and notes that the same roll brings stale machine configuration up to date and gives GPU pools their NVIDIA kernel modules. The pre-upgrade checklist gains an entry for it. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- content/en/docs/next/operations/cluster/upgrade-to-v1.7.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/content/en/docs/next/operations/cluster/upgrade-to-v1.7.md b/content/en/docs/next/operations/cluster/upgrade-to-v1.7.md index 977c0842..f4c5a1fb 100644 --- a/content/en/docs/next/operations/cluster/upgrade-to-v1.7.md +++ b/content/en/docs/next/operations/cluster/upgrade-to-v1.7.md @@ -18,6 +18,7 @@ On every other variant the v1.7 platform release runs platform migrations 54 to - Every node: add `drbd_transport_tcp` to the Talos kernel modules before the node moves to Talos v1.14. See [Talos v1.14](#talos-v114). - `isp-full` and `isp-full-generic`: check that `networking.podCIDR` has a prefix of /20 or shorter. See [Cilium address block](#cilium-address-block). - Tenant Kubernetes clusters: remove `controlPlane.apiServer.extraArgs` entries that name a flag the tenant control plane sets itself. See [Kamaji and tenant control planes](#kamaji-and-tenant-control-planes). +- Tenant Kubernetes clusters: plan for the Talos workers of every pool to be replaced once. See [Worker pools become KubernetesNodes resources](#worker-pools-become-kubernetesnodes-resources). - Multus: decide whether the reference CNI plugins may be written to `/opt/cni/bin` on every node. See [Multus stages CNI plugins](#multus-stages-cni-plugins). - FoundationDB: upgrade every cluster that runs 7.1.x to 7.3. See [FoundationDB 7.1](#foundationdb-71). - PostgreSQL with `external: true`: find clients that disable TLS. See [PostgreSQL](#postgresql). @@ -84,7 +85,9 @@ Tenant Kubernetes clusters are not part of this: their workers keep the Talos ve The `Kubernetes` resource no longer describes worker nodes. `spec.nodeGroups` and `spec.nodeHealthCheck` are removed, and each worker pool is a `KubernetesNodes` resource (`apps.cozystack.io/v1alpha1`) in the same namespace, named `-`, whose `spec.cluster` names the parent cluster. Pool size and machine settings, the health-check settings (`maxUnhealthy`, `nodeStartupTimeout`, `readyUnknownTimeout`, `readyFalseTimeout`) and the autoscaler budget `maxNodeProvisionTime` are set per pool. `cluster` and `storageClass` are immutable once the pool exists. -Platform migration 56 moves the pools of existing clusters. For every pool in `spec.nodeGroups` it creates a `kubernetes-nodes--` release, which appears as a `KubernetesNodes` resource, and hands the pool's existing worker objects over to it. A cluster that had no `nodeGroups` gets a pool named `md0` with the defaults the old chart used for it: `u1.medium` workers, 0 to 10 replicas, a 20Gi disk and the `ingress-nginx` role. The adoption changes the bootstrap configuration template each pool's `MachineDeployment` refers to, so plan the upgrade with room for the workers of a pool to be replaced. A pool that holds `nvidia.com/*` GPUs and does not set `kernelModules` rolls in any case, because v1.7 adds the NVIDIA kernel modules to its machine configuration. +Platform migration 56 moves the pools of existing clusters. For every pool in `spec.nodeGroups` it creates a `kubernetes-nodes--` release, which appears as a `KubernetesNodes` resource, and hands the pool's existing worker objects over to it. A cluster that had no `nodeGroups` gets a pool named `md0` with the defaults the old chart used for it: `u1.medium` workers, 0 to 10 replicas, a 20Gi disk and the `ingress-nginx` role. + +The upgrade then rolls the Talos workers of every pool once. This is expected. v1.7 names each pool's `TalosConfigTemplate` after a hash of its content, `kubernetes---`, instead of the fixed `kubernetes--`, so the pool's `MachineDeployment` refers to a new template and Cluster API replaces the workers under the pool's rolling-update strategy: up to `maxReplicas` new machines at once, and at most one old machine unavailable at a time. No action is needed on the pools; plan the upgrade window for the workers of every tenant cluster to be replaced. Data on a worker's own system disk, such as local-path volumes, does not survive the replacement, as with any roll of the pool. The same roll brings each pool's machine configuration up to date where the old fixed-name template could not follow a later change, and gives a pool that holds `nvidia.com/*` GPUs the NVIDIA kernel modules that v1.7 declares in its machine configuration: a Talos system extension installs a module but does not load it. While the platform upgrade runs, do not edit `Kubernetes` resources: a change in that window renders the old chart, and its apply fails until the new chart arrives. From 7ffed72628365edeb1badcedbd407b26599173ed Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Fri, 16 Oct 2026 17:19:17 +0500 Subject: [PATCH 6/6] docs(operations): note the PostgreSQL restart in the v1.7 upgrade notes Instances with a memory limit below 512Mi and no explicit shared_buffers get a new default that needs a PostgreSQL restart, unless the cluster already runs v1.6.5. Signed-off-by: Myasnikov Daniil Assisted-by: LLM --- content/en/docs/next/operations/cluster/upgrade-to-v1.7.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/en/docs/next/operations/cluster/upgrade-to-v1.7.md b/content/en/docs/next/operations/cluster/upgrade-to-v1.7.md index f4c5a1fb..690247e5 100644 --- a/content/en/docs/next/operations/cluster/upgrade-to-v1.7.md +++ b/content/en/docs/next/operations/cluster/upgrade-to-v1.7.md @@ -226,6 +226,8 @@ An external PostgreSQL instance now refuses sessions that are not encrypted. The The rule covers every client of such an instance, in-cluster ones included. libpq's default `sslmode=prefer` already negotiates TLS, so only clients that disable TLS explicitly, with `sslmode=disable`, are refused. An instance that was external before the upgrade refuses those clients afterwards; update them, or set `tls.required: false` to keep accepting them. +An instance with a memory limit below 512Mi and no explicit `shared_buffers` (the `t1.nano` preset and the default `t1.micro`) now gets `shared_buffers` set to a quarter of the limit, 64MB for `t1.micro`, where PostgreSQL used its built-in 128MB before. The setting needs a restart, so on upgrade CloudNativePG restarts the replicas first and then the primary in place: writes pause for one restart, and an instance with a single replica is unavailable during it. A cluster that already runs v1.6.5 or later has this value from that patch and is not restarted for it again. Set `postgresql.parameters.shared_buffers` before the upgrade to take the restart on your own schedule. + ### PostgreSQL and MariaDB passwords User passwords can no longer be set from values: `users..password` is removed from both applications, and every password, MariaDB's `root` included, is generated into the `-credentials` Secret. On upgrade the charts keep the passwords already stored in that Secret, so existing logins keep working. A `password` left in values is ignored with an admission warning, but the value it set stays the live password until it is rotated. Read a user's password from the Secret: