diff --git a/.claude/skills/pattern-author/SKILL.md b/.claude/skills/pattern-author/SKILL.md index 281eef3b4..0634687a8 100644 --- a/.claude/skills/pattern-author/SKILL.md +++ b/.claude/skills/pattern-author/SKILL.md @@ -26,7 +26,7 @@ When creating a new Pattern: 1. **Initialize** — Determine the Pattern name and whether secrets are needed. Run `pattern init` or `pattern init --with-secrets` in the Pattern directory. 2. **Identify requirements** — What operators, applications, and custom helm charts does this Pattern need? -3. **Define namespaces** — Add all required namespaces to the clustergroup values file (`values-.yaml`). Include OperatorGroup configuration for operator namespaces. +3. **Define namespaces** — Add all required namespaces to the clustergroup values file (`variants//values-.yaml`). Include OperatorGroup configuration for operator namespaces. 4. **Define subscriptions** — Add operator subscriptions with at minimum the operator `name`. Set `namespace`, `channel`, and `source` as needed. 5. **Define applications** — Wire in helm charts as applications: - Local charts: set `path` to the chart location in the repository @@ -234,7 +234,7 @@ Define a custom global variable and use `sharedValueFiles`: global: device: gpu -# values-.yaml +# variants//values-.yaml clusterGroup: sharedValueFiles: - /overrides/values-{{ $.Values.global.device }}.yaml diff --git a/.claude/skills/pattern-author/reference.md b/.claude/skills/pattern-author/reference.md index 1a24ddaf5..10b6c6886 100644 --- a/.claude/skills/pattern-author/reference.md +++ b/.claude/skills/pattern-author/reference.md @@ -79,7 +79,7 @@ global: secretLoader: disabled: false main: - clusterGroupName: prod + variant: prod multiSourceConfig: enabled: true clusterGroupChartVersion: 0.9.* @@ -93,7 +93,7 @@ main: `global.secretLoader.disabled` — When `true`, skip loading secrets. Useful for Patterns that don't use the secrets framework. -`main.clusterGroupName` — The name of the primary clustergroup. Determines which `values-.yaml` file defines the main cluster. If you change this value, the next `pattern init` run creates the corresponding values file (you would need to manually delete the old unused one). +`main.variant` — The name of the primary clustergroup. Determines which `values-.yaml` file defines the main cluster, located at `variants//values-.yaml` (see [Variants folder structure](#variants-folder-structure)). If you change this value, the next `pattern init` run creates the corresponding values file (you would need to manually delete the old unused one). This replaces the deprecated `main.clusterGroupName` field (deprecated by the Validated Patterns operator in version 0.0.78); patternizer automatically migrates `main.clusterGroupName` to `main.variant` on `init`. `main.multiSourceConfig.enabled` — Enable ArgoCD multi-source applications. Should always be `true` for modern Patterns. @@ -111,6 +111,27 @@ There are also some global options that can be set in `values-global.yaml` but a `values-prod.yaml` is the values file that defines the main clustergroup of the Pattern. In hub/spoke Patterns (see [Spoke clusters](#spoke-clusters)) this would be the hub cluster. In single cluster Patterns this defines the solitary cluster. If there is one file to look at which defines the Pattern, it is this one. Its contents are explored in [The clustergroup values](#the-clustergroup-values) section. +With the [variants folder structure](#variants-folder-structure) (the layout patternizer now generates), this file lives at `variants//values-.yaml` (e.g. `variants/prod/values-prod.yaml`) rather than at the repository root. + +### Variants folder structure + +Per-cluster (per-variant) values files live under a top-level `variants/` directory, with one subfolder per variant named after the variant: + +```console +. +├── values-global.yaml +├── charts/ +└── variants/ + ├── prod/ + │ └── values-prod.yaml + └── standalone/ + └── values-standalone.yaml +``` + +The Validated Patterns operator auto-detects a top-level `variants/` directory and, when `main.variant` is set, looks for per-cluster values files under `variants//` instead of the repository root. This keeps the root uncluttered as you add more variants. This structure was introduced in operator version 0.0.78; see the [variants folder structure blog post](https://validatedpatterns.io/blog/2026-08-04-variants-folder-structure/) for details. + +`patternizer init` creates and uses this structure automatically, migrating a legacy root-level `values-.yaml` into `variants//` when present. The flat (root-level) layout is still supported by the operator; `patternizer upgrade` does not move files but prints a deprecation warning when the variant's file is missing from the variants folder. + ### The Pattern CR When you run `./pattern.sh make install` to deploy the Pattern, the values in `values-global.yaml` are passed to the [pattern-install chart](https://github.com/validatedpatterns/pattern-install-chart) by the [rhvp.cluster_utils.install](https://github.com/validatedpatterns/rhvp.cluster_utils/blob/main/playbooks/install.yml) playbook. This chart creates a subscription for the Validated Patterns Operator, a configmap with default operator configuration, and the Pattern CR: @@ -152,6 +173,8 @@ The following values files are automatically included for each application, in t ArgoCD is configured with `ignoreMissingValueFiles: true`, so it silently skips any of these files that do not exist. Only create the files you actually need. +When the [variants folder structure](#variants-folder-structure) is in use, the variant-specific files above (those named after the clustergroup/variant, such as `values-.yaml`) are resolved from `variants//` instead of the repository root. + In multi-source mode (the default), values files from the Pattern repository are prefixed with `$patternref/` to tell ArgoCD which source they come from. This is handled automatically by the clustergroup chart. Some of these values files are cross-clustergroup. For example, `values-AWS.yaml` would apply to both hub and spoke clustergroups running on AWS. @@ -184,7 +207,7 @@ global: pattern: ai-quickstart-rag device: cpu # one of 'cpu' (no GPU) or 'gpu' (NVIDIA GPU) main: - clusterGroupName: prod + variant: prod multiSourceConfig: enabled: true clusterGroupChartVersion: 0.9.* diff --git a/.cursor/skills/pattern-author/SKILL.md b/.cursor/skills/pattern-author/SKILL.md index 281eef3b4..0634687a8 100644 --- a/.cursor/skills/pattern-author/SKILL.md +++ b/.cursor/skills/pattern-author/SKILL.md @@ -26,7 +26,7 @@ When creating a new Pattern: 1. **Initialize** — Determine the Pattern name and whether secrets are needed. Run `pattern init` or `pattern init --with-secrets` in the Pattern directory. 2. **Identify requirements** — What operators, applications, and custom helm charts does this Pattern need? -3. **Define namespaces** — Add all required namespaces to the clustergroup values file (`values-.yaml`). Include OperatorGroup configuration for operator namespaces. +3. **Define namespaces** — Add all required namespaces to the clustergroup values file (`variants//values-.yaml`). Include OperatorGroup configuration for operator namespaces. 4. **Define subscriptions** — Add operator subscriptions with at minimum the operator `name`. Set `namespace`, `channel`, and `source` as needed. 5. **Define applications** — Wire in helm charts as applications: - Local charts: set `path` to the chart location in the repository @@ -234,7 +234,7 @@ Define a custom global variable and use `sharedValueFiles`: global: device: gpu -# values-.yaml +# variants//values-.yaml clusterGroup: sharedValueFiles: - /overrides/values-{{ $.Values.global.device }}.yaml diff --git a/.cursor/skills/pattern-author/reference.md b/.cursor/skills/pattern-author/reference.md index 1a24ddaf5..10b6c6886 100644 --- a/.cursor/skills/pattern-author/reference.md +++ b/.cursor/skills/pattern-author/reference.md @@ -79,7 +79,7 @@ global: secretLoader: disabled: false main: - clusterGroupName: prod + variant: prod multiSourceConfig: enabled: true clusterGroupChartVersion: 0.9.* @@ -93,7 +93,7 @@ main: `global.secretLoader.disabled` — When `true`, skip loading secrets. Useful for Patterns that don't use the secrets framework. -`main.clusterGroupName` — The name of the primary clustergroup. Determines which `values-.yaml` file defines the main cluster. If you change this value, the next `pattern init` run creates the corresponding values file (you would need to manually delete the old unused one). +`main.variant` — The name of the primary clustergroup. Determines which `values-.yaml` file defines the main cluster, located at `variants//values-.yaml` (see [Variants folder structure](#variants-folder-structure)). If you change this value, the next `pattern init` run creates the corresponding values file (you would need to manually delete the old unused one). This replaces the deprecated `main.clusterGroupName` field (deprecated by the Validated Patterns operator in version 0.0.78); patternizer automatically migrates `main.clusterGroupName` to `main.variant` on `init`. `main.multiSourceConfig.enabled` — Enable ArgoCD multi-source applications. Should always be `true` for modern Patterns. @@ -111,6 +111,27 @@ There are also some global options that can be set in `values-global.yaml` but a `values-prod.yaml` is the values file that defines the main clustergroup of the Pattern. In hub/spoke Patterns (see [Spoke clusters](#spoke-clusters)) this would be the hub cluster. In single cluster Patterns this defines the solitary cluster. If there is one file to look at which defines the Pattern, it is this one. Its contents are explored in [The clustergroup values](#the-clustergroup-values) section. +With the [variants folder structure](#variants-folder-structure) (the layout patternizer now generates), this file lives at `variants//values-.yaml` (e.g. `variants/prod/values-prod.yaml`) rather than at the repository root. + +### Variants folder structure + +Per-cluster (per-variant) values files live under a top-level `variants/` directory, with one subfolder per variant named after the variant: + +```console +. +├── values-global.yaml +├── charts/ +└── variants/ + ├── prod/ + │ └── values-prod.yaml + └── standalone/ + └── values-standalone.yaml +``` + +The Validated Patterns operator auto-detects a top-level `variants/` directory and, when `main.variant` is set, looks for per-cluster values files under `variants//` instead of the repository root. This keeps the root uncluttered as you add more variants. This structure was introduced in operator version 0.0.78; see the [variants folder structure blog post](https://validatedpatterns.io/blog/2026-08-04-variants-folder-structure/) for details. + +`patternizer init` creates and uses this structure automatically, migrating a legacy root-level `values-.yaml` into `variants//` when present. The flat (root-level) layout is still supported by the operator; `patternizer upgrade` does not move files but prints a deprecation warning when the variant's file is missing from the variants folder. + ### The Pattern CR When you run `./pattern.sh make install` to deploy the Pattern, the values in `values-global.yaml` are passed to the [pattern-install chart](https://github.com/validatedpatterns/pattern-install-chart) by the [rhvp.cluster_utils.install](https://github.com/validatedpatterns/rhvp.cluster_utils/blob/main/playbooks/install.yml) playbook. This chart creates a subscription for the Validated Patterns Operator, a configmap with default operator configuration, and the Pattern CR: @@ -152,6 +173,8 @@ The following values files are automatically included for each application, in t ArgoCD is configured with `ignoreMissingValueFiles: true`, so it silently skips any of these files that do not exist. Only create the files you actually need. +When the [variants folder structure](#variants-folder-structure) is in use, the variant-specific files above (those named after the clustergroup/variant, such as `values-.yaml`) are resolved from `variants//` instead of the repository root. + In multi-source mode (the default), values files from the Pattern repository are prefixed with `$patternref/` to tell ArgoCD which source they come from. This is handled automatically by the clustergroup chart. Some of these values files are cross-clustergroup. For example, `values-AWS.yaml` would apply to both hub and spoke clustergroups running on AWS. @@ -184,7 +207,7 @@ global: pattern: ai-quickstart-rag device: cpu # one of 'cpu' (no GPU) or 'gpu' (NVIDIA GPU) main: - clusterGroupName: prod + variant: prod multiSourceConfig: enabled: true clusterGroupChartVersion: 0.9.* diff --git a/.github/workflows/ansible-lint.yml b/.github/workflows/ansible-lint.yml index ca45d9fb5..ac6509177 100644 --- a/.github/workflows/ansible-lint.yml +++ b/.github/workflows/ansible-lint.yml @@ -15,4 +15,4 @@ jobs: persist-credentials: false - name: Lint Ansible Playbook - uses: ansible/ansible-lint@262624cd0ab22a4221293216856c59671ce7aa5e + uses: ansible/ansible-lint@fac2489f3e4ccffa83a40b839770abbf4ab52327 diff --git a/.github/workflows/superlinter.yml b/.github/workflows/superlinter.yml index ac8b63ebf..ec50089a7 100644 --- a/.github/workflows/superlinter.yml +++ b/.github/workflows/superlinter.yml @@ -25,7 +25,7 @@ jobs: # Run Linter against code base # ################################ - name: Lint Code Base - uses: super-linter/super-linter/slim@4ce20838b8ab83717e78138c5b3a1407148e0918 # v8.7.0 + uses: super-linter/super-linter/slim@2da136927bd4a73596db63044b504547c62cb854 # v9.0.0 env: VALIDATE_ALL_CODEBASE: true DEFAULT_BRANCH: main @@ -37,6 +37,7 @@ jobs: VALIDATE_JSCPD: false VALIDATE_JSON_PRETTIER: false VALIDATE_MARKDOWN_PRETTIER: false + VALIDATE_PYTHON_ISORT: false VALIDATE_PYTHON_PYLINT: false VALIDATE_SHELL_SHFMT: false VALIDATE_YAML: false diff --git a/.github/workflows/sync-rhdp-branch.yml b/.github/workflows/sync-rhdp-branch.yml index e93a041ba..1cd135b0c 100644 --- a/.github/workflows/sync-rhdp-branch.yml +++ b/.github/workflows/sync-rhdp-branch.yml @@ -22,7 +22,7 @@ jobs: with: persist-credentials: false - name: Set up Node - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 + uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0 with: node-version: 20 - name: Opening pull request diff --git a/Makefile-common b/Makefile-common index 36cae531c..d68f1f0fd 100644 --- a/Makefile-common +++ b/Makefile-common @@ -18,7 +18,8 @@ operator-deploy operator-upgrade: ## Installs/updates the pattern on a cluster ( @$(ANSIBLE_RUN) rhvp.cluster_utils.operator_deploy .PHONY: install -install: pattern-install ## Installs the pattern onto a cluster (Loads secrets as well if configured) +install: ## Installs the pattern onto a cluster (Loads secrets as well if configured) + @$(ANSIBLE_RUN) rhvp.cluster_utils.install .PHONY: uninstall uninstall: ## (EXPERIMENTAL) See https://validatedpatterns.io/blog/2026-02-16-pattern-uninstall/. @@ -32,6 +33,11 @@ pattern-install: load-secrets: ## Loads secrets onto the cluster (unless explicitly disabled in values-global.yaml) @$(ANSIBLE_RUN) rhvp.cluster_utils.load_secrets +##@ Debug Tasks +.PHONY: display-secrets-info +display-secrets-info: ## Display your secret material on terminal or show secret loading error. Use with caution! + @$(ANSIBLE_RUN) rhvp.cluster_utils.display_secrets_info + ##@ Validation Tasks .PHONY: validate-prereq validate-prereq: ## verify pre-requisites @@ -53,6 +59,7 @@ validate-schema: ## validates values files against schema in common/clustergroup argo-healthcheck: ## Checks if all argo applications are synced @$(ANSIBLE_RUN) rhvp.cluster_utils.argo_healthcheck +##@ Testing (CI) Tasks .PHONY: run-ci-tests run-ci-tests: ## To run ci-tests, set any needed env vars @$(ANSIBLE_RUN) rhvp.cluster_utils.run_ci_tests diff --git a/pattern-metadata.yaml b/pattern-metadata.yaml index 06fbcfcfe..437c77919 100644 --- a/pattern-metadata.yaml +++ b/pattern-metadata.yaml @@ -23,7 +23,7 @@ requirements: type: n1-standard-8 azure: replicas: 3 - type: Standard_D8s_v3 + type: Standard_D8s_v4 aws: replicas: 3 type: m5.2xlarge @@ -34,7 +34,7 @@ requirements: type: n1-standard-4 azure: replicas: 3 - type: Standard_D4s_v3 + type: Standard_D4s_v4 aws: replicas: 3 type: m5.xlarge diff --git a/pattern.sh b/pattern.sh index b2172fd4f..7d8937c54 100755 --- a/pattern.sh +++ b/pattern.sh @@ -17,6 +17,46 @@ function is_container() { return 1 } +function verify_image() { + local image="$1" + + case "${image}" in + quay.io/validatedpatterns/*|quay.io/hybridcloudpatterns/*) + ;; + *) + echo "Skipping image verification for third-party registry" + return 0 + ;; + esac + + if ! command -v cosign >/dev/null 2>&1; then + echo "WARNING: cosign is not installed, cannot verify image signature" + echo "Install cosign to enable image verification: https://docs.sigstore.dev/cosign/system_config/installation/" + return 0 + fi + + echo "Verifying image signature for ${image}..." + local output rc + local oidc_issuer="${VP_COSIGN_OIDC_ISSUER:-https://token.actions.githubusercontent.com}" + local cert_identity="${VP_COSIGN_CERT_IDENTITY:-https://github.com/validatedpatterns/utility-container/.*}" + output=$(cosign verify \ + --certificate-oidc-issuer "${oidc_issuer}" \ + --certificate-identity-regexp "${cert_identity}" \ + "${image}" 2>&1) && rc=$? || rc=$? + + if [ "${rc}" -eq 0 ]; then + echo "Image signature verified successfully" + elif [ "${rc}" -ge 10 ] && [ "${rc}" -le 13 ]; then + echo "ERROR: Image signature verification failed for ${image} (exit code ${rc})" + echo "${output}" + echo "Set VP_VERIFY_IMAGE=false to skip this check" + exit 1 + else + echo "WARNING: Could not verify image signature for ${image} (likely a network issue)" + echo "Set VP_VERIFY_IMAGE=false to skip this check" + fi +} + if is_container; then echo "Already running in a container" exec "$@" @@ -99,6 +139,10 @@ if [ -n "${EXTRA_ARGS:-}" ]; then EXTRA_ARGS_ARRAY=(${EXTRA_ARGS}) fi +if [ "${VP_VERIFY_IMAGE:-true}" != "false" ]; then + verify_image "$PATTERN_UTILITY_CONTAINER" +fi + # Copy Kubeconfig from current environment. The utilities will pick up ~/.kube/config if set so it's not mandatory # $HOME is mounted as itself for any files that are referenced with absolute paths # $HOME is mounted to /root because the UID in the container is 0 and that's where SSH looks for credentials @@ -129,12 +173,12 @@ podman run -it --rm --pull=newer \ -e UUID_FILE \ -e VALUES_SECRET \ -e 'VP_*' \ - "${PKI_HOST_MOUNT_ARGS[@]}" \ + ${PKI_HOST_MOUNT_ARGS[@]+"${PKI_HOST_MOUNT_ARGS[@]}"} \ -v "$(pwd -P)":"$(pwd -P)" \ -v "${HOME}":"${HOME}" \ -v "${HOME}":/pattern-home \ "${PODMAN_ARGS[@]}" \ - "${EXTRA_ARGS_ARRAY[@]}" \ + ${EXTRA_ARGS_ARRAY[@]+"${EXTRA_ARGS_ARRAY[@]}"} \ -w "$(pwd -P)" \ "$PATTERN_UTILITY_CONTAINER" \ "$@" diff --git a/tests/ci/conftest.py b/tests/ci/conftest.py index 12cc3c06c..5af1bde1b 100644 --- a/tests/ci/conftest.py +++ b/tests/ci/conftest.py @@ -1 +1 @@ -from validatedpatterns_tests.interop.conftest_openshift import * # noqa: F401,F403 +from validatedpatterns_tests.interop.conftest_openshift import * # noqa: F401,F403,RUF100 diff --git a/tests/interop/conftest.py b/tests/interop/conftest.py index fb301d579..7192e6c6d 100644 --- a/tests/interop/conftest.py +++ b/tests/interop/conftest.py @@ -1,2 +1,2 @@ -from validatedpatterns_tests.interop.conftest_logger import * # noqa: F401, F403 -from validatedpatterns_tests.interop.conftest_openshift import * # noqa: F401, F403 +from validatedpatterns_tests.interop.conftest_logger import * # noqa: F401, F403, RUF100 +from validatedpatterns_tests.interop.conftest_openshift import * # noqa: F401, F403, RUF100 diff --git a/tests/interop/create_ci_badge.py b/tests/interop/create_ci_badge.py index 717b47d1e..8e0ec789f 100644 --- a/tests/interop/create_ci_badge.py +++ b/tests/interop/create_ci_badge.py @@ -2,7 +2,7 @@ import os import subprocess import sys -from datetime import datetime +from datetime import UTC, datetime from junitparser import JUnitXml @@ -20,7 +20,7 @@ "patternName": os.environ.get("PATTERN_NAME"), "patternRepo": "", "patternBranch": "", - "date": datetime.today().strftime("%Y-%m-%d"), + "date": datetime.now(UTC).strftime("%Y-%m-%d"), "testSource": "Community", "debugInfo": None, } @@ -58,10 +58,16 @@ def get_openshift_version(): ci_badge["openshiftVersion"] = versions[0] pattern_repo = subprocess.run( - ["git", "config", "--get", "remote.origin.url"], capture_output=True, text=True + ["git", "config", "--get", "remote.origin.url"], + capture_output=True, + text=True, + check=False, ) pattern_branch = subprocess.run( - ["git", "branch", "--show-current"], capture_output=True, text=True + ["git", "branch", "--show-current"], + capture_output=True, + text=True, + check=False, ) ci_badge["patternRepo"] = pattern_repo.stdout.strip() diff --git a/tests/interop/test_modify_web_content.py b/tests/interop/test_modify_web_content.py index 3250c02ce..25ef0d30d 100644 --- a/tests/interop/test_modify_web_content.py +++ b/tests/interop/test_modify_web_content.py @@ -70,7 +70,11 @@ def test_modify_web_content(openshift_dyn_client): patterns_repo = PATTERNS_REPO_PATH if os.getenv("EXTERNAL_TEST") != "true": git_add = subprocess.run( - ["git", "add", chart], cwd=patterns_repo, capture_output=True, text=True + ["git", "add", chart], + cwd=patterns_repo, + capture_output=True, + text=True, + check=False, ) if git_add.returncode != 0: logger.error(f"git add failed: {git_add.stderr}") @@ -80,15 +84,25 @@ def test_modify_web_content(openshift_dyn_client): cwd=patterns_repo, capture_output=True, text=True, + check=False, ) if git_commit.returncode != 0: logger.warning(f"git commit returned non-zero: {git_commit.stderr}") push = subprocess.run( - ["git", "push"], cwd=patterns_repo, capture_output=True, text=True + ["git", "push"], + cwd=patterns_repo, + capture_output=True, + text=True, + check=False, ) else: - git_add = subprocess.run(["git", "add", chart], capture_output=True, text=True) + git_add = subprocess.run( + ["git", "add", chart], + capture_output=True, + text=True, + check=False, + ) if git_add.returncode != 0: logger.error(f"git add failed: {git_add.stderr}") @@ -96,11 +110,14 @@ def test_modify_web_content(openshift_dyn_client): ["git", "commit", "-m", "Updating 'hello-world'"], capture_output=True, text=True, + check=False, ) if git_commit.returncode != 0: logger.warning(f"git commit returned non-zero: {git_commit.stderr}") - push = subprocess.run(["git", "push"], capture_output=True, text=True) + push = subprocess.run( + ["git", "push"], capture_output=True, text=True, check=False + ) if push.returncode != 0: logger.error(f"git push failed with return code {push.returncode}")