diff --git a/docs/user/reference/config/components.md b/docs/user/reference/config/components.md index ce06e6f60..6966adde7 100644 --- a/docs/user/reference/config/components.md +++ b/docs/user/reference/config/components.md @@ -11,7 +11,7 @@ A component definition tells azldev where to find the spec file, how to customiz | Spec source | `spec` | [SpecSource](#spec-source) | No | Where to find the spec file for this component. Inherited from distro defaults if not specified. | | Release config | `release` | [ReleaseConfig](#release-configuration) | No | Controls how the Release tag is managed during rendering | | Overlays | `overlays` | array of [Overlay](overlays.md) | No | Modifications to apply to the spec and/or source files | -| Overlay files | `overlay-files` | array of string | No | Path or glob patterns that load per-file overlay documents after component config resolution. Inherited relative patterns are resolved from the concrete component's config file, or from the matched spec file's directory for spec-discovered components. Use `[]` to disable inherited patterns. See [Per-file overlay format](overlays.md#per-file-overlay-format). | +| Overlay files | `overlay-files` | array of string | No | Path or glob patterns that load per-file overlay documents after component config resolution. Allowed in project defaults and component settings, but not distro-version or component-group defaults (including `[]`). Inherited relative patterns are resolved from the concrete component's config file, or from the matched spec file's directory for spec-discovered components. Use `[]` on a component to disable inherited patterns. See [Per-file overlay format](overlays.md#per-file-overlay-format). | | Build config | `build` | [BuildConfig](#build-configuration) | No | Build-time options (macros, conditionals, check config) | | Render config | `render` | [RenderConfig](#render-configuration) | No | Options controlling spec rendering behavior | | Source files | `source-files` | array of [SourceFileReference](#source-file-references) | No | Additional source files to download for this component | diff --git a/docs/user/reference/config/overlays.md b/docs/user/reference/config/overlays.md index f252e9d91..e1355e11b 100644 --- a/docs/user/reference/config/overlays.md +++ b/docs/user/reference/config/overlays.md @@ -241,7 +241,9 @@ When a single logical change (a CVE backport, a feature disablement, a Fedora ch Set `overlay-files` on the component to one or more globs (relative to the component config) and drop one overlay file per logical change into a directory of your choosing. The conventional layout uses a sibling `overlays/` directory and a `*.overlay.toml` filename suffix, but neither is required — `overlay-files` is just a glob, so any layout you can describe with `**`/`*` patterns works. -`overlay-files` can also be inherited from `default-component-config` at the project, distro, or component-group level. Inherited relative patterns are still resolved for each concrete component: from its component config file when it has one, or from the matched spec file's directory when it is discovered by a component group's `specs` pattern. This makes defaults useful for component-local discovery patterns such as `overlay-files = ["overlays/*.overlay.toml"]`. If a component sets `overlay-files`, that value replaces the inherited list; use `overlay-files = []` to disable inherited overlay files for a component, or include both patterns explicitly when you want to keep default discovery and add component-specific locations. +`overlay-files` on `[components.X]` is a plain glob resolved relative to the component's config file. Plain legacy patterns such as `overlays/*.overlay.toml` and explicit empty lists remain supported in project-level defaults and component settings, with unchanged path resolution and precedence. Inherited plain patterns resolve relative to the concrete component config file or matched spec directory. Project-level defaults can use [shared patterns with `{component}`](#shared-patterns-with-component-in-overlay-files) to locate overlays for known components. A component that inherits such a pattern substitutes its own name for `{component}`; use `overlay-files = []` to disable the inherited list, or set your own list to replace it wholesale. + +`overlay-files` is only settable at project scope (the top-level `default-component-config`) or at component scope (`[components.X]`). Setting it inside a distro-version or component-group `default-component-config` is rejected at load time — even an empty list (`overlay-files = []`). ``` base/comps/mypackage/ @@ -293,6 +295,80 @@ lines = ["- Fix CVE-2024-1234"] Use `azldev component metadata` to list documentation metadata for one or more components. By default it lists metadata from both sources — component overlays and the component groups each component belongs to — tagging each row with its `source`, and shows only entries that carry a `[metadata]` block. Pass `--overlays` or `--groups` to restrict to one source. The command is read-only and supports `--category` and `--upstream-status` filters; output is available as a table (default) or JSON (`-O json`). +### Shared patterns with `{component}` in `overlay-files` + +For projects that keep each component's overlay files in a predictable per-component directory (e.g. `base/comps//overlays/*.overlay.toml`), declare components centrally and use a shared **project-level** `default-component-config.overlay-files` pattern containing `{component}`. This removes the need for declaration-only `.comp.toml` files without removing the declarations themselves. + +**Overlay patterns only apply to known components.** A component must have a `[components.]` declaration or be identified by another supported mechanism, such as a component-group `specs` glob. An entry that uses `{component}` **must** contain it exactly once, as a whole path segment, and **must** have at least one more path segment after it (the placeholder names a component directory, not a file). Additional constraints for placeholder entries enforced at load time: + +* Use `/` as the path separator (backslashes are rejected; Windows is not supported). +* No glob metacharacters (`*`, `?`, `[`, `{`) before `{component}` — the prefix must be literal. +* No additional `{` or `}` after `{component}`, including brace alternatives such as `{base,extra}`. Use separate list entries for multiple directories. Ordinary patterns without `{component}` retain their existing brace support. +* `..` path segments are allowed before `{component}`, but never after it. For example, `../comps/{component}/overlays/*.overlay.toml` is valid. Both `comps/{component}/../overlays/*.overlay.toml` and `comps/{component}/nested/../overlays/*.overlay.toml` are rejected. Write the latter as `comps/{component}/overlays/*.overlay.toml` instead. +* The rest of the entry (after `{component}/`) must be a valid `doublestar` glob (validated with `doublestar.ValidatePattern`). + +For example, a resolved component named `curl` substitutes its name into `comps/{component}/overlays/*.overlay.toml` and loads `comps/curl/overlays/fix.overlay.toml`. A matching directory for an undeclared component does not create a new component. + +Each component inherits the project-level `overlay-files` list, including any shared patterns and `{component}` patterns, unless it sets its own: a component-specific list replaces the project list, and `[]` disables inherited overlay files. + +An inherited `{component}` entry that matches no files is skipped, even if it ends in a literal filename such as `{component}/overlays.toml`. An ordinary literal path without `{component}` must still exist. + +```toml +# base/comps/components.toml (included at project scope) +[default-component-config] +overlay-files = ["{component}/overlays/*.overlay.toml"] + +[components.anaconda] + +[components.curl] +``` + +The pattern is relative to the file that declares it. If declared in the root +`azldev.toml` instead, use `base/comps/{component}/overlays/*.overlay.toml`. +With the layout below, the central declarations load their respective overlay files. +Do not create an empty `anaconda.comp.toml`; keep `[components.anaconda]` in the +central file instead. + +``` +base/comps/ +├── components.toml +├── anaconda/ +│ └── overlays/ +│ ├── 0001-cve-2024-1234.overlay.toml +│ └── 0002-disable-fips-tests.overlay.toml +└── curl/ + └── overlays/ + └── 0001-cve-2024-5678.overlay.toml +``` + +**Where `{component}` is allowed.** Only in **project-level** `default-component-config`, where plain and placeholder entries may coexist. Component settings accept plain patterns and `[]`, but reject `{component}`. Distro-version and component-group defaults must omit `overlay-files` entirely, including empty lists. A component-specific list replaces the project list rather than merging with it; an empty component list disables inherited overlay files. + +**Inheritance with `{component}`.** A component that does not override `overlay-files` inherits the project entry and expands `{component}` with its own name at glob time. This also works for components identified by component-group spec globs. + +A component that sets its own `overlay-files = [...]` replaces the inherited value; the placeholder pattern no longer applies to that component. Setting `overlay-files = []` disables inheritance entirely for that component. + +Multiple patterns may select overlays for the same component. Files are loaded in pattern order, with duplicate file matches loaded only once. + +**Explicit component-group members need a `[components.]` table.** A component group can enumerate its members two ways: via a `specs` glob (spec files on disk) or via an explicit `components = ["", ...]` list. Every name in the *explicit list* must have a matching `[components.]` table in the project (validated at load time, error `ErrUndefinedComponent`). For example, retain `anaconda` centrally rather than adding an empty dedicated file: + +```toml +# base/comps/components.toml (included at project scope) +[default-component-config] +overlay-files = ["{component}/overlays/*.overlay.toml"] + +[components.anaconda] + +[component-groups.installer] +components = ["anaconda"] +``` + +Once the table exists, the component inherits the project-level `{component}` overlay pattern normally. Group members matched via `specs` (not the explicit list) do **not** need a `.comp.toml` — the resolver synthesizes an empty config for them and inheritance still applies. + +Keep a dedicated `.comp.toml` when it contains actual component-specific settings, +such as upstream pins, build options, inline overlays, or local-spec configuration. +Components with or without matching overlay files need an explicit declaration or +another supported mechanism, such as a group's `specs` glob. + ## Examples ### Adding a Build Dependency diff --git a/internal/app/azldev/agentskill/agentskill_test.go b/internal/app/azldev/agentskill/agentskill_test.go index e53c985a8..a943bd060 100644 --- a/internal/app/azldev/agentskill/agentskill_test.go +++ b/internal/app/azldev/agentskill/agentskill_test.go @@ -480,6 +480,48 @@ func TestFilesGitHubLayout(t *testing.T) { assert.Contains(t, mockFile.Content, "name: azldev-mock") } +func TestComponentSkillsExplainCentralDeclarationsAndPlaceholderOverlays(t *testing.T) { + for _, mode := range []struct { + name string + withoutLockfile bool + }{ + {name: "default"}, + {name: "without-lockfile", withoutLockfile: true}, + } { + t.Run(mode.name, func(t *testing.T) { + catalog := agentskill.NewCatalog(mode.withoutLockfile) + + for _, name := range []string{"azldev-add-component", "azldev-comp-toml", "azldev-overlays"} { + t.Run(name, func(t *testing.T) { + doc, err := catalog.SkillDocument(name, testParams()) + require.NoError(t, err) + + doc = strings.Join(strings.Fields(doc), " ") + assert.Contains(t, doc, "name: "+name) + assert.Contains(t, doc, "base/comps/{component}/overlays/*.overlay.toml") + assert.Contains(t, doc, "do not create an empty") + assert.Contains(t, doc, "base/comps/components.toml") + assert.Contains(t, doc, "without matching overlay files") + assert.Contains(t, doc, "upstream pins, build options, inline overlays,") + assert.Contains(t, doc, "do not discover undeclared components") + assert.NotContains(t, doc, "Declaration-free") + assert.NotContains(t, doc, "Discovery is opt-in.") + assert.Contains(t, doc, "unchanged path resolution and precedence") + assert.Contains(t, doc, + "Distro-version and component-group defaults must omit `overlay-files`, including empty lists.") + assert.NotContains(t, doc, "Every component lives under") + assert.NotContains(t, doc, "anything that needs overlays") + + if name == "azldev-overlays" { + assert.Contains(t, doc, "Additional braces are forbidden after `{component}`") + assert.Contains(t, doc, "use separate list entries for multiple directories") + } + }) + } + }) + } +} + // schemaEnum extracts values from an authoritative jsonschema enum tag. func schemaEnum(t *testing.T, structType reflect.Type, fieldName string) []string { t.Helper() diff --git a/internal/app/azldev/agentskill/content/add-component.md.tmpl b/internal/app/azldev/agentskill/content/add-component.md.tmpl index e21d45798..ae50328b7 100644 --- a/internal/app/azldev/agentskill/content/add-component.md.tmpl +++ b/internal/app/azldev/agentskill/content/add-component.md.tmpl @@ -39,7 +39,27 @@ dist-git often fail bot detection): 4. Read the spec and plan any overlays. -## Inline vs dedicated file +## Central or dedicated declarations + +For components whose only customization is per-file overlays, add files under +`base/comps//overlays/`; do not create an empty `.comp.toml`. This requires +a shared project-level `default-component-config.overlay-files` pattern such as +`base/comps/{component}/overlays/*.overlay.toml` in the root config. The `{component}` +pattern substitutes each resolved component's name to load its overlays. +Retain `[components.]` in `base/comps/components.toml`. +Overlay files do not discover undeclared components. + +Plain legacy patterns such as `overlays/*.overlay.toml` and +`[]` remain supported in project defaults and component settings, with unchanged +path resolution and precedence; they do not discover components. Distro-version +and component-group defaults must omit `overlay-files`, including empty lists. +Only project-level defaults may use `{component}`. + +If a component is explicitly listed in a component group, retain `[components.]` +in `base/comps/components.toml` for membership validation. Components with or +without matching overlay files still need an +explicit declaration or another supported discovery mechanism (such as a group's +`specs` glob). - **Inline** — a bare upstream import with no changes stays in a shared config file: @@ -47,9 +67,10 @@ dist-git often fail bot detection): [components.jq] ``` -- **Dedicated** — anything that needs overlays, build config, or a local spec gets its - own `/.comp.toml`. Rule of thumb: more than `[components.]` earns a - dedicated file. An `includes = ["**/*.comp.toml"]` glob picks it up automatically. +- **Dedicated** — keep `/.comp.toml` when it contains actual + component-specific settings, such as upstream pins, build options, inline overlays, + or local-spec configuration. An `includes = ["**/*.comp.toml"]` glob picks it up + automatically. `azldev comp add [...]` adds bare `[components.]` entries that inherit the distro defaults; it writes to the **root** config file and does not scaffold spec or diff --git a/internal/app/azldev/agentskill/content/comp-toml.md.tmpl b/internal/app/azldev/agentskill/content/comp-toml.md.tmpl index 0def4bf59..43929d63c 100644 --- a/internal/app/azldev/agentskill/content/comp-toml.md.tmpl +++ b/internal/app/azldev/agentskill/content/comp-toml.md.tmpl @@ -6,7 +6,9 @@ description: {{ printf "%q" .Description }} # Component definition files (`*.comp.toml`) A component definition tells azldev where a package's spec comes from and how to -customize it for your distro. Every component lives under `[components.]`. +customize it for your distro. Explicit definitions live under `[components.]`; +components may also be identified by component-group spec globs, but overlay files +do not discover undeclared components. Get the authoritative, always-current field list from the schema: @@ -102,11 +104,28 @@ the rare spec whose `Source`/`Patch` filenames use macros the filter cannot expa ## File organization +Plain legacy patterns such as `overlays/*.overlay.toml` and +`[]` remain supported in project defaults and component settings, with unchanged +path resolution and precedence; they do not discover components. Distro-version +and component-group defaults must omit `overlay-files`, including empty lists. +Only project-level defaults may use `{component}`. + +- **Central declarations** — for components whose only customization is per-file + overlays, add files under `base/comps//overlays/`; do not create an empty + `.comp.toml`. A shared project-level `default-component-config.overlay-files` + pattern such as `base/comps/{component}/overlays/*.overlay.toml` in the root config + substitutes the component's name to load its overlays. Retain + `[components.]` in `base/comps/components.toml`. +- **Explicit group members** — if a component is explicitly listed in a component + group's `components` list, retain `[components.]` in + `base/comps/components.toml`. Components with or without matching overlay files still need an explicit + declaration or another supported discovery mechanism (such as a group's `specs` + glob). - **Inline** — put simple, customization-free components directly in a shared config file (e.g. `[components.jq]`). -- **Dedicated** — give a component its own `/.comp.toml` once it needs - overlays, build config, or a local spec. Rule of thumb: anything more than - `[components.]` earns a dedicated file. +- **Dedicated** — keep `/.comp.toml` when it contains actual + component-specific settings, such as upstream pins, build options, inline overlays, + or local-spec configuration. - A parent config picks up dedicated files through an `includes` glob, for example `includes = ["**/*.comp.toml"]`. @@ -117,7 +136,7 @@ Start with `azldev comp list -p -q -O json`, then use For a change review, focus on the diff while checking enough surrounding context to ensure it fits the component and repository conventions. -- **Organization:** The component follows the repository's inline-versus-dedicated-file +- **Organization:** The component follows the repository's central or dedicated-file convention; its name matches upstream or sets `spec.upstream-name`; no stale or orphaned component files remain. - **Spec source:** The default upstream source is preferred. Pins explain why they are diff --git a/internal/app/azldev/agentskill/content/overlays.md.tmpl b/internal/app/azldev/agentskill/content/overlays.md.tmpl index 23b8b1121..65e6ba23d 100644 --- a/internal/app/azldev/agentskill/content/overlays.md.tmpl +++ b/internal/app/azldev/agentskill/content/overlays.md.tmpl @@ -13,12 +13,12 @@ overwritten. ## The inner loop -Overlays live in the component's TOML config — inline `[[components..overlays]]` +Overlays can live in the component's TOML config — inline `[[components..overlays]]` entries, or per-file overlay documents referenced by the component's `overlay-files` glob. They apply **in order** and are **non-atomic**: if one fails part-way, the overlays before it stay applied. -1. Add or edit the overlay in the component config. +1. Add or edit the inline overlay or per-file overlay document. 2. Re-render and inspect the result: ```sh @@ -33,6 +33,58 @@ overlays before it stay applied. Config errors reference the offending overlay by its `description`, so give every overlay a short, specific `description`. +## Shared patterns and central declarations + +For components whose only customization is per-file overlays, add files under +`base/comps//overlays/`; do not create an empty `.comp.toml`. Configure +the shared pattern at project scope (paths are relative to the declaring config): + +```toml +# Root config, e.g. azldev.toml +[default-component-config] +overlay-files = ["base/comps/{component}/overlays/*.overlay.toml"] + +# These declarations can instead live in included base/comps/components.toml. +[components.anaconda] +``` + +The `{component}` segment substitutes the resolved component's name. For example, +`anaconda` loads `base/comps/anaconda/overlays/0001-branding.overlay.toml`. +Overlay files do not discover undeclared components. A component with +no `overlay-files` of its own inherits the pattern; its own list replaces it, and +`overlay-files = []` disables inheritance. + +If a component is explicitly listed in a component group's `components` list, +retain `[components.]` in `base/comps/components.toml` for membership +validation. Components with or without matching +overlay files still need an explicit declaration or another supported discovery +mechanism (such as a group's `specs` glob). Keep a dedicated `.comp.toml` when it +contains actual component-specific settings, such as upstream pins, build options, +inline overlays, or local-spec configuration. + +Plain legacy patterns such as `overlays/*.overlay.toml` and +`[]` remain supported in project defaults and component settings, with unchanged +path resolution and precedence; they do not discover components. Distro-version +and component-group defaults must omit `overlay-files`, including empty lists. +Inherited plain globs resolve relative to the concrete component config file or +matched spec directory. A higher-priority list replaces a lower-priority one; +an empty list disables inherited overlay files. + +Only project-level defaults may use `{component}`. Placeholder entries require +it exactly once as a whole directory segment, with no glob metacharacters +(`*`, `?`, `[`, `{`) before it and a file pattern after it. Additional braces +are forbidden after `{component}`; use separate list entries for multiple directories. +Ordinary patterns without `{component}` retain their existing brace support. +`..` path segments are allowed before +`{component}`, but never after it. For example, +`../comps/{component}/overlays/*.overlay.toml` is valid; simplify +`comps/{component}/nested/../overlays/*.overlay.toml` to +`comps/{component}/overlays/*.overlay.toml`. Use `/` separators. Component settings +allow plain patterns and `[]` but reject the placeholder. +Components use the complete inherited project overlay list. +An inherited placeholder entry with no matching +files is optional, even with a literal suffix; an ordinary literal path must exist. + ## Diagnose common failures Start with `azldev comp diff-sources -p ` to see the exact overlay effect. diff --git a/internal/app/azldev/agentskill/content/withoutlockfile/add-component.md.tmpl b/internal/app/azldev/agentskill/content/withoutlockfile/add-component.md.tmpl index 27c94feb6..404c76e51 100644 --- a/internal/app/azldev/agentskill/content/withoutlockfile/add-component.md.tmpl +++ b/internal/app/azldev/agentskill/content/withoutlockfile/add-component.md.tmpl @@ -39,7 +39,27 @@ dist-git often fail bot detection): 4. Read the spec and plan any overlays. -## Inline vs dedicated file +## Central or dedicated declarations + +For components whose only customization is per-file overlays, add files under +`base/comps//overlays/`; do not create an empty `.comp.toml`. This requires +a shared project-level `default-component-config.overlay-files` pattern such as +`base/comps/{component}/overlays/*.overlay.toml` in the root config. The `{component}` +pattern substitutes each resolved component's name to load its overlays. +Retain `[components.]` in `base/comps/components.toml`. +Overlay files do not discover undeclared components. + +Plain legacy patterns such as `overlays/*.overlay.toml` and +`[]` remain supported in project defaults and component settings, with unchanged +path resolution and precedence; they do not discover components. Distro-version +and component-group defaults must omit `overlay-files`, including empty lists. +Only project-level defaults may use `{component}`. + +If a component is explicitly listed in a component group, retain `[components.]` +in `base/comps/components.toml` for membership validation. Components with or +without matching overlay files still need an +explicit declaration or another supported discovery mechanism (such as a group's +`specs` glob). - **Inline** — a bare upstream import with no changes stays in a shared config file: @@ -47,9 +67,10 @@ dist-git often fail bot detection): [components.jq] ``` -- **Dedicated** — anything that needs overlays, build config, or a local spec gets its - own `/.comp.toml`. Rule of thumb: more than `[components.]` earns a - dedicated file. An `includes = ["**/*.comp.toml"]` glob picks it up automatically. +- **Dedicated** — keep `/.comp.toml` when it contains actual + component-specific settings, such as upstream pins, build options, inline overlays, + or local-spec configuration. An `includes = ["**/*.comp.toml"]` glob picks it up + automatically. `azldev comp add [...]` adds bare `[components.]` entries that inherit the distro defaults; it writes to the **root** config file and does not scaffold spec or diff --git a/internal/app/azldev/agentskill/content/withoutlockfile/comp-toml.md.tmpl b/internal/app/azldev/agentskill/content/withoutlockfile/comp-toml.md.tmpl index 0e33a9e6d..91bd6536d 100644 --- a/internal/app/azldev/agentskill/content/withoutlockfile/comp-toml.md.tmpl +++ b/internal/app/azldev/agentskill/content/withoutlockfile/comp-toml.md.tmpl @@ -6,7 +6,9 @@ description: {{ printf "%q" .Description }} # Component definition files (`*.comp.toml`) A component definition tells azldev where a package's spec comes from and how to -customize it for your distro. Every component lives under `[components.]`. +customize it for your distro. Explicit definitions live under `[components.]`; +components may also be identified by component-group spec globs, but overlay files +do not discover undeclared components. Get the authoritative, always-current field list from the schema: @@ -92,11 +94,28 @@ the rare spec whose `Source`/`Patch` filenames use macros the filter cannot expa ## File organization +Plain legacy patterns such as `overlays/*.overlay.toml` and +`[]` remain supported in project defaults and component settings, with unchanged +path resolution and precedence; they do not discover components. Distro-version +and component-group defaults must omit `overlay-files`, including empty lists. +Only project-level defaults may use `{component}`. + +- **Central declarations** — for components whose only customization is per-file + overlays, add files under `base/comps//overlays/`; do not create an empty + `.comp.toml`. A shared project-level `default-component-config.overlay-files` + pattern such as `base/comps/{component}/overlays/*.overlay.toml` in the root config + substitutes the component's name to load its overlays. Retain + `[components.]` in `base/comps/components.toml`. +- **Explicit group members** — if a component is explicitly listed in a component + group's `components` list, retain `[components.]` in + `base/comps/components.toml`. Components with or without matching overlay files still need an explicit + declaration or another supported discovery mechanism (such as a group's `specs` + glob). - **Inline** — put simple, customization-free components directly in a shared config file (e.g. `[components.jq]`). -- **Dedicated** — give a component its own `/.comp.toml` once it needs - overlays, build config, or a local spec. Rule of thumb: anything more than - `[components.]` earns a dedicated file. +- **Dedicated** — keep `/.comp.toml` when it contains actual + component-specific settings, such as upstream pins, build options, inline overlays, + or local-spec configuration. - A parent config picks up dedicated files through an `includes` glob, for example `includes = ["**/*.comp.toml"]`. @@ -108,7 +127,7 @@ details, inspect the rendered spec under the diff while checking enough surrounding context to ensure it fits the component and repository conventions. -- **Organization:** The component follows the repository's inline-versus-dedicated-file +- **Organization:** The component follows the repository's central or dedicated-file convention; its name matches upstream or sets `spec.upstream-name`; no stale or orphaned component files remain. - **Spec source:** The default upstream source is preferred. Pins explain why they are diff --git a/internal/app/azldev/agentskill/content/withoutlockfile/overlays.md.tmpl b/internal/app/azldev/agentskill/content/withoutlockfile/overlays.md.tmpl index 2235eb559..e1ec9dd94 100644 --- a/internal/app/azldev/agentskill/content/withoutlockfile/overlays.md.tmpl +++ b/internal/app/azldev/agentskill/content/withoutlockfile/overlays.md.tmpl @@ -13,12 +13,12 @@ overwritten. ## The inner loop -Overlays live in the component's TOML config — inline `[[components..overlays]]` +Overlays can live in the component's TOML config — inline `[[components..overlays]]` entries, or per-file overlay documents referenced by the component's `overlay-files` glob. They apply **in order** and are **non-atomic**: if one fails part-way, the overlays before it stay applied. -1. Add or edit the overlay in the component config. +1. Add or edit the inline overlay or per-file overlay document. 2. Re-render and inspect the result: ```sh @@ -33,6 +33,58 @@ overlays before it stay applied. Config errors reference the offending overlay by its `description`, so give every overlay a short, specific `description`. +## Shared patterns and central declarations + +For components whose only customization is per-file overlays, add files under +`base/comps//overlays/`; do not create an empty `.comp.toml`. Configure +the shared pattern at project scope (paths are relative to the declaring config): + +```toml +# Root config, e.g. azldev.toml +[default-component-config] +overlay-files = ["base/comps/{component}/overlays/*.overlay.toml"] + +# These declarations can instead live in included base/comps/components.toml. +[components.anaconda] +``` + +The `{component}` segment substitutes the resolved component's name. For example, +`anaconda` loads `base/comps/anaconda/overlays/0001-branding.overlay.toml`. +Overlay files do not discover undeclared components. A component with +no `overlay-files` of its own inherits the pattern; its own list replaces it, and +`overlay-files = []` disables inheritance. + +If a component is explicitly listed in a component group's `components` list, +retain `[components.]` in `base/comps/components.toml` for membership +validation. Components with or without matching +overlay files still need an explicit declaration or another supported discovery +mechanism (such as a group's `specs` glob). Keep a dedicated `.comp.toml` when it +contains actual component-specific settings, such as upstream pins, build options, +inline overlays, or local-spec configuration. + +Plain legacy patterns such as `overlays/*.overlay.toml` and +`[]` remain supported in project defaults and component settings, with unchanged +path resolution and precedence; they do not discover components. Distro-version +and component-group defaults must omit `overlay-files`, including empty lists. +Inherited plain globs resolve relative to the concrete component config file or +matched spec directory. A higher-priority list replaces a lower-priority one; +an empty list disables inherited overlay files. + +Only project-level defaults may use `{component}`. Placeholder entries require +it exactly once as a whole directory segment, with no glob metacharacters +(`*`, `?`, `[`, `{`) before it and a file pattern after it. Additional braces +are forbidden after `{component}`; use separate list entries for multiple directories. +Ordinary patterns without `{component}` retain their existing brace support. +`..` path segments are allowed before +`{component}`, but never after it. For example, +`../comps/{component}/overlays/*.overlay.toml` is valid; simplify +`comps/{component}/nested/../overlays/*.overlay.toml` to +`comps/{component}/overlays/*.overlay.toml`. Use `/` separators. Component settings +allow plain patterns and `[]` but reject the placeholder. +Components use the complete inherited project overlay list. +An inherited placeholder entry with no matching +files is optional, even with a literal suffix; an ordinary literal path must exist. + ## Diagnose common failures Start with `azldev comp diff-sources -p ` to see the exact overlay effect. diff --git a/internal/app/azldev/core/components/resolver.go b/internal/app/azldev/core/components/resolver.go index fad536a40..bec9fb447 100644 --- a/internal/app/azldev/core/components/resolver.go +++ b/internal/app/azldev/core/components/resolver.go @@ -109,7 +109,9 @@ func (r *Resolver) FindComponents(filter *ComponentFilter) (components *Componen return components, r.validateLockFiles(components, false, skipValidation) } -// Finds *all* components defined in the environment. +// Finds *all* components defined in the environment. This unions two sources: +// - components reached via component-group spec-path patterns, +// - loose components declared in the project config. func (r *Resolver) FindAllComponents() (components *ComponentSet, err error) { components = NewComponentSet() diff --git a/internal/app/azldev/core/components/resolver_test.go b/internal/app/azldev/core/components/resolver_test.go index bf3b6280f..19aff1139 100644 --- a/internal/app/azldev/core/components/resolver_test.go +++ b/internal/app/azldev/core/components/resolver_test.go @@ -25,6 +25,27 @@ func touchFile(t *testing.T, testEnv *testutils.TestEnv, path string) { require.NoError(t, fileutils.WriteFile(testEnv.TestFS, path, []byte{}, fileperms.PrivateFile)) } +const patternTestOverlayContent = ` +[metadata] +category = "azl-branding-policy" +upstream-status = "inapplicable" + +[[overlays]] +description = "Bump release" +type = "spec-search-replace" +section = "%install" +regex = "unused" +` + +func writeOverlayFile(t *testing.T, env *testutils.TestEnv, path string) { + t.Helper() + + require.NoError(t, fileutils.MkdirAll(env.TestFS, filepath.Dir(path))) + require.NoError(t, fileutils.WriteFile( + env.TestFS, path, []byte(patternTestOverlayContent), fileperms.PrivateFile, + )) +} + // Creates a test spec and returns its expected config. func setupTestSpec(t *testing.T, testEnv *testutils.TestEnv, path string) projectconfig.ComponentConfig { t.Helper() @@ -724,10 +745,8 @@ value = "Microsoft" env.Config.ComponentGroups["specs"] = projectconfig.ComponentGroupConfig{ SpecPathPatterns: []string{"/specs/**/*.spec"}, - DefaultComponentConfig: projectconfig.ComponentConfig{ - OverlayFiles: []string{"overlays/*.overlay.toml"}, - }, } + env.Config.DefaultComponentConfig.OverlayFiles = []string{"overlays/*.overlay.toml"} filter := &components.ComponentFilter{ IncludeAllComponents: true, @@ -749,7 +768,7 @@ value = "Microsoft" assert.Equal(t, projectconfig.OverlayCategoryAZLBrandingPolicy, overlays[0].Metadata.Category) } -func TestFindComponents_SpecPathFilterInheritsOverlayFilesFromGroupDefaults(t *testing.T) { +func TestFindComponents_SpecPathFilterInheritsOverlayFilesFromProjectDefaults(t *testing.T) { env := testutils.NewTestEnv(t) touchFile(t, env, "/specs/ant/ant.spec") @@ -768,10 +787,8 @@ value = "Microsoft" env.Config.ComponentGroups["specs"] = projectconfig.ComponentGroupConfig{ SpecPathPatterns: []string{"/specs/**/*.spec"}, - DefaultComponentConfig: projectconfig.ComponentConfig{ - OverlayFiles: []string{"overlays/*.overlay.toml"}, - }, } + env.Config.DefaultComponentConfig.OverlayFiles = []string{"overlays/*.overlay.toml"} filter := &components.ComponentFilter{ SpecPaths: []string{"/specs/ant/ant.spec"}, @@ -793,6 +810,183 @@ value = "Microsoft" assert.Equal(t, projectconfig.OverlayCategoryAZLBrandingPolicy, overlays[0].Metadata.Category) } +func TestFindAllComponents_PlaceholderOnlyAppliesToKnownComponents(t *testing.T) { + for _, mode := range []struct { + name string + newEnv func(*testing.T) *testutils.TestEnv + }{ + {name: "locked", newEnv: testutils.NewTestEnv}, + {name: "without-lockfile", newEnv: testutils.NewTestEnvWithoutLockfile}, + } { + t.Run(mode.name, func(t *testing.T) { + env := mode.newEnv(t) + require.NoError(t, fileutils.WriteFile(env.TestFS, "/project/azldev.toml", + []byte("includes = ['base/comps/components.toml']\n"), fileperms.PrivateFile)) + require.NoError(t, fileutils.WriteFile(env.TestFS, "/project/base/comps/components.toml", + []byte("[default-component-config]\noverlay-files = ['{component}/overlays/*.overlay.toml']\n"+ + "[components.foo]\n[components.no-overlays]\n"), fileperms.PrivateFile)) + + _, loaded, err := projectconfig.LoadProjectConfig( + env.TestFS, env.TestOSEnv, "/project", true, "/tmp", nil, false, env.Env.WithoutLockfile()) + require.NoError(t, err) + + env.Config.Components = loaded.Components + env.Config.DefaultComponentConfig = loaded.DefaultComponentConfig + + writeOverlayFile(t, env, "/project/base/comps/foo/overlays/0001.overlay.toml") + writeOverlayFile(t, env, "/project/base/comps/foo/overlays/0002.overlay.toml") + // An undeclared directory must not add a component or cause its invalid file to be parsed. + require.NoError(t, fileutils.WriteFile(env.TestFS, + "/project/base/comps/bar/overlays/0001.overlay.toml", []byte("invalid TOML"), fileperms.PrivateFile)) + + resolver := components.NewResolver(env.Env) + all, err := resolver.FindAllComponents() + require.NoError(t, err) + assert.ElementsMatch(t, []string{"foo", "no-overlays"}, all.Names()) + + foo, ok := all.TryGet("foo") + require.True(t, ok) + assert.Len(t, foo.GetConfig().Overlays, 2) + + noOverlays, ok := all.TryGet("no-overlays") + require.True(t, ok) + assert.Empty(t, noOverlays.GetConfig().Overlays) + + _, err = resolver.GetComponentByName("bar") + require.ErrorContains(t, err, "component not found") + }) + } +} + +func TestFindAllComponents_PlaceholderCompleteInheritance(t *testing.T) { + env := testutils.NewTestEnv(t) + env.Config.Components["curl"] = projectconfig.ComponentConfig{Name: "curl"} + env.Config.DefaultComponentConfig.OverlayFiles = []string{ + "/project/shared/*.overlay.toml", + "/project/comps/{component}/overlays/*.overlay.toml", + } + + for path, value := range map[string]string{ + "/project/shared/shared.overlay.toml": "shared", + "/project/comps/curl/overlays/fix[1].overlay.toml": "brackets", + "/project/comps/curl/overlays/fix1.overlay.toml": "sibling", + } { + require.NoError(t, fileutils.WriteFile(env.TestFS, path, + []byte(patternTestOverlayContent+"\nreplacement = \""+value+"\"\n"), fileperms.PrivateFile)) + } + + all, err := components.NewResolver(env.Env).FindAllComponents() + require.NoError(t, err) + + found, ok := all.TryGet("curl") + require.True(t, ok) + require.Nil(t, found.GetConfig().SourceConfigFile) + + values := make([]string, 0, len(found.GetConfig().Overlays)) + for _, overlay := range found.GetConfig().Overlays { + values = append(values, overlay.Replacement) + } + + assert.Equal(t, []string{"shared", "sibling", "brackets"}, values) +} + +// Declared components with no 'overlay-files' inherit the project pattern; the +// placeholder is substituted with the component's own name at glob time. +func TestFindAllComponents_DeclaredComponentInheritsProjectPattern(t *testing.T) { + env := testutils.NewTestEnv(t) + + env.Config.DefaultComponentConfig.OverlayFiles = []string{ + "/project/comps/{component}/overlays/*.overlay.toml", + } + + writeOverlayFile(t, env, "/project/comps/foo/overlays/0001.overlay.toml") + writeOverlayFile(t, env, "/project/comps/foo/overlays/0002.overlay.toml") + + env.Config.Components["foo"] = projectconfig.ComponentConfig{Name: "foo"} + + all, err := components.NewResolver(env.Env).FindAllComponents() + require.NoError(t, err) + + foo, ok := all.TryGet("foo") + require.True(t, ok) + assert.Len(t, foo.GetConfig().Overlays, 2, + "foo should inherit the project pattern with {component}=foo") +} + +// A component that sets its own 'overlay-files' replaces the inherited project +// pattern wholesale; the placeholder pattern no longer applies. +func TestFindAllComponents_ComponentOverlayFilesReplacesInherited(t *testing.T) { + env := testutils.NewTestEnv(t) + + env.Config.DefaultComponentConfig.OverlayFiles = []string{ + "/project/comps/{component}/overlays/*.overlay.toml", + } + + // This file would match the inherited pattern (component=bar). + writeOverlayFile(t, env, "/project/comps/bar/overlays/0001.overlay.toml") + + // But this file lives elsewhere and is the only one bar's config points to. + writeOverlayFile(t, env, "/project/custom/bar/0001.overlay.toml") + + env.Config.Components["bar"] = projectconfig.ComponentConfig{ + Name: "bar", + OverlayFiles: []string{"/project/custom/bar/*.overlay.toml"}, + } + + all, err := components.NewResolver(env.Env).FindAllComponents() + require.NoError(t, err) + + bar, ok := all.TryGet("bar") + require.True(t, ok) + assert.Len(t, bar.GetConfig().Overlays, 1, + "bar overrides overlay-files wholesale; inherited pattern must not apply") +} + +// Empty per-component 'overlay-files' disables the inherited project pattern. +func TestFindAllComponents_ComponentOverlayFilesEmptyDisablesInheritance(t *testing.T) { + env := testutils.NewTestEnv(t) + + env.Config.DefaultComponentConfig.OverlayFiles = []string{ + "/project/comps/{component}/overlays/*.overlay.toml", + } + + writeOverlayFile(t, env, "/project/comps/baz/overlays/0001.overlay.toml") + + env.Config.Components["baz"] = projectconfig.ComponentConfig{ + Name: "baz", + OverlayFiles: []string{}, + } + + all, err := components.NewResolver(env.Env).FindAllComponents() + require.NoError(t, err) + + baz, ok := all.TryGet("baz") + require.True(t, ok) + assert.Empty(t, baz.GetConfig().Overlays, + "empty overlay-files must disable inherited pattern") +} + +func TestFindAllComponents_MultiplePlaceholderPatterns(t *testing.T) { + env := testutils.NewTestEnv(t) + env.Config.Components["dup"] = projectconfig.ComponentConfig{Name: "dup"} + + env.Config.DefaultComponentConfig.OverlayFiles = []string{ + "/project/a/{component}/*.overlay.toml", + "/project/b/{component}/*.overlay.toml", + "/project/a/{component}/*.overlay.toml", + } + + writeOverlayFile(t, env, "/project/a/dup/0001.overlay.toml") + writeOverlayFile(t, env, "/project/b/dup/0002.overlay.toml") + + all, err := components.NewResolver(env.Env).FindAllComponents() + require.NoError(t, err) + assert.Equal(t, []string{"dup"}, all.Names()) + dup, ok := all.TryGet("dup") + require.True(t, ok) + assert.Len(t, dup.GetConfig().Overlays, 2, "overlapping patterns load each file once") +} + // When a lock file exists for a component, the resolver should attach all of // its data (commit, import-commit, manual-bump, fingerprint) to the resolved // component via the Locked field — without touching the original Spec config. diff --git a/internal/app/azldev/core/sources/synthistory.go b/internal/app/azldev/core/sources/synthistory.go index 0c957c81d..4f23960b7 100644 --- a/internal/app/azldev/core/sources/synthistory.go +++ b/internal/app/azldev/core/sources/synthistory.go @@ -547,8 +547,8 @@ func updateHead(repo *gogit.Repository, commitHash plumbing.Hash) error { } // buildSyntheticCommits resolves the project repository from the component's -// config file, walks the lock file's git history for fingerprint changes, and -// returns the matching [FingerprintChange] entries sorted chronologically. +// config file or configured lock directory, walks the lock file's git history +// for fingerprint changes, and returns matching entries sorted chronologically. // Returns (nil, "", nil) when there are no changes to represent — either the // lock file is missing, has no fingerprint changes (with a warning), or the // current fingerprint matches the committed one. @@ -568,7 +568,7 @@ func buildSyntheticCommits( lockDir string, currentFingerprint string, ) (changes []FingerprintChange, importCommit string, err error) { - projectRepo, projectRepoDir, err := openProjectRepo(config, componentName) + projectRepo, projectRepoDir, err := openProjectRepo(config, componentName, lockDir) if err != nil { return nil, "", err } @@ -681,19 +681,31 @@ func BuildDirtyChange( } // openProjectRepo opens the git repository that contains the component's -// config file and returns both the [gogit.Repository] and the worktree root -// directory. Returns (nil, "", nil) when the config file path cannot be -// resolved, indicating that synthetic commits should be skipped. +// config file, or its configured lock directory for declaration-free components. +// The lock directory anchors history to the repository containing the committed +// lock, without inventing config provenance or using the process working directory. +// Returns (nil, "", nil) for undeclared components with no resolved lock data. func openProjectRepo( config *projectconfig.ComponentConfig, - componentName string, + componentName, lockDir string, ) (*gogit.Repository, string, error) { - var configFilePath string - if config.SourceConfigFile != nil { - configFilePath = config.SourceConfigFile.SourcePath() + if config.SourceConfigFile != nil && config.SourceConfigFile.SourcePath() != "" { + return openProjectRepoForConfigFile(config.SourceConfigFile.SourcePath(), componentName) + } + + if config.Locked == nil { + slog.Debug("Undeclared component has no lock data; skipping synthetic history", + "component", componentName) + + return nil, "", nil + } + + if !filepath.IsAbs(lockDir) { + return nil, "", fmt.Errorf("cannot locate project repository for component %#q: absolute lock directory required", + componentName) } - return openProjectRepoForConfigFile(configFilePath, componentName) + return openProjectRepoAtDir(lockDir) } // openProjectRepoForConfigFile opens the git repository containing configFilePath @@ -711,10 +723,14 @@ func openProjectRepoForConfigFile( return nil, "", nil } - repo, err := git.OpenProjectRepo(filepath.Dir(configFilePath)) + return openProjectRepoAtDir(filepath.Dir(configFilePath)) +} + +func openProjectRepoAtDir(directory string) (*gogit.Repository, string, error) { + repo, err := git.OpenProjectRepo(directory) if err != nil { - return nil, "", fmt.Errorf("failed to find project repository for config file %#q:\n%w", - configFilePath, err) + return nil, "", fmt.Errorf("failed to find project repository from directory %#q:\n%w", + directory, err) } worktree, err := repo.Worktree() diff --git a/internal/app/azldev/core/sources/synthistory_internal_test.go b/internal/app/azldev/core/sources/synthistory_internal_test.go index 303c2de52..a0c2522dc 100644 --- a/internal/app/azldev/core/sources/synthistory_internal_test.go +++ b/internal/app/azldev/core/sources/synthistory_internal_test.go @@ -4,6 +4,7 @@ package sources import ( + "fmt" "os" "os/exec" "path/filepath" @@ -20,6 +21,7 @@ import ( "github.com/spf13/afero" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" + "go.uber.org/mock/gomock" ) func TestBuildSyntheticCommits_SymlinkedProjectPath(t *testing.T) { @@ -68,6 +70,20 @@ func TestBuildSyntheticCommits_SymlinkedProjectPath(t *testing.T) { }) require.NoError(t, err) + lockPath := filepath.Join(realProjectDir, "locks", componentName+".lock") + lockContent, err := fileutils.ReadFile(osFS, lockPath) + require.NoError(t, err) + require.NoError(t, fileutils.WriteFile(osFS, lockPath, + []byte(strings.ReplaceAll(string(lockContent), "sha256:abc", "sha256:def")), fileperms.PublicFile)) + require.NoError(t, worktree.AddWithOptions(&gogit.AddOptions{All: true})) + secondHash, err := worktree.Commit("update curl overlay", &gogit.CommitOptions{ + Author: &object.Signature{ + Name: "azldev", Email: "azldev@local", + When: time.Date(2024, 6, 2, 0, 0, 0, 0, time.UTC), + }, + }) + require.NoError(t, err) + _, config, err := projectconfig.LoadProjectConfig( osFS, testctx.NewTestOSEnv(), linkedProjectDir, true, t.TempDir(), nil, false, false, @@ -79,26 +95,70 @@ func TestBuildSyntheticCommits_SymlinkedProjectPath(t *testing.T) { component, ok := config.Components[componentName] require.True(t, ok) - var loggedPaths []string + for _, declarationFree := range []bool{false, true} { + t.Run(fmt.Sprintf("declaration-free=%t", declarationFree), func(t *testing.T) { + component := component + if declarationFree { + component.SourceConfigFile = nil + component.Locked = &projectconfig.ComponentLockData{ + UpstreamCommit: upstreamCommit, + ImportCommit: importCommit, + } + } + + var loggedPaths []string + + cmdFactory := testctx.NewTestCmdFactory() + cmdFactory.RunAndGetOutputHandler = func(cmd *exec.Cmd) (string, error) { + loggedPaths = append(loggedPaths, cmd.Args[len(cmd.Args)-1]) + + return strings.Join([]string{ + secondHash.String(), "azldev", "azldev@local", "1717286400", "update curl overlay", + }, "\x00") + "\x01" + strings.Join([]string{ + commitHash.String(), "azldev", "azldev@local", "1717200000", "add curl lock", + }, "\x00") + "\x01", nil + } + + changes, gotImportCommit, err := buildSyntheticCommits( + t.Context(), cmdFactory, &component, componentName, config.Project.LockDir, "", + ) + require.NoError(t, err) + assert.Equal(t, []string{filepath.Join("locks", componentName+".lock")}, loggedPaths) + assert.Equal(t, importCommit, gotImportCommit) + require.Len(t, changes, 2) + assert.Equal(t, commitHash.String(), changes[0].Hash) + assert.Equal(t, secondHash.String(), changes[1].Hash) + assert.Equal(t, upstreamCommit, changes[1].UpstreamCommit) + + ctx := testctx.NewCtx() + specPath := writeStaticReleaseTestSpec(t, ctx.FS(), componentName, "1%{?dist}") + component.Release.Calculation = projectconfig.ReleaseCalculationStatic + comp := mockComponent(gomock.NewController(t), componentName, &component) + preparer := newTestPreparer(ctx.FS()) + require.NoError(t, preparer.tryBumpRelease( + t.Context(), comp, filepath.Dir(specPath), changes)) + release, err := GetReleaseTagValue(ctx.FS(), specPath) + require.NoError(t, err) + assert.Equal(t, "3%{?dist}", release) + }) + } - cmdFactory := testctx.NewTestCmdFactory() - cmdFactory.RunAndGetOutputHandler = func(cmd *exec.Cmd) (string, error) { - loggedPaths = append(loggedPaths, cmd.Args[len(cmd.Args)-1]) + lockedConfig := &projectconfig.ComponentConfig{Locked: &projectconfig.ComponentLockData{}} + _, _, err = openProjectRepo(lockedConfig, componentName, tempDir) + require.ErrorContains(t, err, "failed to find project repository") + _, _, err = openProjectRepo(lockedConfig, componentName, "") + require.ErrorContains(t, err, "absolute lock directory required") +} - return strings.Join([]string{ - commitHash.String(), "azldev", "azldev@local", "1717200000", "add curl lock", - }, "\x00") + "\x01", nil +func TestBuildSyntheticCommits_UndeclaredComponentWithoutLock(t *testing.T) { + config := &projectconfig.ComponentConfig{ + Name: "local", + Spec: projectconfig.SpecSource{SourceType: projectconfig.SpecSourceTypeLocal}, } - - changes, gotImportCommit, err := buildSyntheticCommits( - t.Context(), cmdFactory, &component, componentName, config.Project.LockDir, "", + changes, importCommit, err := buildSyntheticCommits( + t.Context(), testctx.NewTestCmdFactory(), config, config.Name, "/not-a-project/locks", "", ) require.NoError(t, err) - - assert.Equal(t, []string{filepath.Join("locks", componentName+".lock")}, loggedPaths) - assert.Equal(t, importCommit, gotImportCommit) - - require.Len(t, changes, 1) - assert.Equal(t, commitHash.String(), changes[0].Hash) - assert.Equal(t, upstreamCommit, changes[0].UpstreamCommit) + assert.Empty(t, changes) + assert.Empty(t, importCommit) } diff --git a/internal/projectconfig/component.go b/internal/projectconfig/component.go index 6b56f9d8e..49501fbe6 100644 --- a/internal/projectconfig/component.go +++ b/internal/projectconfig/component.go @@ -408,20 +408,38 @@ type ComponentConfig struct { // Overlays to apply to sources after they've been acquired. May mutate the spec as well as sources. Overlays []ComponentOverlay `toml:"overlays,omitempty" json:"overlays,omitempty" table:"-" jsonschema:"title=Overlays,description=Overlays to apply to this component's spec and/or sources"` - // OverlayFiles, if set, lists path or glob patterns (relative to this component config file) - // matched against the filesystem after component config resolution to locate per-file overlay documents. - // Each matched file is parsed as an [OverlayFile]: one logical change consisting of a - // file-level `[metadata]` block plus an ordered list of `[[overlays]]`. The per-file - // metadata is stamped onto every overlay in the file. Matches are concatenated in the - // order patterns are declared; within a single pattern, matches are applied in - // filename (lexicographic) order, using the full path as a tie-breaker when - // filenames match. Duplicate matches are de-duplicated, preserving first - // occurrence. The resulting overlays are appended to [ComponentConfig.Overlays] - // after any inline overlays. A value set in a higher-priority config layer replaces - // lower-priority overlay-files values; an explicit empty list disables inherited - // overlay files. Excluded from the fingerprint because the value affects only where - // overlays are sourced from, not their content. - OverlayFiles []string `toml:"overlay-files,omitempty" json:"overlayFiles,omitempty" table:"-" validate:"dive,required" jsonschema:"title=Overlay files,description=Path or glob patterns (relative to the component config file or matched spec directory) matched against the filesystem to locate per-file overlay documents after component config resolution. Use an empty list to disable inherited overlay-file patterns" fingerprint:"-"` + // OverlayFiles lists path or glob patterns matched against the filesystem after + // component config resolution to locate per-file overlay documents. Each matched + // file is parsed as an [OverlayFile]: one logical change consisting of a file-level + // `[metadata]` block plus an ordered list of `[[overlays]]`. The per-file metadata + // is stamped onto every overlay in the file. Matches are concatenated in the order + // entries are declared; within a single entry, matches are applied in filename + // (lexicographic) order, using the full path as a tie-breaker when filenames match. + // Duplicate matches are de-duplicated, preserving first occurrence. The resulting + // overlays are appended to [ComponentConfig.Overlays] after any inline overlays. + // + // A value set in a higher-priority config layer replaces lower-priority values + // wholesale; an explicit empty list disables inherited overlay files. + // + // Plain legacy globs and empty lists remain valid at project and component scopes, + // with unchanged path resolution and precedence. Distro-version and component-group + // defaults must omit 'overlay-files', including empty lists. + // + // Project-level 'default-component-config' may use the '{component}' placeholder + // exactly once as a whole path segment. At glob time it is substituted with + // the resolved component's name. These entries are anchored to the config file + // that declares the default and only apply to already-resolved components. + // '..' path segments may appear before '{component}', but never after it. + // Additional braces are forbidden after '{component}'; use separate list entries. + // + // The placeholder is forbidden at component scope. Plain inherited globs are + // interpreted relative to the concrete component + // config file (or matched spec directory). An explicit 'overlay-files = []' + // disables inherited overlay files. + // + // Excluded from the fingerprint because the value affects only where overlays are + // sourced from, not their content. + OverlayFiles []string `toml:"overlay-files,omitempty" json:"overlayFiles,omitempty" table:"-" validate:"dive,required" jsonschema:"title=Overlay files,description=Path or glob patterns matched against the filesystem to locate per-file overlay documents. Allowed only in project-level 'default-component-config' and explicit component settings. Distro-version and component-group defaults must omit 'overlay-files' including empty lists. Plain legacy patterns and empty lists remain supported at project and component scope with unchanged path resolution and precedence. Project-level defaults may use {component} exactly once as a whole directory segment to substitute the resolved component's name. Placeholder patterns are relative to the config file declaring the default and only apply to already-resolved components. '..' path segments are allowed before {component} but forbidden after it. Additional braces are forbidden after {component}; use separate list entries for multiple directories. The placeholder is forbidden at component scope. Plain inherited globs resolve relative to the concrete component config file or matched spec directory. Use an empty list at component scope to disable inherited overlay files." fingerprint:"-"` // Configuration for building the component. Build ComponentBuildConfig `toml:"build,omitempty" json:"build,omitempty" table:"-" jsonschema:"title=Build configuration,description=Configuration for building the component"` @@ -637,10 +655,12 @@ func (c *ComponentConfig) WithAbsolutePaths(referenceDir string) *ComponentConfi SourceFiles: deep.MustCopy(c.SourceFiles), Packages: deep.MustCopy(c.Packages), Publish: deep.MustCopy(c.Publish), - // OverlayFiles is consumed after component config resolution; preserve it verbatim - // here so inherited patterns can be interpreted relative to the concrete component - // config file. - OverlayFiles: slices.Clone(c.OverlayFiles), + // OverlayFiles: entries with the {component} placeholder are project-scope + // shared patterns and are absolutized now so downstream code sees stable + // absolute paths regardless of where the config was inherited from. Non-placeholder + // entries are left verbatim so component-scope patterns are still interpreted + // relative to the concrete component config file at glob time. + OverlayFiles: absolutizeOverlayFilesPlaceholderEntries(referenceDir, c.OverlayFiles), Tests: deep.MustCopy(c.Tests), } diff --git a/internal/projectconfig/component_test.go b/internal/projectconfig/component_test.go index 178af07f2..3cb78e2a6 100644 --- a/internal/projectconfig/component_test.go +++ b/internal/projectconfig/component_test.go @@ -612,13 +612,9 @@ func TestResolveComponentConfig(t *testing.T) { assert.Equal(t, "comp-commit", resolved.Spec.UpstreamCommit) }) - t.Run("component empty overlay files clears group defaults", func(t *testing.T) { - groups := map[string]projectconfig.ComponentGroupConfig{ - "core": { - DefaultComponentConfig: projectconfig.ComponentConfig{ - OverlayFiles: []string{"overlays/*.overlay.toml"}, - }, - }, + t.Run("component empty overlay files clears project defaults", func(t *testing.T) { + projectDefaults := projectconfig.ComponentConfig{ + OverlayFiles: []string{"overlays/*.overlay.toml"}, } comp := projectconfig.ComponentConfig{ Name: "curl", @@ -626,7 +622,7 @@ func TestResolveComponentConfig(t *testing.T) { } resolved, err := projectconfig.ResolveComponentConfig( - comp, projectconfig.ComponentConfig{}, distroDefaults, groups, []string{"core"}, + comp, projectDefaults, distroDefaults, nil, nil, ) require.NoError(t, err) require.NotNil(t, resolved.OverlayFiles) diff --git a/internal/projectconfig/configfile.go b/internal/projectconfig/configfile.go index ee6f07c73..6f98c74cd 100644 --- a/internal/projectconfig/configfile.go +++ b/internal/projectconfig/configfile.go @@ -111,6 +111,10 @@ func (f ConfigFile) Validate() error { } } + if err := f.validateOverlayFilesByScope(); err != nil { + return err + } + // Validate component group metadata. if err := validateComponentGroupMetadata(f.ComponentGroups); err != nil { return err @@ -163,6 +167,12 @@ func validateComponentConfigs( customScriptPathsResolved bool, ) error { for componentName, component := range components { + if err := validateOverlayFilesEntries( + component.OverlayFiles, validateComponentOverlayFilesEntry, + ); err != nil { + return fmt.Errorf("invalid 'overlay-files' for component %#q:\n%w", componentName, err) + } + for i, overlay := range component.Overlays { err := overlay.Validate() if err != nil { @@ -516,6 +526,82 @@ func dedupTestRefSKUGroups( return nil } +// validateOverlayFilesByScope restricts default 'overlay-files' lists to project scope. +// +// Distro-version and component-group defaults must omit the field, including empty lists. +// Project defaults accept plain patterns and may substitute component names with '{component}'. +func (f ConfigFile) validateOverlayFilesByScope() error { + if f.DefaultComponentConfig != nil { + if err := validateOverlayFilesEntries( + f.DefaultComponentConfig.OverlayFiles, validateProjectOverlayFilesEntry, + ); err != nil { + return fmt.Errorf("invalid project 'default-component-config':\n%w", err) + } + } + + for distroName, distro := range f.Distros { + for versionName, version := range distro.Versions { + if version.DefaultComponentConfig.OverlayFiles != nil { + return fmt.Errorf( + "%w: 'overlay-files' is not allowed in 'default-component-config' for distro %#q version %#q", + ErrInvalidOverlayFilesEntry, distroName, versionName, + ) + } + } + } + + for groupName, group := range f.ComponentGroups { + if group.DefaultComponentConfig.OverlayFiles != nil { + return fmt.Errorf( + "%w: 'overlay-files' is not allowed in 'default-component-config' for component group %#q", + ErrInvalidOverlayFilesEntry, groupName, + ) + } + } + + return nil +} + +// overlayFilesValidator validates a single 'overlay-files' entry against +// the placeholder rules of a particular config scope. +type overlayFilesValidator func(entry string) error + +// validateProjectOverlayFilesEntry validates entries in the project-level +// default-component-config. Legacy plain globs remain valid; entries containing +// '{component}' substitute the resolved component's name and must obey the placeholder rules. +func validateProjectOverlayFilesEntry(entry string) error { + if !hasOverlayFilesPlaceholder(entry) { + return nil + } + + return validateOverlayFilesPlaceholder(entry) +} + +// validateComponentOverlayFilesEntry validates plain globs at component scope, +// where '{component}' is forbidden. +func validateComponentOverlayFilesEntry(entry string) error { + if !hasOverlayFilesPlaceholder(entry) { + return nil + } + + return fmt.Errorf( + "%w: %q is only allowed in project-level 'default-component-config' entries; entry %#q", + ErrInvalidOverlayFilesEntry, OverlayFilesComponentPlaceholder, entry, + ) +} + +// validateOverlayFilesEntries runs validate against each entry in overlayFiles, +// annotating the returned error with the offending index. +func validateOverlayFilesEntries(overlayFiles []string, validate overlayFilesValidator) error { + for idx, entry := range overlayFiles { + if err := validate(entry); err != nil { + return fmt.Errorf("'overlay-files'[%d]: %w", idx, err) + } + } + + return nil +} + // validateSourceFiles checks 'source-files' configuration for a component: // - All filenames must be unique. // - Hash type must be a supported algorithm when specified. diff --git a/internal/projectconfig/configfile_test.go b/internal/projectconfig/configfile_test.go index b10216d17..4dbdae5df 100644 --- a/internal/projectconfig/configfile_test.go +++ b/internal/projectconfig/configfile_test.go @@ -1709,3 +1709,114 @@ func TestValidateCustomSourceRef_InvalidInputFilename(t *testing.T) { assert.Contains(t, err.Error(), "'inputs'") assert.Contains(t, err.Error(), "escape.tar.gz") } + +func TestProjectConfigFileValidation_OverlayFilesPlaceholderOnProjectDefault(t *testing.T) { + file := projectconfig.ConfigFile{ + DefaultComponentConfig: &projectconfig.ComponentConfig{ + OverlayFiles: []string{"comps/{component}/overlays/*.overlay.toml"}, + }, + } + assert.NoError(t, file.Validate()) +} + +func TestProjectConfigFileValidation_OverlayFilesPlaceholderOnDistroDefaultRejected(t *testing.T) { + file := projectconfig.ConfigFile{ + Distros: map[string]projectconfig.DistroDefinition{ + "my-distro": { + Versions: map[string]projectconfig.DistroVersionDefinition{ + "1.0": { + ReleaseVer: "1.0", + DefaultComponentConfig: projectconfig.ComponentConfig{ + OverlayFiles: []string{"distro/{component}/*.overlay.toml"}, + }, + }, + }, + }, + }, + } + err := file.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "my-distro") + assert.Contains(t, err.Error(), "overlay-files") +} + +func TestProjectConfigFileValidation_OverlayFilesPlaceholderOnGroupDefaultRejected(t *testing.T) { + file := projectconfig.ConfigFile{ + ComponentGroups: map[string]projectconfig.ComponentGroupConfig{ + "core": { + DefaultComponentConfig: projectconfig.ComponentConfig{ + OverlayFiles: []string{"core/{component}/*.overlay.toml"}, + }, + }, + }, + } + err := file.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "core") + assert.Contains(t, err.Error(), "overlay-files") +} + +func TestProjectConfigFileValidation_OverlayFilesPlaceholderOnComponentRejected(t *testing.T) { + file := projectconfig.ConfigFile{ + Components: map[string]projectconfig.ComponentConfig{ + "my-comp": { + OverlayFiles: []string{"comps/{component}/overlays/*.overlay.toml"}, + }, + }, + } + err := file.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "my-comp") + assert.Contains(t, err.Error(), "{component}") +} + +func TestProjectConfigFileValidation_PlainGlobOnProjectDefaultAllowed(t *testing.T) { + file := projectconfig.ConfigFile{ + DefaultComponentConfig: &projectconfig.ComponentConfig{ + OverlayFiles: []string{"comps/no-placeholder/*.overlay.toml"}, + }, + } + assert.NoError(t, file.Validate()) +} + +func TestProjectConfigFileValidation_MalformedPlaceholderOnProjectDefault(t *testing.T) { + file := projectconfig.ConfigFile{ + DefaultComponentConfig: &projectconfig.ComponentConfig{ + OverlayFiles: []string{"prefix-{component}/*.overlay.toml"}, + }, + } + err := file.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "whole path segment") +} + +func TestProjectConfigFileValidation_EmptyOverlayFilesOnDistroDefaultRejected(t *testing.T) { + file := projectconfig.ConfigFile{ + Distros: map[string]projectconfig.DistroDefinition{ + "my-distro": { + Versions: map[string]projectconfig.DistroVersionDefinition{ + "1.0": { + ReleaseVer: "1.0", + DefaultComponentConfig: projectconfig.ComponentConfig{ + OverlayFiles: []string{}, + }, + }, + }, + }, + }, + } + require.ErrorIs(t, file.Validate(), projectconfig.ErrInvalidOverlayFilesEntry) +} + +func TestProjectConfigFileValidation_EmptyOverlayFilesOnGroupDefaultRejected(t *testing.T) { + file := projectconfig.ConfigFile{ + ComponentGroups: map[string]projectconfig.ComponentGroupConfig{ + "core": { + DefaultComponentConfig: projectconfig.ComponentConfig{ + OverlayFiles: []string{}, + }, + }, + }, + } + require.ErrorIs(t, file.Validate(), projectconfig.ErrInvalidOverlayFilesEntry) +} diff --git a/internal/projectconfig/overlay_file.go b/internal/projectconfig/overlay_file.go index eb04ff258..240feb52d 100644 --- a/internal/projectconfig/overlay_file.go +++ b/internal/projectconfig/overlay_file.go @@ -56,6 +56,11 @@ type OverlayFile struct { // [OverlayFile], stamps the per-file metadata onto each overlay, appends the // resulting overlays after inline overlays, and clears [ComponentConfig.OverlayFiles] // so the returned config is not expanded twice. +// +// Entries containing [OverlayFilesComponentPlaceholder] have the placeholder +// substituted with the component's own name before globbing; those entries are +// inherited from a project-level default and become concrete per-component +// globs here. func ExpandResolvedOverlayFiles( fs opctx.FS, component ComponentConfig, referenceDir string, permissiveConfigParsing bool, ) (ComponentConfig, error) { @@ -63,14 +68,19 @@ func ExpandResolvedOverlayFiles( return component, nil } - if referenceDir == "" && overlayFilesHasRelativePattern(component.OverlayFiles) { + expanded := make([]string, len(component.OverlayFiles)) + for i, entry := range component.OverlayFiles { + expanded[i] = substituteOverlayFilesPlaceholder(entry, component.Name) + } + + if referenceDir == "" && overlayFilesHasRelativePattern(expanded) { return ComponentConfig{}, fmt.Errorf( "component %#q has relative 'overlay-files' entries but no reference directory", component.Name, ) } - loaded, err := loadOverlayFiles(fs, referenceDir, component.OverlayFiles, permissiveConfigParsing) + loaded, err := loadOverlayFiles(fs, referenceDir, component.OverlayFiles, permissiveConfigParsing, component.Name) if err != nil { return ComponentConfig{}, fmt.Errorf("component %#q overlay-files:\n%w", component.Name, err) } @@ -101,9 +111,11 @@ func overlayFilesHasRelativePattern(patterns []string) bool { // pattern, matches are applied in filename (lexicographic) order, using the full // path as a tie-breaker when filenames match. Duplicate matches across patterns // are de-duplicated, preserving first occurrence. Glob patterns that match no -// files contribute no overlays; literal paths must match a file. +// files contribute no overlays; explicit literal paths must match a file. +// Inherited placeholder entries may match no files even with a literal suffix. func loadOverlayFiles( fs opctx.FS, referenceDir string, patterns []string, permissiveConfigParsing bool, + componentName string, ) ([]ComponentOverlay, error) { var ( ordered []string @@ -111,6 +123,9 @@ func loadOverlayFiles( ) for _, pattern := range patterns { + inheritedPlaceholder := hasOverlayFilesPlaceholder(pattern) + pattern = substituteOverlayFilesPlaceholder(pattern, componentName) + absPattern := pattern if !filepath.IsAbs(absPattern) { absPattern = path.Join(referenceDir, absPattern) @@ -125,7 +140,7 @@ func loadOverlayFiles( switch { case err != nil: return nil, fmt.Errorf("failed to scan for overlay files matching %q:\n%w", pattern, err) - case len(matches) == 0 && !containsPattern(pattern): + case len(matches) == 0 && !inheritedPlaceholder && !containsPattern(pattern): return nil, fmt.Errorf("%w: %q", ErrOverlayFilesNoMatches, pattern) case len(matches) == 0: continue diff --git a/internal/projectconfig/overlay_file_test.go b/internal/projectconfig/overlay_file_test.go index 05207c3c0..dfbfaf162 100644 --- a/internal/projectconfig/overlay_file_test.go +++ b/internal/projectconfig/overlay_file_test.go @@ -65,7 +65,7 @@ value = "Microsoft" filepath.Join(overlayDir, "0001-backport-drop-openjdk21.overlay.toml"), []byte(validBackportOverlayFile), fileperms.PrivateFile)) - loaded, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false) + loaded, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false, "ant") require.NoError(t, err) require.Len(t, loaded, 3, "two overlays from 0001 + one from 0002") @@ -146,7 +146,8 @@ tag = "Order" value = "fourth" `), fileperms.PrivateFile)) - loaded, err := loadOverlayFiles(ctx.FS(), "/project", []string{"/project/comps/ant/overlays/**/*.overlay.toml"}, false) + loaded, err := loadOverlayFiles( + ctx.FS(), "/project", []string{"/project/comps/ant/overlays/**/*.overlay.toml"}, false, "ant") require.NoError(t, err) require.Len(t, loaded, 4) assert.Equal(t, "first", loaded[0].Value) @@ -165,7 +166,7 @@ func TestLoadOverlayFiles_NoMatchesIsNoop(t *testing.T) { filepath.Join(overlayDir, "README.md"), []byte("not an overlay"), fileperms.PrivateFile)) - loaded, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false) + loaded, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false, "ant") require.NoError(t, err) assert.Empty(t, loaded) } @@ -175,7 +176,7 @@ func TestLoadOverlayFiles_MissingDirIsNoop(t *testing.T) { loaded, err := loadOverlayFiles( ctx.FS(), "/project", - []string{overlayGlob("/project/comps/does-not-exist/overlays")}, false, + []string{overlayGlob("/project/comps/does-not-exist/overlays")}, false, "ant", ) require.NoError(t, err) assert.Empty(t, loaded) @@ -186,7 +187,7 @@ func TestLoadOverlayFiles_MissingLiteralPathErrors(t *testing.T) { _, err := loadOverlayFiles( ctx.FS(), "/project", - []string{"/project/comps/does-not-exist/overlays/0001-missing.overlay.toml"}, false, + []string{"/project/comps/does-not-exist/overlays/0001-missing.overlay.toml"}, false, "ant", ) require.ErrorIs(t, err, ErrOverlayFilesNoMatches) } @@ -202,7 +203,7 @@ func TestLoadOverlayFiles_NoMatchDoesNotBlockLaterMatches(t *testing.T) { loaded, err := loadOverlayFiles(ctx.FS(), "/project", []string{ overlayGlob("/project/comps/does-not-exist/overlays"), overlayGlob(overlayDir), - }, false) + }, false, "ant") require.NoError(t, err) require.Len(t, loaded, 2) } @@ -225,7 +226,7 @@ value = "Microsoft" metadata = { category = "azl-branding-policy", upstream-status = "inapplicable" } `), fileperms.PrivateFile)) - _, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false) + _, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false, "ant") require.ErrorIs(t, err, ErrOverlayFilePerOverlayMetadata) } @@ -241,7 +242,7 @@ category = "azl-compatibility" upstream-status = "inapplicable" `), fileperms.PrivateFile)) - _, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false) + _, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false, "ant") require.ErrorIs(t, err, ErrOverlayFileEmpty) } @@ -262,7 +263,7 @@ tag = "Vendor" value = "Microsoft" `), fileperms.PrivateFile)) - _, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false) + _, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false, "ant") require.Error(t, err) assert.Contains(t, err.Error(), "category") } @@ -283,7 +284,7 @@ type = "spec-set-tag" tag = "Vendor" `), fileperms.PrivateFile)) - _, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false) + _, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false, "ant") require.Error(t, err) assert.Contains(t, err.Error(), "invalid overlay 1") assert.Contains(t, err.Error(), "requires") @@ -336,7 +337,7 @@ type = "patch-add" source = "patches/fix.patch" `), fileperms.PrivateFile)) - loaded, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false) + loaded, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false, "ant") require.NoError(t, err) require.Len(t, loaded, 1) assert.Equal(t, @@ -358,7 +359,7 @@ func TestLoadOverlayFiles_DedupsAcrossPatterns(t *testing.T) { loaded, err := loadOverlayFiles(ctx.FS(), "/project", []string{ overlayGlob(overlayDir), filepath.Join(overlayDir, "0001-*.overlay.toml"), - }, false) + }, false, "ant") require.NoError(t, err) require.Len(t, loaded, 2, "overlapping globs must not double-apply matched files") } @@ -373,7 +374,7 @@ func TestLoadOverlayFiles_DoubleStarMatchesNested(t *testing.T) { loaded, err := loadOverlayFiles( ctx.FS(), "/project", - []string{filepath.Join(root, "**", "*.overlay.toml")}, false, + []string{filepath.Join(root, "**", "*.overlay.toml")}, false, "ant", ) require.NoError(t, err) require.Len(t, loaded, 2, "**-style glob must descend into subdirectories") @@ -441,6 +442,65 @@ func TestExpandResolvedOverlayFiles_NoopWhenUnset(t *testing.T) { require.Len(t, expanded.Overlays, 1, "no overlay-files, no merging") } +func TestExpandResolvedOverlayFiles_UnmatchedPlaceholderLiteral(t *testing.T) { + tests := []struct { + name string + patterns []string + wantErr bool + }{ + { + name: "unrelated explicit component inherits optional placeholder", + patterns: []string{"comps/{component}/overlays.toml"}, + }, + { + name: "explicit missing literal remains an error", + patterns: []string{"comps/ant/overlays.toml"}, + wantErr: true, + }, + { + name: "placeholder before equivalent explicit literal", + patterns: []string{"comps/{component}/overlays.toml", "comps/ant/overlays.toml"}, + wantErr: true, + }, + { + name: "explicit literal before equivalent placeholder", + patterns: []string{"comps/ant/overlays.toml", "comps/{component}/overlays.toml"}, + wantErr: true, + }, + } + + for _, testCase := range tests { + t.Run(testCase.name, func(t *testing.T) { + ctx := testctx.NewCtx() + component := ComponentConfig{Name: "ant", OverlayFiles: testCase.patterns} + + expanded, err := ExpandResolvedOverlayFiles(ctx.FS(), component, "/project", false) + if testCase.wantErr { + require.ErrorIs(t, err, ErrOverlayFilesNoMatches) + + return + } + + require.NoError(t, err) + assert.Empty(t, expanded.Overlays) + assert.Empty(t, expanded.OverlayFiles) + }) + } +} + +func TestExpandResolvedOverlayFiles_EquivalentPlaceholderEntriesDeduplicate(t *testing.T) { + ctx := testctx.NewCtx() + require.NoError(t, fileutils.WriteFile(ctx.FS(), "/project/comps/ant/overlays.toml", + []byte(validBackportOverlayFile), fileperms.PrivateFile)) + + expanded, err := ExpandResolvedOverlayFiles(ctx.FS(), ComponentConfig{ + Name: "ant", + OverlayFiles: []string{"comps/{component}/overlays.toml", "comps/ant/overlays.toml"}, + }, "/project", false) + require.NoError(t, err) + require.Len(t, expanded.Overlays, 2) +} + func TestExpandResolvedOverlayFiles_AcceptsAbsoluteGlob(t *testing.T) { ctx := testctx.NewCtx() absDir := "/elsewhere/overlays" diff --git a/internal/projectconfig/overlay_files_placeholder.go b/internal/projectconfig/overlay_files_placeholder.go new file mode 100644 index 000000000..a2c81864e --- /dev/null +++ b/internal/projectconfig/overlay_files_placeholder.go @@ -0,0 +1,168 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +package projectconfig + +import ( + "errors" + "fmt" + "slices" + "strings" + + "github.com/bmatcuk/doublestar/v4" +) + +// OverlayFilesComponentPlaceholder is the single supported placeholder in an +// [ComponentConfig.OverlayFiles] entry. Project-level defaults may substitute +// the resolved component's name with this placeholder; components permit only plain patterns, and +// distro-version and component-group defaults must omit 'overlay-files'. +// When present it must appear exactly once and must be a whole path segment. +const OverlayFilesComponentPlaceholder = "{component}" + +// ErrInvalidOverlayFilesEntry is returned when an entry in +// [ComponentConfig.OverlayFiles] does not conform to the placeholder rules +// documented on that field. +var ErrInvalidOverlayFilesEntry = errors.New("invalid overlay-files entry") + +// hasOverlayFilesPlaceholder reports whether entry contains at least one +// occurrence of [OverlayFilesComponentPlaceholder]. +func hasOverlayFilesPlaceholder(entry string) bool { + return strings.Contains(entry, OverlayFilesComponentPlaceholder) +} + +// validateOverlayFilesPlaceholder verifies that entry contains exactly one +// [OverlayFilesComponentPlaceholder] and that the placeholder is a whole path +// segment. Path separators must be '/' (POSIX-style), and no glob +// metacharacters ('*', '?', '[', '{') may appear before the placeholder. +// The prefix must be literal. Glob metacharacters after the placeholder are +// allowed except for braces. '..' path segments are allowed only before the +// placeholder. Returns nil when the entry is well-formed. +func validateOverlayFilesPlaceholder(entry string) error { + if entry == "" { + return fmt.Errorf("%w: entry is empty", ErrInvalidOverlayFilesEntry) + } + + count := strings.Count(entry, OverlayFilesComponentPlaceholder) + if count == 0 { + return fmt.Errorf( + "%w: entry %#q must contain %q exactly once", + ErrInvalidOverlayFilesEntry, entry, OverlayFilesComponentPlaceholder, + ) + } + + if count > 1 { + return fmt.Errorf( + "%w: entry %#q contains %q %d times; it must appear exactly once", + ErrInvalidOverlayFilesEntry, entry, OverlayFilesComponentPlaceholder, count, + ) + } + + if strings.ContainsRune(entry, '\\') { + return fmt.Errorf( + "%w: entry %#q must use %q as the path separator (backslash is not accepted)", + ErrInvalidOverlayFilesEntry, entry, "/", + ) + } + + idx := strings.Index(entry, OverlayFilesComponentPlaceholder) + if idx > 0 && entry[idx-1] != '/' { + return fmt.Errorf( + "%w: %q in entry %#q must be a whole path segment (preceded by %q or start of entry)", + ErrInvalidOverlayFilesEntry, OverlayFilesComponentPlaceholder, entry, "/", + ) + } + + tail := idx + len(OverlayFilesComponentPlaceholder) + if tail == len(entry) { + return fmt.Errorf( + "%w: %q in entry %#q must be followed by %q; it names a directory, not a file", + ErrInvalidOverlayFilesEntry, OverlayFilesComponentPlaceholder, entry, "/", + ) + } + + if entry[tail] != '/' { + return fmt.Errorf( + "%w: %q in entry %#q must be a whole path segment (followed by %q)", + ErrInvalidOverlayFilesEntry, OverlayFilesComponentPlaceholder, entry, "/", + ) + } + + if tail+1 == len(entry) { + return fmt.Errorf( + "%w: %q in entry %#q must be followed by at least one more path segment; "+ + "it names a directory, not a file", + ErrInvalidOverlayFilesEntry, OverlayFilesComponentPlaceholder, entry, + ) + } + + if err := validateOverlayFilesPlaceholderSuffix(entry, entry[tail+1:]); err != nil { + return err + } + + prefix := entry[:idx] + if globMetaIdx := strings.IndexAny(prefix, "*?[{"); globMetaIdx >= 0 { + return fmt.Errorf( + "%w: entry %#q must not contain glob metacharacters (%q) before %q; "+ + "the prefix must be literal", + ErrInvalidOverlayFilesEntry, entry, string(prefix[globMetaIdx]), OverlayFilesComponentPlaceholder, + ) + } + + // Validate the glob syntax independently of the resolved component's name. + globPattern := prefix + "*" + entry[tail:] + if !doublestar.ValidatePattern(globPattern) { + return fmt.Errorf( + "%w: entry %#q expands to invalid glob %#q", + ErrInvalidOverlayFilesEntry, entry, globPattern, + ) + } + + return nil +} + +func validateOverlayFilesPlaceholderSuffix(entry, suffix string) error { + if strings.ContainsAny(suffix, "{}") { + return fmt.Errorf("%w: entry %#q must not contain braces after %q", + ErrInvalidOverlayFilesEntry, entry, OverlayFilesComponentPlaceholder) + } + + if slices.Contains(strings.Split(suffix, "/"), "..") { + return fmt.Errorf("%w: entry %#q must not contain '..' path segments after %q", + ErrInvalidOverlayFilesEntry, entry, OverlayFilesComponentPlaceholder) + } + + return nil +} + +// substituteOverlayFilesPlaceholder returns entry with the single +// [OverlayFilesComponentPlaceholder] replaced by componentName. Entries with +// no placeholder are returned unchanged. +func substituteOverlayFilesPlaceholder(entry, componentName string) string { + if !hasOverlayFilesPlaceholder(entry) { + return entry + } + + return strings.Replace(entry, OverlayFilesComponentPlaceholder, componentName, 1) +} + +// absolutizeOverlayFilesPlaceholderEntries returns a copy of entries where every +// entry containing [OverlayFilesComponentPlaceholder] has been absolutized +// against referenceDir. Non-placeholder entries are cloned verbatim so they can +// still be resolved relative to a per-component reference directory later. +func absolutizeOverlayFilesPlaceholderEntries(referenceDir string, entries []string) []string { + if entries == nil { + return nil + } + + result := make([]string, len(entries)) + + for i, entry := range entries { + if hasOverlayFilesPlaceholder(entry) { + result[i] = makeAbsolute(referenceDir, entry) + } else { + result[i] = entry + } + } + + return result +} diff --git a/internal/projectconfig/overlay_files_placeholder_test.go b/internal/projectconfig/overlay_files_placeholder_test.go new file mode 100644 index 000000000..d77d34f86 --- /dev/null +++ b/internal/projectconfig/overlay_files_placeholder_test.go @@ -0,0 +1,272 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +//nolint:testpackage // Allow to test unexported helpers. +package projectconfig + +import ( + "path/filepath" + "strings" + "testing" + + "github.com/microsoft/azure-linux-dev-tools/internal/global/testctx" + "github.com/microsoft/azure-linux-dev-tools/internal/utils/fileperms" + "github.com/microsoft/azure-linux-dev-tools/internal/utils/fileutils" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestValidateOverlayFilesPlaceholder(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + entry string + wantErr bool + }{ + {name: "empty", entry: "", wantErr: true}, + {name: "missing placeholder", entry: "comps/*/overlays/*.overlay.toml", wantErr: true}, + {name: "multiple placeholders", entry: "comps/{component}/{component}/x.toml", wantErr: true}, + {name: "placeholder is prefix substring", entry: "prefix-{component}/x.toml", wantErr: true}, + {name: "placeholder is suffix substring", entry: "{component}-suffix/x.toml", wantErr: true}, + {name: "backslash separator rejected", entry: "comps\\{component}\\x.toml", wantErr: true}, + {name: "glob metachar before placeholder", entry: "**/{component}/*.overlay.toml", wantErr: true}, + {name: "star before placeholder", entry: "comps*/{component}/*.overlay.toml", wantErr: true}, + {name: "brace alternatives before placeholder", entry: "{base,extra}/{component}/*.overlay.toml", wantErr: true}, + {name: "brace suffix rejected", entry: "comps/{component}/{base,extra}/*.overlay.toml", wantErr: true}, + {name: "brace parent alternative rejected", entry: "comps/{component}/{..,overlays}/*.overlay.toml", wantErr: true}, + { + name: "nested brace alternatives rejected", entry: "comps/{component}/{base,{..,extra}}/*.overlay.toml", + wantErr: true, + }, + { + name: "concatenated brace alternatives rejected", entry: "comps/{component}/{.,x}{.,y}/*.overlay.toml", + wantErr: true, + }, + {name: "unmatched opening brace rejected", entry: "comps/{component}/fix{.overlay.toml", wantErr: true}, + {name: "unmatched closing brace rejected", entry: "comps/{component}/fix}.overlay.toml", wantErr: true}, + {name: "whole segment at start", entry: "{component}/overlays/*.overlay.toml", wantErr: false}, + {name: "whole segment in middle", entry: "comps/{component}/overlays/*.overlay.toml", wantErr: false}, + {name: "placeholder at end rejected", entry: "comps/overlays/{component}", wantErr: true}, + {name: "placeholder followed by only slash rejected", entry: "comps/{component}/", wantErr: true}, + {name: "doublestar in suffix allowed", entry: "comps/{component}/**/*.overlay.toml", wantErr: false}, + {name: "absolute path", entry: "/project/base/comps/{component}/overlays/*.overlay.toml", wantErr: false}, + {name: "parent directory in prefix allowed", entry: "../comps/{component}/overlays/*.overlay.toml"}, + {name: "parent after placeholder rejected", entry: "comps/{component}/../overlays/*.overlay.toml", wantErr: true}, + {name: "nested parent rejected", entry: "comps/{component}/nested/../overlays/*.overlay.toml", wantErr: true}, + {name: "nested escape rejected", entry: "comps/{component}/nested/../../overlays/*.overlay.toml", wantErr: true}, + {name: "trailing parent rejected", entry: "comps/{component}/overlays/..", wantErr: true}, + {name: "two dots in filename allowed", entry: "comps/{component}/overlays/fix..overlay.toml"}, + } + + for _, testCase := range tests { + t.Run(testCase.name, func(t *testing.T) { + t.Parallel() + + err := validateOverlayFilesPlaceholder(testCase.entry) + if testCase.wantErr { + require.Error(t, err) + require.ErrorIs(t, err, ErrInvalidOverlayFilesEntry, + "expected ErrInvalidOverlayFilesEntry, got %v", err) + } else { + assert.NoError(t, err) + } + }) + } +} + +func TestSubstituteOverlayFilesPlaceholder(t *testing.T) { + t.Parallel() + + assert.Equal(t, + "comps/foo/overlays/*.toml", + substituteOverlayFilesPlaceholder("comps/{component}/overlays/*.toml", "foo")) + + // No placeholder → unchanged. + assert.Equal(t, + "overlays/*.toml", + substituteOverlayFilesPlaceholder("overlays/*.toml", "foo")) +} + +func TestLoadProjectConfig_OverlayFilesPlaceholderPaths(t *testing.T) { + testCases := []struct { + name string + pattern string + expected string + wantErr bool + braces bool + }{ + { + name: "parent in placeholder prefix", + pattern: "../comps/{component}/overlays/*.overlay.toml", + expected: "/test/comps/{component}/overlays/*.overlay.toml", + }, + { + name: "parent removes placeholder", + pattern: "comps/{component}/../overlays/*.overlay.toml", + wantErr: true, + }, + { + name: "nested parents remove placeholder", + pattern: "comps/{component}/nested/../../overlays/*.overlay.toml", + wantErr: true, + }, + { + name: "parent after nested directory", + pattern: "comps/{component}/nested/../overlays/*.overlay.toml", + wantErr: true, + }, + { + name: "plain legacy pattern unchanged", + pattern: "nested/../overlays/*.overlay.toml", + expected: "nested/../overlays/*.overlay.toml", + }, + { + name: "safe brace alternatives rejected", + pattern: "comps/{component}/{base,extra}/*.overlay.toml", + wantErr: true, + braces: true, + }, + { + name: "brace parent bypass rejected", + pattern: "comps/{component}/{..,overlays}/*.overlay.toml", + wantErr: true, + braces: true, + }, + { + name: "plain legacy brace pattern unchanged", + pattern: "{base,extra}/*.overlay.toml", + expected: "{base,extra}/*.overlay.toml", + }, + } + + for _, withoutLockfile := range []bool{false, true} { + mode := "locked" + if withoutLockfile { + mode = "lockless" + } + + t.Run(mode, func(t *testing.T) { + for _, testCase := range testCases { + t.Run(testCase.name, func(t *testing.T) { + const projectDir = "/test/project" + + osEnv := testctx.NewTestOSEnv() + osEnv.SetEnv("XDG_CONFIG_HOME", "/test/xdghome") + ctx := testctx.NewCtx(testctx.WithOSEnv(osEnv)) + require.NoError(t, fileutils.MkdirAll(ctx.FS(), projectDir)) + require.NoError(t, fileutils.WriteFile(ctx.FS(), + filepath.Join(projectDir, DefaultConfigFileName), + []byte("[default-component-config]\noverlay-files = [\""+testCase.pattern+"\"]\n"), + fileperms.PublicFile)) + + _, config, err := LoadProjectConfig( + ctx.FS(), ctx.OSEnv(), projectDir, true, "/test/tmp", nil, false, withoutLockfile, + ) + if testCase.wantErr { + require.ErrorIs(t, err, ErrInvalidOverlayFilesEntry) + + if testCase.braces { + require.ErrorContains(t, err, "must not contain braces after") + } else { + require.ErrorContains(t, err, "must not contain '..' path segments after") + } + + return + } + + require.NoError(t, err) + assert.Equal(t, []string{testCase.expected}, config.DefaultComponentConfig.OverlayFiles) + }) + } + }) + } +} + +func TestLoadAndResolveProjectConfig_OverlayFilesDefaultScopes(t *testing.T) { + scopes := []struct { + name string + lines []string + entries func(*ProjectConfig) []string + }{ + { + name: "project", + lines: []string{"[default-component-config]"}, + entries: func(config *ProjectConfig) []string { + return config.DefaultComponentConfig.OverlayFiles + }, + }, + { + name: "distro", + lines: []string{ + "[distros.test.versions.'1']", + "release-ver = '1'", + "[distros.test.versions.'1'.default-component-config]", + }, + entries: func(config *ProjectConfig) []string { + return config.Distros["test"].Versions["1"].DefaultComponentConfig.OverlayFiles + }, + }, + { + name: "group", + lines: []string{"[component-groups.test.default-component-config]"}, + entries: func(config *ProjectConfig) []string { + return config.ComponentGroups["test"].DefaultComponentConfig.OverlayFiles + }, + }, + } + + for _, scope := range scopes { + t.Run(scope.name, func(t *testing.T) { + for _, value := range []struct { + name string + toml string + expected []string + }{ + {name: "omitted"}, + {name: "plain-glob", toml: "['overlays/*.overlay.toml']", expected: []string{"overlays/*.overlay.toml"}}, + {name: "empty-list", toml: "[]", expected: []string{}}, + { + name: "placeholder", toml: "['comps/{component}/overlays/*.overlay.toml']", + expected: []string{"/project/comps/{component}/overlays/*.overlay.toml"}, + }, + } { + t.Run(value.name, func(t *testing.T) { + for _, mode := range []struct { + name string + withoutLockfile bool + }{ + {name: "default"}, + {name: "without-lockfile", withoutLockfile: true}, + } { + t.Run(mode.name, func(t *testing.T) { + ctx := testctx.NewCtx() + configPath := filepath.Join("/project", DefaultConfigFileName) + + lines := append([]string{}, scope.lines...) + if value.toml != "" { + lines = append(lines, "overlay-files = "+value.toml) + } + + require.NoError(t, fileutils.WriteFile(ctx.FS(), configPath, + []byte(strings.Join(lines, "\n")+"\n"), fileperms.PrivateFile)) + + config, err := loadAndResolveProjectConfig(ctx.FS(), + loadOptions{withoutLockfile: mode.withoutLockfile}, configPath) + if scope.name != "project" && value.toml != "" { + require.ErrorIs(t, err, ErrInvalidOverlayFilesEntry) + require.ErrorContains(t, err, scope.name) + assert.Nil(t, config) + + return + } + + require.NoError(t, err) + assert.Equal(t, value.expected, scope.entries(config)) + }) + } + }) + } + }) + } +} diff --git a/scenario/__snapshots__/TestSnapshotsContainer_config_generate-schema_stdout_1.snap b/scenario/__snapshots__/TestSnapshotsContainer_config_generate-schema_stdout_1.snap index 9cf47bd35..39870154d 100755 --- a/scenario/__snapshots__/TestSnapshotsContainer_config_generate-schema_stdout_1.snap +++ b/scenario/__snapshots__/TestSnapshotsContainer_config_generate-schema_stdout_1.snap @@ -130,7 +130,7 @@ }, "type": "array", "title": "Overlay files", - "description": "Path or glob patterns (relative to the component config file or matched spec directory) matched against the filesystem to locate per-file overlay documents after component config resolution. Use an empty list to disable inherited overlay-file patterns" + "description": "Path or glob patterns matched against the filesystem to locate per-file overlay documents. Allowed only in project-level 'default-component-config' and explicit component settings. Distro-version and component-group defaults must omit 'overlay-files' including empty lists. Plain legacy patterns and empty lists remain supported at project and component scope with unchanged path resolution and precedence. Project-level defaults may use {component} exactly once as a whole directory segment to substitute the resolved component's name. Placeholder patterns are relative to the config file declaring the default and only apply to already-resolved components. '..' path segments are allowed before {component} but forbidden after it. Additional braces are forbidden after {component}; use separate list entries for multiple directories. The placeholder is forbidden at component scope. Plain inherited globs resolve relative to the concrete component config file or matched spec directory. Use an empty list at component scope to disable inherited overlay files." }, "build": { "$ref": "#/$defs/ComponentBuildConfig", diff --git a/scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap b/scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap index 9cf47bd35..39870154d 100755 --- a/scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap +++ b/scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap @@ -130,7 +130,7 @@ }, "type": "array", "title": "Overlay files", - "description": "Path or glob patterns (relative to the component config file or matched spec directory) matched against the filesystem to locate per-file overlay documents after component config resolution. Use an empty list to disable inherited overlay-file patterns" + "description": "Path or glob patterns matched against the filesystem to locate per-file overlay documents. Allowed only in project-level 'default-component-config' and explicit component settings. Distro-version and component-group defaults must omit 'overlay-files' including empty lists. Plain legacy patterns and empty lists remain supported at project and component scope with unchanged path resolution and precedence. Project-level defaults may use {component} exactly once as a whole directory segment to substitute the resolved component's name. Placeholder patterns are relative to the config file declaring the default and only apply to already-resolved components. '..' path segments are allowed before {component} but forbidden after it. Additional braces are forbidden after {component}; use separate list entries for multiple directories. The placeholder is forbidden at component scope. Plain inherited globs resolve relative to the concrete component config file or matched spec directory. Use an empty list at component scope to disable inherited overlay files." }, "build": { "$ref": "#/$defs/ComponentBuildConfig", diff --git a/schemas/azldev.schema.json b/schemas/azldev.schema.json index 9cf47bd35..39870154d 100644 --- a/schemas/azldev.schema.json +++ b/schemas/azldev.schema.json @@ -130,7 +130,7 @@ }, "type": "array", "title": "Overlay files", - "description": "Path or glob patterns (relative to the component config file or matched spec directory) matched against the filesystem to locate per-file overlay documents after component config resolution. Use an empty list to disable inherited overlay-file patterns" + "description": "Path or glob patterns matched against the filesystem to locate per-file overlay documents. Allowed only in project-level 'default-component-config' and explicit component settings. Distro-version and component-group defaults must omit 'overlay-files' including empty lists. Plain legacy patterns and empty lists remain supported at project and component scope with unchanged path resolution and precedence. Project-level defaults may use {component} exactly once as a whole directory segment to substitute the resolved component's name. Placeholder patterns are relative to the config file declaring the default and only apply to already-resolved components. '..' path segments are allowed before {component} but forbidden after it. Additional braces are forbidden after {component}; use separate list entries for multiple directories. The placeholder is forbidden at component scope. Plain inherited globs resolve relative to the concrete component config file or matched spec directory. Use an empty list at component scope to disable inherited overlay files." }, "build": { "$ref": "#/$defs/ComponentBuildConfig",