From 13406af204ba40c598b3ef5c94d24a7837f526be Mon Sep 17 00:00:00 2001 From: Nan Liu Date: Tue, 7 Jul 2026 20:45:05 +0000 Subject: [PATCH 1/4] feat(overlay-files): support {component} placeholder for pattern-discovered components Add '{component}' placeholder in project-level overlay-files entries. The resolver globs matching paths, extracts each captured segment as a component name, and synthesizes components not otherwise declared. Explicit component declarations shadow pattern-discovered matches (warning emitted). - projectconfig: validate placeholder syntax per scope; reject backslash separators and glob metachars before the placeholder - components: pattern_discovery walks overlay-files entries; Resolver caches discovery results per instance - schema/docs: user-facing overlays.md updated --- docs/user/reference/config/overlays.md | 58 ++++- .../core/components/pattern_discovery.go | 217 ++++++++++++++++++ .../core/components/pattern_discovery_test.go | 191 +++++++++++++++ .../app/azldev/core/components/resolver.go | 98 +++++++- internal/projectconfig/component.go | 63 +++-- internal/projectconfig/configfile.go | 100 ++++++++ internal/projectconfig/configfile_test.go | 120 ++++++++++ internal/projectconfig/overlay_file.go | 14 +- .../overlay_files_placeholder.go | 167 ++++++++++++++ .../overlay_files_placeholder_test.go | 115 ++++++++++ ...ainer_config_generate-schema_stdout_1.snap | 2 +- ...shots_config_generate-schema_stdout_1.snap | 2 +- schemas/azldev.schema.json | 2 +- 13 files changed, 1123 insertions(+), 26 deletions(-) create mode 100644 internal/app/azldev/core/components/pattern_discovery.go create mode 100644 internal/app/azldev/core/components/pattern_discovery_test.go create mode 100644 internal/projectconfig/overlay_files_placeholder.go create mode 100644 internal/projectconfig/overlay_files_placeholder_test.go diff --git a/docs/user/reference/config/overlays.md b/docs/user/reference/config/overlays.md index f252e9d91..996a8defc 100644 --- a/docs/user/reference/config/overlays.md +++ b/docs/user/reference/config/overlays.md @@ -241,7 +241,9 @@ When a single logical change (a CVE backport, a feature disablement, a Fedora ch Set `overlay-files` on the component to one or more globs (relative to the component config) and drop one overlay file per logical change into a directory of your choosing. The conventional layout uses a sibling `overlays/` directory and a `*.overlay.toml` filename suffix, but neither is required — `overlay-files` is just a glob, so any layout you can describe with `**`/`*` patterns works. -`overlay-files` can also be inherited from `default-component-config` at the project, distro, or component-group level. Inherited relative patterns are still resolved for each concrete component: from its component config file when it has one, or from the matched spec file's directory when it is discovered by a component group's `specs` pattern. This makes defaults useful for component-local discovery patterns such as `overlay-files = ["overlays/*.overlay.toml"]`. If a component sets `overlay-files`, that value replaces the inherited list; use `overlay-files = []` to disable inherited overlay files for a component, or include both patterns explicitly when you want to keep default discovery and add component-specific locations. +`overlay-files` on `[components.X]` is a plain glob resolved relative to the component's config file. `overlay-files` can also be set on the project-level `default-component-config`, where **every entry must contain the `{component}` placeholder** (see [Discovering components by pattern](#discovering-components-by-pattern-component-in-overlay-files) below). A component that doesn't declare its own `overlay-files` inherits the project pattern with `{component}` substituted for its own name; use `overlay-files = []` at component scope to disable the inherited list, or set your own list to replace it wholesale. + +`overlay-files` is only settable at project scope (the top-level `default-component-config`) or at component scope (`[components.X]`). Setting it inside a distro-version or component-group `default-component-config` is rejected at load time — even an empty list (`overlay-files = []`). ``` base/comps/mypackage/ @@ -293,6 +295,60 @@ lines = ["- Fix CVE-2024-1234"] Use `azldev component metadata` to list documentation metadata for one or more components. By default it lists metadata from both sources — component overlays and the component groups each component belongs to — tagging each row with its `source`, and shows only entries that carry a `[metadata]` block. Pass `--overlays` or `--groups` to restrict to one source. The command is read-only and supports `--category` and `--upstream-status` filters; output is available as a table (default) or JSON (`-O json`). +### Discovering components by pattern (`{component}` in `overlay-files`) + +For projects that keep each component's overlay files in a predictable per-component directory (e.g. `base/comps//overlays/*.overlay.toml`), you can skip declaring a one-line `[components.]` table for every component. Set `overlay-files` on the **project-level** `default-component-config` to a pattern containing `{component}`, and every match becomes an implicit component. + +Project-scope `overlay-files` entries **must** contain `{component}` exactly once, as a whole path segment, and **must** have at least one more path segment after it (the placeholder names a component directory, not a file). Additional constraints enforced at load time: + +* Use `/` as the path separator (backslashes are rejected, even on Windows). +* No glob metacharacters (`*`, `?`, `[`) before `{component}` — the prefix is a literal used to capture the component name. +* The rest of the entry (after `{component}/`) must be a valid `doublestar` glob (validated with `doublestar.ValidatePattern`). +* The captured segment must be a safe component name (no spaces, no path traversal, no absolute paths); matches with unsafe names are skipped with a `slog.Debug` entry. + +For each entry, `{component}` is expanded to `*` for globbing, and the captured path segment becomes the discovered component's name. The matched files are attached to that component (as if it had declared `overlay-files = []` explicitly). Explicit components with no `overlay-files` of their own inherit the same pattern and re-expand it with their own name substituted for `{component}`. + +```toml +# components.toml (project-level) +[default-component-config] +overlay-files = ["base/comps/{component}/overlays/*.overlay.toml"] +``` + +With the layout below, no per-component `[components.]` table is needed — `openssl` and `curl` are discovered automatically, each carrying its overlay files. + +``` +base/comps/ +├── openssl/ +│ └── overlays/ +│ ├── 0001-cve-2024-1234.overlay.toml +│ └── 0002-disable-fips-tests.overlay.toml +└── curl/ + └── overlays/ + └── 0001-cve-2024-5678.overlay.toml +``` + +**Where `{component}` is allowed.** Only in **project-level** `default-component-config`. At every project-level `overlay-files` entry the placeholder is *required* — plain globs at that scope are rejected because they would apply the same files to every component in the project, which is almost never what you want. At component scope, `overlay-files` entries are plain globs relative to the declaring config file; `{component}` is rejected there. `overlay-files` cannot be set at all in distro-version or component-group `default-component-config` — those broad-scope defaults would silently displace project-level discovery. + +**Inheritance with `{component}`.** A project-level `overlay-files` entry containing `{component}` serves two purposes at once: + +1. **Discovery.** Every captured directory becomes an implicit component unless there is already an explicit `[components.]` table with the same name. +2. **Inheritance.** An explicitly-declared component that does not override `overlay-files` inherits the project entry and expands `{component}` with its own name at glob time. + +A component that sets its own `overlay-files = [...]` replaces the inherited value; the placeholder pattern no longer applies to that component. Setting `overlay-files = []` disables inheritance entirely for that component. + +**Same-scope collisions are a hard error.** If two project-scope `overlay-files` entries both produce the same component name, the resolver fails with a diagnostic listing both entries. Restrict one of the entries or rename one of the directories. + +**Shadowing.** An explicit `[components.]` declaration supersedes pattern discovery for the same name. If that declaration also sets a non-empty `overlay-files` list (which replaces inheritance with a different set of files), `azldev` emits an `slog.Warn` naming the shadowed pattern so you can spot the genuine displacement. An explicit declaration that leaves `overlay-files` unset still inherits the project pattern, so the discovered files apply and no warning is emitted. An explicit empty list (`overlay-files = []`) is the documented off-switch for a component and is not treated as shadowing. + +**Limitation: explicit component-group members need a `[components.]` table.** A component group can enumerate its members two ways: via a `specs` glob (spec files on disk) or via an explicit `components = ["", ...]` list. Only the *explicit list* interacts with pattern discovery: every name in that list must have a matching `[components.]` table in the project (validated at load time, error `ErrUndefinedComponent`). Even a one-liner is enough: + +```toml +# base/comps/openssl/openssl.comp.toml +[components.openssl] +``` + +Once the table exists, the component inherits the project-level `{component}` overlay pattern normally. Group members discovered via `specs` (not the explicit list) do **not** need a `.comp.toml` — the resolver synthesizes an empty config for them and inheritance still applies. + ## Examples ### Adding a Build Dependency diff --git a/internal/app/azldev/core/components/pattern_discovery.go b/internal/app/azldev/core/components/pattern_discovery.go new file mode 100644 index 000000000..b94b0bde9 --- /dev/null +++ b/internal/app/azldev/core/components/pattern_discovery.go @@ -0,0 +1,217 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +package components + +import ( + "errors" + "fmt" + "log/slog" + "path" + "path/filepath" + "slices" + "strings" + + "github.com/bmatcuk/doublestar/v4" + "github.com/microsoft/azure-linux-dev-tools/internal/app/azldev" + "github.com/microsoft/azure-linux-dev-tools/internal/projectconfig" + "github.com/microsoft/azure-linux-dev-tools/internal/utils/fileutils" +) + +// patternDiscoveredComponent captures the result of expanding a single +// project-level [projectconfig.ComponentConfig.OverlayFiles] entry that +// contains the `{component}` placeholder. +type patternDiscoveredComponent struct { + // Name of the discovered component (the captured {component} segment). + Name string + // ReferenceDir is the absolute directory the pattern captured for this + // component (i.e. the prefix directory + captured name). Used as the + // component's reference directory for downstream operations. + ReferenceDir string + // OverlayFiles is the deduplicated, sorted list of absolute overlay file + // paths matched by the winning pattern for this component. + OverlayFiles []string + // SourcePattern is the exact entry string that produced this match, used + // in shadow-lint diagnostics. + SourcePattern string +} + +// discoverComponentsFromPatterns walks the project-level +// default-component-config, expands every 'overlay-files' entry that contains +// the `{component}` placeholder, and returns the resulting pattern-discovered +// components. Two entries at project scope producing the same component name +// is a hard error. +// +// The caller is responsible for further precedence against explicitly-declared +// components (that override happens at the resolver layer; see +// [Resolver.patternDiscoveredComponents]). +func discoverComponentsFromPatterns(env *azldev.Env) (map[string]*patternDiscoveredComponent, error) { + result := make(map[string]*patternDiscoveredComponent) + + config := env.Config() + if config == nil { + return result, nil + } + + var patterns []string + + for _, entry := range config.DefaultComponentConfig.OverlayFiles { + if strings.Contains(entry, projectconfig.OverlayFilesComponentPlaceholder) { + patterns = append(patterns, entry) + } + } + + if len(patterns) == 0 { + return result, nil + } + + for _, pattern := range patterns { + byName, err := expandSinglePattern(env, pattern) + if err != nil { + return nil, fmt.Errorf("project 'overlay-files': %w", err) + } + + for name, match := range byName { + if existing, ok := result[name]; ok { + return nil, fmt.Errorf( + "%w: component %#q is discovered by both %#q and %#q; "+ + "restrict one of the patterns or rename one of the directories", + ErrPatternDiscoveryCollision, name, existing.SourcePattern, match.SourcePattern, + ) + } + + result[name] = match + } + } + + return result, nil +} + +// expandSinglePattern expands a single validated pattern and groups matched +// files by captured component name. +func expandSinglePattern(env *azldev.Env, pattern string) (map[string]*patternDiscoveredComponent, error) { + prefix, suffix := projectconfig.SplitOverlayFilesPlaceholder(pattern) + + // Substitute a single-segment wildcard for {component} to build the actual + // glob. Because {component} is validated as a whole path segment (see + // ConfigFile.Validate), replacing it with "*" produces a well-formed + // doublestar pattern that captures exactly one path segment. + globPattern := prefix + "*" + suffix + + // doublestar.WithFailOnIOErrors() is intentionally not set, mirroring + // findComponentGroupSpecPaths: we may legitimately hit unreadable subtrees + // under the project root. Non-IO errors (bad pattern, etc.) are still + // returned. + matches, err := fileutils.Glob(env.FS(), globPattern, doublestar.WithFilesOnly()) + if err != nil { + return nil, fmt.Errorf("failed to expand pattern %#q:\n%w", pattern, err) + } + + byName := make(map[string]*patternDiscoveredComponent) + + for _, match := range matches { + name, referenceDir, err := extractCapturedName(pattern, prefix, match) + if err != nil { + // Skip matches we can't parse (should not happen with a validated + // pattern, but guard anyway). + slog.Debug("pattern-discovery: skipping unparseable match", + "pattern", pattern, "match", match, "err", err) + + continue + } + + // Reject matches whose captured segment is not a safe component name + // (spaces, path traversal, absolute paths, etc). Downstream code uses + // the name as a filesystem path fragment (see fileutils.ValidateFilename) + // so we fail fast rather than let it explode later. + if err := fileutils.ValidateFilename(name); err != nil { + slog.Debug("pattern-discovery: skipping match with unsafe captured name", + "pattern", pattern, "match", match, "name", name, "err", err) + + continue + } + + bucket, ok := byName[name] + if !ok { + bucket = &patternDiscoveredComponent{ + Name: name, + ReferenceDir: referenceDir, + SourcePattern: pattern, + } + byName[name] = bucket + } + + bucket.OverlayFiles = append(bucket.OverlayFiles, match) + } + + // Sort the overlay-files list per component (filename first, then full path). + for _, bucket := range byName { + slices.SortFunc(bucket.OverlayFiles, func(left, right string) int { + if result := strings.Compare(filepath.Base(left), filepath.Base(right)); result != 0 { + return result + } + + return strings.Compare(left, right) + }) + } + + return byName, nil +} + +// extractCapturedName strips prefix from match and returns the first path +// segment (the captured component name) together with the concrete directory +// that captured segment lives in. The returned ReferenceDir is the natural +// per-component anchor for downstream operations. +func extractCapturedName(pattern, prefix, match string) (name, referenceDir string, err error) { + rel := match + if prefix != "" { + if !strings.HasPrefix(match, prefix) { + return "", "", fmt.Errorf( + "match %#q does not start with expected prefix %#q for pattern %#q", + match, prefix, pattern, + ) + } + + rel = match[len(prefix):] + } + + // The captured segment is everything up to (but not including) the first + // path separator in the residual, or the entire residual if there is no + // suffix. Patterns are validated to use '/' only. + sep := strings.IndexByte(rel, '/') + if sep < 0 { + name = rel + } else { + name = rel[:sep] + } + + if name == "" { + return "", "", fmt.Errorf( + "empty captured component name for match %#q against pattern %#q", + match, pattern, + ) + } + + // The reference dir is the prefix + captured segment (no trailing separator). + // Use path.Clean (POSIX '/') rather than filepath.Clean so ReferenceDir stays + // consistent with the placeholder contract, which validates and operates on + // POSIX-style separators only (filepath.Clean would emit '\' on Windows). + referenceDir = path.Clean(prefix + name) + + return name, referenceDir, nil +} + +// logShadowedByDeclaration emits a warning when an explicit component +// declaration overrides pattern-discovered overlay files for the same name. +func logShadowedByDeclaration(shadowed *patternDiscoveredComponent) { + slog.Warn( + "project 'overlay-files' pattern match shadowed by explicit component declaration; overlay files will not be applied", + "component", shadowed.Name, + "shadowed_pattern", shadowed.SourcePattern, + "shadowed_files", shadowed.OverlayFiles, + ) +} + +// ErrPatternDiscoveryCollision wraps project-scope pattern collision errors so +// tests can assert on them. +var ErrPatternDiscoveryCollision = errors.New("project 'overlay-files' pattern collision") diff --git a/internal/app/azldev/core/components/pattern_discovery_test.go b/internal/app/azldev/core/components/pattern_discovery_test.go new file mode 100644 index 000000000..5b18d37ea --- /dev/null +++ b/internal/app/azldev/core/components/pattern_discovery_test.go @@ -0,0 +1,191 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +package components_test + +import ( + "testing" + + "github.com/microsoft/azure-linux-dev-tools/internal/app/azldev/core/components" + "github.com/microsoft/azure-linux-dev-tools/internal/app/azldev/core/testutils" + "github.com/microsoft/azure-linux-dev-tools/internal/projectconfig" + "github.com/microsoft/azure-linux-dev-tools/internal/utils/fileperms" + "github.com/microsoft/azure-linux-dev-tools/internal/utils/fileutils" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +const patternTestOverlayContent = ` +[metadata] +category = "azl-branding-policy" + +[[overlays]] +description = "Bump release" +type = "spec-search-replace" +section = "%install" +regex = "unused" +` + +// writeOverlayFile writes a minimal but valid overlay TOML at path. +func writeOverlayFile(t *testing.T, env *testutils.TestEnv, path string) { + t.Helper() + + require.NoError(t, fileutils.MkdirAll(env.TestFS, dirOf(path))) + require.NoError(t, fileutils.WriteFile( + env.TestFS, path, []byte(patternTestOverlayContent), fileperms.PrivateFile, + )) +} + +func dirOf(path string) string { + for i := len(path) - 1; i >= 0; i-- { + if path[i] == '/' { + return path[:i] + } + } + + return "." +} + +func TestFindAllComponents_PatternDiscoveryAtProjectScope(t *testing.T) { + env := testutils.NewTestEnv(t) + + env.Config.DefaultComponentConfig.OverlayFiles = []string{ + "/project/comps/{component}/overlays/*.overlay.toml", + } + + writeOverlayFile(t, env, "/project/comps/foo/overlays/0001.overlay.toml") + writeOverlayFile(t, env, "/project/comps/foo/overlays/0002.overlay.toml") + writeOverlayFile(t, env, "/project/comps/bar/overlays/0001.overlay.toml") + + all, err := components.NewResolver(env.Env).FindAllComponents() + require.NoError(t, err) + + assert.ElementsMatch(t, []string{"foo", "bar"}, all.Names()) + + foo, ok := all.TryGet("foo") + require.True(t, ok) + assert.Len(t, foo.GetConfig().Overlays, 2, "both foo overlay files should attach") + + bar, ok := all.TryGet("bar") + require.True(t, ok) + assert.Len(t, bar.GetConfig().Overlays, 1) +} + +// Declared components with no 'overlay-files' inherit the project pattern; the +// placeholder is substituted with the component's own name at glob time. +func TestFindAllComponents_DeclaredComponentInheritsProjectPattern(t *testing.T) { + env := testutils.NewTestEnv(t) + + env.Config.DefaultComponentConfig.OverlayFiles = []string{ + "/project/comps/{component}/overlays/*.overlay.toml", + } + + writeOverlayFile(t, env, "/project/comps/foo/overlays/0001.overlay.toml") + writeOverlayFile(t, env, "/project/comps/foo/overlays/0002.overlay.toml") + + env.Config.Components["foo"] = projectconfig.ComponentConfig{Name: "foo"} + + all, err := components.NewResolver(env.Env).FindAllComponents() + require.NoError(t, err) + + foo, ok := all.TryGet("foo") + require.True(t, ok) + assert.Len(t, foo.GetConfig().Overlays, 2, + "foo should inherit the project pattern with {component}=foo") +} + +// A component that sets its own 'overlay-files' replaces the inherited project +// pattern wholesale; the placeholder pattern no longer applies. +func TestFindAllComponents_ComponentOverlayFilesReplacesInherited(t *testing.T) { + env := testutils.NewTestEnv(t) + + env.Config.DefaultComponentConfig.OverlayFiles = []string{ + "/project/comps/{component}/overlays/*.overlay.toml", + } + + // This file would match the inherited pattern (component=bar). + writeOverlayFile(t, env, "/project/comps/bar/overlays/0001.overlay.toml") + + // But this file lives elsewhere and is the only one bar's config points to. + writeOverlayFile(t, env, "/project/custom/bar/0001.overlay.toml") + + env.Config.Components["bar"] = projectconfig.ComponentConfig{ + Name: "bar", + OverlayFiles: []string{"/project/custom/bar/*.overlay.toml"}, + } + + all, err := components.NewResolver(env.Env).FindAllComponents() + require.NoError(t, err) + + bar, ok := all.TryGet("bar") + require.True(t, ok) + assert.Len(t, bar.GetConfig().Overlays, 1, + "bar overrides overlay-files wholesale; inherited pattern must not apply") +} + +// Empty per-component 'overlay-files' disables the inherited project pattern. +func TestFindAllComponents_ComponentOverlayFilesEmptyDisablesInheritance(t *testing.T) { + env := testutils.NewTestEnv(t) + + env.Config.DefaultComponentConfig.OverlayFiles = []string{ + "/project/comps/{component}/overlays/*.overlay.toml", + } + + writeOverlayFile(t, env, "/project/comps/baz/overlays/0001.overlay.toml") + + env.Config.Components["baz"] = projectconfig.ComponentConfig{ + Name: "baz", + OverlayFiles: []string{}, + } + + all, err := components.NewResolver(env.Env).FindAllComponents() + require.NoError(t, err) + + baz, ok := all.TryGet("baz") + require.True(t, ok) + assert.Empty(t, baz.GetConfig().Overlays, + "empty overlay-files must disable inherited pattern") +} + +func TestFindAllComponents_PatternDiscoverySameScopeCollisionErrors(t *testing.T) { + env := testutils.NewTestEnv(t) + + env.Config.DefaultComponentConfig.OverlayFiles = []string{ + "/project/a/{component}/*.overlay.toml", + "/project/b/{component}/*.overlay.toml", + } + + writeOverlayFile(t, env, "/project/a/dup/0001.overlay.toml") + writeOverlayFile(t, env, "/project/b/dup/0002.overlay.toml") + + _, err := components.NewResolver(env.Env).FindAllComponents() + require.Error(t, err) + require.ErrorIs(t, err, components.ErrPatternDiscoveryCollision, + "expected ErrPatternDiscoveryCollision, got %v", err) + assert.Contains(t, err.Error(), "dup") +} + +// A pattern match whose captured segment isn't a safe component name +// (spaces, dot-directory, etc.) is skipped without failing overall discovery. +func TestFindAllComponents_PatternDiscoverySkipsUnsafeCapturedNames(t *testing.T) { + env := testutils.NewTestEnv(t) + + env.Config.DefaultComponentConfig.OverlayFiles = []string{ + "/project/comps/{component}/overlays/*.overlay.toml", + } + + // Well-formed match — should be discovered. + writeOverlayFile(t, env, "/project/comps/openssl/overlays/0001.overlay.toml") + + // Unsafe captured segments — should be skipped without erroring the whole + // discovery. These simulate directories a user might create that don't + // correspond to valid component names. + writeOverlayFile(t, env, "/project/comps/has space/overlays/0001.overlay.toml") + writeOverlayFile(t, env, "/project/comps/./overlays/0001.overlay.toml") + + all, err := components.NewResolver(env.Env).FindAllComponents() + require.NoError(t, err) + + assert.ElementsMatch(t, []string{"openssl"}, all.Names(), + "only the safely-named directory should be discovered") +} diff --git a/internal/app/azldev/core/components/resolver.go b/internal/app/azldev/core/components/resolver.go index fad536a40..0a891a832 100644 --- a/internal/app/azldev/core/components/resolver.go +++ b/internal/app/azldev/core/components/resolver.go @@ -7,10 +7,12 @@ import ( "errors" "fmt" "log/slog" + "maps" "path" "path/filepath" "slices" "strings" + "sync" "github.com/bmatcuk/doublestar/v4" "github.com/microsoft/azure-linux-dev-tools/internal/app/azldev" @@ -42,13 +44,22 @@ type Resolver struct { // Only 'component update' enables this — it uses the freshness status to // decide which components need re-resolution and which can be skipped. CheckFreshness bool + + // patternDiscovery lazily holds the result of pattern-based component + // discovery so repeated resolver calls don't re-glob the filesystem. + patternDiscovery func() (map[string]*patternDiscoveredComponent, error) } // NewResolver constructs a new [Resolver] for the given environment. func NewResolver(env *azldev.Env) *Resolver { - return &Resolver{ + resolver := &Resolver{ env: env, } + resolver.patternDiscovery = sync.OnceValues(func() (map[string]*patternDiscoveredComponent, error) { + return discoverComponentsFromPatterns(env) + }) + + return resolver } // Given a component filter, finds all components defined in the environment that match the filter. @@ -109,7 +120,14 @@ func (r *Resolver) FindComponents(filter *ComponentFilter) (components *Componen return components, r.validateLockFiles(components, false, skipValidation) } -// Finds *all* components defined in the environment. +// Finds *all* components defined in the environment. This unions three sources: +// - components reached via component-group spec-path patterns, +// - loose components declared in the project config, and +// - components discovered by a project-level 'overlay-files' entry containing +// the '{component}' placeholder. +// +// Explicit declarations shadow pattern-discovered matches of the same name; a +// warning is emitted for the shadowed entry. func (r *Resolver) FindAllComponents() (components *ComponentSet, err error) { components = NewComponentSet() @@ -167,6 +185,49 @@ func (r *Resolver) FindAllComponents() (components *ComponentSet, err error) { components.Add(comp) } + // Finally, add pattern-discovered components (from project-level 'overlay-files' + // entries containing '{component}') that were not already covered by an explicit + // component declaration. When an explicit declaration exists but doesn't set its + // own overlay-files, it inherits the project pattern and still receives the + // discovered files (no warning). A warning is only emitted when the explicit + // declaration replaces overlay-files with a non-empty list, genuinely displacing + // the discovered files. An explicit empty list ('overlay-files = []') is the + // documented off-switch and is not treated as shadowing. + discovered, err := r.patternDiscoveredComponents() + if err != nil { + return components, err + } + + for _, name := range slices.Sorted(maps.Keys(discovered)) { + entry := discovered[name] + + if components.Contains(name) { + // Only warn when the explicit declaration provides a non-empty + // 'overlay-files' list, which replaces inheritance with a different + // set of files (see [projectconfig.ComponentConfig.MergeUpdatesFrom]). + // A declaration with no 'overlay-files' inherits the pattern and the + // discovered files still apply. An explicit empty list is the + // documented way to opt out of overlays for a component and is not + // treated as shadowing. + // + // 'overlay-files' can only be set at project or component scope + // (validated by [projectconfig.ConfigFile.validateOverlayFilesByScope]), + // so a shadowing declaration always lives in Config().Components. + if declared, ok := r.env.Config().Components[name]; ok && len(declared.OverlayFiles) > 0 { + logShadowedByDeclaration(entry) + } + + continue + } + + comp, addErr := r.componentFromPatternDiscovered(entry, nil) + if addErr != nil { + return components, addErr + } + + components.Add(comp) + } + return components, nil } @@ -274,6 +335,10 @@ func (r *Resolver) GetComponentGroupByName(componentGroupName string) (component componentGroup.Components = append(componentGroup.Components, groupEntry) } + // Pattern-discovered components (from a project-level 'overlay-files' entry + // containing '{component}') do not belong to any specific component group; + // they are added to the project-wide component set in FindAllComponents. + return componentGroup, nil } @@ -892,6 +957,35 @@ func componentGroupNames(env *azldev.Env, componentName string, extraGroupNames return groupNames } +// patternDiscoveredComponents returns the (lazily-computed) map of components +// discovered by a project-level 'overlay-files' entry containing the +// '{component}' placeholder. Returns an empty (non-nil) map and nil error when +// the project default-component-config declares no such entry. +func (r *Resolver) patternDiscoveredComponents() (map[string]*patternDiscoveredComponent, error) { + return r.patternDiscovery() +} + +// componentFromPatternDiscovered synthesizes a [Component] for a pattern +// discovered entry, applying inherited defaults with the entry's captured +// directory as the overlay-files reference dir. +func (r *Resolver) componentFromPatternDiscovered( + entry *patternDiscoveredComponent, groupNames []string, +) (Component, error) { + synth := projectconfig.ComponentConfig{ + Name: entry.Name, + OverlayFiles: slices.Clone(entry.OverlayFiles), + } + + updated, err := applyInheritedDefaultsToComponent(r.env, synth, entry.ReferenceDir, groupNames) + if err != nil { + return nil, fmt.Errorf( + "applying defaults to pattern-discovered component %#q:\n%w", entry.Name, err, + ) + } + + return r.createComponentFromConfig(updated) +} + // validateLockFiles checks lock file consistency against the resolved component // set. Skipped when skipValidation is true (set per-filter or via the global // '--skip-lock-validation' flag). diff --git a/internal/projectconfig/component.go b/internal/projectconfig/component.go index 6b56f9d8e..e6c5d6709 100644 --- a/internal/projectconfig/component.go +++ b/internal/projectconfig/component.go @@ -408,20 +408,42 @@ type ComponentConfig struct { // Overlays to apply to sources after they've been acquired. May mutate the spec as well as sources. Overlays []ComponentOverlay `toml:"overlays,omitempty" json:"overlays,omitempty" table:"-" jsonschema:"title=Overlays,description=Overlays to apply to this component's spec and/or sources"` - // OverlayFiles, if set, lists path or glob patterns (relative to this component config file) - // matched against the filesystem after component config resolution to locate per-file overlay documents. - // Each matched file is parsed as an [OverlayFile]: one logical change consisting of a - // file-level `[metadata]` block plus an ordered list of `[[overlays]]`. The per-file - // metadata is stamped onto every overlay in the file. Matches are concatenated in the - // order patterns are declared; within a single pattern, matches are applied in - // filename (lexicographic) order, using the full path as a tie-breaker when - // filenames match. Duplicate matches are de-duplicated, preserving first - // occurrence. The resulting overlays are appended to [ComponentConfig.Overlays] - // after any inline overlays. A value set in a higher-priority config layer replaces - // lower-priority overlay-files values; an explicit empty list disables inherited - // overlay files. Excluded from the fingerprint because the value affects only where - // overlays are sourced from, not their content. - OverlayFiles []string `toml:"overlay-files,omitempty" json:"overlayFiles,omitempty" table:"-" validate:"dive,required" jsonschema:"title=Overlay files,description=Path or glob patterns (relative to the component config file or matched spec directory) matched against the filesystem to locate per-file overlay documents after component config resolution. Use an empty list to disable inherited overlay-file patterns" fingerprint:"-"` + // OverlayFiles lists path or glob patterns matched against the filesystem after + // component config resolution to locate per-file overlay documents. Each matched + // file is parsed as an [OverlayFile]: one logical change consisting of a file-level + // `[metadata]` block plus an ordered list of `[[overlays]]`. The per-file metadata + // is stamped onto every overlay in the file. Matches are concatenated in the order + // entries are declared; within a single entry, matches are applied in filename + // (lexicographic) order, using the full path as a tie-breaker when filenames match. + // Duplicate matches are de-duplicated, preserving first occurrence. The resulting + // overlays are appended to [ComponentConfig.Overlays] after any inline overlays. + // + // A value set in a higher-priority config layer replaces lower-priority values + // wholesale; an explicit empty list disables inherited overlay files. + // + // `overlay-files` may only be set at project scope (the top-level + // `default-component-config`) or at component scope (`[components.X]`). + // Distro-version and component-group `default-component-config` tables must + // not set this field: broad-scope defaults for overlay files silently + // displace project-level discovery and per-component overrides, so we + // require callers to place the list at project or component scope where the + // intent is unambiguous. + // + // At project scope, every entry MUST contain the `{component}` placeholder exactly + // once as a whole path segment: those entries drive component discovery (each + // captured segment becomes an implicit component) and, when inherited by an explicit + // component that does not override, `{component}` is substituted with that + // component's name at glob time. Two project-scope entries that produce the same + // component name is a hard error. + // + // At component scope, entries are plain globs interpreted relative to the + // declaring config file (or the matched spec directory); `{component}` is + // not allowed. Setting an empty list on a component (`overlay-files = []`) + // disables inheritance from the project default. + // + // Excluded from the fingerprint because the value affects only where overlays are + // sourced from, not their content. + OverlayFiles []string `toml:"overlay-files,omitempty" json:"overlayFiles,omitempty" table:"-" validate:"dive,required" jsonschema:"title=Overlay files,description=Path or glob patterns matched against the filesystem to locate per-file overlay documents. Only settable at project scope (top-level 'default-component-config') or at component scope ('[components.X]'). At project scope every entry must contain the {component} placeholder as a whole path segment (drives discovery and per-component substitution). At component scope {component} is not allowed and entries are plain globs relative to the config file. Distro-version and component-group 'default-component-config' tables must not set this field. Use an empty list at component scope to disable inheritance." fingerprint:"-"` // Configuration for building the component. Build ComponentBuildConfig `toml:"build,omitempty" json:"build,omitempty" table:"-" jsonschema:"title=Build configuration,description=Configuration for building the component"` @@ -637,10 +659,15 @@ func (c *ComponentConfig) WithAbsolutePaths(referenceDir string) *ComponentConfi SourceFiles: deep.MustCopy(c.SourceFiles), Packages: deep.MustCopy(c.Packages), Publish: deep.MustCopy(c.Publish), - // OverlayFiles is consumed after component config resolution; preserve it verbatim - // here so inherited patterns can be interpreted relative to the concrete component - // config file. - OverlayFiles: slices.Clone(c.OverlayFiles), + // OverlayFiles: entries with the {component} placeholder are project-scope + // discovery patterns and are absolutized now so downstream code sees stable + // absolute paths regardless of where the config was inherited from. Non-placeholder + // entries are left verbatim so component-scope patterns are still interpreted + // relative to the concrete component config file at glob time. (Project-scope + // entries must all contain the placeholder; distro-version and component-group + // scopes may not set overlay-files at all — see + // [ConfigFile.validateOverlayFilesByScope].) + OverlayFiles: absolutizeOverlayFilesPlaceholderEntries(referenceDir, c.OverlayFiles), Tests: deep.MustCopy(c.Tests), } diff --git a/internal/projectconfig/configfile.go b/internal/projectconfig/configfile.go index ee6f07c73..5db9b88da 100644 --- a/internal/projectconfig/configfile.go +++ b/internal/projectconfig/configfile.go @@ -111,6 +111,10 @@ func (f ConfigFile) Validate() error { } } + if err := f.validateOverlayFilesByScope(); err != nil { + return err + } + // Validate component group metadata. if err := validateComponentGroupMetadata(f.ComponentGroups); err != nil { return err @@ -163,6 +167,12 @@ func validateComponentConfigs( customScriptPathsResolved bool, ) error { for componentName, component := range components { + if err := validateOverlayFilesEntries( + component.OverlayFiles, validateComponentOverlayFilesEntry, + ); err != nil { + return fmt.Errorf("invalid 'overlay-files' for component %#q:\n%w", componentName, err) + } + for i, overlay := range component.Overlays { err := overlay.Validate() if err != nil { @@ -516,6 +526,96 @@ func dedupTestRefSKUGroups( return nil } +// validateOverlayFilesByScope enforces where 'overlay-files' may appear. +// +// - Project scope (top-level 'default-component-config'): each entry must +// contain the '{component}' placeholder as a whole path segment; the +// placeholder drives pattern-based component discovery. +// - Component scope ('[components.X]'): plain globs; the '{component}' +// placeholder is forbidden (the name is already fixed). +// - Distro-version and component-group 'default-component-config': +// 'overlay-files' is not allowed at all. Broad-scope defaults for this +// field silently displace project-level discovery and per-component +// overrides, so we require callers to place the list at project or +// component scope where the intent is unambiguous. +func (f ConfigFile) validateOverlayFilesByScope() error { + if f.DefaultComponentConfig != nil { + if err := validateOverlayFilesEntries( + f.DefaultComponentConfig.OverlayFiles, validateProjectOverlayFilesEntry, + ); err != nil { + return fmt.Errorf("invalid project 'default-component-config':\n%w", err) + } + } + + for distroName, distro := range f.Distros { + for versionName, version := range distro.Versions { + if version.DefaultComponentConfig.OverlayFiles != nil { + return fmt.Errorf( + "invalid 'default-component-config' for distro %#q version %#q:\n"+ + "%w: 'overlay-files' is only allowed on the project-level "+ + "'default-component-config' (with '%s' patterns) or on individual "+ + "'[components.X]' entries", + distroName, versionName, + ErrInvalidOverlayFilesEntry, OverlayFilesComponentPlaceholder, + ) + } + } + } + + for groupName, group := range f.ComponentGroups { + if group.DefaultComponentConfig.OverlayFiles != nil { + return fmt.Errorf( + "invalid 'default-component-config' for component group %#q:\n"+ + "%w: 'overlay-files' is only allowed on the project-level "+ + "'default-component-config' (with '%s' patterns) or on individual "+ + "'[components.X]' entries", + groupName, + ErrInvalidOverlayFilesEntry, OverlayFilesComponentPlaceholder, + ) + } + } + + return nil +} + +// overlayFilesValidator validates a single 'overlay-files' entry against +// the placeholder rules of a particular config scope. +type overlayFilesValidator func(entry string) error + +// validateProjectOverlayFilesEntry validates entries in the project-level +// default-component-config. Every entry MUST contain the '{component}' +// placeholder exactly once as a whole path segment — it's the discovery +// mechanism. +func validateProjectOverlayFilesEntry(entry string) error { + return validateOverlayFilesPlaceholder(entry) +} + +// validateComponentOverlayFilesEntry validates entries in a '[components.X]' +// table. Entries are plain globs; '{component}' is forbidden because the +// component name is already fixed by the table key. +func validateComponentOverlayFilesEntry(entry string) error { + if !hasOverlayFilesPlaceholder(entry) { + return nil + } + + return fmt.Errorf( + "%w: %q is only allowed in project-level 'default-component-config' entries; entry %#q", + ErrInvalidOverlayFilesEntry, OverlayFilesComponentPlaceholder, entry, + ) +} + +// validateOverlayFilesEntries runs validate against each entry in overlayFiles, +// annotating the returned error with the offending index. +func validateOverlayFilesEntries(overlayFiles []string, validate overlayFilesValidator) error { + for idx, entry := range overlayFiles { + if err := validate(entry); err != nil { + return fmt.Errorf("'overlay-files'[%d]: %w", idx, err) + } + } + + return nil +} + // validateSourceFiles checks 'source-files' configuration for a component: // - All filenames must be unique. // - Hash type must be a supported algorithm when specified. diff --git a/internal/projectconfig/configfile_test.go b/internal/projectconfig/configfile_test.go index b10216d17..2dcc27cf5 100644 --- a/internal/projectconfig/configfile_test.go +++ b/internal/projectconfig/configfile_test.go @@ -1709,3 +1709,123 @@ func TestValidateCustomSourceRef_InvalidInputFilename(t *testing.T) { assert.Contains(t, err.Error(), "'inputs'") assert.Contains(t, err.Error(), "escape.tar.gz") } + +func TestProjectConfigFileValidation_OverlayFilesPlaceholderOnProjectDefault(t *testing.T) { + file := projectconfig.ConfigFile{ + DefaultComponentConfig: &projectconfig.ComponentConfig{ + OverlayFiles: []string{"comps/{component}/overlays/*.overlay.toml"}, + }, + } + assert.NoError(t, file.Validate()) +} + +func TestProjectConfigFileValidation_OverlayFilesOnDistroDefaultRejected(t *testing.T) { + file := projectconfig.ConfigFile{ + Distros: map[string]projectconfig.DistroDefinition{ + "my-distro": { + Versions: map[string]projectconfig.DistroVersionDefinition{ + "1.0": { + ReleaseVer: "1.0", + DefaultComponentConfig: projectconfig.ComponentConfig{ + OverlayFiles: []string{"distro/*.overlay.toml"}, + }, + }, + }, + }, + }, + } + err := file.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "my-distro") + assert.Contains(t, err.Error(), "overlay-files") +} + +func TestProjectConfigFileValidation_OverlayFilesOnGroupDefaultRejected(t *testing.T) { + file := projectconfig.ConfigFile{ + ComponentGroups: map[string]projectconfig.ComponentGroupConfig{ + "core": { + DefaultComponentConfig: projectconfig.ComponentConfig{ + OverlayFiles: []string{"core/*.overlay.toml"}, + }, + }, + }, + } + err := file.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "core") + assert.Contains(t, err.Error(), "overlay-files") +} + +func TestProjectConfigFileValidation_OverlayFilesPlaceholderOnComponentRejected(t *testing.T) { + file := projectconfig.ConfigFile{ + Components: map[string]projectconfig.ComponentConfig{ + "my-comp": { + OverlayFiles: []string{"comps/{component}/overlays/*.overlay.toml"}, + }, + }, + } + err := file.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "my-comp") + assert.Contains(t, err.Error(), "{component}") +} + +func TestProjectConfigFileValidation_PlainGlobOnProjectDefaultRejected(t *testing.T) { + file := projectconfig.ConfigFile{ + DefaultComponentConfig: &projectconfig.ComponentConfig{ + OverlayFiles: []string{"comps/no-placeholder/*.overlay.toml"}, + }, + } + err := file.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "invalid project 'default-component-config'") + assert.Contains(t, err.Error(), "{component}") +} + +func TestProjectConfigFileValidation_MalformedPlaceholderOnProjectDefault(t *testing.T) { + file := projectconfig.ConfigFile{ + DefaultComponentConfig: &projectconfig.ComponentConfig{ + OverlayFiles: []string{"prefix-{component}/*.overlay.toml"}, + }, + } + err := file.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "whole path segment") +} + +func TestProjectConfigFileValidation_EmptyOverlayFilesOnDistroDefaultRejected(t *testing.T) { + file := projectconfig.ConfigFile{ + Distros: map[string]projectconfig.DistroDefinition{ + "my-distro": { + Versions: map[string]projectconfig.DistroVersionDefinition{ + "1.0": { + ReleaseVer: "1.0", + DefaultComponentConfig: projectconfig.ComponentConfig{ + OverlayFiles: []string{}, + }, + }, + }, + }, + }, + } + err := file.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "my-distro") + assert.Contains(t, err.Error(), "overlay-files") +} + +func TestProjectConfigFileValidation_EmptyOverlayFilesOnGroupDefaultRejected(t *testing.T) { + file := projectconfig.ConfigFile{ + ComponentGroups: map[string]projectconfig.ComponentGroupConfig{ + "core": { + DefaultComponentConfig: projectconfig.ComponentConfig{ + OverlayFiles: []string{}, + }, + }, + }, + } + err := file.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "core") + assert.Contains(t, err.Error(), "overlay-files") +} diff --git a/internal/projectconfig/overlay_file.go b/internal/projectconfig/overlay_file.go index eb04ff258..82ebccdb7 100644 --- a/internal/projectconfig/overlay_file.go +++ b/internal/projectconfig/overlay_file.go @@ -56,6 +56,11 @@ type OverlayFile struct { // [OverlayFile], stamps the per-file metadata onto each overlay, appends the // resulting overlays after inline overlays, and clears [ComponentConfig.OverlayFiles] // so the returned config is not expanded twice. +// +// Entries containing [OverlayFilesComponentPlaceholder] have the placeholder +// substituted with the component's own name before globbing; those entries are +// inherited from a project-level default and become concrete per-component +// globs here. func ExpandResolvedOverlayFiles( fs opctx.FS, component ComponentConfig, referenceDir string, permissiveConfigParsing bool, ) (ComponentConfig, error) { @@ -63,14 +68,19 @@ func ExpandResolvedOverlayFiles( return component, nil } - if referenceDir == "" && overlayFilesHasRelativePattern(component.OverlayFiles) { + expanded := make([]string, len(component.OverlayFiles)) + for i, entry := range component.OverlayFiles { + expanded[i] = substituteOverlayFilesPlaceholder(entry, component.Name) + } + + if referenceDir == "" && overlayFilesHasRelativePattern(expanded) { return ComponentConfig{}, fmt.Errorf( "component %#q has relative 'overlay-files' entries but no reference directory", component.Name, ) } - loaded, err := loadOverlayFiles(fs, referenceDir, component.OverlayFiles, permissiveConfigParsing) + loaded, err := loadOverlayFiles(fs, referenceDir, expanded, permissiveConfigParsing) if err != nil { return ComponentConfig{}, fmt.Errorf("component %#q overlay-files:\n%w", component.Name, err) } diff --git a/internal/projectconfig/overlay_files_placeholder.go b/internal/projectconfig/overlay_files_placeholder.go new file mode 100644 index 000000000..23578869e --- /dev/null +++ b/internal/projectconfig/overlay_files_placeholder.go @@ -0,0 +1,167 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +package projectconfig + +import ( + "errors" + "fmt" + "strings" + + "github.com/bmatcuk/doublestar/v4" +) + +// OverlayFilesComponentPlaceholder is the single supported placeholder in an +// [ComponentConfig.OverlayFiles] entry. It is required in every project-scope +// entry (discovery) and forbidden in distro/component-group/component-scope +// entries; when present it must appear exactly once and must be a whole path +// segment. +const OverlayFilesComponentPlaceholder = "{component}" + +// ErrInvalidOverlayFilesEntry is returned when an entry in +// [ComponentConfig.OverlayFiles] does not conform to the placeholder rules +// documented on that field. +var ErrInvalidOverlayFilesEntry = errors.New("invalid overlay-files entry") + +// hasOverlayFilesPlaceholder reports whether entry contains at least one +// occurrence of [OverlayFilesComponentPlaceholder]. +func hasOverlayFilesPlaceholder(entry string) bool { + return strings.Contains(entry, OverlayFilesComponentPlaceholder) +} + +// validateOverlayFilesPlaceholder verifies that entry contains exactly one +// [OverlayFilesComponentPlaceholder] and that the placeholder is a whole path +// segment. Path separators must be '/' (POSIX-style), and no glob +// metacharacters ('*', '?', '[') may appear before the placeholder because +// pattern-discovery extracts the captured component name by literal-prefix +// matching against the concrete glob result. Metacharacters after the +// placeholder are allowed. Returns nil when the entry is well-formed. +func validateOverlayFilesPlaceholder(entry string) error { + if entry == "" { + return fmt.Errorf("%w: entry is empty", ErrInvalidOverlayFilesEntry) + } + + count := strings.Count(entry, OverlayFilesComponentPlaceholder) + if count == 0 { + return fmt.Errorf( + "%w: entry %#q must contain %q exactly once", + ErrInvalidOverlayFilesEntry, entry, OverlayFilesComponentPlaceholder, + ) + } + + if count > 1 { + return fmt.Errorf( + "%w: entry %#q contains %q %d times; it must appear exactly once", + ErrInvalidOverlayFilesEntry, entry, OverlayFilesComponentPlaceholder, count, + ) + } + + if strings.ContainsRune(entry, '\\') { + return fmt.Errorf( + "%w: entry %#q must use %q as the path separator (backslash is not accepted)", + ErrInvalidOverlayFilesEntry, entry, "/", + ) + } + + idx := strings.Index(entry, OverlayFilesComponentPlaceholder) + if idx > 0 && entry[idx-1] != '/' { + return fmt.Errorf( + "%w: %q in entry %#q must be a whole path segment (preceded by %q or start of entry)", + ErrInvalidOverlayFilesEntry, OverlayFilesComponentPlaceholder, entry, "/", + ) + } + + tail := idx + len(OverlayFilesComponentPlaceholder) + if tail == len(entry) { + return fmt.Errorf( + "%w: %q in entry %#q must be followed by %q; it names a directory, not a file", + ErrInvalidOverlayFilesEntry, OverlayFilesComponentPlaceholder, entry, "/", + ) + } + + if entry[tail] != '/' { + return fmt.Errorf( + "%w: %q in entry %#q must be a whole path segment (followed by %q)", + ErrInvalidOverlayFilesEntry, OverlayFilesComponentPlaceholder, entry, "/", + ) + } + + if tail+1 == len(entry) { + return fmt.Errorf( + "%w: %q in entry %#q must be followed by at least one more path segment; "+ + "it names a directory, not a file", + ErrInvalidOverlayFilesEntry, OverlayFilesComponentPlaceholder, entry, + ) + } + + prefix := entry[:idx] + if globMetaIdx := strings.IndexAny(prefix, "*?["); globMetaIdx >= 0 { + return fmt.Errorf( + "%w: entry %#q must not contain glob metacharacters (%q) before %q; "+ + "the discovery capture uses a literal prefix", + ErrInvalidOverlayFilesEntry, entry, string(prefix[globMetaIdx]), OverlayFilesComponentPlaceholder, + ) + } + + // Substitute '*' for {component} to build the concrete glob that + // pattern-discovery will hand to doublestar. + globPattern := prefix + "*" + entry[tail:] + if !doublestar.ValidatePattern(globPattern) { + return fmt.Errorf( + "%w: entry %#q expands to invalid glob %#q", + ErrInvalidOverlayFilesEntry, entry, globPattern, + ) + } + + return nil +} + +// SplitOverlayFilesPlaceholder splits an entry at its +// [OverlayFilesComponentPlaceholder] and returns the prefix and suffix. The +// caller can then substitute a concrete component name (or a glob wildcard) +// for the placeholder. Passing an entry without a placeholder is a programming +// error and panics with a diagnostic message. +func SplitOverlayFilesPlaceholder(entry string) (prefix, suffix string) { + idx := strings.Index(entry, OverlayFilesComponentPlaceholder) + if idx < 0 { + panic(fmt.Sprintf( + "SplitOverlayFilesPlaceholder: entry %q does not contain %q", + entry, OverlayFilesComponentPlaceholder, + )) + } + + return entry[:idx], entry[idx+len(OverlayFilesComponentPlaceholder):] +} + +// substituteOverlayFilesPlaceholder returns entry with the single +// [OverlayFilesComponentPlaceholder] replaced by componentName. Entries with +// no placeholder are returned unchanged. +func substituteOverlayFilesPlaceholder(entry, componentName string) string { + if !hasOverlayFilesPlaceholder(entry) { + return entry + } + + return strings.Replace(entry, OverlayFilesComponentPlaceholder, componentName, 1) +} + +// absolutizeOverlayFilesPlaceholderEntries returns a copy of entries where every +// entry containing [OverlayFilesComponentPlaceholder] has been absolutized +// against referenceDir. Non-placeholder entries are cloned verbatim so they can +// still be resolved relative to a per-component reference directory later. +func absolutizeOverlayFilesPlaceholderEntries(referenceDir string, entries []string) []string { + if entries == nil { + return nil + } + + result := make([]string, len(entries)) + + for i, entry := range entries { + if hasOverlayFilesPlaceholder(entry) { + result[i] = makeAbsolute(referenceDir, entry) + } else { + result[i] = entry + } + } + + return result +} diff --git a/internal/projectconfig/overlay_files_placeholder_test.go b/internal/projectconfig/overlay_files_placeholder_test.go new file mode 100644 index 000000000..95b00707c --- /dev/null +++ b/internal/projectconfig/overlay_files_placeholder_test.go @@ -0,0 +1,115 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +//nolint:testpackage // Allow to test unexported helpers. +package projectconfig + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestValidateOverlayFilesPlaceholder(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + entry string + wantErr bool + }{ + {name: "empty", entry: "", wantErr: true}, + {name: "missing placeholder", entry: "comps/*/overlays/*.overlay.toml", wantErr: true}, + {name: "multiple placeholders", entry: "comps/{component}/{component}/x.toml", wantErr: true}, + {name: "placeholder is prefix substring", entry: "prefix-{component}/x.toml", wantErr: true}, + {name: "placeholder is suffix substring", entry: "{component}-suffix/x.toml", wantErr: true}, + {name: "backslash separator rejected", entry: "comps\\{component}\\x.toml", wantErr: true}, + {name: "glob metachar before placeholder", entry: "**/{component}/*.overlay.toml", wantErr: true}, + {name: "star before placeholder", entry: "comps*/{component}/*.overlay.toml", wantErr: true}, + {name: "whole segment at start", entry: "{component}/overlays/*.overlay.toml", wantErr: false}, + {name: "whole segment in middle", entry: "comps/{component}/overlays/*.overlay.toml", wantErr: false}, + {name: "placeholder at end rejected", entry: "comps/overlays/{component}", wantErr: true}, + {name: "placeholder followed by only slash rejected", entry: "comps/{component}/", wantErr: true}, + {name: "doublestar in suffix allowed", entry: "comps/{component}/**/*.overlay.toml", wantErr: false}, + {name: "absolute path", entry: "/project/base/comps/{component}/overlays/*.overlay.toml", wantErr: false}, + } + + for _, testCase := range tests { + t.Run(testCase.name, func(t *testing.T) { + t.Parallel() + + err := validateOverlayFilesPlaceholder(testCase.entry) + if testCase.wantErr { + require.Error(t, err) + require.ErrorIs(t, err, ErrInvalidOverlayFilesEntry, + "expected ErrInvalidOverlayFilesEntry, got %v", err) + } else { + assert.NoError(t, err) + } + }) + } +} + +func TestSplitOverlayFilesPlaceholder(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + entry string + wantPrefix string + wantSuffix string + }{ + { + name: "middle", + entry: "comps/{component}/overlays/*.overlay.toml", + wantPrefix: "comps/", + wantSuffix: "/overlays/*.overlay.toml", + }, + { + name: "start", + entry: "{component}/overlays/*.overlay.toml", + wantPrefix: "", + wantSuffix: "/overlays/*.overlay.toml", + }, + { + name: "end", + entry: "comps/overlays/{component}", + wantPrefix: "comps/overlays/", + wantSuffix: "", + }, + } + + for _, testCase := range tests { + t.Run(testCase.name, func(t *testing.T) { + t.Parallel() + + prefix, suffix := SplitOverlayFilesPlaceholder(testCase.entry) + assert.Equal(t, testCase.wantPrefix, prefix) + assert.Equal(t, testCase.wantSuffix, suffix) + }) + } +} + +func TestSplitOverlayFilesPlaceholder_PanicsWithoutPlaceholder(t *testing.T) { + t.Parallel() + + assert.PanicsWithValue(t, + `SplitOverlayFilesPlaceholder: entry "comps/foo/overlays/x.toml" does not contain "{component}"`, + func() { + SplitOverlayFilesPlaceholder("comps/foo/overlays/x.toml") + }) +} + +func TestSubstituteOverlayFilesPlaceholder(t *testing.T) { + t.Parallel() + + assert.Equal(t, + "comps/foo/overlays/*.toml", + substituteOverlayFilesPlaceholder("comps/{component}/overlays/*.toml", "foo")) + + // No placeholder → unchanged. + assert.Equal(t, + "overlays/*.toml", + substituteOverlayFilesPlaceholder("overlays/*.toml", "foo")) +} diff --git a/scenario/__snapshots__/TestSnapshotsContainer_config_generate-schema_stdout_1.snap b/scenario/__snapshots__/TestSnapshotsContainer_config_generate-schema_stdout_1.snap index 9cf47bd35..d71968059 100755 --- a/scenario/__snapshots__/TestSnapshotsContainer_config_generate-schema_stdout_1.snap +++ b/scenario/__snapshots__/TestSnapshotsContainer_config_generate-schema_stdout_1.snap @@ -130,7 +130,7 @@ }, "type": "array", "title": "Overlay files", - "description": "Path or glob patterns (relative to the component config file or matched spec directory) matched against the filesystem to locate per-file overlay documents after component config resolution. Use an empty list to disable inherited overlay-file patterns" + "description": "Path or glob patterns matched against the filesystem to locate per-file overlay documents. Only settable at project scope (top-level 'default-component-config') or at component scope ('[components.X]'). At project scope every entry must contain the {component} placeholder as a whole path segment (drives discovery and per-component substitution). At component scope {component} is not allowed and entries are plain globs relative to the config file. Distro-version and component-group 'default-component-config' tables must not set this field. Use an empty list at component scope to disable inheritance." }, "build": { "$ref": "#/$defs/ComponentBuildConfig", diff --git a/scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap b/scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap index 9cf47bd35..d71968059 100755 --- a/scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap +++ b/scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap @@ -130,7 +130,7 @@ }, "type": "array", "title": "Overlay files", - "description": "Path or glob patterns (relative to the component config file or matched spec directory) matched against the filesystem to locate per-file overlay documents after component config resolution. Use an empty list to disable inherited overlay-file patterns" + "description": "Path or glob patterns matched against the filesystem to locate per-file overlay documents. Only settable at project scope (top-level 'default-component-config') or at component scope ('[components.X]'). At project scope every entry must contain the {component} placeholder as a whole path segment (drives discovery and per-component substitution). At component scope {component} is not allowed and entries are plain globs relative to the config file. Distro-version and component-group 'default-component-config' tables must not set this field. Use an empty list at component scope to disable inheritance." }, "build": { "$ref": "#/$defs/ComponentBuildConfig", diff --git a/schemas/azldev.schema.json b/schemas/azldev.schema.json index 9cf47bd35..d71968059 100644 --- a/schemas/azldev.schema.json +++ b/schemas/azldev.schema.json @@ -130,7 +130,7 @@ }, "type": "array", "title": "Overlay files", - "description": "Path or glob patterns (relative to the component config file or matched spec directory) matched against the filesystem to locate per-file overlay documents after component config resolution. Use an empty list to disable inherited overlay-file patterns" + "description": "Path or glob patterns matched against the filesystem to locate per-file overlay documents. Only settable at project scope (top-level 'default-component-config') or at component scope ('[components.X]'). At project scope every entry must contain the {component} placeholder as a whole path segment (drives discovery and per-component substitution). At component scope {component} is not allowed and entries are plain globs relative to the config file. Distro-version and component-group 'default-component-config' tables must not set this field. Use an empty list at component scope to disable inheritance." }, "build": { "$ref": "#/$defs/ComponentBuildConfig", From 733fe5811d189bde28f8a43bacd5c092f54b00f1 Mon Sep 17 00:00:00 2001 From: Nan Liu Date: Thu, 12 Nov 2026 00:44:48 +0000 Subject: [PATCH 2/4] docs(overlays): explain declaration-free component discovery Document shared placeholder patterns and explicit-group declarations in both skill modes and the overlay reference. Update the rebased discovery fixture for required upstream-status metadata. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- docs/user/reference/config/overlays.md | 29 +++++++++++---- .../app/azldev/agentskill/agentskill_test.go | 31 ++++++++++++++++ .../agentskill/content/add-component.md.tmpl | 22 +++++++++--- .../agentskill/content/comp-toml.md.tmpl | 22 +++++++++--- .../agentskill/content/overlays.md.tmpl | 35 +++++++++++++++++-- .../withoutlockfile/add-component.md.tmpl | 22 +++++++++--- .../content/withoutlockfile/comp-toml.md.tmpl | 22 +++++++++--- .../content/withoutlockfile/overlays.md.tmpl | 35 +++++++++++++++++-- .../core/components/pattern_discovery_test.go | 1 + 9 files changed, 190 insertions(+), 29 deletions(-) diff --git a/docs/user/reference/config/overlays.md b/docs/user/reference/config/overlays.md index 996a8defc..c7694e589 100644 --- a/docs/user/reference/config/overlays.md +++ b/docs/user/reference/config/overlays.md @@ -309,16 +309,20 @@ Project-scope `overlay-files` entries **must** contain `{component}` exactly onc For each entry, `{component}` is expanded to `*` for globbing, and the captured path segment becomes the discovered component's name. The matched files are attached to that component (as if it had declared `overlay-files = []` explicitly). Explicit components with no `overlay-files` of their own inherit the same pattern and re-expand it with their own name substituted for `{component}`. ```toml -# components.toml (project-level) +# base/comps/components.toml (included at project scope) [default-component-config] -overlay-files = ["base/comps/{component}/overlays/*.overlay.toml"] +overlay-files = ["{component}/overlays/*.overlay.toml"] ``` -With the layout below, no per-component `[components.]` table is needed — `openssl` and `curl` are discovered automatically, each carrying its overlay files. +The pattern is relative to the file that declares it. If declared in the root +`azldev.toml` instead, use `base/comps/{component}/overlays/*.overlay.toml`. +With the layout below, no per-component `[components.]` table is needed — +`anaconda` and `curl` are discovered automatically, each carrying its overlay files. +This is the declaration-free case: do not create an empty `anaconda.comp.toml`. ``` base/comps/ -├── openssl/ +├── anaconda/ │ └── overlays/ │ ├── 0001-cve-2024-1234.overlay.toml │ └── 0002-disable-fips-tests.overlay.toml @@ -340,15 +344,26 @@ A component that sets its own `overlay-files = [...]` replaces the inherited val **Shadowing.** An explicit `[components.]` declaration supersedes pattern discovery for the same name. If that declaration also sets a non-empty `overlay-files` list (which replaces inheritance with a different set of files), `azldev` emits an `slog.Warn` naming the shadowed pattern so you can spot the genuine displacement. An explicit declaration that leaves `overlay-files` unset still inherits the project pattern, so the discovered files apply and no warning is emitted. An explicit empty list (`overlay-files = []`) is the documented off-switch for a component and is not treated as shadowing. -**Limitation: explicit component-group members need a `[components.]` table.** A component group can enumerate its members two ways: via a `specs` glob (spec files on disk) or via an explicit `components = ["", ...]` list. Only the *explicit list* interacts with pattern discovery: every name in that list must have a matching `[components.]` table in the project (validated at load time, error `ErrUndefinedComponent`). Even a one-liner is enough: +**Limitation: explicit component-group members need a `[components.]` table.** A component group can enumerate its members two ways: via a `specs` glob (spec files on disk) or via an explicit `components = ["", ...]` list. Every name in the *explicit list* must have a matching `[components.]` table in the project (validated at load time, error `ErrUndefinedComponent`); pattern discovery alone does not satisfy this validation. For example, if `anaconda` is explicitly listed, retain its declaration centrally rather than adding an empty dedicated file: ```toml -# base/comps/openssl/openssl.comp.toml -[components.openssl] +# base/comps/components.toml (included at project scope) +[default-component-config] +overlay-files = ["{component}/overlays/*.overlay.toml"] + +[components.anaconda] + +[component-groups.installer] +components = ["anaconda"] ``` Once the table exists, the component inherits the project-level `{component}` overlay pattern normally. Group members discovered via `specs` (not the explicit list) do **not** need a `.comp.toml` — the resolver synthesizes an empty config for them and inheritance still applies. +Keep a dedicated `.comp.toml` when it contains actual component-specific settings, +such as upstream pins, build options, inline overlays, or local-spec configuration. +Components without matching overlay files still need an explicit declaration or +another supported discovery mechanism, such as a group's `specs` glob. + ## Examples ### Adding a Build Dependency diff --git a/internal/app/azldev/agentskill/agentskill_test.go b/internal/app/azldev/agentskill/agentskill_test.go index e53c985a8..589ccc5b3 100644 --- a/internal/app/azldev/agentskill/agentskill_test.go +++ b/internal/app/azldev/agentskill/agentskill_test.go @@ -480,6 +480,37 @@ func TestFilesGitHubLayout(t *testing.T) { assert.Contains(t, mockFile.Content, "name: azldev-mock") } +func TestComponentSkillsExplainDeclarationFreeOverlays(t *testing.T) { + for _, mode := range []struct { + name string + withoutLockfile bool + }{ + {name: "default"}, + {name: "without-lockfile", withoutLockfile: true}, + } { + t.Run(mode.name, func(t *testing.T) { + catalog := agentskill.NewCatalog(mode.withoutLockfile) + + for _, name := range []string{"azldev-add-component", "azldev-comp-toml", "azldev-overlays"} { + t.Run(name, func(t *testing.T) { + doc, err := catalog.SkillDocument(name, testParams()) + require.NoError(t, err) + + doc = strings.Join(strings.Fields(doc), " ") + assert.Contains(t, doc, "name: "+name) + assert.Contains(t, doc, "base/comps/{component}/overlays/*.overlay.toml") + assert.Contains(t, doc, "do not create an empty") + assert.Contains(t, doc, "base/comps/components.toml") + assert.Contains(t, doc, "without matching overlay files") + assert.Contains(t, doc, "upstream pins, build options, inline overlays,") + assert.NotContains(t, doc, "Every component lives under") + assert.NotContains(t, doc, "anything that needs overlays") + }) + } + }) + } +} + // schemaEnum extracts values from an authoritative jsonschema enum tag. func schemaEnum(t *testing.T, structType reflect.Type, fieldName string) []string { t.Helper() diff --git a/internal/app/azldev/agentskill/content/add-component.md.tmpl b/internal/app/azldev/agentskill/content/add-component.md.tmpl index e21d45798..cbac1aa7b 100644 --- a/internal/app/azldev/agentskill/content/add-component.md.tmpl +++ b/internal/app/azldev/agentskill/content/add-component.md.tmpl @@ -39,7 +39,20 @@ dist-git often fail bot detection): 4. Read the spec and plan any overlays. -## Inline vs dedicated file +## Declaration-free, inline, or dedicated + +For components whose only customization is per-file overlays, add files under +`base/comps//overlays/`; do not create an empty `.comp.toml`. This requires +a shared project-level `default-component-config.overlay-files` pattern such as +`base/comps/{component}/overlays/*.overlay.toml` in the root config. The `{component}` +pattern discovers components from matching files and attaches their overlays. +Remove any provisional bare entry once discovery supplies it. + +If a component is explicitly listed in a component group, retain `[components.]` +in `base/comps/components.toml`: pattern discovery does not satisfy explicit group +membership validation. Components without matching overlay files still need an +explicit declaration or another supported discovery mechanism (such as a group's +`specs` glob). - **Inline** — a bare upstream import with no changes stays in a shared config file: @@ -47,9 +60,10 @@ dist-git often fail bot detection): [components.jq] ``` -- **Dedicated** — anything that needs overlays, build config, or a local spec gets its - own `/.comp.toml`. Rule of thumb: more than `[components.]` earns a - dedicated file. An `includes = ["**/*.comp.toml"]` glob picks it up automatically. +- **Dedicated** — keep `/.comp.toml` when it contains actual + component-specific settings, such as upstream pins, build options, inline overlays, + or local-spec configuration. An `includes = ["**/*.comp.toml"]` glob picks it up + automatically. `azldev comp add [...]` adds bare `[components.]` entries that inherit the distro defaults; it writes to the **root** config file and does not scaffold spec or diff --git a/internal/app/azldev/agentskill/content/comp-toml.md.tmpl b/internal/app/azldev/agentskill/content/comp-toml.md.tmpl index 0def4bf59..734da1c71 100644 --- a/internal/app/azldev/agentskill/content/comp-toml.md.tmpl +++ b/internal/app/azldev/agentskill/content/comp-toml.md.tmpl @@ -6,7 +6,8 @@ description: {{ printf "%q" .Description }} # Component definition files (`*.comp.toml`) A component definition tells azldev where a package's spec comes from and how to -customize it for your distro. Every component lives under `[components.]`. +customize it for your distro. Explicit definitions live under `[components.]`; +components can also be discovered from matching overlay files without a declaration. Get the authoritative, always-current field list from the schema: @@ -102,11 +103,22 @@ the rare spec whose `Source`/`Patch` filenames use macros the filter cannot expa ## File organization +- **Declaration-free** — for components whose only customization is per-file + overlays, add files under `base/comps//overlays/`; do not create an empty + `.comp.toml`. A shared project-level `default-component-config.overlay-files` + pattern such as `base/comps/{component}/overlays/*.overlay.toml` in the root config + discovers these components and attaches their overlays. +- **Explicit group members** — if a component is explicitly listed in a component + group's `components` list, retain `[components.]` in + `base/comps/components.toml`. Pattern discovery alone does not satisfy that + validation. Components without matching overlay files still need an explicit + declaration or another supported discovery mechanism (such as a group's `specs` + glob). - **Inline** — put simple, customization-free components directly in a shared config file (e.g. `[components.jq]`). -- **Dedicated** — give a component its own `/.comp.toml` once it needs - overlays, build config, or a local spec. Rule of thumb: anything more than - `[components.]` earns a dedicated file. +- **Dedicated** — keep `/.comp.toml` when it contains actual + component-specific settings, such as upstream pins, build options, inline overlays, + or local-spec configuration. - A parent config picks up dedicated files through an `includes` glob, for example `includes = ["**/*.comp.toml"]`. @@ -117,7 +129,7 @@ Start with `azldev comp list -p -q -O json`, then use For a change review, focus on the diff while checking enough surrounding context to ensure it fits the component and repository conventions. -- **Organization:** The component follows the repository's inline-versus-dedicated-file +- **Organization:** The component follows the repository's declaration-free, inline, or dedicated-file convention; its name matches upstream or sets `spec.upstream-name`; no stale or orphaned component files remain. - **Spec source:** The default upstream source is preferred. Pins explain why they are diff --git a/internal/app/azldev/agentskill/content/overlays.md.tmpl b/internal/app/azldev/agentskill/content/overlays.md.tmpl index 23b8b1121..a4fecd158 100644 --- a/internal/app/azldev/agentskill/content/overlays.md.tmpl +++ b/internal/app/azldev/agentskill/content/overlays.md.tmpl @@ -13,12 +13,12 @@ overwritten. ## The inner loop -Overlays live in the component's TOML config — inline `[[components..overlays]]` +Overlays can live in the component's TOML config — inline `[[components..overlays]]` entries, or per-file overlay documents referenced by the component's `overlay-files` glob. They apply **in order** and are **non-atomic**: if one fails part-way, the overlays before it stay applied. -1. Add or edit the overlay in the component config. +1. Add or edit the inline overlay or per-file overlay document. 2. Re-render and inspect the result: ```sh @@ -33,6 +33,37 @@ overlays before it stay applied. Config errors reference the offending overlay by its `description`, so give every overlay a short, specific `description`. +## Shared patterns and declaration-free components + +For components whose only customization is per-file overlays, add files under +`base/comps//overlays/`; do not create an empty `.comp.toml`. Configure +the shared pattern at project scope (paths are relative to the declaring config): + +```toml +# Root config, e.g. azldev.toml +[default-component-config] +overlay-files = ["base/comps/{component}/overlays/*.overlay.toml"] +``` + +The `{component}` segment discovers components from matching files and attaches +their overlays. For example, `base/comps/anaconda/overlays/0001-branding.overlay.toml` +discovers `anaconda` without `[components.anaconda]`. An explicit component with +no `overlay-files` of its own inherits the pattern; its own list replaces it, and +`overlay-files = []` disables inheritance. + +If a component is explicitly listed in a component group's `components` list, +retain `[components.]` in `base/comps/components.toml`: pattern discovery does +not satisfy explicit group membership validation. Components without matching +overlay files still need an explicit declaration or another supported discovery +mechanism (such as a group's `specs` glob). Keep a dedicated `.comp.toml` when it +contains actual component-specific settings, such as upstream pins, build options, +inline overlays, or local-spec configuration. + +Project-level `overlay-files` entries require `{component}` exactly once as a whole +directory segment, with no glob metacharacters before it and a file glob after it. +Use `/` separators. Component-level entries are plain globs (no placeholder); +distro-version and component-group defaults cannot set `overlay-files`. + ## Diagnose common failures Start with `azldev comp diff-sources -p ` to see the exact overlay effect. diff --git a/internal/app/azldev/agentskill/content/withoutlockfile/add-component.md.tmpl b/internal/app/azldev/agentskill/content/withoutlockfile/add-component.md.tmpl index 27c94feb6..4001813e1 100644 --- a/internal/app/azldev/agentskill/content/withoutlockfile/add-component.md.tmpl +++ b/internal/app/azldev/agentskill/content/withoutlockfile/add-component.md.tmpl @@ -39,7 +39,20 @@ dist-git often fail bot detection): 4. Read the spec and plan any overlays. -## Inline vs dedicated file +## Declaration-free, inline, or dedicated + +For components whose only customization is per-file overlays, add files under +`base/comps//overlays/`; do not create an empty `.comp.toml`. This requires +a shared project-level `default-component-config.overlay-files` pattern such as +`base/comps/{component}/overlays/*.overlay.toml` in the root config. The `{component}` +pattern discovers components from matching files and attaches their overlays. +Remove any provisional bare entry once discovery supplies it. + +If a component is explicitly listed in a component group, retain `[components.]` +in `base/comps/components.toml`: pattern discovery does not satisfy explicit group +membership validation. Components without matching overlay files still need an +explicit declaration or another supported discovery mechanism (such as a group's +`specs` glob). - **Inline** — a bare upstream import with no changes stays in a shared config file: @@ -47,9 +60,10 @@ dist-git often fail bot detection): [components.jq] ``` -- **Dedicated** — anything that needs overlays, build config, or a local spec gets its - own `/.comp.toml`. Rule of thumb: more than `[components.]` earns a - dedicated file. An `includes = ["**/*.comp.toml"]` glob picks it up automatically. +- **Dedicated** — keep `/.comp.toml` when it contains actual + component-specific settings, such as upstream pins, build options, inline overlays, + or local-spec configuration. An `includes = ["**/*.comp.toml"]` glob picks it up + automatically. `azldev comp add [...]` adds bare `[components.]` entries that inherit the distro defaults; it writes to the **root** config file and does not scaffold spec or diff --git a/internal/app/azldev/agentskill/content/withoutlockfile/comp-toml.md.tmpl b/internal/app/azldev/agentskill/content/withoutlockfile/comp-toml.md.tmpl index 0e33a9e6d..c6288abf6 100644 --- a/internal/app/azldev/agentskill/content/withoutlockfile/comp-toml.md.tmpl +++ b/internal/app/azldev/agentskill/content/withoutlockfile/comp-toml.md.tmpl @@ -6,7 +6,8 @@ description: {{ printf "%q" .Description }} # Component definition files (`*.comp.toml`) A component definition tells azldev where a package's spec comes from and how to -customize it for your distro. Every component lives under `[components.]`. +customize it for your distro. Explicit definitions live under `[components.]`; +components can also be discovered from matching overlay files without a declaration. Get the authoritative, always-current field list from the schema: @@ -92,11 +93,22 @@ the rare spec whose `Source`/`Patch` filenames use macros the filter cannot expa ## File organization +- **Declaration-free** — for components whose only customization is per-file + overlays, add files under `base/comps//overlays/`; do not create an empty + `.comp.toml`. A shared project-level `default-component-config.overlay-files` + pattern such as `base/comps/{component}/overlays/*.overlay.toml` in the root config + discovers these components and attaches their overlays. +- **Explicit group members** — if a component is explicitly listed in a component + group's `components` list, retain `[components.]` in + `base/comps/components.toml`. Pattern discovery alone does not satisfy that + validation. Components without matching overlay files still need an explicit + declaration or another supported discovery mechanism (such as a group's `specs` + glob). - **Inline** — put simple, customization-free components directly in a shared config file (e.g. `[components.jq]`). -- **Dedicated** — give a component its own `/.comp.toml` once it needs - overlays, build config, or a local spec. Rule of thumb: anything more than - `[components.]` earns a dedicated file. +- **Dedicated** — keep `/.comp.toml` when it contains actual + component-specific settings, such as upstream pins, build options, inline overlays, + or local-spec configuration. - A parent config picks up dedicated files through an `includes` glob, for example `includes = ["**/*.comp.toml"]`. @@ -108,7 +120,7 @@ details, inspect the rendered spec under the diff while checking enough surrounding context to ensure it fits the component and repository conventions. -- **Organization:** The component follows the repository's inline-versus-dedicated-file +- **Organization:** The component follows the repository's declaration-free, inline, or dedicated-file convention; its name matches upstream or sets `spec.upstream-name`; no stale or orphaned component files remain. - **Spec source:** The default upstream source is preferred. Pins explain why they are diff --git a/internal/app/azldev/agentskill/content/withoutlockfile/overlays.md.tmpl b/internal/app/azldev/agentskill/content/withoutlockfile/overlays.md.tmpl index 2235eb559..ca484c932 100644 --- a/internal/app/azldev/agentskill/content/withoutlockfile/overlays.md.tmpl +++ b/internal/app/azldev/agentskill/content/withoutlockfile/overlays.md.tmpl @@ -13,12 +13,12 @@ overwritten. ## The inner loop -Overlays live in the component's TOML config — inline `[[components..overlays]]` +Overlays can live in the component's TOML config — inline `[[components..overlays]]` entries, or per-file overlay documents referenced by the component's `overlay-files` glob. They apply **in order** and are **non-atomic**: if one fails part-way, the overlays before it stay applied. -1. Add or edit the overlay in the component config. +1. Add or edit the inline overlay or per-file overlay document. 2. Re-render and inspect the result: ```sh @@ -33,6 +33,37 @@ overlays before it stay applied. Config errors reference the offending overlay by its `description`, so give every overlay a short, specific `description`. +## Shared patterns and declaration-free components + +For components whose only customization is per-file overlays, add files under +`base/comps//overlays/`; do not create an empty `.comp.toml`. Configure +the shared pattern at project scope (paths are relative to the declaring config): + +```toml +# Root config, e.g. azldev.toml +[default-component-config] +overlay-files = ["base/comps/{component}/overlays/*.overlay.toml"] +``` + +The `{component}` segment discovers components from matching files and attaches +their overlays. For example, `base/comps/anaconda/overlays/0001-branding.overlay.toml` +discovers `anaconda` without `[components.anaconda]`. An explicit component with +no `overlay-files` of its own inherits the pattern; its own list replaces it, and +`overlay-files = []` disables inheritance. + +If a component is explicitly listed in a component group's `components` list, +retain `[components.]` in `base/comps/components.toml`: pattern discovery does +not satisfy explicit group membership validation. Components without matching +overlay files still need an explicit declaration or another supported discovery +mechanism (such as a group's `specs` glob). Keep a dedicated `.comp.toml` when it +contains actual component-specific settings, such as upstream pins, build options, +inline overlays, or local-spec configuration. + +Project-level `overlay-files` entries require `{component}` exactly once as a whole +directory segment, with no glob metacharacters before it and a file glob after it. +Use `/` separators. Component-level entries are plain globs (no placeholder); +distro-version and component-group defaults cannot set `overlay-files`. + ## Diagnose common failures Start with `azldev comp diff-sources -p ` to see the exact overlay effect. diff --git a/internal/app/azldev/core/components/pattern_discovery_test.go b/internal/app/azldev/core/components/pattern_discovery_test.go index 5b18d37ea..514339d46 100644 --- a/internal/app/azldev/core/components/pattern_discovery_test.go +++ b/internal/app/azldev/core/components/pattern_discovery_test.go @@ -18,6 +18,7 @@ import ( const patternTestOverlayContent = ` [metadata] category = "azl-branding-policy" +upstream-status = "inapplicable" [[overlays]] description = "Bump release" From bf8910db830a59b39bbf41f6818468e6e2874083 Mon Sep 17 00:00:00 2001 From: Nan Liu Date: Thu, 12 Nov 2026 06:12:34 +0000 Subject: [PATCH 3/4] fix(overlay-files): preserve compatibility and discovery behavior Preserve plain overlay-file patterns and empty-list overrides in project defaults and component settings, with opt-in {component} discovery. Reject overlay-files entirely in distro-version and component-group defaults, including empty lists. Preserve complete inherited overlay lists and optional placeholder matches, and restore locked synthetic history without forcing Git lookups for undeclared components without resolved lock data. Reject brace-alternative prefixes and parent-directory segments after {component}, while allowing parent-directory segments before it. Update configuration references, both emitted skill modes, schema descriptions and snapshots, and regression coverage for scope validation, discovery, inheritance, filenames, optional matches, history and no-lock behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- docs/user/reference/config/components.md | 2 +- docs/user/reference/config/overlays.md | 17 ++-- .../app/azldev/agentskill/agentskill_test.go | 4 + .../agentskill/content/add-component.md.tmpl | 6 ++ .../agentskill/content/comp-toml.md.tmpl | 6 ++ .../agentskill/content/overlays.md.tmpl | 24 ++++- .../withoutlockfile/add-component.md.tmpl | 6 ++ .../content/withoutlockfile/comp-toml.md.tmpl | 6 ++ .../content/withoutlockfile/overlays.md.tmpl | 24 ++++- .../core/components/pattern_discovery_test.go | 31 ++++++ .../app/azldev/core/components/resolver.go | 9 +- .../azldev/core/components/resolver_test.go | 10 +- .../app/azldev/core/sources/synthistory.go | 44 ++++++--- .../core/sources/synthistory_internal_test.go | 94 +++++++++++++++---- internal/projectconfig/component.go | 31 +++--- internal/projectconfig/component_test.go | 12 +-- internal/projectconfig/config_test.go | 64 +++++++++++++ internal/projectconfig/configfile.go | 44 +++------ internal/projectconfig/configfile_test.go | 25 ++--- internal/projectconfig/loader_test.go | 88 +++++++++++++++++ internal/projectconfig/overlay_file.go | 11 ++- internal/projectconfig/overlay_file_test.go | 86 ++++++++++++++--- .../overlay_files_placeholder.go | 21 +++-- .../overlay_files_placeholder_test.go | 8 ++ ...ainer_config_generate-schema_stdout_1.snap | 2 +- ...shots_config_generate-schema_stdout_1.snap | 2 +- schemas/azldev.schema.json | 2 +- 27 files changed, 521 insertions(+), 158 deletions(-) diff --git a/docs/user/reference/config/components.md b/docs/user/reference/config/components.md index ce06e6f60..6966adde7 100644 --- a/docs/user/reference/config/components.md +++ b/docs/user/reference/config/components.md @@ -11,7 +11,7 @@ A component definition tells azldev where to find the spec file, how to customiz | Spec source | `spec` | [SpecSource](#spec-source) | No | Where to find the spec file for this component. Inherited from distro defaults if not specified. | | Release config | `release` | [ReleaseConfig](#release-configuration) | No | Controls how the Release tag is managed during rendering | | Overlays | `overlays` | array of [Overlay](overlays.md) | No | Modifications to apply to the spec and/or source files | -| Overlay files | `overlay-files` | array of string | No | Path or glob patterns that load per-file overlay documents after component config resolution. Inherited relative patterns are resolved from the concrete component's config file, or from the matched spec file's directory for spec-discovered components. Use `[]` to disable inherited patterns. See [Per-file overlay format](overlays.md#per-file-overlay-format). | +| Overlay files | `overlay-files` | array of string | No | Path or glob patterns that load per-file overlay documents after component config resolution. Allowed in project defaults and component settings, but not distro-version or component-group defaults (including `[]`). Inherited relative patterns are resolved from the concrete component's config file, or from the matched spec file's directory for spec-discovered components. Use `[]` on a component to disable inherited patterns. See [Per-file overlay format](overlays.md#per-file-overlay-format). | | Build config | `build` | [BuildConfig](#build-configuration) | No | Build-time options (macros, conditionals, check config) | | Render config | `render` | [RenderConfig](#render-configuration) | No | Options controlling spec rendering behavior | | Source files | `source-files` | array of [SourceFileReference](#source-file-references) | No | Additional source files to download for this component | diff --git a/docs/user/reference/config/overlays.md b/docs/user/reference/config/overlays.md index c7694e589..67d8bb8bf 100644 --- a/docs/user/reference/config/overlays.md +++ b/docs/user/reference/config/overlays.md @@ -241,7 +241,7 @@ When a single logical change (a CVE backport, a feature disablement, a Fedora ch Set `overlay-files` on the component to one or more globs (relative to the component config) and drop one overlay file per logical change into a directory of your choosing. The conventional layout uses a sibling `overlays/` directory and a `*.overlay.toml` filename suffix, but neither is required — `overlay-files` is just a glob, so any layout you can describe with `**`/`*` patterns works. -`overlay-files` on `[components.X]` is a plain glob resolved relative to the component's config file. `overlay-files` can also be set on the project-level `default-component-config`, where **every entry must contain the `{component}` placeholder** (see [Discovering components by pattern](#discovering-components-by-pattern-component-in-overlay-files) below). A component that doesn't declare its own `overlay-files` inherits the project pattern with `{component}` substituted for its own name; use `overlay-files = []` at component scope to disable the inherited list, or set your own list to replace it wholesale. +`overlay-files` on `[components.X]` is a plain glob resolved relative to the component's config file. Plain legacy patterns such as `overlays/*.overlay.toml` and explicit empty lists remain supported in project-level defaults and component settings, with unchanged path resolution and precedence. Inherited plain patterns resolve relative to the concrete component config file or matched spec directory; they do not discover components. Project-level defaults can optionally use `{component}` to opt into [component discovery](#discovering-components-by-pattern-component-in-overlay-files). A component that inherits such a pattern substitutes its own name for `{component}`; use `overlay-files = []` to disable the inherited list, or set your own list to replace it wholesale. `overlay-files` is only settable at project scope (the top-level `default-component-config`) or at component scope (`[components.X]`). Setting it inside a distro-version or component-group `default-component-config` is rejected at load time — even an empty list (`overlay-files = []`). @@ -299,14 +299,19 @@ Use `azldev component metadata` to list documentation metadata for one or more c For projects that keep each component's overlay files in a predictable per-component directory (e.g. `base/comps//overlays/*.overlay.toml`), you can skip declaring a one-line `[components.]` table for every component. Set `overlay-files` on the **project-level** `default-component-config` to a pattern containing `{component}`, and every match becomes an implicit component. -Project-scope `overlay-files` entries **must** contain `{component}` exactly once, as a whole path segment, and **must** have at least one more path segment after it (the placeholder names a component directory, not a file). Additional constraints enforced at load time: +Discovery is **opt-in**: plain project-scope patterns remain valid and do not discover components. An entry that uses `{component}` **must** contain it exactly once, as a whole path segment, and **must** have at least one more path segment after it (the placeholder names a component directory, not a file). Additional constraints for placeholder entries enforced at load time: -* Use `/` as the path separator (backslashes are rejected, even on Windows). -* No glob metacharacters (`*`, `?`, `[`) before `{component}` — the prefix is a literal used to capture the component name. +* Use `/` as the path separator (backslashes are rejected; Windows is not supported). +* No glob metacharacters (`*`, `?`, `[`, `{`) before `{component}` — the prefix is a literal used to capture the component name. Brace alternatives are allowed only after the placeholder. +* `..` path segments are allowed before `{component}`, but never after it. For example, `../comps/{component}/overlays/*.overlay.toml` is valid. Both `comps/{component}/../overlays/*.overlay.toml` and `comps/{component}/nested/../overlays/*.overlay.toml` are rejected. Write the latter as `comps/{component}/overlays/*.overlay.toml` instead. * The rest of the entry (after `{component}/`) must be a valid `doublestar` glob (validated with `doublestar.ValidatePattern`). * The captured segment must be a safe component name (no spaces, no path traversal, no absolute paths); matches with unsafe names are skipped with a `slog.Debug` entry. -For each entry, `{component}` is expanded to `*` for globbing, and the captured path segment becomes the discovered component's name. The matched files are attached to that component (as if it had declared `overlay-files = []` explicitly). Explicit components with no `overlay-files` of their own inherit the same pattern and re-expand it with their own name substituted for `{component}`. +For example, `comps/{component}/overlays/*.overlay.toml` discovers `curl` when it finds `comps/curl/overlays/fix.overlay.toml`. `{component}` takes its name from that directory. + +After discovery, `curl` inherits the project-level `overlay-files` list, including any shared patterns and `{component}` patterns. An explicitly declared component also inherits that list unless it sets its own: a component-specific list replaces the project list, and `[]` disables inherited overlay files. When loading its overlays, `{component}` is replaced with the component's name. + +An inherited `{component}` entry that matches no files is skipped, even if it ends in a literal filename such as `{component}/overlays.toml`. An ordinary literal path without `{component}` must still exist. ```toml # base/comps/components.toml (included at project scope) @@ -331,7 +336,7 @@ base/comps/ └── 0001-cve-2024-5678.overlay.toml ``` -**Where `{component}` is allowed.** Only in **project-level** `default-component-config`. At every project-level `overlay-files` entry the placeholder is *required* — plain globs at that scope are rejected because they would apply the same files to every component in the project, which is almost never what you want. At component scope, `overlay-files` entries are plain globs relative to the declaring config file; `{component}` is rejected there. `overlay-files` cannot be set at all in distro-version or component-group `default-component-config` — those broad-scope defaults would silently displace project-level discovery. +**Where `{component}` is allowed.** Only in **project-level** `default-component-config`, where plain and placeholder entries may coexist. Component settings accept plain patterns and `[]`, but reject `{component}`. Distro-version and component-group defaults must omit `overlay-files` entirely, including empty lists. A component-specific list replaces the project list rather than merging with it; an empty component list disables inherited overlay files. **Inheritance with `{component}`.** A project-level `overlay-files` entry containing `{component}` serves two purposes at once: diff --git a/internal/app/azldev/agentskill/agentskill_test.go b/internal/app/azldev/agentskill/agentskill_test.go index 589ccc5b3..db2ea2f04 100644 --- a/internal/app/azldev/agentskill/agentskill_test.go +++ b/internal/app/azldev/agentskill/agentskill_test.go @@ -503,6 +503,10 @@ func TestComponentSkillsExplainDeclarationFreeOverlays(t *testing.T) { assert.Contains(t, doc, "base/comps/components.toml") assert.Contains(t, doc, "without matching overlay files") assert.Contains(t, doc, "upstream pins, build options, inline overlays,") + assert.Contains(t, doc, "Discovery is opt-in.") + assert.Contains(t, doc, "unchanged path resolution and precedence") + assert.Contains(t, doc, + "Distro-version and component-group defaults must omit `overlay-files`, including empty lists.") assert.NotContains(t, doc, "Every component lives under") assert.NotContains(t, doc, "anything that needs overlays") }) diff --git a/internal/app/azldev/agentskill/content/add-component.md.tmpl b/internal/app/azldev/agentskill/content/add-component.md.tmpl index cbac1aa7b..7405b5e3f 100644 --- a/internal/app/azldev/agentskill/content/add-component.md.tmpl +++ b/internal/app/azldev/agentskill/content/add-component.md.tmpl @@ -48,6 +48,12 @@ a shared project-level `default-component-config.overlay-files` pattern such as pattern discovers components from matching files and attaches their overlays. Remove any provisional bare entry once discovery supplies it. +Discovery is opt-in. Plain legacy patterns such as `overlays/*.overlay.toml` and +`[]` remain supported in project defaults and component settings, with unchanged +path resolution and precedence; they do not discover components. Distro-version +and component-group defaults must omit `overlay-files`, including empty lists. +Only project-level defaults may use `{component}`. + If a component is explicitly listed in a component group, retain `[components.]` in `base/comps/components.toml`: pattern discovery does not satisfy explicit group membership validation. Components without matching overlay files still need an diff --git a/internal/app/azldev/agentskill/content/comp-toml.md.tmpl b/internal/app/azldev/agentskill/content/comp-toml.md.tmpl index 734da1c71..c049a5bfd 100644 --- a/internal/app/azldev/agentskill/content/comp-toml.md.tmpl +++ b/internal/app/azldev/agentskill/content/comp-toml.md.tmpl @@ -103,6 +103,12 @@ the rare spec whose `Source`/`Patch` filenames use macros the filter cannot expa ## File organization +Discovery is opt-in. Plain legacy patterns such as `overlays/*.overlay.toml` and +`[]` remain supported in project defaults and component settings, with unchanged +path resolution and precedence; they do not discover components. Distro-version +and component-group defaults must omit `overlay-files`, including empty lists. +Only project-level defaults may use `{component}`. + - **Declaration-free** — for components whose only customization is per-file overlays, add files under `base/comps//overlays/`; do not create an empty `.comp.toml`. A shared project-level `default-component-config.overlay-files` diff --git a/internal/app/azldev/agentskill/content/overlays.md.tmpl b/internal/app/azldev/agentskill/content/overlays.md.tmpl index a4fecd158..3213e78a3 100644 --- a/internal/app/azldev/agentskill/content/overlays.md.tmpl +++ b/internal/app/azldev/agentskill/content/overlays.md.tmpl @@ -59,10 +59,26 @@ mechanism (such as a group's `specs` glob). Keep a dedicated `.comp.toml` when i contains actual component-specific settings, such as upstream pins, build options, inline overlays, or local-spec configuration. -Project-level `overlay-files` entries require `{component}` exactly once as a whole -directory segment, with no glob metacharacters before it and a file glob after it. -Use `/` separators. Component-level entries are plain globs (no placeholder); -distro-version and component-group defaults cannot set `overlay-files`. +Discovery is opt-in. Plain legacy patterns such as `overlays/*.overlay.toml` and +`[]` remain supported in project defaults and component settings, with unchanged +path resolution and precedence; they do not discover components. Distro-version +and component-group defaults must omit `overlay-files`, including empty lists. +Inherited plain globs resolve relative to the concrete component config file or +matched spec directory. A higher-priority list replaces a lower-priority one; +an empty list disables inherited overlay files. + +Only project-level defaults may use `{component}`. Placeholder entries require +it exactly once as a whole directory segment, with no glob metacharacters +(`*`, `?`, `[`, `{`) before it and a file pattern after it. Brace alternatives +are allowed only after the placeholder. `..` path segments are allowed before +`{component}`, but never after it. For example, +`../comps/{component}/overlays/*.overlay.toml` is valid; simplify +`comps/{component}/nested/../overlays/*.overlay.toml` to +`comps/{component}/overlays/*.overlay.toml`. Use `/` separators. Component settings +allow plain patterns and `[]` but reject the placeholder. +Discovered components use the complete inherited project overlay list, +not just the discovery matches. An inherited placeholder entry with no matching +files is optional, even with a literal suffix; an ordinary literal path must exist. ## Diagnose common failures diff --git a/internal/app/azldev/agentskill/content/withoutlockfile/add-component.md.tmpl b/internal/app/azldev/agentskill/content/withoutlockfile/add-component.md.tmpl index 4001813e1..7eb2d7cec 100644 --- a/internal/app/azldev/agentskill/content/withoutlockfile/add-component.md.tmpl +++ b/internal/app/azldev/agentskill/content/withoutlockfile/add-component.md.tmpl @@ -48,6 +48,12 @@ a shared project-level `default-component-config.overlay-files` pattern such as pattern discovers components from matching files and attaches their overlays. Remove any provisional bare entry once discovery supplies it. +Discovery is opt-in. Plain legacy patterns such as `overlays/*.overlay.toml` and +`[]` remain supported in project defaults and component settings, with unchanged +path resolution and precedence; they do not discover components. Distro-version +and component-group defaults must omit `overlay-files`, including empty lists. +Only project-level defaults may use `{component}`. + If a component is explicitly listed in a component group, retain `[components.]` in `base/comps/components.toml`: pattern discovery does not satisfy explicit group membership validation. Components without matching overlay files still need an diff --git a/internal/app/azldev/agentskill/content/withoutlockfile/comp-toml.md.tmpl b/internal/app/azldev/agentskill/content/withoutlockfile/comp-toml.md.tmpl index c6288abf6..9df1cdd12 100644 --- a/internal/app/azldev/agentskill/content/withoutlockfile/comp-toml.md.tmpl +++ b/internal/app/azldev/agentskill/content/withoutlockfile/comp-toml.md.tmpl @@ -93,6 +93,12 @@ the rare spec whose `Source`/`Patch` filenames use macros the filter cannot expa ## File organization +Discovery is opt-in. Plain legacy patterns such as `overlays/*.overlay.toml` and +`[]` remain supported in project defaults and component settings, with unchanged +path resolution and precedence; they do not discover components. Distro-version +and component-group defaults must omit `overlay-files`, including empty lists. +Only project-level defaults may use `{component}`. + - **Declaration-free** — for components whose only customization is per-file overlays, add files under `base/comps//overlays/`; do not create an empty `.comp.toml`. A shared project-level `default-component-config.overlay-files` diff --git a/internal/app/azldev/agentskill/content/withoutlockfile/overlays.md.tmpl b/internal/app/azldev/agentskill/content/withoutlockfile/overlays.md.tmpl index ca484c932..915fa6d8d 100644 --- a/internal/app/azldev/agentskill/content/withoutlockfile/overlays.md.tmpl +++ b/internal/app/azldev/agentskill/content/withoutlockfile/overlays.md.tmpl @@ -59,10 +59,26 @@ mechanism (such as a group's `specs` glob). Keep a dedicated `.comp.toml` when i contains actual component-specific settings, such as upstream pins, build options, inline overlays, or local-spec configuration. -Project-level `overlay-files` entries require `{component}` exactly once as a whole -directory segment, with no glob metacharacters before it and a file glob after it. -Use `/` separators. Component-level entries are plain globs (no placeholder); -distro-version and component-group defaults cannot set `overlay-files`. +Discovery is opt-in. Plain legacy patterns such as `overlays/*.overlay.toml` and +`[]` remain supported in project defaults and component settings, with unchanged +path resolution and precedence; they do not discover components. Distro-version +and component-group defaults must omit `overlay-files`, including empty lists. +Inherited plain globs resolve relative to the concrete component config file or +matched spec directory. A higher-priority list replaces a lower-priority one; +an empty list disables inherited overlay files. + +Only project-level defaults may use `{component}`. Placeholder entries require +it exactly once as a whole directory segment, with no glob metacharacters +(`*`, `?`, `[`, `{`) before it and a file pattern after it. Brace alternatives +are allowed only after the placeholder. `..` path segments are allowed before +`{component}`, but never after it. For example, +`../comps/{component}/overlays/*.overlay.toml` is valid; simplify +`comps/{component}/nested/../overlays/*.overlay.toml` to +`comps/{component}/overlays/*.overlay.toml`. Use `/` separators. Component settings +allow plain patterns and `[]` but reject the placeholder. +Discovered components use the complete inherited project overlay list, +not just the discovery matches. An inherited placeholder entry with no matching +files is optional, even with a literal suffix; an ordinary literal path must exist. ## Diagnose common failures diff --git a/internal/app/azldev/core/components/pattern_discovery_test.go b/internal/app/azldev/core/components/pattern_discovery_test.go index 514339d46..e4dda7c5f 100644 --- a/internal/app/azldev/core/components/pattern_discovery_test.go +++ b/internal/app/azldev/core/components/pattern_discovery_test.go @@ -72,6 +72,37 @@ func TestFindAllComponents_PatternDiscoveryAtProjectScope(t *testing.T) { assert.Len(t, bar.GetConfig().Overlays, 1) } +func TestFindAllComponents_PatternDiscoveryCompleteInheritance(t *testing.T) { + env := testutils.NewTestEnv(t) + env.Config.DefaultComponentConfig.OverlayFiles = []string{ + "/project/shared/*.overlay.toml", + "/project/comps/{component}/overlays/*.overlay.toml", + } + + for path, value := range map[string]string{ + "/project/shared/shared.overlay.toml": "shared", + "/project/comps/curl/overlays/fix[1].overlay.toml": "brackets", + "/project/comps/curl/overlays/fix1.overlay.toml": "sibling", + } { + require.NoError(t, fileutils.WriteFile(env.TestFS, path, + []byte(patternTestOverlayContent+"\nreplacement = \""+value+"\"\n"), fileperms.PrivateFile)) + } + + all, err := components.NewResolver(env.Env).FindAllComponents() + require.NoError(t, err) + + found, ok := all.TryGet("curl") + require.True(t, ok) + require.Nil(t, found.GetConfig().SourceConfigFile) + + values := make([]string, 0, len(found.GetConfig().Overlays)) + for _, overlay := range found.GetConfig().Overlays { + values = append(values, overlay.Replacement) + } + + assert.Equal(t, []string{"shared", "sibling", "brackets"}, values) +} + // Declared components with no 'overlay-files' inherit the project pattern; the // placeholder is substituted with the component's own name at glob time. func TestFindAllComponents_DeclaredComponentInheritsProjectPattern(t *testing.T) { diff --git a/internal/app/azldev/core/components/resolver.go b/internal/app/azldev/core/components/resolver.go index 0a891a832..ba13e96a9 100644 --- a/internal/app/azldev/core/components/resolver.go +++ b/internal/app/azldev/core/components/resolver.go @@ -210,9 +210,7 @@ func (r *Resolver) FindAllComponents() (components *ComponentSet, err error) { // documented way to opt out of overlays for a component and is not // treated as shadowing. // - // 'overlay-files' can only be set at project or component scope - // (validated by [projectconfig.ConfigFile.validateOverlayFilesByScope]), - // so a shadowing declaration always lives in Config().Components. + // Inherited legacy defaults retain their existing precedence. if declared, ok := r.env.Config().Components[name]; ok && len(declared.OverlayFiles) > 0 { logShadowedByDeclaration(entry) } @@ -971,10 +969,7 @@ func (r *Resolver) patternDiscoveredComponents() (map[string]*patternDiscoveredC func (r *Resolver) componentFromPatternDiscovered( entry *patternDiscoveredComponent, groupNames []string, ) (Component, error) { - synth := projectconfig.ComponentConfig{ - Name: entry.Name, - OverlayFiles: slices.Clone(entry.OverlayFiles), - } + synth := projectconfig.ComponentConfig{Name: entry.Name} updated, err := applyInheritedDefaultsToComponent(r.env, synth, entry.ReferenceDir, groupNames) if err != nil { diff --git a/internal/app/azldev/core/components/resolver_test.go b/internal/app/azldev/core/components/resolver_test.go index bf3b6280f..914ee67e4 100644 --- a/internal/app/azldev/core/components/resolver_test.go +++ b/internal/app/azldev/core/components/resolver_test.go @@ -724,10 +724,8 @@ value = "Microsoft" env.Config.ComponentGroups["specs"] = projectconfig.ComponentGroupConfig{ SpecPathPatterns: []string{"/specs/**/*.spec"}, - DefaultComponentConfig: projectconfig.ComponentConfig{ - OverlayFiles: []string{"overlays/*.overlay.toml"}, - }, } + env.Config.DefaultComponentConfig.OverlayFiles = []string{"overlays/*.overlay.toml"} filter := &components.ComponentFilter{ IncludeAllComponents: true, @@ -749,7 +747,7 @@ value = "Microsoft" assert.Equal(t, projectconfig.OverlayCategoryAZLBrandingPolicy, overlays[0].Metadata.Category) } -func TestFindComponents_SpecPathFilterInheritsOverlayFilesFromGroupDefaults(t *testing.T) { +func TestFindComponents_SpecPathFilterInheritsOverlayFilesFromProjectDefaults(t *testing.T) { env := testutils.NewTestEnv(t) touchFile(t, env, "/specs/ant/ant.spec") @@ -768,10 +766,8 @@ value = "Microsoft" env.Config.ComponentGroups["specs"] = projectconfig.ComponentGroupConfig{ SpecPathPatterns: []string{"/specs/**/*.spec"}, - DefaultComponentConfig: projectconfig.ComponentConfig{ - OverlayFiles: []string{"overlays/*.overlay.toml"}, - }, } + env.Config.DefaultComponentConfig.OverlayFiles = []string{"overlays/*.overlay.toml"} filter := &components.ComponentFilter{ SpecPaths: []string{"/specs/ant/ant.spec"}, diff --git a/internal/app/azldev/core/sources/synthistory.go b/internal/app/azldev/core/sources/synthistory.go index 0c957c81d..4f23960b7 100644 --- a/internal/app/azldev/core/sources/synthistory.go +++ b/internal/app/azldev/core/sources/synthistory.go @@ -547,8 +547,8 @@ func updateHead(repo *gogit.Repository, commitHash plumbing.Hash) error { } // buildSyntheticCommits resolves the project repository from the component's -// config file, walks the lock file's git history for fingerprint changes, and -// returns the matching [FingerprintChange] entries sorted chronologically. +// config file or configured lock directory, walks the lock file's git history +// for fingerprint changes, and returns matching entries sorted chronologically. // Returns (nil, "", nil) when there are no changes to represent — either the // lock file is missing, has no fingerprint changes (with a warning), or the // current fingerprint matches the committed one. @@ -568,7 +568,7 @@ func buildSyntheticCommits( lockDir string, currentFingerprint string, ) (changes []FingerprintChange, importCommit string, err error) { - projectRepo, projectRepoDir, err := openProjectRepo(config, componentName) + projectRepo, projectRepoDir, err := openProjectRepo(config, componentName, lockDir) if err != nil { return nil, "", err } @@ -681,19 +681,31 @@ func BuildDirtyChange( } // openProjectRepo opens the git repository that contains the component's -// config file and returns both the [gogit.Repository] and the worktree root -// directory. Returns (nil, "", nil) when the config file path cannot be -// resolved, indicating that synthetic commits should be skipped. +// config file, or its configured lock directory for declaration-free components. +// The lock directory anchors history to the repository containing the committed +// lock, without inventing config provenance or using the process working directory. +// Returns (nil, "", nil) for undeclared components with no resolved lock data. func openProjectRepo( config *projectconfig.ComponentConfig, - componentName string, + componentName, lockDir string, ) (*gogit.Repository, string, error) { - var configFilePath string - if config.SourceConfigFile != nil { - configFilePath = config.SourceConfigFile.SourcePath() + if config.SourceConfigFile != nil && config.SourceConfigFile.SourcePath() != "" { + return openProjectRepoForConfigFile(config.SourceConfigFile.SourcePath(), componentName) + } + + if config.Locked == nil { + slog.Debug("Undeclared component has no lock data; skipping synthetic history", + "component", componentName) + + return nil, "", nil + } + + if !filepath.IsAbs(lockDir) { + return nil, "", fmt.Errorf("cannot locate project repository for component %#q: absolute lock directory required", + componentName) } - return openProjectRepoForConfigFile(configFilePath, componentName) + return openProjectRepoAtDir(lockDir) } // openProjectRepoForConfigFile opens the git repository containing configFilePath @@ -711,10 +723,14 @@ func openProjectRepoForConfigFile( return nil, "", nil } - repo, err := git.OpenProjectRepo(filepath.Dir(configFilePath)) + return openProjectRepoAtDir(filepath.Dir(configFilePath)) +} + +func openProjectRepoAtDir(directory string) (*gogit.Repository, string, error) { + repo, err := git.OpenProjectRepo(directory) if err != nil { - return nil, "", fmt.Errorf("failed to find project repository for config file %#q:\n%w", - configFilePath, err) + return nil, "", fmt.Errorf("failed to find project repository from directory %#q:\n%w", + directory, err) } worktree, err := repo.Worktree() diff --git a/internal/app/azldev/core/sources/synthistory_internal_test.go b/internal/app/azldev/core/sources/synthistory_internal_test.go index 303c2de52..a0c2522dc 100644 --- a/internal/app/azldev/core/sources/synthistory_internal_test.go +++ b/internal/app/azldev/core/sources/synthistory_internal_test.go @@ -4,6 +4,7 @@ package sources import ( + "fmt" "os" "os/exec" "path/filepath" @@ -20,6 +21,7 @@ import ( "github.com/spf13/afero" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" + "go.uber.org/mock/gomock" ) func TestBuildSyntheticCommits_SymlinkedProjectPath(t *testing.T) { @@ -68,6 +70,20 @@ func TestBuildSyntheticCommits_SymlinkedProjectPath(t *testing.T) { }) require.NoError(t, err) + lockPath := filepath.Join(realProjectDir, "locks", componentName+".lock") + lockContent, err := fileutils.ReadFile(osFS, lockPath) + require.NoError(t, err) + require.NoError(t, fileutils.WriteFile(osFS, lockPath, + []byte(strings.ReplaceAll(string(lockContent), "sha256:abc", "sha256:def")), fileperms.PublicFile)) + require.NoError(t, worktree.AddWithOptions(&gogit.AddOptions{All: true})) + secondHash, err := worktree.Commit("update curl overlay", &gogit.CommitOptions{ + Author: &object.Signature{ + Name: "azldev", Email: "azldev@local", + When: time.Date(2024, 6, 2, 0, 0, 0, 0, time.UTC), + }, + }) + require.NoError(t, err) + _, config, err := projectconfig.LoadProjectConfig( osFS, testctx.NewTestOSEnv(), linkedProjectDir, true, t.TempDir(), nil, false, false, @@ -79,26 +95,70 @@ func TestBuildSyntheticCommits_SymlinkedProjectPath(t *testing.T) { component, ok := config.Components[componentName] require.True(t, ok) - var loggedPaths []string + for _, declarationFree := range []bool{false, true} { + t.Run(fmt.Sprintf("declaration-free=%t", declarationFree), func(t *testing.T) { + component := component + if declarationFree { + component.SourceConfigFile = nil + component.Locked = &projectconfig.ComponentLockData{ + UpstreamCommit: upstreamCommit, + ImportCommit: importCommit, + } + } + + var loggedPaths []string + + cmdFactory := testctx.NewTestCmdFactory() + cmdFactory.RunAndGetOutputHandler = func(cmd *exec.Cmd) (string, error) { + loggedPaths = append(loggedPaths, cmd.Args[len(cmd.Args)-1]) + + return strings.Join([]string{ + secondHash.String(), "azldev", "azldev@local", "1717286400", "update curl overlay", + }, "\x00") + "\x01" + strings.Join([]string{ + commitHash.String(), "azldev", "azldev@local", "1717200000", "add curl lock", + }, "\x00") + "\x01", nil + } + + changes, gotImportCommit, err := buildSyntheticCommits( + t.Context(), cmdFactory, &component, componentName, config.Project.LockDir, "", + ) + require.NoError(t, err) + assert.Equal(t, []string{filepath.Join("locks", componentName+".lock")}, loggedPaths) + assert.Equal(t, importCommit, gotImportCommit) + require.Len(t, changes, 2) + assert.Equal(t, commitHash.String(), changes[0].Hash) + assert.Equal(t, secondHash.String(), changes[1].Hash) + assert.Equal(t, upstreamCommit, changes[1].UpstreamCommit) + + ctx := testctx.NewCtx() + specPath := writeStaticReleaseTestSpec(t, ctx.FS(), componentName, "1%{?dist}") + component.Release.Calculation = projectconfig.ReleaseCalculationStatic + comp := mockComponent(gomock.NewController(t), componentName, &component) + preparer := newTestPreparer(ctx.FS()) + require.NoError(t, preparer.tryBumpRelease( + t.Context(), comp, filepath.Dir(specPath), changes)) + release, err := GetReleaseTagValue(ctx.FS(), specPath) + require.NoError(t, err) + assert.Equal(t, "3%{?dist}", release) + }) + } - cmdFactory := testctx.NewTestCmdFactory() - cmdFactory.RunAndGetOutputHandler = func(cmd *exec.Cmd) (string, error) { - loggedPaths = append(loggedPaths, cmd.Args[len(cmd.Args)-1]) + lockedConfig := &projectconfig.ComponentConfig{Locked: &projectconfig.ComponentLockData{}} + _, _, err = openProjectRepo(lockedConfig, componentName, tempDir) + require.ErrorContains(t, err, "failed to find project repository") + _, _, err = openProjectRepo(lockedConfig, componentName, "") + require.ErrorContains(t, err, "absolute lock directory required") +} - return strings.Join([]string{ - commitHash.String(), "azldev", "azldev@local", "1717200000", "add curl lock", - }, "\x00") + "\x01", nil +func TestBuildSyntheticCommits_UndeclaredComponentWithoutLock(t *testing.T) { + config := &projectconfig.ComponentConfig{ + Name: "local", + Spec: projectconfig.SpecSource{SourceType: projectconfig.SpecSourceTypeLocal}, } - - changes, gotImportCommit, err := buildSyntheticCommits( - t.Context(), cmdFactory, &component, componentName, config.Project.LockDir, "", + changes, importCommit, err := buildSyntheticCommits( + t.Context(), testctx.NewTestCmdFactory(), config, config.Name, "/not-a-project/locks", "", ) require.NoError(t, err) - - assert.Equal(t, []string{filepath.Join("locks", componentName+".lock")}, loggedPaths) - assert.Equal(t, importCommit, gotImportCommit) - - require.Len(t, changes, 1) - assert.Equal(t, commitHash.String(), changes[0].Hash) - assert.Equal(t, upstreamCommit, changes[0].UpstreamCommit) + assert.Empty(t, changes) + assert.Empty(t, importCommit) } diff --git a/internal/projectconfig/component.go b/internal/projectconfig/component.go index e6c5d6709..150689991 100644 --- a/internal/projectconfig/component.go +++ b/internal/projectconfig/component.go @@ -421,29 +421,27 @@ type ComponentConfig struct { // A value set in a higher-priority config layer replaces lower-priority values // wholesale; an explicit empty list disables inherited overlay files. // - // `overlay-files` may only be set at project scope (the top-level - // `default-component-config`) or at component scope (`[components.X]`). - // Distro-version and component-group `default-component-config` tables must - // not set this field: broad-scope defaults for overlay files silently - // displace project-level discovery and per-component overrides, so we - // require callers to place the list at project or component scope where the - // intent is unambiguous. + // Plain legacy globs and empty lists remain valid at project and component scopes, + // with unchanged path resolution and precedence. Distro-version and component-group + // defaults must omit 'overlay-files', including empty lists. // - // At project scope, every entry MUST contain the `{component}` placeholder exactly - // once as a whole path segment: those entries drive component discovery (each + // Project-level 'default-component-config' may opt into discovery by using the + // '{component}' placeholder exactly once as a whole path segment. These entries + // drive component discovery (each // captured segment becomes an implicit component) and, when inherited by an explicit // component that does not override, `{component}` is substituted with that // component's name at glob time. Two project-scope entries that produce the same // component name is a hard error. + // '..' path segments may appear before '{component}', but never after it. // - // At component scope, entries are plain globs interpreted relative to the - // declaring config file (or the matched spec directory); `{component}` is - // not allowed. Setting an empty list on a component (`overlay-files = []`) - // disables inheritance from the project default. + // The placeholder is forbidden at component scope. Plain inherited globs are + // interpreted relative to the concrete component + // config file (or matched spec directory). An explicit 'overlay-files = []' + // disables inherited overlay files. // // Excluded from the fingerprint because the value affects only where overlays are // sourced from, not their content. - OverlayFiles []string `toml:"overlay-files,omitempty" json:"overlayFiles,omitempty" table:"-" validate:"dive,required" jsonschema:"title=Overlay files,description=Path or glob patterns matched against the filesystem to locate per-file overlay documents. Only settable at project scope (top-level 'default-component-config') or at component scope ('[components.X]'). At project scope every entry must contain the {component} placeholder as a whole path segment (drives discovery and per-component substitution). At component scope {component} is not allowed and entries are plain globs relative to the config file. Distro-version and component-group 'default-component-config' tables must not set this field. Use an empty list at component scope to disable inheritance." fingerprint:"-"` + OverlayFiles []string `toml:"overlay-files,omitempty" json:"overlayFiles,omitempty" table:"-" validate:"dive,required" jsonschema:"title=Overlay files,description=Path or glob patterns matched against the filesystem to locate per-file overlay documents. Allowed only in project-level 'default-component-config' and explicit component settings. Distro-version and component-group defaults must omit 'overlay-files' including empty lists. Plain legacy patterns and empty lists remain supported at project and component scope with unchanged path resolution and precedence. Project-level defaults may opt into component discovery with {component} exactly once as a whole directory segment. In discovery patterns '..' path segments are allowed before {component} but forbidden after it. The placeholder is forbidden at component scope. Plain inherited globs resolve relative to the concrete component config file or matched spec directory. Use an empty list at component scope to disable inherited overlay files." fingerprint:"-"` // Configuration for building the component. Build ComponentBuildConfig `toml:"build,omitempty" json:"build,omitempty" table:"-" jsonschema:"title=Build configuration,description=Configuration for building the component"` @@ -663,10 +661,7 @@ func (c *ComponentConfig) WithAbsolutePaths(referenceDir string) *ComponentConfi // discovery patterns and are absolutized now so downstream code sees stable // absolute paths regardless of where the config was inherited from. Non-placeholder // entries are left verbatim so component-scope patterns are still interpreted - // relative to the concrete component config file at glob time. (Project-scope - // entries must all contain the placeholder; distro-version and component-group - // scopes may not set overlay-files at all — see - // [ConfigFile.validateOverlayFilesByScope].) + // relative to the concrete component config file at glob time. OverlayFiles: absolutizeOverlayFilesPlaceholderEntries(referenceDir, c.OverlayFiles), Tests: deep.MustCopy(c.Tests), } diff --git a/internal/projectconfig/component_test.go b/internal/projectconfig/component_test.go index 178af07f2..3cb78e2a6 100644 --- a/internal/projectconfig/component_test.go +++ b/internal/projectconfig/component_test.go @@ -612,13 +612,9 @@ func TestResolveComponentConfig(t *testing.T) { assert.Equal(t, "comp-commit", resolved.Spec.UpstreamCommit) }) - t.Run("component empty overlay files clears group defaults", func(t *testing.T) { - groups := map[string]projectconfig.ComponentGroupConfig{ - "core": { - DefaultComponentConfig: projectconfig.ComponentConfig{ - OverlayFiles: []string{"overlays/*.overlay.toml"}, - }, - }, + t.Run("component empty overlay files clears project defaults", func(t *testing.T) { + projectDefaults := projectconfig.ComponentConfig{ + OverlayFiles: []string{"overlays/*.overlay.toml"}, } comp := projectconfig.ComponentConfig{ Name: "curl", @@ -626,7 +622,7 @@ func TestResolveComponentConfig(t *testing.T) { } resolved, err := projectconfig.ResolveComponentConfig( - comp, projectconfig.ComponentConfig{}, distroDefaults, groups, []string{"core"}, + comp, projectDefaults, distroDefaults, nil, nil, ) require.NoError(t, err) require.NotNil(t, resolved.OverlayFiles) diff --git a/internal/projectconfig/config_test.go b/internal/projectconfig/config_test.go index 5912b2c47..b29104b7f 100644 --- a/internal/projectconfig/config_test.go +++ b/internal/projectconfig/config_test.go @@ -173,6 +173,70 @@ description = "`+testUserDesc+`" assert.Equal(t, testUserDesc, config.Project.Description) } +func TestLoadProjectConfig_OverlayFilesParentSegments(t *testing.T) { + testCases := []struct { + name string + pattern string + expected string + wantErr bool + }{ + { + name: "parent in placeholder prefix", + pattern: "../comps/{component}/overlays/*.overlay.toml", + expected: "/test/comps/{component}/overlays/*.overlay.toml", + }, + { + name: "parent removes placeholder", + pattern: "comps/{component}/../overlays/*.overlay.toml", + wantErr: true, + }, + { + name: "nested parents remove placeholder", + pattern: "comps/{component}/nested/../../overlays/*.overlay.toml", + wantErr: true, + }, + { + name: "parent after nested directory", + pattern: "comps/{component}/nested/../overlays/*.overlay.toml", + wantErr: true, + }, + { + name: "plain legacy pattern unchanged", + pattern: "nested/../overlays/*.overlay.toml", + expected: "nested/../overlays/*.overlay.toml", + }, + } + + for _, withoutLockfile := range []bool{false, true} { + mode := "locked" + if withoutLockfile { + mode = "lockless" + } + + t.Run(mode, func(t *testing.T) { + for _, testCase := range testCases { + t.Run(testCase.name, func(t *testing.T) { + ctx := newTestCtxWithXDGConfigHome() + writeProjectConfig(t, ctx, "[default-component-config]\noverlay-files = [\""+testCase.pattern+"\"]\n") + + _, config, err := projectconfig.LoadProjectConfig( + ctx.FS(), ctx.OSEnv(), testProjectDir, true, "/test/tmp", nil, false, withoutLockfile, + ) + if testCase.wantErr { + require.ErrorIs(t, err, projectconfig.ErrInvalidOverlayFilesEntry) + assert.ErrorContains(t, err, "must not contain '..' path segments after") + + return + } + + require.NoError(t, err) + assert.Equal(t, []string{testCase.expected}, config.DefaultComponentConfig.OverlayFiles) + }) + } + }) + } +} + // TestLoadProjectConfig_LockDirByMode verifies that the project's lock directory is // defaulted in azldev's default mode and left unset in lock-file-free mode, where an // explicitly configured 'lock-dir' is accepted but ignored. diff --git a/internal/projectconfig/configfile.go b/internal/projectconfig/configfile.go index 5db9b88da..c583c370b 100644 --- a/internal/projectconfig/configfile.go +++ b/internal/projectconfig/configfile.go @@ -526,18 +526,10 @@ func dedupTestRefSKUGroups( return nil } -// validateOverlayFilesByScope enforces where 'overlay-files' may appear. +// validateOverlayFilesByScope restricts default 'overlay-files' lists to project scope. // -// - Project scope (top-level 'default-component-config'): each entry must -// contain the '{component}' placeholder as a whole path segment; the -// placeholder drives pattern-based component discovery. -// - Component scope ('[components.X]'): plain globs; the '{component}' -// placeholder is forbidden (the name is already fixed). -// - Distro-version and component-group 'default-component-config': -// 'overlay-files' is not allowed at all. Broad-scope defaults for this -// field silently displace project-level discovery and per-component -// overrides, so we require callers to place the list at project or -// component scope where the intent is unambiguous. +// Distro-version and component-group defaults must omit the field, including empty lists. +// Project defaults accept plain patterns and may opt into discovery with '{component}'. func (f ConfigFile) validateOverlayFilesByScope() error { if f.DefaultComponentConfig != nil { if err := validateOverlayFilesEntries( @@ -551,12 +543,8 @@ func (f ConfigFile) validateOverlayFilesByScope() error { for versionName, version := range distro.Versions { if version.DefaultComponentConfig.OverlayFiles != nil { return fmt.Errorf( - "invalid 'default-component-config' for distro %#q version %#q:\n"+ - "%w: 'overlay-files' is only allowed on the project-level "+ - "'default-component-config' (with '%s' patterns) or on individual "+ - "'[components.X]' entries", - distroName, versionName, - ErrInvalidOverlayFilesEntry, OverlayFilesComponentPlaceholder, + "%w: 'overlay-files' is not allowed in 'default-component-config' for distro %#q version %#q", + ErrInvalidOverlayFilesEntry, distroName, versionName, ) } } @@ -565,12 +553,8 @@ func (f ConfigFile) validateOverlayFilesByScope() error { for groupName, group := range f.ComponentGroups { if group.DefaultComponentConfig.OverlayFiles != nil { return fmt.Errorf( - "invalid 'default-component-config' for component group %#q:\n"+ - "%w: 'overlay-files' is only allowed on the project-level "+ - "'default-component-config' (with '%s' patterns) or on individual "+ - "'[components.X]' entries", - groupName, - ErrInvalidOverlayFilesEntry, OverlayFilesComponentPlaceholder, + "%w: 'overlay-files' is not allowed in 'default-component-config' for component group %#q", + ErrInvalidOverlayFilesEntry, groupName, ) } } @@ -583,16 +567,18 @@ func (f ConfigFile) validateOverlayFilesByScope() error { type overlayFilesValidator func(entry string) error // validateProjectOverlayFilesEntry validates entries in the project-level -// default-component-config. Every entry MUST contain the '{component}' -// placeholder exactly once as a whole path segment — it's the discovery -// mechanism. +// default-component-config. Legacy plain globs remain valid; entries containing +// '{component}' opt into discovery and must obey the placeholder rules. func validateProjectOverlayFilesEntry(entry string) error { + if !hasOverlayFilesPlaceholder(entry) { + return nil + } + return validateOverlayFilesPlaceholder(entry) } -// validateComponentOverlayFilesEntry validates entries in a '[components.X]' -// table. Entries are plain globs; '{component}' is forbidden because the -// component name is already fixed by the table key. +// validateComponentOverlayFilesEntry validates plain globs at component scope, +// where '{component}' is forbidden. func validateComponentOverlayFilesEntry(entry string) error { if !hasOverlayFilesPlaceholder(entry) { return nil diff --git a/internal/projectconfig/configfile_test.go b/internal/projectconfig/configfile_test.go index 2dcc27cf5..4dbdae5df 100644 --- a/internal/projectconfig/configfile_test.go +++ b/internal/projectconfig/configfile_test.go @@ -1719,7 +1719,7 @@ func TestProjectConfigFileValidation_OverlayFilesPlaceholderOnProjectDefault(t * assert.NoError(t, file.Validate()) } -func TestProjectConfigFileValidation_OverlayFilesOnDistroDefaultRejected(t *testing.T) { +func TestProjectConfigFileValidation_OverlayFilesPlaceholderOnDistroDefaultRejected(t *testing.T) { file := projectconfig.ConfigFile{ Distros: map[string]projectconfig.DistroDefinition{ "my-distro": { @@ -1727,7 +1727,7 @@ func TestProjectConfigFileValidation_OverlayFilesOnDistroDefaultRejected(t *test "1.0": { ReleaseVer: "1.0", DefaultComponentConfig: projectconfig.ComponentConfig{ - OverlayFiles: []string{"distro/*.overlay.toml"}, + OverlayFiles: []string{"distro/{component}/*.overlay.toml"}, }, }, }, @@ -1740,12 +1740,12 @@ func TestProjectConfigFileValidation_OverlayFilesOnDistroDefaultRejected(t *test assert.Contains(t, err.Error(), "overlay-files") } -func TestProjectConfigFileValidation_OverlayFilesOnGroupDefaultRejected(t *testing.T) { +func TestProjectConfigFileValidation_OverlayFilesPlaceholderOnGroupDefaultRejected(t *testing.T) { file := projectconfig.ConfigFile{ ComponentGroups: map[string]projectconfig.ComponentGroupConfig{ "core": { DefaultComponentConfig: projectconfig.ComponentConfig{ - OverlayFiles: []string{"core/*.overlay.toml"}, + OverlayFiles: []string{"core/{component}/*.overlay.toml"}, }, }, }, @@ -1770,16 +1770,13 @@ func TestProjectConfigFileValidation_OverlayFilesPlaceholderOnComponentRejected( assert.Contains(t, err.Error(), "{component}") } -func TestProjectConfigFileValidation_PlainGlobOnProjectDefaultRejected(t *testing.T) { +func TestProjectConfigFileValidation_PlainGlobOnProjectDefaultAllowed(t *testing.T) { file := projectconfig.ConfigFile{ DefaultComponentConfig: &projectconfig.ComponentConfig{ OverlayFiles: []string{"comps/no-placeholder/*.overlay.toml"}, }, } - err := file.Validate() - require.Error(t, err) - assert.Contains(t, err.Error(), "invalid project 'default-component-config'") - assert.Contains(t, err.Error(), "{component}") + assert.NoError(t, file.Validate()) } func TestProjectConfigFileValidation_MalformedPlaceholderOnProjectDefault(t *testing.T) { @@ -1808,10 +1805,7 @@ func TestProjectConfigFileValidation_EmptyOverlayFilesOnDistroDefaultRejected(t }, }, } - err := file.Validate() - require.Error(t, err) - assert.Contains(t, err.Error(), "my-distro") - assert.Contains(t, err.Error(), "overlay-files") + require.ErrorIs(t, file.Validate(), projectconfig.ErrInvalidOverlayFilesEntry) } func TestProjectConfigFileValidation_EmptyOverlayFilesOnGroupDefaultRejected(t *testing.T) { @@ -1824,8 +1818,5 @@ func TestProjectConfigFileValidation_EmptyOverlayFilesOnGroupDefaultRejected(t * }, }, } - err := file.Validate() - require.Error(t, err) - assert.Contains(t, err.Error(), "core") - assert.Contains(t, err.Error(), "overlay-files") + require.ErrorIs(t, file.Validate(), projectconfig.ErrInvalidOverlayFilesEntry) } diff --git a/internal/projectconfig/loader_test.go b/internal/projectconfig/loader_test.go index ec9f4168d..45b939eef 100644 --- a/internal/projectconfig/loader_test.go +++ b/internal/projectconfig/loader_test.go @@ -7,6 +7,7 @@ package projectconfig import ( "os" "path/filepath" + "strings" "testing" "github.com/microsoft/azure-linux-dev-tools/internal/global/testctx" @@ -27,6 +28,93 @@ func TestLoadAndResolveProjectConfig(t *testing.T) { assert.Nil(t, config) } +func TestLoadAndResolveProjectConfig_OverlayFilesDefaultScopes(t *testing.T) { + scopes := []struct { + name string + lines []string + entries func(*ProjectConfig) []string + }{ + { + name: "project", + lines: []string{"[default-component-config]"}, + entries: func(config *ProjectConfig) []string { + return config.DefaultComponentConfig.OverlayFiles + }, + }, + { + name: "distro", + lines: []string{ + "[distros.test.versions.'1']", + "release-ver = '1'", + "[distros.test.versions.'1'.default-component-config]", + }, + entries: func(config *ProjectConfig) []string { + return config.Distros["test"].Versions["1"].DefaultComponentConfig.OverlayFiles + }, + }, + { + name: "group", + lines: []string{"[component-groups.test.default-component-config]"}, + entries: func(config *ProjectConfig) []string { + return config.ComponentGroups["test"].DefaultComponentConfig.OverlayFiles + }, + }, + } + + for _, scope := range scopes { + t.Run(scope.name, func(t *testing.T) { + for _, value := range []struct { + name string + toml string + expected []string + }{ + {name: "omitted"}, + {name: "plain-glob", toml: "['overlays/*.overlay.toml']", expected: []string{"overlays/*.overlay.toml"}}, + {name: "empty-list", toml: "[]", expected: []string{}}, + { + name: "placeholder", toml: "['comps/{component}/overlays/*.overlay.toml']", + expected: []string{"/project/comps/{component}/overlays/*.overlay.toml"}, + }, + } { + t.Run(value.name, func(t *testing.T) { + for _, mode := range []struct { + name string + withoutLockfile bool + }{ + {name: "default"}, + {name: "without-lockfile", withoutLockfile: true}, + } { + t.Run(mode.name, func(t *testing.T) { + ctx := testctx.NewCtx() + + lines := append([]string{}, scope.lines...) + if value.toml != "" { + lines = append(lines, "overlay-files = "+value.toml) + } + + require.NoError(t, fileutils.WriteFile(ctx.FS(), testConfigPath, + []byte(strings.Join(lines, "\n")+"\n"), fileperms.PrivateFile)) + + config, err := loadAndResolveProjectConfig(ctx.FS(), + loadOptions{withoutLockfile: mode.withoutLockfile}, testConfigPath) + if scope.name != "project" && value.toml != "" { + require.ErrorIs(t, err, ErrInvalidOverlayFilesEntry) + require.ErrorContains(t, err, scope.name) + assert.Nil(t, config) + + return + } + + require.NoError(t, err) + assert.Equal(t, value.expected, scope.entries(config)) + }) + } + }) + } + }) + } +} + func TestLoadAndResolveProjectConfig_SyntaxError(t *testing.T) { ctx := testctx.NewCtx() require.NoError(t, fileutils.WriteFile(ctx.FS(), testConfigPath, []byte("///"), fileperms.PrivateFile)) diff --git a/internal/projectconfig/overlay_file.go b/internal/projectconfig/overlay_file.go index 82ebccdb7..240feb52d 100644 --- a/internal/projectconfig/overlay_file.go +++ b/internal/projectconfig/overlay_file.go @@ -80,7 +80,7 @@ func ExpandResolvedOverlayFiles( ) } - loaded, err := loadOverlayFiles(fs, referenceDir, expanded, permissiveConfigParsing) + loaded, err := loadOverlayFiles(fs, referenceDir, component.OverlayFiles, permissiveConfigParsing, component.Name) if err != nil { return ComponentConfig{}, fmt.Errorf("component %#q overlay-files:\n%w", component.Name, err) } @@ -111,9 +111,11 @@ func overlayFilesHasRelativePattern(patterns []string) bool { // pattern, matches are applied in filename (lexicographic) order, using the full // path as a tie-breaker when filenames match. Duplicate matches across patterns // are de-duplicated, preserving first occurrence. Glob patterns that match no -// files contribute no overlays; literal paths must match a file. +// files contribute no overlays; explicit literal paths must match a file. +// Inherited placeholder entries may match no files even with a literal suffix. func loadOverlayFiles( fs opctx.FS, referenceDir string, patterns []string, permissiveConfigParsing bool, + componentName string, ) ([]ComponentOverlay, error) { var ( ordered []string @@ -121,6 +123,9 @@ func loadOverlayFiles( ) for _, pattern := range patterns { + inheritedPlaceholder := hasOverlayFilesPlaceholder(pattern) + pattern = substituteOverlayFilesPlaceholder(pattern, componentName) + absPattern := pattern if !filepath.IsAbs(absPattern) { absPattern = path.Join(referenceDir, absPattern) @@ -135,7 +140,7 @@ func loadOverlayFiles( switch { case err != nil: return nil, fmt.Errorf("failed to scan for overlay files matching %q:\n%w", pattern, err) - case len(matches) == 0 && !containsPattern(pattern): + case len(matches) == 0 && !inheritedPlaceholder && !containsPattern(pattern): return nil, fmt.Errorf("%w: %q", ErrOverlayFilesNoMatches, pattern) case len(matches) == 0: continue diff --git a/internal/projectconfig/overlay_file_test.go b/internal/projectconfig/overlay_file_test.go index 05207c3c0..dfbfaf162 100644 --- a/internal/projectconfig/overlay_file_test.go +++ b/internal/projectconfig/overlay_file_test.go @@ -65,7 +65,7 @@ value = "Microsoft" filepath.Join(overlayDir, "0001-backport-drop-openjdk21.overlay.toml"), []byte(validBackportOverlayFile), fileperms.PrivateFile)) - loaded, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false) + loaded, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false, "ant") require.NoError(t, err) require.Len(t, loaded, 3, "two overlays from 0001 + one from 0002") @@ -146,7 +146,8 @@ tag = "Order" value = "fourth" `), fileperms.PrivateFile)) - loaded, err := loadOverlayFiles(ctx.FS(), "/project", []string{"/project/comps/ant/overlays/**/*.overlay.toml"}, false) + loaded, err := loadOverlayFiles( + ctx.FS(), "/project", []string{"/project/comps/ant/overlays/**/*.overlay.toml"}, false, "ant") require.NoError(t, err) require.Len(t, loaded, 4) assert.Equal(t, "first", loaded[0].Value) @@ -165,7 +166,7 @@ func TestLoadOverlayFiles_NoMatchesIsNoop(t *testing.T) { filepath.Join(overlayDir, "README.md"), []byte("not an overlay"), fileperms.PrivateFile)) - loaded, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false) + loaded, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false, "ant") require.NoError(t, err) assert.Empty(t, loaded) } @@ -175,7 +176,7 @@ func TestLoadOverlayFiles_MissingDirIsNoop(t *testing.T) { loaded, err := loadOverlayFiles( ctx.FS(), "/project", - []string{overlayGlob("/project/comps/does-not-exist/overlays")}, false, + []string{overlayGlob("/project/comps/does-not-exist/overlays")}, false, "ant", ) require.NoError(t, err) assert.Empty(t, loaded) @@ -186,7 +187,7 @@ func TestLoadOverlayFiles_MissingLiteralPathErrors(t *testing.T) { _, err := loadOverlayFiles( ctx.FS(), "/project", - []string{"/project/comps/does-not-exist/overlays/0001-missing.overlay.toml"}, false, + []string{"/project/comps/does-not-exist/overlays/0001-missing.overlay.toml"}, false, "ant", ) require.ErrorIs(t, err, ErrOverlayFilesNoMatches) } @@ -202,7 +203,7 @@ func TestLoadOverlayFiles_NoMatchDoesNotBlockLaterMatches(t *testing.T) { loaded, err := loadOverlayFiles(ctx.FS(), "/project", []string{ overlayGlob("/project/comps/does-not-exist/overlays"), overlayGlob(overlayDir), - }, false) + }, false, "ant") require.NoError(t, err) require.Len(t, loaded, 2) } @@ -225,7 +226,7 @@ value = "Microsoft" metadata = { category = "azl-branding-policy", upstream-status = "inapplicable" } `), fileperms.PrivateFile)) - _, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false) + _, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false, "ant") require.ErrorIs(t, err, ErrOverlayFilePerOverlayMetadata) } @@ -241,7 +242,7 @@ category = "azl-compatibility" upstream-status = "inapplicable" `), fileperms.PrivateFile)) - _, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false) + _, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false, "ant") require.ErrorIs(t, err, ErrOverlayFileEmpty) } @@ -262,7 +263,7 @@ tag = "Vendor" value = "Microsoft" `), fileperms.PrivateFile)) - _, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false) + _, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false, "ant") require.Error(t, err) assert.Contains(t, err.Error(), "category") } @@ -283,7 +284,7 @@ type = "spec-set-tag" tag = "Vendor" `), fileperms.PrivateFile)) - _, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false) + _, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false, "ant") require.Error(t, err) assert.Contains(t, err.Error(), "invalid overlay 1") assert.Contains(t, err.Error(), "requires") @@ -336,7 +337,7 @@ type = "patch-add" source = "patches/fix.patch" `), fileperms.PrivateFile)) - loaded, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false) + loaded, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false, "ant") require.NoError(t, err) require.Len(t, loaded, 1) assert.Equal(t, @@ -358,7 +359,7 @@ func TestLoadOverlayFiles_DedupsAcrossPatterns(t *testing.T) { loaded, err := loadOverlayFiles(ctx.FS(), "/project", []string{ overlayGlob(overlayDir), filepath.Join(overlayDir, "0001-*.overlay.toml"), - }, false) + }, false, "ant") require.NoError(t, err) require.Len(t, loaded, 2, "overlapping globs must not double-apply matched files") } @@ -373,7 +374,7 @@ func TestLoadOverlayFiles_DoubleStarMatchesNested(t *testing.T) { loaded, err := loadOverlayFiles( ctx.FS(), "/project", - []string{filepath.Join(root, "**", "*.overlay.toml")}, false, + []string{filepath.Join(root, "**", "*.overlay.toml")}, false, "ant", ) require.NoError(t, err) require.Len(t, loaded, 2, "**-style glob must descend into subdirectories") @@ -441,6 +442,65 @@ func TestExpandResolvedOverlayFiles_NoopWhenUnset(t *testing.T) { require.Len(t, expanded.Overlays, 1, "no overlay-files, no merging") } +func TestExpandResolvedOverlayFiles_UnmatchedPlaceholderLiteral(t *testing.T) { + tests := []struct { + name string + patterns []string + wantErr bool + }{ + { + name: "unrelated explicit component inherits optional placeholder", + patterns: []string{"comps/{component}/overlays.toml"}, + }, + { + name: "explicit missing literal remains an error", + patterns: []string{"comps/ant/overlays.toml"}, + wantErr: true, + }, + { + name: "placeholder before equivalent explicit literal", + patterns: []string{"comps/{component}/overlays.toml", "comps/ant/overlays.toml"}, + wantErr: true, + }, + { + name: "explicit literal before equivalent placeholder", + patterns: []string{"comps/ant/overlays.toml", "comps/{component}/overlays.toml"}, + wantErr: true, + }, + } + + for _, testCase := range tests { + t.Run(testCase.name, func(t *testing.T) { + ctx := testctx.NewCtx() + component := ComponentConfig{Name: "ant", OverlayFiles: testCase.patterns} + + expanded, err := ExpandResolvedOverlayFiles(ctx.FS(), component, "/project", false) + if testCase.wantErr { + require.ErrorIs(t, err, ErrOverlayFilesNoMatches) + + return + } + + require.NoError(t, err) + assert.Empty(t, expanded.Overlays) + assert.Empty(t, expanded.OverlayFiles) + }) + } +} + +func TestExpandResolvedOverlayFiles_EquivalentPlaceholderEntriesDeduplicate(t *testing.T) { + ctx := testctx.NewCtx() + require.NoError(t, fileutils.WriteFile(ctx.FS(), "/project/comps/ant/overlays.toml", + []byte(validBackportOverlayFile), fileperms.PrivateFile)) + + expanded, err := ExpandResolvedOverlayFiles(ctx.FS(), ComponentConfig{ + Name: "ant", + OverlayFiles: []string{"comps/{component}/overlays.toml", "comps/ant/overlays.toml"}, + }, "/project", false) + require.NoError(t, err) + require.Len(t, expanded.Overlays, 2) +} + func TestExpandResolvedOverlayFiles_AcceptsAbsoluteGlob(t *testing.T) { ctx := testctx.NewCtx() absDir := "/elsewhere/overlays" diff --git a/internal/projectconfig/overlay_files_placeholder.go b/internal/projectconfig/overlay_files_placeholder.go index 23578869e..ca359d8dd 100644 --- a/internal/projectconfig/overlay_files_placeholder.go +++ b/internal/projectconfig/overlay_files_placeholder.go @@ -6,16 +6,17 @@ package projectconfig import ( "errors" "fmt" + "slices" "strings" "github.com/bmatcuk/doublestar/v4" ) // OverlayFilesComponentPlaceholder is the single supported placeholder in an -// [ComponentConfig.OverlayFiles] entry. It is required in every project-scope -// entry (discovery) and forbidden in distro/component-group/component-scope -// entries; when present it must appear exactly once and must be a whole path -// segment. +// [ComponentConfig.OverlayFiles] entry. Project-level defaults may opt into +// discovery with this placeholder; components permit only plain patterns, and +// distro-version and component-group defaults must omit 'overlay-files'. +// When present it must appear exactly once and must be a whole path segment. const OverlayFilesComponentPlaceholder = "{component}" // ErrInvalidOverlayFilesEntry is returned when an entry in @@ -32,10 +33,11 @@ func hasOverlayFilesPlaceholder(entry string) bool { // validateOverlayFilesPlaceholder verifies that entry contains exactly one // [OverlayFilesComponentPlaceholder] and that the placeholder is a whole path // segment. Path separators must be '/' (POSIX-style), and no glob -// metacharacters ('*', '?', '[') may appear before the placeholder because +// metacharacters ('*', '?', '[', '{') may appear before the placeholder because // pattern-discovery extracts the captured component name by literal-prefix // matching against the concrete glob result. Metacharacters after the -// placeholder are allowed. Returns nil when the entry is well-formed. +// placeholder are allowed, but '..' path segments are allowed only before the +// placeholder. Returns nil when the entry is well-formed. func validateOverlayFilesPlaceholder(entry string) error { if entry == "" { return fmt.Errorf("%w: entry is empty", ErrInvalidOverlayFilesEntry) @@ -94,8 +96,13 @@ func validateOverlayFilesPlaceholder(entry string) error { ) } + if slices.Contains(strings.Split(entry[tail+1:], "/"), "..") { + return fmt.Errorf("%w: entry %#q must not contain '..' path segments after %q", + ErrInvalidOverlayFilesEntry, entry, OverlayFilesComponentPlaceholder) + } + prefix := entry[:idx] - if globMetaIdx := strings.IndexAny(prefix, "*?["); globMetaIdx >= 0 { + if globMetaIdx := strings.IndexAny(prefix, "*?[{"); globMetaIdx >= 0 { return fmt.Errorf( "%w: entry %#q must not contain glob metacharacters (%q) before %q; "+ "the discovery capture uses a literal prefix", diff --git a/internal/projectconfig/overlay_files_placeholder_test.go b/internal/projectconfig/overlay_files_placeholder_test.go index 95b00707c..19a04f2f5 100644 --- a/internal/projectconfig/overlay_files_placeholder_test.go +++ b/internal/projectconfig/overlay_files_placeholder_test.go @@ -27,12 +27,20 @@ func TestValidateOverlayFilesPlaceholder(t *testing.T) { {name: "backslash separator rejected", entry: "comps\\{component}\\x.toml", wantErr: true}, {name: "glob metachar before placeholder", entry: "**/{component}/*.overlay.toml", wantErr: true}, {name: "star before placeholder", entry: "comps*/{component}/*.overlay.toml", wantErr: true}, + {name: "brace alternatives before placeholder", entry: "{base,extra}/{component}/*.overlay.toml", wantErr: true}, + {name: "brace suffix allowed", entry: "comps/{component}/{base,extra}/*.overlay.toml", wantErr: false}, {name: "whole segment at start", entry: "{component}/overlays/*.overlay.toml", wantErr: false}, {name: "whole segment in middle", entry: "comps/{component}/overlays/*.overlay.toml", wantErr: false}, {name: "placeholder at end rejected", entry: "comps/overlays/{component}", wantErr: true}, {name: "placeholder followed by only slash rejected", entry: "comps/{component}/", wantErr: true}, {name: "doublestar in suffix allowed", entry: "comps/{component}/**/*.overlay.toml", wantErr: false}, {name: "absolute path", entry: "/project/base/comps/{component}/overlays/*.overlay.toml", wantErr: false}, + {name: "parent directory in prefix allowed", entry: "../comps/{component}/overlays/*.overlay.toml"}, + {name: "parent after placeholder rejected", entry: "comps/{component}/../overlays/*.overlay.toml", wantErr: true}, + {name: "nested parent rejected", entry: "comps/{component}/nested/../overlays/*.overlay.toml", wantErr: true}, + {name: "nested escape rejected", entry: "comps/{component}/nested/../../overlays/*.overlay.toml", wantErr: true}, + {name: "trailing parent rejected", entry: "comps/{component}/overlays/..", wantErr: true}, + {name: "two dots in filename allowed", entry: "comps/{component}/overlays/fix..overlay.toml"}, } for _, testCase := range tests { diff --git a/scenario/__snapshots__/TestSnapshotsContainer_config_generate-schema_stdout_1.snap b/scenario/__snapshots__/TestSnapshotsContainer_config_generate-schema_stdout_1.snap index d71968059..ffe768777 100755 --- a/scenario/__snapshots__/TestSnapshotsContainer_config_generate-schema_stdout_1.snap +++ b/scenario/__snapshots__/TestSnapshotsContainer_config_generate-schema_stdout_1.snap @@ -130,7 +130,7 @@ }, "type": "array", "title": "Overlay files", - "description": "Path or glob patterns matched against the filesystem to locate per-file overlay documents. Only settable at project scope (top-level 'default-component-config') or at component scope ('[components.X]'). At project scope every entry must contain the {component} placeholder as a whole path segment (drives discovery and per-component substitution). At component scope {component} is not allowed and entries are plain globs relative to the config file. Distro-version and component-group 'default-component-config' tables must not set this field. Use an empty list at component scope to disable inheritance." + "description": "Path or glob patterns matched against the filesystem to locate per-file overlay documents. Allowed only in project-level 'default-component-config' and explicit component settings. Distro-version and component-group defaults must omit 'overlay-files' including empty lists. Plain legacy patterns and empty lists remain supported at project and component scope with unchanged path resolution and precedence. Project-level defaults may opt into component discovery with {component} exactly once as a whole directory segment. In discovery patterns '..' path segments are allowed before {component} but forbidden after it. The placeholder is forbidden at component scope. Plain inherited globs resolve relative to the concrete component config file or matched spec directory. Use an empty list at component scope to disable inherited overlay files." }, "build": { "$ref": "#/$defs/ComponentBuildConfig", diff --git a/scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap b/scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap index d71968059..ffe768777 100755 --- a/scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap +++ b/scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap @@ -130,7 +130,7 @@ }, "type": "array", "title": "Overlay files", - "description": "Path or glob patterns matched against the filesystem to locate per-file overlay documents. Only settable at project scope (top-level 'default-component-config') or at component scope ('[components.X]'). At project scope every entry must contain the {component} placeholder as a whole path segment (drives discovery and per-component substitution). At component scope {component} is not allowed and entries are plain globs relative to the config file. Distro-version and component-group 'default-component-config' tables must not set this field. Use an empty list at component scope to disable inheritance." + "description": "Path or glob patterns matched against the filesystem to locate per-file overlay documents. Allowed only in project-level 'default-component-config' and explicit component settings. Distro-version and component-group defaults must omit 'overlay-files' including empty lists. Plain legacy patterns and empty lists remain supported at project and component scope with unchanged path resolution and precedence. Project-level defaults may opt into component discovery with {component} exactly once as a whole directory segment. In discovery patterns '..' path segments are allowed before {component} but forbidden after it. The placeholder is forbidden at component scope. Plain inherited globs resolve relative to the concrete component config file or matched spec directory. Use an empty list at component scope to disable inherited overlay files." }, "build": { "$ref": "#/$defs/ComponentBuildConfig", diff --git a/schemas/azldev.schema.json b/schemas/azldev.schema.json index d71968059..ffe768777 100644 --- a/schemas/azldev.schema.json +++ b/schemas/azldev.schema.json @@ -130,7 +130,7 @@ }, "type": "array", "title": "Overlay files", - "description": "Path or glob patterns matched against the filesystem to locate per-file overlay documents. Only settable at project scope (top-level 'default-component-config') or at component scope ('[components.X]'). At project scope every entry must contain the {component} placeholder as a whole path segment (drives discovery and per-component substitution). At component scope {component} is not allowed and entries are plain globs relative to the config file. Distro-version and component-group 'default-component-config' tables must not set this field. Use an empty list at component scope to disable inheritance." + "description": "Path or glob patterns matched against the filesystem to locate per-file overlay documents. Allowed only in project-level 'default-component-config' and explicit component settings. Distro-version and component-group defaults must omit 'overlay-files' including empty lists. Plain legacy patterns and empty lists remain supported at project and component scope with unchanged path resolution and precedence. Project-level defaults may opt into component discovery with {component} exactly once as a whole directory segment. In discovery patterns '..' path segments are allowed before {component} but forbidden after it. The placeholder is forbidden at component scope. Plain inherited globs resolve relative to the concrete component config file or matched spec directory. Use an empty list at component scope to disable inherited overlay files." }, "build": { "$ref": "#/$defs/ComponentBuildConfig", From aab618f153d20c9c1b3e7bbefbd2f2fade174b84 Mon Sep 17 00:00:00 2001 From: Nan Liu Date: Sat, 14 Nov 2026 01:34:06 +0000 Subject: [PATCH 4/4] refactor(overlay-files): remove automatic component discovery Keep shared {component} path substitution for declared and spec-discovered components, while retaining central component declarations. Remove overlay-driven discovery, caching, shadow warnings, and the unused splitting helper. Update emitted skills, configuration guidance, schema descriptions, and regression tests to describe declaration-based overlay loading. --- docs/user/reference/config/overlays.md | 44 ++-- .../app/azldev/agentskill/agentskill_test.go | 11 +- .../agentskill/content/add-component.md.tmpl | 13 +- .../agentskill/content/comp-toml.md.tmpl | 15 +- .../agentskill/content/overlays.md.tmpl | 27 +- .../withoutlockfile/add-component.md.tmpl | 13 +- .../content/withoutlockfile/comp-toml.md.tmpl | 15 +- .../content/withoutlockfile/overlays.md.tmpl | 27 +- .../core/components/pattern_discovery.go | 217 ---------------- .../core/components/pattern_discovery_test.go | 223 ----------------- .../app/azldev/core/components/resolver.go | 93 +------ .../azldev/core/components/resolver_test.go | 198 +++++++++++++++ internal/projectconfig/component.go | 16 +- internal/projectconfig/config_test.go | 64 ----- internal/projectconfig/configfile.go | 4 +- internal/projectconfig/loader_test.go | 88 ------- .../overlay_files_placeholder.go | 44 ++-- .../overlay_files_placeholder_test.go | 233 ++++++++++++++---- ...ainer_config_generate-schema_stdout_1.snap | 2 +- ...shots_config_generate-schema_stdout_1.snap | 2 +- schemas/azldev.schema.json | 2 +- 21 files changed, 516 insertions(+), 835 deletions(-) delete mode 100644 internal/app/azldev/core/components/pattern_discovery.go delete mode 100644 internal/app/azldev/core/components/pattern_discovery_test.go diff --git a/docs/user/reference/config/overlays.md b/docs/user/reference/config/overlays.md index 67d8bb8bf..e1355e11b 100644 --- a/docs/user/reference/config/overlays.md +++ b/docs/user/reference/config/overlays.md @@ -241,7 +241,7 @@ When a single logical change (a CVE backport, a feature disablement, a Fedora ch Set `overlay-files` on the component to one or more globs (relative to the component config) and drop one overlay file per logical change into a directory of your choosing. The conventional layout uses a sibling `overlays/` directory and a `*.overlay.toml` filename suffix, but neither is required — `overlay-files` is just a glob, so any layout you can describe with `**`/`*` patterns works. -`overlay-files` on `[components.X]` is a plain glob resolved relative to the component's config file. Plain legacy patterns such as `overlays/*.overlay.toml` and explicit empty lists remain supported in project-level defaults and component settings, with unchanged path resolution and precedence. Inherited plain patterns resolve relative to the concrete component config file or matched spec directory; they do not discover components. Project-level defaults can optionally use `{component}` to opt into [component discovery](#discovering-components-by-pattern-component-in-overlay-files). A component that inherits such a pattern substitutes its own name for `{component}`; use `overlay-files = []` to disable the inherited list, or set your own list to replace it wholesale. +`overlay-files` on `[components.X]` is a plain glob resolved relative to the component's config file. Plain legacy patterns such as `overlays/*.overlay.toml` and explicit empty lists remain supported in project-level defaults and component settings, with unchanged path resolution and precedence. Inherited plain patterns resolve relative to the concrete component config file or matched spec directory. Project-level defaults can use [shared patterns with `{component}`](#shared-patterns-with-component-in-overlay-files) to locate overlays for known components. A component that inherits such a pattern substitutes its own name for `{component}`; use `overlay-files = []` to disable the inherited list, or set your own list to replace it wholesale. `overlay-files` is only settable at project scope (the top-level `default-component-config`) or at component scope (`[components.X]`). Setting it inside a distro-version or component-group `default-component-config` is rejected at load time — even an empty list (`overlay-files = []`). @@ -295,21 +295,21 @@ lines = ["- Fix CVE-2024-1234"] Use `azldev component metadata` to list documentation metadata for one or more components. By default it lists metadata from both sources — component overlays and the component groups each component belongs to — tagging each row with its `source`, and shows only entries that carry a `[metadata]` block. Pass `--overlays` or `--groups` to restrict to one source. The command is read-only and supports `--category` and `--upstream-status` filters; output is available as a table (default) or JSON (`-O json`). -### Discovering components by pattern (`{component}` in `overlay-files`) +### Shared patterns with `{component}` in `overlay-files` -For projects that keep each component's overlay files in a predictable per-component directory (e.g. `base/comps//overlays/*.overlay.toml`), you can skip declaring a one-line `[components.]` table for every component. Set `overlay-files` on the **project-level** `default-component-config` to a pattern containing `{component}`, and every match becomes an implicit component. +For projects that keep each component's overlay files in a predictable per-component directory (e.g. `base/comps//overlays/*.overlay.toml`), declare components centrally and use a shared **project-level** `default-component-config.overlay-files` pattern containing `{component}`. This removes the need for declaration-only `.comp.toml` files without removing the declarations themselves. -Discovery is **opt-in**: plain project-scope patterns remain valid and do not discover components. An entry that uses `{component}` **must** contain it exactly once, as a whole path segment, and **must** have at least one more path segment after it (the placeholder names a component directory, not a file). Additional constraints for placeholder entries enforced at load time: +**Overlay patterns only apply to known components.** A component must have a `[components.]` declaration or be identified by another supported mechanism, such as a component-group `specs` glob. An entry that uses `{component}` **must** contain it exactly once, as a whole path segment, and **must** have at least one more path segment after it (the placeholder names a component directory, not a file). Additional constraints for placeholder entries enforced at load time: * Use `/` as the path separator (backslashes are rejected; Windows is not supported). -* No glob metacharacters (`*`, `?`, `[`, `{`) before `{component}` — the prefix is a literal used to capture the component name. Brace alternatives are allowed only after the placeholder. +* No glob metacharacters (`*`, `?`, `[`, `{`) before `{component}` — the prefix must be literal. +* No additional `{` or `}` after `{component}`, including brace alternatives such as `{base,extra}`. Use separate list entries for multiple directories. Ordinary patterns without `{component}` retain their existing brace support. * `..` path segments are allowed before `{component}`, but never after it. For example, `../comps/{component}/overlays/*.overlay.toml` is valid. Both `comps/{component}/../overlays/*.overlay.toml` and `comps/{component}/nested/../overlays/*.overlay.toml` are rejected. Write the latter as `comps/{component}/overlays/*.overlay.toml` instead. * The rest of the entry (after `{component}/`) must be a valid `doublestar` glob (validated with `doublestar.ValidatePattern`). -* The captured segment must be a safe component name (no spaces, no path traversal, no absolute paths); matches with unsafe names are skipped with a `slog.Debug` entry. -For example, `comps/{component}/overlays/*.overlay.toml` discovers `curl` when it finds `comps/curl/overlays/fix.overlay.toml`. `{component}` takes its name from that directory. +For example, a resolved component named `curl` substitutes its name into `comps/{component}/overlays/*.overlay.toml` and loads `comps/curl/overlays/fix.overlay.toml`. A matching directory for an undeclared component does not create a new component. -After discovery, `curl` inherits the project-level `overlay-files` list, including any shared patterns and `{component}` patterns. An explicitly declared component also inherits that list unless it sets its own: a component-specific list replaces the project list, and `[]` disables inherited overlay files. When loading its overlays, `{component}` is replaced with the component's name. +Each component inherits the project-level `overlay-files` list, including any shared patterns and `{component}` patterns, unless it sets its own: a component-specific list replaces the project list, and `[]` disables inherited overlay files. An inherited `{component}` entry that matches no files is skipped, even if it ends in a literal filename such as `{component}/overlays.toml`. An ordinary literal path without `{component}` must still exist. @@ -317,16 +317,21 @@ An inherited `{component}` entry that matches no files is skipped, even if it en # base/comps/components.toml (included at project scope) [default-component-config] overlay-files = ["{component}/overlays/*.overlay.toml"] + +[components.anaconda] + +[components.curl] ``` The pattern is relative to the file that declares it. If declared in the root `azldev.toml` instead, use `base/comps/{component}/overlays/*.overlay.toml`. -With the layout below, no per-component `[components.]` table is needed — -`anaconda` and `curl` are discovered automatically, each carrying its overlay files. -This is the declaration-free case: do not create an empty `anaconda.comp.toml`. +With the layout below, the central declarations load their respective overlay files. +Do not create an empty `anaconda.comp.toml`; keep `[components.anaconda]` in the +central file instead. ``` base/comps/ +├── components.toml ├── anaconda/ │ └── overlays/ │ ├── 0001-cve-2024-1234.overlay.toml @@ -338,18 +343,13 @@ base/comps/ **Where `{component}` is allowed.** Only in **project-level** `default-component-config`, where plain and placeholder entries may coexist. Component settings accept plain patterns and `[]`, but reject `{component}`. Distro-version and component-group defaults must omit `overlay-files` entirely, including empty lists. A component-specific list replaces the project list rather than merging with it; an empty component list disables inherited overlay files. -**Inheritance with `{component}`.** A project-level `overlay-files` entry containing `{component}` serves two purposes at once: - -1. **Discovery.** Every captured directory becomes an implicit component unless there is already an explicit `[components.]` table with the same name. -2. **Inheritance.** An explicitly-declared component that does not override `overlay-files` inherits the project entry and expands `{component}` with its own name at glob time. +**Inheritance with `{component}`.** A component that does not override `overlay-files` inherits the project entry and expands `{component}` with its own name at glob time. This also works for components identified by component-group spec globs. A component that sets its own `overlay-files = [...]` replaces the inherited value; the placeholder pattern no longer applies to that component. Setting `overlay-files = []` disables inheritance entirely for that component. -**Same-scope collisions are a hard error.** If two project-scope `overlay-files` entries both produce the same component name, the resolver fails with a diagnostic listing both entries. Restrict one of the entries or rename one of the directories. - -**Shadowing.** An explicit `[components.]` declaration supersedes pattern discovery for the same name. If that declaration also sets a non-empty `overlay-files` list (which replaces inheritance with a different set of files), `azldev` emits an `slog.Warn` naming the shadowed pattern so you can spot the genuine displacement. An explicit declaration that leaves `overlay-files` unset still inherits the project pattern, so the discovered files apply and no warning is emitted. An explicit empty list (`overlay-files = []`) is the documented off-switch for a component and is not treated as shadowing. +Multiple patterns may select overlays for the same component. Files are loaded in pattern order, with duplicate file matches loaded only once. -**Limitation: explicit component-group members need a `[components.]` table.** A component group can enumerate its members two ways: via a `specs` glob (spec files on disk) or via an explicit `components = ["", ...]` list. Every name in the *explicit list* must have a matching `[components.]` table in the project (validated at load time, error `ErrUndefinedComponent`); pattern discovery alone does not satisfy this validation. For example, if `anaconda` is explicitly listed, retain its declaration centrally rather than adding an empty dedicated file: +**Explicit component-group members need a `[components.]` table.** A component group can enumerate its members two ways: via a `specs` glob (spec files on disk) or via an explicit `components = ["", ...]` list. Every name in the *explicit list* must have a matching `[components.]` table in the project (validated at load time, error `ErrUndefinedComponent`). For example, retain `anaconda` centrally rather than adding an empty dedicated file: ```toml # base/comps/components.toml (included at project scope) @@ -362,12 +362,12 @@ overlay-files = ["{component}/overlays/*.overlay.toml"] components = ["anaconda"] ``` -Once the table exists, the component inherits the project-level `{component}` overlay pattern normally. Group members discovered via `specs` (not the explicit list) do **not** need a `.comp.toml` — the resolver synthesizes an empty config for them and inheritance still applies. +Once the table exists, the component inherits the project-level `{component}` overlay pattern normally. Group members matched via `specs` (not the explicit list) do **not** need a `.comp.toml` — the resolver synthesizes an empty config for them and inheritance still applies. Keep a dedicated `.comp.toml` when it contains actual component-specific settings, such as upstream pins, build options, inline overlays, or local-spec configuration. -Components without matching overlay files still need an explicit declaration or -another supported discovery mechanism, such as a group's `specs` glob. +Components with or without matching overlay files need an explicit declaration or +another supported mechanism, such as a group's `specs` glob. ## Examples diff --git a/internal/app/azldev/agentskill/agentskill_test.go b/internal/app/azldev/agentskill/agentskill_test.go index db2ea2f04..a943bd060 100644 --- a/internal/app/azldev/agentskill/agentskill_test.go +++ b/internal/app/azldev/agentskill/agentskill_test.go @@ -480,7 +480,7 @@ func TestFilesGitHubLayout(t *testing.T) { assert.Contains(t, mockFile.Content, "name: azldev-mock") } -func TestComponentSkillsExplainDeclarationFreeOverlays(t *testing.T) { +func TestComponentSkillsExplainCentralDeclarationsAndPlaceholderOverlays(t *testing.T) { for _, mode := range []struct { name string withoutLockfile bool @@ -503,12 +503,19 @@ func TestComponentSkillsExplainDeclarationFreeOverlays(t *testing.T) { assert.Contains(t, doc, "base/comps/components.toml") assert.Contains(t, doc, "without matching overlay files") assert.Contains(t, doc, "upstream pins, build options, inline overlays,") - assert.Contains(t, doc, "Discovery is opt-in.") + assert.Contains(t, doc, "do not discover undeclared components") + assert.NotContains(t, doc, "Declaration-free") + assert.NotContains(t, doc, "Discovery is opt-in.") assert.Contains(t, doc, "unchanged path resolution and precedence") assert.Contains(t, doc, "Distro-version and component-group defaults must omit `overlay-files`, including empty lists.") assert.NotContains(t, doc, "Every component lives under") assert.NotContains(t, doc, "anything that needs overlays") + + if name == "azldev-overlays" { + assert.Contains(t, doc, "Additional braces are forbidden after `{component}`") + assert.Contains(t, doc, "use separate list entries for multiple directories") + } }) } }) diff --git a/internal/app/azldev/agentskill/content/add-component.md.tmpl b/internal/app/azldev/agentskill/content/add-component.md.tmpl index 7405b5e3f..ae50328b7 100644 --- a/internal/app/azldev/agentskill/content/add-component.md.tmpl +++ b/internal/app/azldev/agentskill/content/add-component.md.tmpl @@ -39,24 +39,25 @@ dist-git often fail bot detection): 4. Read the spec and plan any overlays. -## Declaration-free, inline, or dedicated +## Central or dedicated declarations For components whose only customization is per-file overlays, add files under `base/comps//overlays/`; do not create an empty `.comp.toml`. This requires a shared project-level `default-component-config.overlay-files` pattern such as `base/comps/{component}/overlays/*.overlay.toml` in the root config. The `{component}` -pattern discovers components from matching files and attaches their overlays. -Remove any provisional bare entry once discovery supplies it. +pattern substitutes each resolved component's name to load its overlays. +Retain `[components.]` in `base/comps/components.toml`. +Overlay files do not discover undeclared components. -Discovery is opt-in. Plain legacy patterns such as `overlays/*.overlay.toml` and +Plain legacy patterns such as `overlays/*.overlay.toml` and `[]` remain supported in project defaults and component settings, with unchanged path resolution and precedence; they do not discover components. Distro-version and component-group defaults must omit `overlay-files`, including empty lists. Only project-level defaults may use `{component}`. If a component is explicitly listed in a component group, retain `[components.]` -in `base/comps/components.toml`: pattern discovery does not satisfy explicit group -membership validation. Components without matching overlay files still need an +in `base/comps/components.toml` for membership validation. Components with or +without matching overlay files still need an explicit declaration or another supported discovery mechanism (such as a group's `specs` glob). diff --git a/internal/app/azldev/agentskill/content/comp-toml.md.tmpl b/internal/app/azldev/agentskill/content/comp-toml.md.tmpl index c049a5bfd..43929d63c 100644 --- a/internal/app/azldev/agentskill/content/comp-toml.md.tmpl +++ b/internal/app/azldev/agentskill/content/comp-toml.md.tmpl @@ -7,7 +7,8 @@ description: {{ printf "%q" .Description }} A component definition tells azldev where a package's spec comes from and how to customize it for your distro. Explicit definitions live under `[components.]`; -components can also be discovered from matching overlay files without a declaration. +components may also be identified by component-group spec globs, but overlay files +do not discover undeclared components. Get the authoritative, always-current field list from the schema: @@ -103,21 +104,21 @@ the rare spec whose `Source`/`Patch` filenames use macros the filter cannot expa ## File organization -Discovery is opt-in. Plain legacy patterns such as `overlays/*.overlay.toml` and +Plain legacy patterns such as `overlays/*.overlay.toml` and `[]` remain supported in project defaults and component settings, with unchanged path resolution and precedence; they do not discover components. Distro-version and component-group defaults must omit `overlay-files`, including empty lists. Only project-level defaults may use `{component}`. -- **Declaration-free** — for components whose only customization is per-file +- **Central declarations** — for components whose only customization is per-file overlays, add files under `base/comps//overlays/`; do not create an empty `.comp.toml`. A shared project-level `default-component-config.overlay-files` pattern such as `base/comps/{component}/overlays/*.overlay.toml` in the root config - discovers these components and attaches their overlays. + substitutes the component's name to load its overlays. Retain + `[components.]` in `base/comps/components.toml`. - **Explicit group members** — if a component is explicitly listed in a component group's `components` list, retain `[components.]` in - `base/comps/components.toml`. Pattern discovery alone does not satisfy that - validation. Components without matching overlay files still need an explicit + `base/comps/components.toml`. Components with or without matching overlay files still need an explicit declaration or another supported discovery mechanism (such as a group's `specs` glob). - **Inline** — put simple, customization-free components directly in a shared config @@ -135,7 +136,7 @@ Start with `azldev comp list -p -q -O json`, then use For a change review, focus on the diff while checking enough surrounding context to ensure it fits the component and repository conventions. -- **Organization:** The component follows the repository's declaration-free, inline, or dedicated-file +- **Organization:** The component follows the repository's central or dedicated-file convention; its name matches upstream or sets `spec.upstream-name`; no stale or orphaned component files remain. - **Spec source:** The default upstream source is preferred. Pins explain why they are diff --git a/internal/app/azldev/agentskill/content/overlays.md.tmpl b/internal/app/azldev/agentskill/content/overlays.md.tmpl index 3213e78a3..65e6ba23d 100644 --- a/internal/app/azldev/agentskill/content/overlays.md.tmpl +++ b/internal/app/azldev/agentskill/content/overlays.md.tmpl @@ -33,7 +33,7 @@ overlays before it stay applied. Config errors reference the offending overlay by its `description`, so give every overlay a short, specific `description`. -## Shared patterns and declaration-free components +## Shared patterns and central declarations For components whose only customization is per-file overlays, add files under `base/comps//overlays/`; do not create an empty `.comp.toml`. Configure @@ -43,23 +43,26 @@ the shared pattern at project scope (paths are relative to the declaring config) # Root config, e.g. azldev.toml [default-component-config] overlay-files = ["base/comps/{component}/overlays/*.overlay.toml"] + +# These declarations can instead live in included base/comps/components.toml. +[components.anaconda] ``` -The `{component}` segment discovers components from matching files and attaches -their overlays. For example, `base/comps/anaconda/overlays/0001-branding.overlay.toml` -discovers `anaconda` without `[components.anaconda]`. An explicit component with +The `{component}` segment substitutes the resolved component's name. For example, +`anaconda` loads `base/comps/anaconda/overlays/0001-branding.overlay.toml`. +Overlay files do not discover undeclared components. A component with no `overlay-files` of its own inherits the pattern; its own list replaces it, and `overlay-files = []` disables inheritance. If a component is explicitly listed in a component group's `components` list, -retain `[components.]` in `base/comps/components.toml`: pattern discovery does -not satisfy explicit group membership validation. Components without matching +retain `[components.]` in `base/comps/components.toml` for membership +validation. Components with or without matching overlay files still need an explicit declaration or another supported discovery mechanism (such as a group's `specs` glob). Keep a dedicated `.comp.toml` when it contains actual component-specific settings, such as upstream pins, build options, inline overlays, or local-spec configuration. -Discovery is opt-in. Plain legacy patterns such as `overlays/*.overlay.toml` and +Plain legacy patterns such as `overlays/*.overlay.toml` and `[]` remain supported in project defaults and component settings, with unchanged path resolution and precedence; they do not discover components. Distro-version and component-group defaults must omit `overlay-files`, including empty lists. @@ -69,15 +72,17 @@ an empty list disables inherited overlay files. Only project-level defaults may use `{component}`. Placeholder entries require it exactly once as a whole directory segment, with no glob metacharacters -(`*`, `?`, `[`, `{`) before it and a file pattern after it. Brace alternatives -are allowed only after the placeholder. `..` path segments are allowed before +(`*`, `?`, `[`, `{`) before it and a file pattern after it. Additional braces +are forbidden after `{component}`; use separate list entries for multiple directories. +Ordinary patterns without `{component}` retain their existing brace support. +`..` path segments are allowed before `{component}`, but never after it. For example, `../comps/{component}/overlays/*.overlay.toml` is valid; simplify `comps/{component}/nested/../overlays/*.overlay.toml` to `comps/{component}/overlays/*.overlay.toml`. Use `/` separators. Component settings allow plain patterns and `[]` but reject the placeholder. -Discovered components use the complete inherited project overlay list, -not just the discovery matches. An inherited placeholder entry with no matching +Components use the complete inherited project overlay list. +An inherited placeholder entry with no matching files is optional, even with a literal suffix; an ordinary literal path must exist. ## Diagnose common failures diff --git a/internal/app/azldev/agentskill/content/withoutlockfile/add-component.md.tmpl b/internal/app/azldev/agentskill/content/withoutlockfile/add-component.md.tmpl index 7eb2d7cec..404c76e51 100644 --- a/internal/app/azldev/agentskill/content/withoutlockfile/add-component.md.tmpl +++ b/internal/app/azldev/agentskill/content/withoutlockfile/add-component.md.tmpl @@ -39,24 +39,25 @@ dist-git often fail bot detection): 4. Read the spec and plan any overlays. -## Declaration-free, inline, or dedicated +## Central or dedicated declarations For components whose only customization is per-file overlays, add files under `base/comps//overlays/`; do not create an empty `.comp.toml`. This requires a shared project-level `default-component-config.overlay-files` pattern such as `base/comps/{component}/overlays/*.overlay.toml` in the root config. The `{component}` -pattern discovers components from matching files and attaches their overlays. -Remove any provisional bare entry once discovery supplies it. +pattern substitutes each resolved component's name to load its overlays. +Retain `[components.]` in `base/comps/components.toml`. +Overlay files do not discover undeclared components. -Discovery is opt-in. Plain legacy patterns such as `overlays/*.overlay.toml` and +Plain legacy patterns such as `overlays/*.overlay.toml` and `[]` remain supported in project defaults and component settings, with unchanged path resolution and precedence; they do not discover components. Distro-version and component-group defaults must omit `overlay-files`, including empty lists. Only project-level defaults may use `{component}`. If a component is explicitly listed in a component group, retain `[components.]` -in `base/comps/components.toml`: pattern discovery does not satisfy explicit group -membership validation. Components without matching overlay files still need an +in `base/comps/components.toml` for membership validation. Components with or +without matching overlay files still need an explicit declaration or another supported discovery mechanism (such as a group's `specs` glob). diff --git a/internal/app/azldev/agentskill/content/withoutlockfile/comp-toml.md.tmpl b/internal/app/azldev/agentskill/content/withoutlockfile/comp-toml.md.tmpl index 9df1cdd12..91bd6536d 100644 --- a/internal/app/azldev/agentskill/content/withoutlockfile/comp-toml.md.tmpl +++ b/internal/app/azldev/agentskill/content/withoutlockfile/comp-toml.md.tmpl @@ -7,7 +7,8 @@ description: {{ printf "%q" .Description }} A component definition tells azldev where a package's spec comes from and how to customize it for your distro. Explicit definitions live under `[components.]`; -components can also be discovered from matching overlay files without a declaration. +components may also be identified by component-group spec globs, but overlay files +do not discover undeclared components. Get the authoritative, always-current field list from the schema: @@ -93,21 +94,21 @@ the rare spec whose `Source`/`Patch` filenames use macros the filter cannot expa ## File organization -Discovery is opt-in. Plain legacy patterns such as `overlays/*.overlay.toml` and +Plain legacy patterns such as `overlays/*.overlay.toml` and `[]` remain supported in project defaults and component settings, with unchanged path resolution and precedence; they do not discover components. Distro-version and component-group defaults must omit `overlay-files`, including empty lists. Only project-level defaults may use `{component}`. -- **Declaration-free** — for components whose only customization is per-file +- **Central declarations** — for components whose only customization is per-file overlays, add files under `base/comps//overlays/`; do not create an empty `.comp.toml`. A shared project-level `default-component-config.overlay-files` pattern such as `base/comps/{component}/overlays/*.overlay.toml` in the root config - discovers these components and attaches their overlays. + substitutes the component's name to load its overlays. Retain + `[components.]` in `base/comps/components.toml`. - **Explicit group members** — if a component is explicitly listed in a component group's `components` list, retain `[components.]` in - `base/comps/components.toml`. Pattern discovery alone does not satisfy that - validation. Components without matching overlay files still need an explicit + `base/comps/components.toml`. Components with or without matching overlay files still need an explicit declaration or another supported discovery mechanism (such as a group's `specs` glob). - **Inline** — put simple, customization-free components directly in a shared config @@ -126,7 +127,7 @@ details, inspect the rendered spec under the diff while checking enough surrounding context to ensure it fits the component and repository conventions. -- **Organization:** The component follows the repository's declaration-free, inline, or dedicated-file +- **Organization:** The component follows the repository's central or dedicated-file convention; its name matches upstream or sets `spec.upstream-name`; no stale or orphaned component files remain. - **Spec source:** The default upstream source is preferred. Pins explain why they are diff --git a/internal/app/azldev/agentskill/content/withoutlockfile/overlays.md.tmpl b/internal/app/azldev/agentskill/content/withoutlockfile/overlays.md.tmpl index 915fa6d8d..e1ec9dd94 100644 --- a/internal/app/azldev/agentskill/content/withoutlockfile/overlays.md.tmpl +++ b/internal/app/azldev/agentskill/content/withoutlockfile/overlays.md.tmpl @@ -33,7 +33,7 @@ overlays before it stay applied. Config errors reference the offending overlay by its `description`, so give every overlay a short, specific `description`. -## Shared patterns and declaration-free components +## Shared patterns and central declarations For components whose only customization is per-file overlays, add files under `base/comps//overlays/`; do not create an empty `.comp.toml`. Configure @@ -43,23 +43,26 @@ the shared pattern at project scope (paths are relative to the declaring config) # Root config, e.g. azldev.toml [default-component-config] overlay-files = ["base/comps/{component}/overlays/*.overlay.toml"] + +# These declarations can instead live in included base/comps/components.toml. +[components.anaconda] ``` -The `{component}` segment discovers components from matching files and attaches -their overlays. For example, `base/comps/anaconda/overlays/0001-branding.overlay.toml` -discovers `anaconda` without `[components.anaconda]`. An explicit component with +The `{component}` segment substitutes the resolved component's name. For example, +`anaconda` loads `base/comps/anaconda/overlays/0001-branding.overlay.toml`. +Overlay files do not discover undeclared components. A component with no `overlay-files` of its own inherits the pattern; its own list replaces it, and `overlay-files = []` disables inheritance. If a component is explicitly listed in a component group's `components` list, -retain `[components.]` in `base/comps/components.toml`: pattern discovery does -not satisfy explicit group membership validation. Components without matching +retain `[components.]` in `base/comps/components.toml` for membership +validation. Components with or without matching overlay files still need an explicit declaration or another supported discovery mechanism (such as a group's `specs` glob). Keep a dedicated `.comp.toml` when it contains actual component-specific settings, such as upstream pins, build options, inline overlays, or local-spec configuration. -Discovery is opt-in. Plain legacy patterns such as `overlays/*.overlay.toml` and +Plain legacy patterns such as `overlays/*.overlay.toml` and `[]` remain supported in project defaults and component settings, with unchanged path resolution and precedence; they do not discover components. Distro-version and component-group defaults must omit `overlay-files`, including empty lists. @@ -69,15 +72,17 @@ an empty list disables inherited overlay files. Only project-level defaults may use `{component}`. Placeholder entries require it exactly once as a whole directory segment, with no glob metacharacters -(`*`, `?`, `[`, `{`) before it and a file pattern after it. Brace alternatives -are allowed only after the placeholder. `..` path segments are allowed before +(`*`, `?`, `[`, `{`) before it and a file pattern after it. Additional braces +are forbidden after `{component}`; use separate list entries for multiple directories. +Ordinary patterns without `{component}` retain their existing brace support. +`..` path segments are allowed before `{component}`, but never after it. For example, `../comps/{component}/overlays/*.overlay.toml` is valid; simplify `comps/{component}/nested/../overlays/*.overlay.toml` to `comps/{component}/overlays/*.overlay.toml`. Use `/` separators. Component settings allow plain patterns and `[]` but reject the placeholder. -Discovered components use the complete inherited project overlay list, -not just the discovery matches. An inherited placeholder entry with no matching +Components use the complete inherited project overlay list. +An inherited placeholder entry with no matching files is optional, even with a literal suffix; an ordinary literal path must exist. ## Diagnose common failures diff --git a/internal/app/azldev/core/components/pattern_discovery.go b/internal/app/azldev/core/components/pattern_discovery.go deleted file mode 100644 index b94b0bde9..000000000 --- a/internal/app/azldev/core/components/pattern_discovery.go +++ /dev/null @@ -1,217 +0,0 @@ -// Copyright (c) Microsoft Corporation. -// Licensed under the MIT License. - -package components - -import ( - "errors" - "fmt" - "log/slog" - "path" - "path/filepath" - "slices" - "strings" - - "github.com/bmatcuk/doublestar/v4" - "github.com/microsoft/azure-linux-dev-tools/internal/app/azldev" - "github.com/microsoft/azure-linux-dev-tools/internal/projectconfig" - "github.com/microsoft/azure-linux-dev-tools/internal/utils/fileutils" -) - -// patternDiscoveredComponent captures the result of expanding a single -// project-level [projectconfig.ComponentConfig.OverlayFiles] entry that -// contains the `{component}` placeholder. -type patternDiscoveredComponent struct { - // Name of the discovered component (the captured {component} segment). - Name string - // ReferenceDir is the absolute directory the pattern captured for this - // component (i.e. the prefix directory + captured name). Used as the - // component's reference directory for downstream operations. - ReferenceDir string - // OverlayFiles is the deduplicated, sorted list of absolute overlay file - // paths matched by the winning pattern for this component. - OverlayFiles []string - // SourcePattern is the exact entry string that produced this match, used - // in shadow-lint diagnostics. - SourcePattern string -} - -// discoverComponentsFromPatterns walks the project-level -// default-component-config, expands every 'overlay-files' entry that contains -// the `{component}` placeholder, and returns the resulting pattern-discovered -// components. Two entries at project scope producing the same component name -// is a hard error. -// -// The caller is responsible for further precedence against explicitly-declared -// components (that override happens at the resolver layer; see -// [Resolver.patternDiscoveredComponents]). -func discoverComponentsFromPatterns(env *azldev.Env) (map[string]*patternDiscoveredComponent, error) { - result := make(map[string]*patternDiscoveredComponent) - - config := env.Config() - if config == nil { - return result, nil - } - - var patterns []string - - for _, entry := range config.DefaultComponentConfig.OverlayFiles { - if strings.Contains(entry, projectconfig.OverlayFilesComponentPlaceholder) { - patterns = append(patterns, entry) - } - } - - if len(patterns) == 0 { - return result, nil - } - - for _, pattern := range patterns { - byName, err := expandSinglePattern(env, pattern) - if err != nil { - return nil, fmt.Errorf("project 'overlay-files': %w", err) - } - - for name, match := range byName { - if existing, ok := result[name]; ok { - return nil, fmt.Errorf( - "%w: component %#q is discovered by both %#q and %#q; "+ - "restrict one of the patterns or rename one of the directories", - ErrPatternDiscoveryCollision, name, existing.SourcePattern, match.SourcePattern, - ) - } - - result[name] = match - } - } - - return result, nil -} - -// expandSinglePattern expands a single validated pattern and groups matched -// files by captured component name. -func expandSinglePattern(env *azldev.Env, pattern string) (map[string]*patternDiscoveredComponent, error) { - prefix, suffix := projectconfig.SplitOverlayFilesPlaceholder(pattern) - - // Substitute a single-segment wildcard for {component} to build the actual - // glob. Because {component} is validated as a whole path segment (see - // ConfigFile.Validate), replacing it with "*" produces a well-formed - // doublestar pattern that captures exactly one path segment. - globPattern := prefix + "*" + suffix - - // doublestar.WithFailOnIOErrors() is intentionally not set, mirroring - // findComponentGroupSpecPaths: we may legitimately hit unreadable subtrees - // under the project root. Non-IO errors (bad pattern, etc.) are still - // returned. - matches, err := fileutils.Glob(env.FS(), globPattern, doublestar.WithFilesOnly()) - if err != nil { - return nil, fmt.Errorf("failed to expand pattern %#q:\n%w", pattern, err) - } - - byName := make(map[string]*patternDiscoveredComponent) - - for _, match := range matches { - name, referenceDir, err := extractCapturedName(pattern, prefix, match) - if err != nil { - // Skip matches we can't parse (should not happen with a validated - // pattern, but guard anyway). - slog.Debug("pattern-discovery: skipping unparseable match", - "pattern", pattern, "match", match, "err", err) - - continue - } - - // Reject matches whose captured segment is not a safe component name - // (spaces, path traversal, absolute paths, etc). Downstream code uses - // the name as a filesystem path fragment (see fileutils.ValidateFilename) - // so we fail fast rather than let it explode later. - if err := fileutils.ValidateFilename(name); err != nil { - slog.Debug("pattern-discovery: skipping match with unsafe captured name", - "pattern", pattern, "match", match, "name", name, "err", err) - - continue - } - - bucket, ok := byName[name] - if !ok { - bucket = &patternDiscoveredComponent{ - Name: name, - ReferenceDir: referenceDir, - SourcePattern: pattern, - } - byName[name] = bucket - } - - bucket.OverlayFiles = append(bucket.OverlayFiles, match) - } - - // Sort the overlay-files list per component (filename first, then full path). - for _, bucket := range byName { - slices.SortFunc(bucket.OverlayFiles, func(left, right string) int { - if result := strings.Compare(filepath.Base(left), filepath.Base(right)); result != 0 { - return result - } - - return strings.Compare(left, right) - }) - } - - return byName, nil -} - -// extractCapturedName strips prefix from match and returns the first path -// segment (the captured component name) together with the concrete directory -// that captured segment lives in. The returned ReferenceDir is the natural -// per-component anchor for downstream operations. -func extractCapturedName(pattern, prefix, match string) (name, referenceDir string, err error) { - rel := match - if prefix != "" { - if !strings.HasPrefix(match, prefix) { - return "", "", fmt.Errorf( - "match %#q does not start with expected prefix %#q for pattern %#q", - match, prefix, pattern, - ) - } - - rel = match[len(prefix):] - } - - // The captured segment is everything up to (but not including) the first - // path separator in the residual, or the entire residual if there is no - // suffix. Patterns are validated to use '/' only. - sep := strings.IndexByte(rel, '/') - if sep < 0 { - name = rel - } else { - name = rel[:sep] - } - - if name == "" { - return "", "", fmt.Errorf( - "empty captured component name for match %#q against pattern %#q", - match, pattern, - ) - } - - // The reference dir is the prefix + captured segment (no trailing separator). - // Use path.Clean (POSIX '/') rather than filepath.Clean so ReferenceDir stays - // consistent with the placeholder contract, which validates and operates on - // POSIX-style separators only (filepath.Clean would emit '\' on Windows). - referenceDir = path.Clean(prefix + name) - - return name, referenceDir, nil -} - -// logShadowedByDeclaration emits a warning when an explicit component -// declaration overrides pattern-discovered overlay files for the same name. -func logShadowedByDeclaration(shadowed *patternDiscoveredComponent) { - slog.Warn( - "project 'overlay-files' pattern match shadowed by explicit component declaration; overlay files will not be applied", - "component", shadowed.Name, - "shadowed_pattern", shadowed.SourcePattern, - "shadowed_files", shadowed.OverlayFiles, - ) -} - -// ErrPatternDiscoveryCollision wraps project-scope pattern collision errors so -// tests can assert on them. -var ErrPatternDiscoveryCollision = errors.New("project 'overlay-files' pattern collision") diff --git a/internal/app/azldev/core/components/pattern_discovery_test.go b/internal/app/azldev/core/components/pattern_discovery_test.go deleted file mode 100644 index e4dda7c5f..000000000 --- a/internal/app/azldev/core/components/pattern_discovery_test.go +++ /dev/null @@ -1,223 +0,0 @@ -// Copyright (c) Microsoft Corporation. -// Licensed under the MIT License. - -package components_test - -import ( - "testing" - - "github.com/microsoft/azure-linux-dev-tools/internal/app/azldev/core/components" - "github.com/microsoft/azure-linux-dev-tools/internal/app/azldev/core/testutils" - "github.com/microsoft/azure-linux-dev-tools/internal/projectconfig" - "github.com/microsoft/azure-linux-dev-tools/internal/utils/fileperms" - "github.com/microsoft/azure-linux-dev-tools/internal/utils/fileutils" - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" -) - -const patternTestOverlayContent = ` -[metadata] -category = "azl-branding-policy" -upstream-status = "inapplicable" - -[[overlays]] -description = "Bump release" -type = "spec-search-replace" -section = "%install" -regex = "unused" -` - -// writeOverlayFile writes a minimal but valid overlay TOML at path. -func writeOverlayFile(t *testing.T, env *testutils.TestEnv, path string) { - t.Helper() - - require.NoError(t, fileutils.MkdirAll(env.TestFS, dirOf(path))) - require.NoError(t, fileutils.WriteFile( - env.TestFS, path, []byte(patternTestOverlayContent), fileperms.PrivateFile, - )) -} - -func dirOf(path string) string { - for i := len(path) - 1; i >= 0; i-- { - if path[i] == '/' { - return path[:i] - } - } - - return "." -} - -func TestFindAllComponents_PatternDiscoveryAtProjectScope(t *testing.T) { - env := testutils.NewTestEnv(t) - - env.Config.DefaultComponentConfig.OverlayFiles = []string{ - "/project/comps/{component}/overlays/*.overlay.toml", - } - - writeOverlayFile(t, env, "/project/comps/foo/overlays/0001.overlay.toml") - writeOverlayFile(t, env, "/project/comps/foo/overlays/0002.overlay.toml") - writeOverlayFile(t, env, "/project/comps/bar/overlays/0001.overlay.toml") - - all, err := components.NewResolver(env.Env).FindAllComponents() - require.NoError(t, err) - - assert.ElementsMatch(t, []string{"foo", "bar"}, all.Names()) - - foo, ok := all.TryGet("foo") - require.True(t, ok) - assert.Len(t, foo.GetConfig().Overlays, 2, "both foo overlay files should attach") - - bar, ok := all.TryGet("bar") - require.True(t, ok) - assert.Len(t, bar.GetConfig().Overlays, 1) -} - -func TestFindAllComponents_PatternDiscoveryCompleteInheritance(t *testing.T) { - env := testutils.NewTestEnv(t) - env.Config.DefaultComponentConfig.OverlayFiles = []string{ - "/project/shared/*.overlay.toml", - "/project/comps/{component}/overlays/*.overlay.toml", - } - - for path, value := range map[string]string{ - "/project/shared/shared.overlay.toml": "shared", - "/project/comps/curl/overlays/fix[1].overlay.toml": "brackets", - "/project/comps/curl/overlays/fix1.overlay.toml": "sibling", - } { - require.NoError(t, fileutils.WriteFile(env.TestFS, path, - []byte(patternTestOverlayContent+"\nreplacement = \""+value+"\"\n"), fileperms.PrivateFile)) - } - - all, err := components.NewResolver(env.Env).FindAllComponents() - require.NoError(t, err) - - found, ok := all.TryGet("curl") - require.True(t, ok) - require.Nil(t, found.GetConfig().SourceConfigFile) - - values := make([]string, 0, len(found.GetConfig().Overlays)) - for _, overlay := range found.GetConfig().Overlays { - values = append(values, overlay.Replacement) - } - - assert.Equal(t, []string{"shared", "sibling", "brackets"}, values) -} - -// Declared components with no 'overlay-files' inherit the project pattern; the -// placeholder is substituted with the component's own name at glob time. -func TestFindAllComponents_DeclaredComponentInheritsProjectPattern(t *testing.T) { - env := testutils.NewTestEnv(t) - - env.Config.DefaultComponentConfig.OverlayFiles = []string{ - "/project/comps/{component}/overlays/*.overlay.toml", - } - - writeOverlayFile(t, env, "/project/comps/foo/overlays/0001.overlay.toml") - writeOverlayFile(t, env, "/project/comps/foo/overlays/0002.overlay.toml") - - env.Config.Components["foo"] = projectconfig.ComponentConfig{Name: "foo"} - - all, err := components.NewResolver(env.Env).FindAllComponents() - require.NoError(t, err) - - foo, ok := all.TryGet("foo") - require.True(t, ok) - assert.Len(t, foo.GetConfig().Overlays, 2, - "foo should inherit the project pattern with {component}=foo") -} - -// A component that sets its own 'overlay-files' replaces the inherited project -// pattern wholesale; the placeholder pattern no longer applies. -func TestFindAllComponents_ComponentOverlayFilesReplacesInherited(t *testing.T) { - env := testutils.NewTestEnv(t) - - env.Config.DefaultComponentConfig.OverlayFiles = []string{ - "/project/comps/{component}/overlays/*.overlay.toml", - } - - // This file would match the inherited pattern (component=bar). - writeOverlayFile(t, env, "/project/comps/bar/overlays/0001.overlay.toml") - - // But this file lives elsewhere and is the only one bar's config points to. - writeOverlayFile(t, env, "/project/custom/bar/0001.overlay.toml") - - env.Config.Components["bar"] = projectconfig.ComponentConfig{ - Name: "bar", - OverlayFiles: []string{"/project/custom/bar/*.overlay.toml"}, - } - - all, err := components.NewResolver(env.Env).FindAllComponents() - require.NoError(t, err) - - bar, ok := all.TryGet("bar") - require.True(t, ok) - assert.Len(t, bar.GetConfig().Overlays, 1, - "bar overrides overlay-files wholesale; inherited pattern must not apply") -} - -// Empty per-component 'overlay-files' disables the inherited project pattern. -func TestFindAllComponents_ComponentOverlayFilesEmptyDisablesInheritance(t *testing.T) { - env := testutils.NewTestEnv(t) - - env.Config.DefaultComponentConfig.OverlayFiles = []string{ - "/project/comps/{component}/overlays/*.overlay.toml", - } - - writeOverlayFile(t, env, "/project/comps/baz/overlays/0001.overlay.toml") - - env.Config.Components["baz"] = projectconfig.ComponentConfig{ - Name: "baz", - OverlayFiles: []string{}, - } - - all, err := components.NewResolver(env.Env).FindAllComponents() - require.NoError(t, err) - - baz, ok := all.TryGet("baz") - require.True(t, ok) - assert.Empty(t, baz.GetConfig().Overlays, - "empty overlay-files must disable inherited pattern") -} - -func TestFindAllComponents_PatternDiscoverySameScopeCollisionErrors(t *testing.T) { - env := testutils.NewTestEnv(t) - - env.Config.DefaultComponentConfig.OverlayFiles = []string{ - "/project/a/{component}/*.overlay.toml", - "/project/b/{component}/*.overlay.toml", - } - - writeOverlayFile(t, env, "/project/a/dup/0001.overlay.toml") - writeOverlayFile(t, env, "/project/b/dup/0002.overlay.toml") - - _, err := components.NewResolver(env.Env).FindAllComponents() - require.Error(t, err) - require.ErrorIs(t, err, components.ErrPatternDiscoveryCollision, - "expected ErrPatternDiscoveryCollision, got %v", err) - assert.Contains(t, err.Error(), "dup") -} - -// A pattern match whose captured segment isn't a safe component name -// (spaces, dot-directory, etc.) is skipped without failing overall discovery. -func TestFindAllComponents_PatternDiscoverySkipsUnsafeCapturedNames(t *testing.T) { - env := testutils.NewTestEnv(t) - - env.Config.DefaultComponentConfig.OverlayFiles = []string{ - "/project/comps/{component}/overlays/*.overlay.toml", - } - - // Well-formed match — should be discovered. - writeOverlayFile(t, env, "/project/comps/openssl/overlays/0001.overlay.toml") - - // Unsafe captured segments — should be skipped without erroring the whole - // discovery. These simulate directories a user might create that don't - // correspond to valid component names. - writeOverlayFile(t, env, "/project/comps/has space/overlays/0001.overlay.toml") - writeOverlayFile(t, env, "/project/comps/./overlays/0001.overlay.toml") - - all, err := components.NewResolver(env.Env).FindAllComponents() - require.NoError(t, err) - - assert.ElementsMatch(t, []string{"openssl"}, all.Names(), - "only the safely-named directory should be discovered") -} diff --git a/internal/app/azldev/core/components/resolver.go b/internal/app/azldev/core/components/resolver.go index ba13e96a9..bec9fb447 100644 --- a/internal/app/azldev/core/components/resolver.go +++ b/internal/app/azldev/core/components/resolver.go @@ -7,12 +7,10 @@ import ( "errors" "fmt" "log/slog" - "maps" "path" "path/filepath" "slices" "strings" - "sync" "github.com/bmatcuk/doublestar/v4" "github.com/microsoft/azure-linux-dev-tools/internal/app/azldev" @@ -44,22 +42,13 @@ type Resolver struct { // Only 'component update' enables this — it uses the freshness status to // decide which components need re-resolution and which can be skipped. CheckFreshness bool - - // patternDiscovery lazily holds the result of pattern-based component - // discovery so repeated resolver calls don't re-glob the filesystem. - patternDiscovery func() (map[string]*patternDiscoveredComponent, error) } // NewResolver constructs a new [Resolver] for the given environment. func NewResolver(env *azldev.Env) *Resolver { - resolver := &Resolver{ + return &Resolver{ env: env, } - resolver.patternDiscovery = sync.OnceValues(func() (map[string]*patternDiscoveredComponent, error) { - return discoverComponentsFromPatterns(env) - }) - - return resolver } // Given a component filter, finds all components defined in the environment that match the filter. @@ -120,14 +109,9 @@ func (r *Resolver) FindComponents(filter *ComponentFilter) (components *Componen return components, r.validateLockFiles(components, false, skipValidation) } -// Finds *all* components defined in the environment. This unions three sources: +// Finds *all* components defined in the environment. This unions two sources: // - components reached via component-group spec-path patterns, -// - loose components declared in the project config, and -// - components discovered by a project-level 'overlay-files' entry containing -// the '{component}' placeholder. -// -// Explicit declarations shadow pattern-discovered matches of the same name; a -// warning is emitted for the shadowed entry. +// - loose components declared in the project config. func (r *Resolver) FindAllComponents() (components *ComponentSet, err error) { components = NewComponentSet() @@ -185,47 +169,6 @@ func (r *Resolver) FindAllComponents() (components *ComponentSet, err error) { components.Add(comp) } - // Finally, add pattern-discovered components (from project-level 'overlay-files' - // entries containing '{component}') that were not already covered by an explicit - // component declaration. When an explicit declaration exists but doesn't set its - // own overlay-files, it inherits the project pattern and still receives the - // discovered files (no warning). A warning is only emitted when the explicit - // declaration replaces overlay-files with a non-empty list, genuinely displacing - // the discovered files. An explicit empty list ('overlay-files = []') is the - // documented off-switch and is not treated as shadowing. - discovered, err := r.patternDiscoveredComponents() - if err != nil { - return components, err - } - - for _, name := range slices.Sorted(maps.Keys(discovered)) { - entry := discovered[name] - - if components.Contains(name) { - // Only warn when the explicit declaration provides a non-empty - // 'overlay-files' list, which replaces inheritance with a different - // set of files (see [projectconfig.ComponentConfig.MergeUpdatesFrom]). - // A declaration with no 'overlay-files' inherits the pattern and the - // discovered files still apply. An explicit empty list is the - // documented way to opt out of overlays for a component and is not - // treated as shadowing. - // - // Inherited legacy defaults retain their existing precedence. - if declared, ok := r.env.Config().Components[name]; ok && len(declared.OverlayFiles) > 0 { - logShadowedByDeclaration(entry) - } - - continue - } - - comp, addErr := r.componentFromPatternDiscovered(entry, nil) - if addErr != nil { - return components, addErr - } - - components.Add(comp) - } - return components, nil } @@ -333,10 +276,6 @@ func (r *Resolver) GetComponentGroupByName(componentGroupName string) (component componentGroup.Components = append(componentGroup.Components, groupEntry) } - // Pattern-discovered components (from a project-level 'overlay-files' entry - // containing '{component}') do not belong to any specific component group; - // they are added to the project-wide component set in FindAllComponents. - return componentGroup, nil } @@ -955,32 +894,6 @@ func componentGroupNames(env *azldev.Env, componentName string, extraGroupNames return groupNames } -// patternDiscoveredComponents returns the (lazily-computed) map of components -// discovered by a project-level 'overlay-files' entry containing the -// '{component}' placeholder. Returns an empty (non-nil) map and nil error when -// the project default-component-config declares no such entry. -func (r *Resolver) patternDiscoveredComponents() (map[string]*patternDiscoveredComponent, error) { - return r.patternDiscovery() -} - -// componentFromPatternDiscovered synthesizes a [Component] for a pattern -// discovered entry, applying inherited defaults with the entry's captured -// directory as the overlay-files reference dir. -func (r *Resolver) componentFromPatternDiscovered( - entry *patternDiscoveredComponent, groupNames []string, -) (Component, error) { - synth := projectconfig.ComponentConfig{Name: entry.Name} - - updated, err := applyInheritedDefaultsToComponent(r.env, synth, entry.ReferenceDir, groupNames) - if err != nil { - return nil, fmt.Errorf( - "applying defaults to pattern-discovered component %#q:\n%w", entry.Name, err, - ) - } - - return r.createComponentFromConfig(updated) -} - // validateLockFiles checks lock file consistency against the resolved component // set. Skipped when skipValidation is true (set per-filter or via the global // '--skip-lock-validation' flag). diff --git a/internal/app/azldev/core/components/resolver_test.go b/internal/app/azldev/core/components/resolver_test.go index 914ee67e4..19aff1139 100644 --- a/internal/app/azldev/core/components/resolver_test.go +++ b/internal/app/azldev/core/components/resolver_test.go @@ -25,6 +25,27 @@ func touchFile(t *testing.T, testEnv *testutils.TestEnv, path string) { require.NoError(t, fileutils.WriteFile(testEnv.TestFS, path, []byte{}, fileperms.PrivateFile)) } +const patternTestOverlayContent = ` +[metadata] +category = "azl-branding-policy" +upstream-status = "inapplicable" + +[[overlays]] +description = "Bump release" +type = "spec-search-replace" +section = "%install" +regex = "unused" +` + +func writeOverlayFile(t *testing.T, env *testutils.TestEnv, path string) { + t.Helper() + + require.NoError(t, fileutils.MkdirAll(env.TestFS, filepath.Dir(path))) + require.NoError(t, fileutils.WriteFile( + env.TestFS, path, []byte(patternTestOverlayContent), fileperms.PrivateFile, + )) +} + // Creates a test spec and returns its expected config. func setupTestSpec(t *testing.T, testEnv *testutils.TestEnv, path string) projectconfig.ComponentConfig { t.Helper() @@ -789,6 +810,183 @@ value = "Microsoft" assert.Equal(t, projectconfig.OverlayCategoryAZLBrandingPolicy, overlays[0].Metadata.Category) } +func TestFindAllComponents_PlaceholderOnlyAppliesToKnownComponents(t *testing.T) { + for _, mode := range []struct { + name string + newEnv func(*testing.T) *testutils.TestEnv + }{ + {name: "locked", newEnv: testutils.NewTestEnv}, + {name: "without-lockfile", newEnv: testutils.NewTestEnvWithoutLockfile}, + } { + t.Run(mode.name, func(t *testing.T) { + env := mode.newEnv(t) + require.NoError(t, fileutils.WriteFile(env.TestFS, "/project/azldev.toml", + []byte("includes = ['base/comps/components.toml']\n"), fileperms.PrivateFile)) + require.NoError(t, fileutils.WriteFile(env.TestFS, "/project/base/comps/components.toml", + []byte("[default-component-config]\noverlay-files = ['{component}/overlays/*.overlay.toml']\n"+ + "[components.foo]\n[components.no-overlays]\n"), fileperms.PrivateFile)) + + _, loaded, err := projectconfig.LoadProjectConfig( + env.TestFS, env.TestOSEnv, "/project", true, "/tmp", nil, false, env.Env.WithoutLockfile()) + require.NoError(t, err) + + env.Config.Components = loaded.Components + env.Config.DefaultComponentConfig = loaded.DefaultComponentConfig + + writeOverlayFile(t, env, "/project/base/comps/foo/overlays/0001.overlay.toml") + writeOverlayFile(t, env, "/project/base/comps/foo/overlays/0002.overlay.toml") + // An undeclared directory must not add a component or cause its invalid file to be parsed. + require.NoError(t, fileutils.WriteFile(env.TestFS, + "/project/base/comps/bar/overlays/0001.overlay.toml", []byte("invalid TOML"), fileperms.PrivateFile)) + + resolver := components.NewResolver(env.Env) + all, err := resolver.FindAllComponents() + require.NoError(t, err) + assert.ElementsMatch(t, []string{"foo", "no-overlays"}, all.Names()) + + foo, ok := all.TryGet("foo") + require.True(t, ok) + assert.Len(t, foo.GetConfig().Overlays, 2) + + noOverlays, ok := all.TryGet("no-overlays") + require.True(t, ok) + assert.Empty(t, noOverlays.GetConfig().Overlays) + + _, err = resolver.GetComponentByName("bar") + require.ErrorContains(t, err, "component not found") + }) + } +} + +func TestFindAllComponents_PlaceholderCompleteInheritance(t *testing.T) { + env := testutils.NewTestEnv(t) + env.Config.Components["curl"] = projectconfig.ComponentConfig{Name: "curl"} + env.Config.DefaultComponentConfig.OverlayFiles = []string{ + "/project/shared/*.overlay.toml", + "/project/comps/{component}/overlays/*.overlay.toml", + } + + for path, value := range map[string]string{ + "/project/shared/shared.overlay.toml": "shared", + "/project/comps/curl/overlays/fix[1].overlay.toml": "brackets", + "/project/comps/curl/overlays/fix1.overlay.toml": "sibling", + } { + require.NoError(t, fileutils.WriteFile(env.TestFS, path, + []byte(patternTestOverlayContent+"\nreplacement = \""+value+"\"\n"), fileperms.PrivateFile)) + } + + all, err := components.NewResolver(env.Env).FindAllComponents() + require.NoError(t, err) + + found, ok := all.TryGet("curl") + require.True(t, ok) + require.Nil(t, found.GetConfig().SourceConfigFile) + + values := make([]string, 0, len(found.GetConfig().Overlays)) + for _, overlay := range found.GetConfig().Overlays { + values = append(values, overlay.Replacement) + } + + assert.Equal(t, []string{"shared", "sibling", "brackets"}, values) +} + +// Declared components with no 'overlay-files' inherit the project pattern; the +// placeholder is substituted with the component's own name at glob time. +func TestFindAllComponents_DeclaredComponentInheritsProjectPattern(t *testing.T) { + env := testutils.NewTestEnv(t) + + env.Config.DefaultComponentConfig.OverlayFiles = []string{ + "/project/comps/{component}/overlays/*.overlay.toml", + } + + writeOverlayFile(t, env, "/project/comps/foo/overlays/0001.overlay.toml") + writeOverlayFile(t, env, "/project/comps/foo/overlays/0002.overlay.toml") + + env.Config.Components["foo"] = projectconfig.ComponentConfig{Name: "foo"} + + all, err := components.NewResolver(env.Env).FindAllComponents() + require.NoError(t, err) + + foo, ok := all.TryGet("foo") + require.True(t, ok) + assert.Len(t, foo.GetConfig().Overlays, 2, + "foo should inherit the project pattern with {component}=foo") +} + +// A component that sets its own 'overlay-files' replaces the inherited project +// pattern wholesale; the placeholder pattern no longer applies. +func TestFindAllComponents_ComponentOverlayFilesReplacesInherited(t *testing.T) { + env := testutils.NewTestEnv(t) + + env.Config.DefaultComponentConfig.OverlayFiles = []string{ + "/project/comps/{component}/overlays/*.overlay.toml", + } + + // This file would match the inherited pattern (component=bar). + writeOverlayFile(t, env, "/project/comps/bar/overlays/0001.overlay.toml") + + // But this file lives elsewhere and is the only one bar's config points to. + writeOverlayFile(t, env, "/project/custom/bar/0001.overlay.toml") + + env.Config.Components["bar"] = projectconfig.ComponentConfig{ + Name: "bar", + OverlayFiles: []string{"/project/custom/bar/*.overlay.toml"}, + } + + all, err := components.NewResolver(env.Env).FindAllComponents() + require.NoError(t, err) + + bar, ok := all.TryGet("bar") + require.True(t, ok) + assert.Len(t, bar.GetConfig().Overlays, 1, + "bar overrides overlay-files wholesale; inherited pattern must not apply") +} + +// Empty per-component 'overlay-files' disables the inherited project pattern. +func TestFindAllComponents_ComponentOverlayFilesEmptyDisablesInheritance(t *testing.T) { + env := testutils.NewTestEnv(t) + + env.Config.DefaultComponentConfig.OverlayFiles = []string{ + "/project/comps/{component}/overlays/*.overlay.toml", + } + + writeOverlayFile(t, env, "/project/comps/baz/overlays/0001.overlay.toml") + + env.Config.Components["baz"] = projectconfig.ComponentConfig{ + Name: "baz", + OverlayFiles: []string{}, + } + + all, err := components.NewResolver(env.Env).FindAllComponents() + require.NoError(t, err) + + baz, ok := all.TryGet("baz") + require.True(t, ok) + assert.Empty(t, baz.GetConfig().Overlays, + "empty overlay-files must disable inherited pattern") +} + +func TestFindAllComponents_MultiplePlaceholderPatterns(t *testing.T) { + env := testutils.NewTestEnv(t) + env.Config.Components["dup"] = projectconfig.ComponentConfig{Name: "dup"} + + env.Config.DefaultComponentConfig.OverlayFiles = []string{ + "/project/a/{component}/*.overlay.toml", + "/project/b/{component}/*.overlay.toml", + "/project/a/{component}/*.overlay.toml", + } + + writeOverlayFile(t, env, "/project/a/dup/0001.overlay.toml") + writeOverlayFile(t, env, "/project/b/dup/0002.overlay.toml") + + all, err := components.NewResolver(env.Env).FindAllComponents() + require.NoError(t, err) + assert.Equal(t, []string{"dup"}, all.Names()) + dup, ok := all.TryGet("dup") + require.True(t, ok) + assert.Len(t, dup.GetConfig().Overlays, 2, "overlapping patterns load each file once") +} + // When a lock file exists for a component, the resolver should attach all of // its data (commit, import-commit, manual-bump, fingerprint) to the resolved // component via the Locked field — without touching the original Spec config. diff --git a/internal/projectconfig/component.go b/internal/projectconfig/component.go index 150689991..49501fbe6 100644 --- a/internal/projectconfig/component.go +++ b/internal/projectconfig/component.go @@ -425,14 +425,12 @@ type ComponentConfig struct { // with unchanged path resolution and precedence. Distro-version and component-group // defaults must omit 'overlay-files', including empty lists. // - // Project-level 'default-component-config' may opt into discovery by using the - // '{component}' placeholder exactly once as a whole path segment. These entries - // drive component discovery (each - // captured segment becomes an implicit component) and, when inherited by an explicit - // component that does not override, `{component}` is substituted with that - // component's name at glob time. Two project-scope entries that produce the same - // component name is a hard error. + // Project-level 'default-component-config' may use the '{component}' placeholder + // exactly once as a whole path segment. At glob time it is substituted with + // the resolved component's name. These entries are anchored to the config file + // that declares the default and only apply to already-resolved components. // '..' path segments may appear before '{component}', but never after it. + // Additional braces are forbidden after '{component}'; use separate list entries. // // The placeholder is forbidden at component scope. Plain inherited globs are // interpreted relative to the concrete component @@ -441,7 +439,7 @@ type ComponentConfig struct { // // Excluded from the fingerprint because the value affects only where overlays are // sourced from, not their content. - OverlayFiles []string `toml:"overlay-files,omitempty" json:"overlayFiles,omitempty" table:"-" validate:"dive,required" jsonschema:"title=Overlay files,description=Path or glob patterns matched against the filesystem to locate per-file overlay documents. Allowed only in project-level 'default-component-config' and explicit component settings. Distro-version and component-group defaults must omit 'overlay-files' including empty lists. Plain legacy patterns and empty lists remain supported at project and component scope with unchanged path resolution and precedence. Project-level defaults may opt into component discovery with {component} exactly once as a whole directory segment. In discovery patterns '..' path segments are allowed before {component} but forbidden after it. The placeholder is forbidden at component scope. Plain inherited globs resolve relative to the concrete component config file or matched spec directory. Use an empty list at component scope to disable inherited overlay files." fingerprint:"-"` + OverlayFiles []string `toml:"overlay-files,omitempty" json:"overlayFiles,omitempty" table:"-" validate:"dive,required" jsonschema:"title=Overlay files,description=Path or glob patterns matched against the filesystem to locate per-file overlay documents. Allowed only in project-level 'default-component-config' and explicit component settings. Distro-version and component-group defaults must omit 'overlay-files' including empty lists. Plain legacy patterns and empty lists remain supported at project and component scope with unchanged path resolution and precedence. Project-level defaults may use {component} exactly once as a whole directory segment to substitute the resolved component's name. Placeholder patterns are relative to the config file declaring the default and only apply to already-resolved components. '..' path segments are allowed before {component} but forbidden after it. Additional braces are forbidden after {component}; use separate list entries for multiple directories. The placeholder is forbidden at component scope. Plain inherited globs resolve relative to the concrete component config file or matched spec directory. Use an empty list at component scope to disable inherited overlay files." fingerprint:"-"` // Configuration for building the component. Build ComponentBuildConfig `toml:"build,omitempty" json:"build,omitempty" table:"-" jsonschema:"title=Build configuration,description=Configuration for building the component"` @@ -658,7 +656,7 @@ func (c *ComponentConfig) WithAbsolutePaths(referenceDir string) *ComponentConfi Packages: deep.MustCopy(c.Packages), Publish: deep.MustCopy(c.Publish), // OverlayFiles: entries with the {component} placeholder are project-scope - // discovery patterns and are absolutized now so downstream code sees stable + // shared patterns and are absolutized now so downstream code sees stable // absolute paths regardless of where the config was inherited from. Non-placeholder // entries are left verbatim so component-scope patterns are still interpreted // relative to the concrete component config file at glob time. diff --git a/internal/projectconfig/config_test.go b/internal/projectconfig/config_test.go index b29104b7f..5912b2c47 100644 --- a/internal/projectconfig/config_test.go +++ b/internal/projectconfig/config_test.go @@ -173,70 +173,6 @@ description = "`+testUserDesc+`" assert.Equal(t, testUserDesc, config.Project.Description) } -func TestLoadProjectConfig_OverlayFilesParentSegments(t *testing.T) { - testCases := []struct { - name string - pattern string - expected string - wantErr bool - }{ - { - name: "parent in placeholder prefix", - pattern: "../comps/{component}/overlays/*.overlay.toml", - expected: "/test/comps/{component}/overlays/*.overlay.toml", - }, - { - name: "parent removes placeholder", - pattern: "comps/{component}/../overlays/*.overlay.toml", - wantErr: true, - }, - { - name: "nested parents remove placeholder", - pattern: "comps/{component}/nested/../../overlays/*.overlay.toml", - wantErr: true, - }, - { - name: "parent after nested directory", - pattern: "comps/{component}/nested/../overlays/*.overlay.toml", - wantErr: true, - }, - { - name: "plain legacy pattern unchanged", - pattern: "nested/../overlays/*.overlay.toml", - expected: "nested/../overlays/*.overlay.toml", - }, - } - - for _, withoutLockfile := range []bool{false, true} { - mode := "locked" - if withoutLockfile { - mode = "lockless" - } - - t.Run(mode, func(t *testing.T) { - for _, testCase := range testCases { - t.Run(testCase.name, func(t *testing.T) { - ctx := newTestCtxWithXDGConfigHome() - writeProjectConfig(t, ctx, "[default-component-config]\noverlay-files = [\""+testCase.pattern+"\"]\n") - - _, config, err := projectconfig.LoadProjectConfig( - ctx.FS(), ctx.OSEnv(), testProjectDir, true, "/test/tmp", nil, false, withoutLockfile, - ) - if testCase.wantErr { - require.ErrorIs(t, err, projectconfig.ErrInvalidOverlayFilesEntry) - assert.ErrorContains(t, err, "must not contain '..' path segments after") - - return - } - - require.NoError(t, err) - assert.Equal(t, []string{testCase.expected}, config.DefaultComponentConfig.OverlayFiles) - }) - } - }) - } -} - // TestLoadProjectConfig_LockDirByMode verifies that the project's lock directory is // defaulted in azldev's default mode and left unset in lock-file-free mode, where an // explicitly configured 'lock-dir' is accepted but ignored. diff --git a/internal/projectconfig/configfile.go b/internal/projectconfig/configfile.go index c583c370b..6f98c74cd 100644 --- a/internal/projectconfig/configfile.go +++ b/internal/projectconfig/configfile.go @@ -529,7 +529,7 @@ func dedupTestRefSKUGroups( // validateOverlayFilesByScope restricts default 'overlay-files' lists to project scope. // // Distro-version and component-group defaults must omit the field, including empty lists. -// Project defaults accept plain patterns and may opt into discovery with '{component}'. +// Project defaults accept plain patterns and may substitute component names with '{component}'. func (f ConfigFile) validateOverlayFilesByScope() error { if f.DefaultComponentConfig != nil { if err := validateOverlayFilesEntries( @@ -568,7 +568,7 @@ type overlayFilesValidator func(entry string) error // validateProjectOverlayFilesEntry validates entries in the project-level // default-component-config. Legacy plain globs remain valid; entries containing -// '{component}' opt into discovery and must obey the placeholder rules. +// '{component}' substitute the resolved component's name and must obey the placeholder rules. func validateProjectOverlayFilesEntry(entry string) error { if !hasOverlayFilesPlaceholder(entry) { return nil diff --git a/internal/projectconfig/loader_test.go b/internal/projectconfig/loader_test.go index 45b939eef..ec9f4168d 100644 --- a/internal/projectconfig/loader_test.go +++ b/internal/projectconfig/loader_test.go @@ -7,7 +7,6 @@ package projectconfig import ( "os" "path/filepath" - "strings" "testing" "github.com/microsoft/azure-linux-dev-tools/internal/global/testctx" @@ -28,93 +27,6 @@ func TestLoadAndResolveProjectConfig(t *testing.T) { assert.Nil(t, config) } -func TestLoadAndResolveProjectConfig_OverlayFilesDefaultScopes(t *testing.T) { - scopes := []struct { - name string - lines []string - entries func(*ProjectConfig) []string - }{ - { - name: "project", - lines: []string{"[default-component-config]"}, - entries: func(config *ProjectConfig) []string { - return config.DefaultComponentConfig.OverlayFiles - }, - }, - { - name: "distro", - lines: []string{ - "[distros.test.versions.'1']", - "release-ver = '1'", - "[distros.test.versions.'1'.default-component-config]", - }, - entries: func(config *ProjectConfig) []string { - return config.Distros["test"].Versions["1"].DefaultComponentConfig.OverlayFiles - }, - }, - { - name: "group", - lines: []string{"[component-groups.test.default-component-config]"}, - entries: func(config *ProjectConfig) []string { - return config.ComponentGroups["test"].DefaultComponentConfig.OverlayFiles - }, - }, - } - - for _, scope := range scopes { - t.Run(scope.name, func(t *testing.T) { - for _, value := range []struct { - name string - toml string - expected []string - }{ - {name: "omitted"}, - {name: "plain-glob", toml: "['overlays/*.overlay.toml']", expected: []string{"overlays/*.overlay.toml"}}, - {name: "empty-list", toml: "[]", expected: []string{}}, - { - name: "placeholder", toml: "['comps/{component}/overlays/*.overlay.toml']", - expected: []string{"/project/comps/{component}/overlays/*.overlay.toml"}, - }, - } { - t.Run(value.name, func(t *testing.T) { - for _, mode := range []struct { - name string - withoutLockfile bool - }{ - {name: "default"}, - {name: "without-lockfile", withoutLockfile: true}, - } { - t.Run(mode.name, func(t *testing.T) { - ctx := testctx.NewCtx() - - lines := append([]string{}, scope.lines...) - if value.toml != "" { - lines = append(lines, "overlay-files = "+value.toml) - } - - require.NoError(t, fileutils.WriteFile(ctx.FS(), testConfigPath, - []byte(strings.Join(lines, "\n")+"\n"), fileperms.PrivateFile)) - - config, err := loadAndResolveProjectConfig(ctx.FS(), - loadOptions{withoutLockfile: mode.withoutLockfile}, testConfigPath) - if scope.name != "project" && value.toml != "" { - require.ErrorIs(t, err, ErrInvalidOverlayFilesEntry) - require.ErrorContains(t, err, scope.name) - assert.Nil(t, config) - - return - } - - require.NoError(t, err) - assert.Equal(t, value.expected, scope.entries(config)) - }) - } - }) - } - }) - } -} - func TestLoadAndResolveProjectConfig_SyntaxError(t *testing.T) { ctx := testctx.NewCtx() require.NoError(t, fileutils.WriteFile(ctx.FS(), testConfigPath, []byte("///"), fileperms.PrivateFile)) diff --git a/internal/projectconfig/overlay_files_placeholder.go b/internal/projectconfig/overlay_files_placeholder.go index ca359d8dd..a2c81864e 100644 --- a/internal/projectconfig/overlay_files_placeholder.go +++ b/internal/projectconfig/overlay_files_placeholder.go @@ -13,8 +13,8 @@ import ( ) // OverlayFilesComponentPlaceholder is the single supported placeholder in an -// [ComponentConfig.OverlayFiles] entry. Project-level defaults may opt into -// discovery with this placeholder; components permit only plain patterns, and +// [ComponentConfig.OverlayFiles] entry. Project-level defaults may substitute +// the resolved component's name with this placeholder; components permit only plain patterns, and // distro-version and component-group defaults must omit 'overlay-files'. // When present it must appear exactly once and must be a whole path segment. const OverlayFilesComponentPlaceholder = "{component}" @@ -33,10 +33,9 @@ func hasOverlayFilesPlaceholder(entry string) bool { // validateOverlayFilesPlaceholder verifies that entry contains exactly one // [OverlayFilesComponentPlaceholder] and that the placeholder is a whole path // segment. Path separators must be '/' (POSIX-style), and no glob -// metacharacters ('*', '?', '[', '{') may appear before the placeholder because -// pattern-discovery extracts the captured component name by literal-prefix -// matching against the concrete glob result. Metacharacters after the -// placeholder are allowed, but '..' path segments are allowed only before the +// metacharacters ('*', '?', '[', '{') may appear before the placeholder. +// The prefix must be literal. Glob metacharacters after the placeholder are +// allowed except for braces. '..' path segments are allowed only before the // placeholder. Returns nil when the entry is well-formed. func validateOverlayFilesPlaceholder(entry string) error { if entry == "" { @@ -96,22 +95,20 @@ func validateOverlayFilesPlaceholder(entry string) error { ) } - if slices.Contains(strings.Split(entry[tail+1:], "/"), "..") { - return fmt.Errorf("%w: entry %#q must not contain '..' path segments after %q", - ErrInvalidOverlayFilesEntry, entry, OverlayFilesComponentPlaceholder) + if err := validateOverlayFilesPlaceholderSuffix(entry, entry[tail+1:]); err != nil { + return err } prefix := entry[:idx] if globMetaIdx := strings.IndexAny(prefix, "*?[{"); globMetaIdx >= 0 { return fmt.Errorf( "%w: entry %#q must not contain glob metacharacters (%q) before %q; "+ - "the discovery capture uses a literal prefix", + "the prefix must be literal", ErrInvalidOverlayFilesEntry, entry, string(prefix[globMetaIdx]), OverlayFilesComponentPlaceholder, ) } - // Substitute '*' for {component} to build the concrete glob that - // pattern-discovery will hand to doublestar. + // Validate the glob syntax independently of the resolved component's name. globPattern := prefix + "*" + entry[tail:] if !doublestar.ValidatePattern(globPattern) { return fmt.Errorf( @@ -123,21 +120,18 @@ func validateOverlayFilesPlaceholder(entry string) error { return nil } -// SplitOverlayFilesPlaceholder splits an entry at its -// [OverlayFilesComponentPlaceholder] and returns the prefix and suffix. The -// caller can then substitute a concrete component name (or a glob wildcard) -// for the placeholder. Passing an entry without a placeholder is a programming -// error and panics with a diagnostic message. -func SplitOverlayFilesPlaceholder(entry string) (prefix, suffix string) { - idx := strings.Index(entry, OverlayFilesComponentPlaceholder) - if idx < 0 { - panic(fmt.Sprintf( - "SplitOverlayFilesPlaceholder: entry %q does not contain %q", - entry, OverlayFilesComponentPlaceholder, - )) +func validateOverlayFilesPlaceholderSuffix(entry, suffix string) error { + if strings.ContainsAny(suffix, "{}") { + return fmt.Errorf("%w: entry %#q must not contain braces after %q", + ErrInvalidOverlayFilesEntry, entry, OverlayFilesComponentPlaceholder) } - return entry[:idx], entry[idx+len(OverlayFilesComponentPlaceholder):] + if slices.Contains(strings.Split(suffix, "/"), "..") { + return fmt.Errorf("%w: entry %#q must not contain '..' path segments after %q", + ErrInvalidOverlayFilesEntry, entry, OverlayFilesComponentPlaceholder) + } + + return nil } // substituteOverlayFilesPlaceholder returns entry with the single diff --git a/internal/projectconfig/overlay_files_placeholder_test.go b/internal/projectconfig/overlay_files_placeholder_test.go index 19a04f2f5..d77d34f86 100644 --- a/internal/projectconfig/overlay_files_placeholder_test.go +++ b/internal/projectconfig/overlay_files_placeholder_test.go @@ -5,8 +5,13 @@ package projectconfig import ( + "path/filepath" + "strings" "testing" + "github.com/microsoft/azure-linux-dev-tools/internal/global/testctx" + "github.com/microsoft/azure-linux-dev-tools/internal/utils/fileperms" + "github.com/microsoft/azure-linux-dev-tools/internal/utils/fileutils" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) @@ -28,7 +33,18 @@ func TestValidateOverlayFilesPlaceholder(t *testing.T) { {name: "glob metachar before placeholder", entry: "**/{component}/*.overlay.toml", wantErr: true}, {name: "star before placeholder", entry: "comps*/{component}/*.overlay.toml", wantErr: true}, {name: "brace alternatives before placeholder", entry: "{base,extra}/{component}/*.overlay.toml", wantErr: true}, - {name: "brace suffix allowed", entry: "comps/{component}/{base,extra}/*.overlay.toml", wantErr: false}, + {name: "brace suffix rejected", entry: "comps/{component}/{base,extra}/*.overlay.toml", wantErr: true}, + {name: "brace parent alternative rejected", entry: "comps/{component}/{..,overlays}/*.overlay.toml", wantErr: true}, + { + name: "nested brace alternatives rejected", entry: "comps/{component}/{base,{..,extra}}/*.overlay.toml", + wantErr: true, + }, + { + name: "concatenated brace alternatives rejected", entry: "comps/{component}/{.,x}{.,y}/*.overlay.toml", + wantErr: true, + }, + {name: "unmatched opening brace rejected", entry: "comps/{component}/fix{.overlay.toml", wantErr: true}, + {name: "unmatched closing brace rejected", entry: "comps/{component}/fix}.overlay.toml", wantErr: true}, {name: "whole segment at start", entry: "{component}/overlays/*.overlay.toml", wantErr: false}, {name: "whole segment in middle", entry: "comps/{component}/overlays/*.overlay.toml", wantErr: false}, {name: "placeholder at end rejected", entry: "comps/overlays/{component}", wantErr: true}, @@ -59,65 +75,198 @@ func TestValidateOverlayFilesPlaceholder(t *testing.T) { } } -func TestSplitOverlayFilesPlaceholder(t *testing.T) { +func TestSubstituteOverlayFilesPlaceholder(t *testing.T) { t.Parallel() - tests := []struct { - name string - entry string - wantPrefix string - wantSuffix string + assert.Equal(t, + "comps/foo/overlays/*.toml", + substituteOverlayFilesPlaceholder("comps/{component}/overlays/*.toml", "foo")) + + // No placeholder → unchanged. + assert.Equal(t, + "overlays/*.toml", + substituteOverlayFilesPlaceholder("overlays/*.toml", "foo")) +} + +func TestLoadProjectConfig_OverlayFilesPlaceholderPaths(t *testing.T) { + testCases := []struct { + name string + pattern string + expected string + wantErr bool + braces bool }{ { - name: "middle", - entry: "comps/{component}/overlays/*.overlay.toml", - wantPrefix: "comps/", - wantSuffix: "/overlays/*.overlay.toml", + name: "parent in placeholder prefix", + pattern: "../comps/{component}/overlays/*.overlay.toml", + expected: "/test/comps/{component}/overlays/*.overlay.toml", }, { - name: "start", - entry: "{component}/overlays/*.overlay.toml", - wantPrefix: "", - wantSuffix: "/overlays/*.overlay.toml", + name: "parent removes placeholder", + pattern: "comps/{component}/../overlays/*.overlay.toml", + wantErr: true, }, { - name: "end", - entry: "comps/overlays/{component}", - wantPrefix: "comps/overlays/", - wantSuffix: "", + name: "nested parents remove placeholder", + pattern: "comps/{component}/nested/../../overlays/*.overlay.toml", + wantErr: true, + }, + { + name: "parent after nested directory", + pattern: "comps/{component}/nested/../overlays/*.overlay.toml", + wantErr: true, + }, + { + name: "plain legacy pattern unchanged", + pattern: "nested/../overlays/*.overlay.toml", + expected: "nested/../overlays/*.overlay.toml", + }, + { + name: "safe brace alternatives rejected", + pattern: "comps/{component}/{base,extra}/*.overlay.toml", + wantErr: true, + braces: true, + }, + { + name: "brace parent bypass rejected", + pattern: "comps/{component}/{..,overlays}/*.overlay.toml", + wantErr: true, + braces: true, + }, + { + name: "plain legacy brace pattern unchanged", + pattern: "{base,extra}/*.overlay.toml", + expected: "{base,extra}/*.overlay.toml", }, } - for _, testCase := range tests { - t.Run(testCase.name, func(t *testing.T) { - t.Parallel() + for _, withoutLockfile := range []bool{false, true} { + mode := "locked" + if withoutLockfile { + mode = "lockless" + } + + t.Run(mode, func(t *testing.T) { + for _, testCase := range testCases { + t.Run(testCase.name, func(t *testing.T) { + const projectDir = "/test/project" + + osEnv := testctx.NewTestOSEnv() + osEnv.SetEnv("XDG_CONFIG_HOME", "/test/xdghome") + ctx := testctx.NewCtx(testctx.WithOSEnv(osEnv)) + require.NoError(t, fileutils.MkdirAll(ctx.FS(), projectDir)) + require.NoError(t, fileutils.WriteFile(ctx.FS(), + filepath.Join(projectDir, DefaultConfigFileName), + []byte("[default-component-config]\noverlay-files = [\""+testCase.pattern+"\"]\n"), + fileperms.PublicFile)) - prefix, suffix := SplitOverlayFilesPlaceholder(testCase.entry) - assert.Equal(t, testCase.wantPrefix, prefix) - assert.Equal(t, testCase.wantSuffix, suffix) + _, config, err := LoadProjectConfig( + ctx.FS(), ctx.OSEnv(), projectDir, true, "/test/tmp", nil, false, withoutLockfile, + ) + if testCase.wantErr { + require.ErrorIs(t, err, ErrInvalidOverlayFilesEntry) + + if testCase.braces { + require.ErrorContains(t, err, "must not contain braces after") + } else { + require.ErrorContains(t, err, "must not contain '..' path segments after") + } + + return + } + + require.NoError(t, err) + assert.Equal(t, []string{testCase.expected}, config.DefaultComponentConfig.OverlayFiles) + }) + } }) } } -func TestSplitOverlayFilesPlaceholder_PanicsWithoutPlaceholder(t *testing.T) { - t.Parallel() +func TestLoadAndResolveProjectConfig_OverlayFilesDefaultScopes(t *testing.T) { + scopes := []struct { + name string + lines []string + entries func(*ProjectConfig) []string + }{ + { + name: "project", + lines: []string{"[default-component-config]"}, + entries: func(config *ProjectConfig) []string { + return config.DefaultComponentConfig.OverlayFiles + }, + }, + { + name: "distro", + lines: []string{ + "[distros.test.versions.'1']", + "release-ver = '1'", + "[distros.test.versions.'1'.default-component-config]", + }, + entries: func(config *ProjectConfig) []string { + return config.Distros["test"].Versions["1"].DefaultComponentConfig.OverlayFiles + }, + }, + { + name: "group", + lines: []string{"[component-groups.test.default-component-config]"}, + entries: func(config *ProjectConfig) []string { + return config.ComponentGroups["test"].DefaultComponentConfig.OverlayFiles + }, + }, + } - assert.PanicsWithValue(t, - `SplitOverlayFilesPlaceholder: entry "comps/foo/overlays/x.toml" does not contain "{component}"`, - func() { - SplitOverlayFilesPlaceholder("comps/foo/overlays/x.toml") - }) -} + for _, scope := range scopes { + t.Run(scope.name, func(t *testing.T) { + for _, value := range []struct { + name string + toml string + expected []string + }{ + {name: "omitted"}, + {name: "plain-glob", toml: "['overlays/*.overlay.toml']", expected: []string{"overlays/*.overlay.toml"}}, + {name: "empty-list", toml: "[]", expected: []string{}}, + { + name: "placeholder", toml: "['comps/{component}/overlays/*.overlay.toml']", + expected: []string{"/project/comps/{component}/overlays/*.overlay.toml"}, + }, + } { + t.Run(value.name, func(t *testing.T) { + for _, mode := range []struct { + name string + withoutLockfile bool + }{ + {name: "default"}, + {name: "without-lockfile", withoutLockfile: true}, + } { + t.Run(mode.name, func(t *testing.T) { + ctx := testctx.NewCtx() + configPath := filepath.Join("/project", DefaultConfigFileName) -func TestSubstituteOverlayFilesPlaceholder(t *testing.T) { - t.Parallel() + lines := append([]string{}, scope.lines...) + if value.toml != "" { + lines = append(lines, "overlay-files = "+value.toml) + } - assert.Equal(t, - "comps/foo/overlays/*.toml", - substituteOverlayFilesPlaceholder("comps/{component}/overlays/*.toml", "foo")) + require.NoError(t, fileutils.WriteFile(ctx.FS(), configPath, + []byte(strings.Join(lines, "\n")+"\n"), fileperms.PrivateFile)) - // No placeholder → unchanged. - assert.Equal(t, - "overlays/*.toml", - substituteOverlayFilesPlaceholder("overlays/*.toml", "foo")) + config, err := loadAndResolveProjectConfig(ctx.FS(), + loadOptions{withoutLockfile: mode.withoutLockfile}, configPath) + if scope.name != "project" && value.toml != "" { + require.ErrorIs(t, err, ErrInvalidOverlayFilesEntry) + require.ErrorContains(t, err, scope.name) + assert.Nil(t, config) + + return + } + + require.NoError(t, err) + assert.Equal(t, value.expected, scope.entries(config)) + }) + } + }) + } + }) + } } diff --git a/scenario/__snapshots__/TestSnapshotsContainer_config_generate-schema_stdout_1.snap b/scenario/__snapshots__/TestSnapshotsContainer_config_generate-schema_stdout_1.snap index ffe768777..39870154d 100755 --- a/scenario/__snapshots__/TestSnapshotsContainer_config_generate-schema_stdout_1.snap +++ b/scenario/__snapshots__/TestSnapshotsContainer_config_generate-schema_stdout_1.snap @@ -130,7 +130,7 @@ }, "type": "array", "title": "Overlay files", - "description": "Path or glob patterns matched against the filesystem to locate per-file overlay documents. Allowed only in project-level 'default-component-config' and explicit component settings. Distro-version and component-group defaults must omit 'overlay-files' including empty lists. Plain legacy patterns and empty lists remain supported at project and component scope with unchanged path resolution and precedence. Project-level defaults may opt into component discovery with {component} exactly once as a whole directory segment. In discovery patterns '..' path segments are allowed before {component} but forbidden after it. The placeholder is forbidden at component scope. Plain inherited globs resolve relative to the concrete component config file or matched spec directory. Use an empty list at component scope to disable inherited overlay files." + "description": "Path or glob patterns matched against the filesystem to locate per-file overlay documents. Allowed only in project-level 'default-component-config' and explicit component settings. Distro-version and component-group defaults must omit 'overlay-files' including empty lists. Plain legacy patterns and empty lists remain supported at project and component scope with unchanged path resolution and precedence. Project-level defaults may use {component} exactly once as a whole directory segment to substitute the resolved component's name. Placeholder patterns are relative to the config file declaring the default and only apply to already-resolved components. '..' path segments are allowed before {component} but forbidden after it. Additional braces are forbidden after {component}; use separate list entries for multiple directories. The placeholder is forbidden at component scope. Plain inherited globs resolve relative to the concrete component config file or matched spec directory. Use an empty list at component scope to disable inherited overlay files." }, "build": { "$ref": "#/$defs/ComponentBuildConfig", diff --git a/scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap b/scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap index ffe768777..39870154d 100755 --- a/scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap +++ b/scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap @@ -130,7 +130,7 @@ }, "type": "array", "title": "Overlay files", - "description": "Path or glob patterns matched against the filesystem to locate per-file overlay documents. Allowed only in project-level 'default-component-config' and explicit component settings. Distro-version and component-group defaults must omit 'overlay-files' including empty lists. Plain legacy patterns and empty lists remain supported at project and component scope with unchanged path resolution and precedence. Project-level defaults may opt into component discovery with {component} exactly once as a whole directory segment. In discovery patterns '..' path segments are allowed before {component} but forbidden after it. The placeholder is forbidden at component scope. Plain inherited globs resolve relative to the concrete component config file or matched spec directory. Use an empty list at component scope to disable inherited overlay files." + "description": "Path or glob patterns matched against the filesystem to locate per-file overlay documents. Allowed only in project-level 'default-component-config' and explicit component settings. Distro-version and component-group defaults must omit 'overlay-files' including empty lists. Plain legacy patterns and empty lists remain supported at project and component scope with unchanged path resolution and precedence. Project-level defaults may use {component} exactly once as a whole directory segment to substitute the resolved component's name. Placeholder patterns are relative to the config file declaring the default and only apply to already-resolved components. '..' path segments are allowed before {component} but forbidden after it. Additional braces are forbidden after {component}; use separate list entries for multiple directories. The placeholder is forbidden at component scope. Plain inherited globs resolve relative to the concrete component config file or matched spec directory. Use an empty list at component scope to disable inherited overlay files." }, "build": { "$ref": "#/$defs/ComponentBuildConfig", diff --git a/schemas/azldev.schema.json b/schemas/azldev.schema.json index ffe768777..39870154d 100644 --- a/schemas/azldev.schema.json +++ b/schemas/azldev.schema.json @@ -130,7 +130,7 @@ }, "type": "array", "title": "Overlay files", - "description": "Path or glob patterns matched against the filesystem to locate per-file overlay documents. Allowed only in project-level 'default-component-config' and explicit component settings. Distro-version and component-group defaults must omit 'overlay-files' including empty lists. Plain legacy patterns and empty lists remain supported at project and component scope with unchanged path resolution and precedence. Project-level defaults may opt into component discovery with {component} exactly once as a whole directory segment. In discovery patterns '..' path segments are allowed before {component} but forbidden after it. The placeholder is forbidden at component scope. Plain inherited globs resolve relative to the concrete component config file or matched spec directory. Use an empty list at component scope to disable inherited overlay files." + "description": "Path or glob patterns matched against the filesystem to locate per-file overlay documents. Allowed only in project-level 'default-component-config' and explicit component settings. Distro-version and component-group defaults must omit 'overlay-files' including empty lists. Plain legacy patterns and empty lists remain supported at project and component scope with unchanged path resolution and precedence. Project-level defaults may use {component} exactly once as a whole directory segment to substitute the resolved component's name. Placeholder patterns are relative to the config file declaring the default and only apply to already-resolved components. '..' path segments are allowed before {component} but forbidden after it. Additional braces are forbidden after {component}; use separate list entries for multiple directories. The placeholder is forbidden at component scope. Plain inherited globs resolve relative to the concrete component config file or matched spec directory. Use an empty list at component scope to disable inherited overlay files." }, "build": { "$ref": "#/$defs/ComponentBuildConfig",