Skip to content

feat(overlay-files): support shared patterns with {component} substitution - #269

Open
Nan Liu (liunan-ms) wants to merge 4 commits into
microsoft:mainfrom
liunan-ms:liunan/overlayfiles-pattern
Open

Nan Liu (liunan-ms) wants to merge 4 commits into
microsoft:mainfrom
liunan-ms:liunan/overlayfiles-pattern

Conversation

@liunan-ms

@liunan-ms Nan Liu (liunan-ms) commented Jul 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add {component} substitution to project-level default-component-config.overlay-files, allowing centrally declared components to share a per-component overlay layout without empty dedicated .comp.toml files.

# base/comps/components.toml
[default-component-config]
overlay-files = ["{component}/overlays/*.overlay.toml"]

[components.anaconda]

[components.curl]

The pattern resolves relative to the file declaring the default. Each known component substitutes its own name and loads its matching overlays. Overlay directories do not create components: central declarations remain required.

Behavior and restrictions

  • {component} is permitted only in project-level defaults, exactly once as a complete directory segment followed by a file path or pattern.
  • Placeholder prefixes must be literal. Additional braces and .. segments are forbidden after the placeholder; .. remains allowed before it. Use separate list entries for multiple directories.
  • Distro-version and component-group defaults must omit overlay-files, including [].
  • Plain project/component patterns retain their existing behavior. A component-specific list replaces the inherited list; [] disables inheritance.
  • Unmatched inherited placeholder entries are optional, including literal suffixes. Ordinary literal paths must still exist.
  • Preserve pattern order, filename ordering within each pattern, duplicate-file suppression, and inline-overlay precedence.

Documentation and tests

Update emitted agent-skill templates, configuration references, JSON schema, and scenario snapshots. Consolidate resolver regressions into resolver_test.go and placeholder validation/loading regressions into projectconfig/overlay_files_placeholder_test.go.

Validation

  • mage fix all, mage docs, mage scenarioUpdate, and mage all passed with network-dependent tests disabled.
  • Separately validated the Azure Linux central-declaration migration(refactor(components): centralize declaration-only imports azurelinux#20283): all 157 baseline and migrated renders succeeded; all 1,247 generated files were byte-identical. This is render-only evidence, not RPM-build/runtime validation or coverage of every placeholder edge case.

Copilot AI review requested due to automatic review settings July 7, 2026 22:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a {component} placeholder for project-level default-component-config.overlay-files to support pattern-based discovery of components and per-component overlay-file resolution, and updates schema/snapshots/docs accordingly.

Changes:

  • Add projectconfig placeholder validation/splitting/substitution helpers and scope-based validation ({component} required at project scope, forbidden elsewhere).
  • Add pattern-based component discovery from project-level overlay-files entries with resolver-level caching and deterministic unioning into FindAllComponents.
  • Regenerate JSON schema / scenario snapshot and document the new placeholder behavior in the overlays config reference.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
schemas/azldev.schema.json Updates schema description for overlay-files to document {component} rules.
scenario/snapshots/TestSnapshots_config_generate-schema_stdout_1.snap Updates snapshot to match regenerated schema output.
internal/projectconfig/overlay_files_placeholder.go Implements {component} placeholder validation, splitting, substitution, and absolutization helpers.
internal/projectconfig/overlay_files_placeholder_test.go Unit tests for placeholder validation/splitting/substitution helpers.
internal/projectconfig/overlay_file.go Substitutes {component} with component name before overlay glob expansion.
internal/projectconfig/configfile.go Adds config-file validation enforcing placeholder rules by scope.
internal/projectconfig/configfile_test.go Adds validation tests for allowed/forbidden placeholder usage across scopes.
internal/projectconfig/component.go Updates OverlayFiles docs/schema description and absolutizes placeholder entries during WithAbsolutePaths.
internal/app/azldev/core/components/resolver.go Unions pattern-discovered components into FindAllComponents with caching and shadowing behavior.
internal/app/azldev/core/components/pattern_discovery.go Implements filesystem globbing + capture of {component} segment into synthesized components.
internal/app/azldev/core/components/pattern_discovery_test.go Tests discovery, inheritance/override semantics, and same-scope collision errors.
docs/user/reference/config/overlays.md Documents {component} placeholder discovery/inheritance/shadowing behavior for users.

@liunan-ms
Nan Liu (liunan-ms) force-pushed the liunan/overlayfiles-pattern branch from 4a56921 to f2c8de7 Compare July 7, 2026 23:33
}

count := strings.Count(entry, OverlayFilesComponentPlaceholder)
if count == 0 {

@dmcilvaney Daniel McIlvaney (dmcilvaney) Jul 8, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this stops us having non-component placeholder entries, ie have overlay-files = ["static/overlays/*.toml"]. If this is expected, docs need to be updated (overlays.md:166) to match.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intentional. Project-level entries without {component} are rejected because they would silently apply the same files to every component. Updated the doc.

{name: "star before placeholder", entry: "comps*/{component}/*.overlay.toml", wantErr: true},
{name: "whole segment at start", entry: "{component}/overlays/*.overlay.toml", wantErr: false},
{name: "whole segment in middle", entry: "comps/{component}/overlays/*.overlay.toml", wantErr: false},
{name: "whole segment at end", entry: "comps/overlays/{component}", wantErr: false},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this might be an error, won't it glob files with the name of the component, not directories? I think it probably always needs to be afolder right?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch, updated the check and test.

// declaration overrides pattern-discovered overlay files for the same name.
func logShadowedByDeclaration(shadowed *patternDiscoveredComponent) {
slog.Warn(
"project 'overlay-files' pattern match shadowed by explicit component declaration; overlay files will not be applied",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this might miss shadowing from project defaults by groups. Not sure if thats something worth an error though.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be resolved by the scope restriction, now rejecting overlay-files in distro-version and component-group default-component-config entirely.

// which suppresses inheritance (see [projectconfig.ComponentConfig.MergeUpdatesFrom]).
// A declaration with no overlay-files inherits the pattern and the
// discovered files still apply, so there's nothing to warn about.
if declared, ok := r.env.Config().Components[name]; ok && declared.OverlayFiles != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might want to use len(declared.OverlayFiles) > 0 rather than nil, if I recall from the previous PR the way to disable inheritance is to set overlay-files=[]. 50/50 on this, we could still warn because thats weird, but it is intentionally explicitly saying "no overlays" for a component.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Changed to len(declared.OverlayFiles) > 0.

}

prefix := entry[:idx]
if globMetaIdx := strings.IndexAny(prefix, "*?["); globMetaIdx >= 0 {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be able to add https://pkg.go.dev/github.com/bmatcuk/doublestar/v4#ValidatePattern here as well, get early signal back on a bad glob pattern.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now using doublestar.ValidatePattern on the substituted glob validation.

err error
}

func (c *discoveredComponentsCache) load(env *azldev.Env) (map[string]*patternDiscoveredComponent, error) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found this a while back, might do the trick here: https://pkg.go.dev/sync#OnceValues

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

switched to using sync.OnceValues

}

byName := make(map[string]*patternDiscoveredComponent)
// Deduplicate files per component before appending to preserve stable ordering.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious if this is needed, my understanding was that glob would deduplicate internally.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doublestar.Glob returns unique matches per call so the seen map was dead code and has been removed.

Copilot AI review requested due to automatic review settings July 8, 2026 21:38
@liunan-ms
Nan Liu (liunan-ms) force-pushed the liunan/overlayfiles-pattern branch from f2c8de7 to 1f37e09 Compare July 8, 2026 21:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Comment thread internal/projectconfig/configfile.go Resolved
Comment thread internal/projectconfig/configfile.go Resolved
Comment thread docs/user/reference/config/overlays.md Resolved
Copilot AI review requested due to automatic review settings July 8, 2026 21:46
@liunan-ms
Nan Liu (liunan-ms) force-pushed the liunan/overlayfiles-pattern branch from 1f37e09 to 27e7ffb Compare July 8, 2026 21:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Comment thread docs/user/reference/config/overlays.md Resolved
Copilot AI review requested due to automatic review settings July 8, 2026 21:55
@liunan-ms
Nan Liu (liunan-ms) force-pushed the liunan/overlayfiles-pattern branch from 27e7ffb to 594c51b Compare July 8, 2026 21:55
@liunan-ms

Copy link
Copy Markdown
Contributor Author

Thanks Daniel McIlvaney (@dmcilvaney) for the review! After thinking about the scope and usage of overlay-files in azurelinux, I think it'd be better to restrict overlay-files scope to project + component only, a few reasons:

  • All components now live in base/comps/ with per-component overlays, so a single project-level {component} pattern is enough for every existing case. Broader scopes (distro/group) are out-of-scope for this PR and can be added later if a real requirement appears.
  • Simpler validation and no risk of users accidentally declaring overlay-files at multiple levels and having to reason about merge/override precedence.

I update the implementation and now the scope rule is enforced. overlay-files allowed in:

  • project default-component-config: must contain {component} (pattern-discovery + inheritance)
  • [components.X]: plain globs relative to the declaring .comp.toml
  • Reject (with clear errors) in distro-version and component-group default-component-config.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Copilot AI review requested due to automatic review settings July 8, 2026 22:09
@liunan-ms
Nan Liu (liunan-ms) force-pushed the liunan/overlayfiles-pattern branch from 594c51b to fc2fd2b Compare July 8, 2026 22:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Comment thread internal/projectconfig/component.go Resolved
Comment thread docs/user/reference/config/overlays.md Resolved
Copilot AI review requested due to automatic review settings July 8, 2026 22:29
@liunan-ms
Nan Liu (liunan-ms) force-pushed the liunan/overlayfiles-pattern branch from fc2fd2b to 0573473 Compare July 8, 2026 22:29

@ddstreet Dan Streetman (ddstreet) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question(blocking): Sorry I don't really understand what this is doing or solving, can you please add an example or at least more detail?

@liunan-ms

Copy link
Copy Markdown
Contributor Author

question(blocking): Sorry I don't really understand what this is doing or solving, can you please add an example or at least more detail?

The main goal is to get rid of empty per-component config files when the overlay directory already tells us the component name. In azurelinux repo, 155 of 540 *.comp.toml files (28.7%) contain only a component declaration.

For example, in azurelinux, base/comps/GraphicsMagick/GraphicsMagick.comp.toml contains only:

[components.GraphicsMagick]

The actual changes live under GraphicsMagick/overlays/. With this project default in base/comps/components.toml:

[default-component-config]
overlay-files = ["{component}/overlays/*.overlay.toml"]

azldev would discover GraphicsMagick from those files and apply its overlays, so we could delete the empty .comp.toml. Adding another overlay-only package would just mean adding its overlay files.

For the explicitly listed group members that still need a component declaration. For example, anaconda belongs to base-packages, so we'd suggest to move it into the central components.toml and delete its separate empty file. For the components with actual settings, like upstream pins or build overrides, would keep their configs.

I'll rebase this PR against main and update the guidance in skills and instructions.

Nan Liu (liunan-ms) and others added 2 commits November 12, 2026 00:34
…overed components

Add '{component}' placeholder in project-level overlay-files entries. The resolver globs matching paths, extracts each captured segment as a component name, and synthesizes components not otherwise declared. Explicit component declarations shadow pattern-discovered matches (warning emitted).

- projectconfig: validate placeholder syntax per scope; reject backslash separators and glob metachars before the placeholder

- components: pattern_discovery walks overlay-files entries; Resolver caches discovery results per instance

- schema/docs: user-facing overlays.md updated
Document shared placeholder patterns and explicit-group declarations in both skill modes and the overlay reference. Update the rebased discovery fixture for required upstream-status metadata.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@liunan-ms
Nan Liu (liunan-ms) force-pushed the liunan/overlayfiles-pattern branch from 0573473 to 733fe58 Compare November 12, 2026 00:59
Copilot AI balanced review requested due to automatic review settings November 12, 2026 00:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Pattern-discovered components lack repository provenance, causing default lock-file mode to skip synthetic history and automatic release bumps for overlay changes even when lock-file fingerprints are committed.

6 open issues

Review effort: Balanced

Comment thread internal/projectconfig/overlay_file.go Resolved
Copilot AI balanced review requested due to automatic review settings November 12, 2026 01:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread internal/projectconfig/component.go Resolved
@liunan-ms
Nan Liu (liunan-ms) force-pushed the liunan/overlayfiles-pattern branch from 31c8e8f to 6dadb7f Compare November 12, 2026 06:14
Copilot AI balanced review requested due to automatic review settings November 12, 2026 06:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Preserve plain overlay-file patterns and empty-list overrides in project defaults and component settings, with opt-in {component} discovery. Reject overlay-files entirely in distro-version and component-group defaults, including empty lists.

Preserve complete inherited overlay lists and optional placeholder matches, and restore locked synthetic history without forcing Git lookups for undeclared components without resolved lock data.

Reject brace-alternative prefixes and parent-directory segments after {component}, while allowing parent-directory segments before it.

Update configuration references, both emitted skill modes, schema descriptions and snapshots, and regression coverage for scope validation, discovery, inheritance, filenames, optional matches, history and no-lock behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@liunan-ms
Nan Liu (liunan-ms) force-pushed the liunan/overlayfiles-pattern branch from 6dadb7f to bf8910d Compare November 12, 2026 19:48
Copilot AI balanced review requested due to automatic review settings November 12, 2026 19:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot AI balanced review requested due to automatic review settings November 14, 2026 01:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because path normalization in --without-lockfile mode can bypass component-scope validation and load overlays from the wrong directory.

4 open issues

Review effort: Balanced

Keep shared {component} path substitution for declared and spec-discovered components, while retaining central component declarations. Remove overlay-driven discovery, caching, shadow warnings, and the unused splitting helper.

Update emitted skills, configuration guidance, schema descriptions, and regression tests to describe declaration-based overlay loading.
@liunan-ms
Nan Liu (liunan-ms) force-pushed the liunan/overlayfiles-pattern branch from 9605959 to aab618f Compare November 14, 2026 04:33
Copilot AI balanced review requested due to automatic review settings November 14, 2026 04:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

In --without-lockfile mode, path normalization can erase forbidden component placeholders before validation, allowing invalid entries to pass and load unintended shared overlays.

5 open issues
1 resolved since last review

Review effort: Balanced

Comment on lines +170 to +174
if err := validateOverlayFilesEntries(
component.OverlayFiles, validateComponentOverlayFilesEntry,
); err != nil {
return fmt.Errorf("invalid 'overlay-files' for component %#q:\n%w", componentName, err)
}
@liunan-ms Nan Liu (liunan-ms) changed the title feat(overlay-files): support {component} placeholder for pattern-discovered components feat(overlay-files): support shared patterns with {component} substitution Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants