Repository navigation
feat(overlay-files): support shared patterns with {component} substitution - #269
Nan Liu (liunan-ms) wants to merge 4 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a {component} placeholder for project-level default-component-config.overlay-files to support pattern-based discovery of components and per-component overlay-file resolution, and updates schema/snapshots/docs accordingly.
Changes:
- Add projectconfig placeholder validation/splitting/substitution helpers and scope-based validation (
{component}required at project scope, forbidden elsewhere). - Add pattern-based component discovery from project-level
overlay-filesentries with resolver-level caching and deterministic unioning intoFindAllComponents. - Regenerate JSON schema / scenario snapshot and document the new placeholder behavior in the overlays config reference.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| schemas/azldev.schema.json | Updates schema description for overlay-files to document {component} rules. |
| scenario/snapshots/TestSnapshots_config_generate-schema_stdout_1.snap | Updates snapshot to match regenerated schema output. |
| internal/projectconfig/overlay_files_placeholder.go | Implements {component} placeholder validation, splitting, substitution, and absolutization helpers. |
| internal/projectconfig/overlay_files_placeholder_test.go | Unit tests for placeholder validation/splitting/substitution helpers. |
| internal/projectconfig/overlay_file.go | Substitutes {component} with component name before overlay glob expansion. |
| internal/projectconfig/configfile.go | Adds config-file validation enforcing placeholder rules by scope. |
| internal/projectconfig/configfile_test.go | Adds validation tests for allowed/forbidden placeholder usage across scopes. |
| internal/projectconfig/component.go | Updates OverlayFiles docs/schema description and absolutizes placeholder entries during WithAbsolutePaths. |
| internal/app/azldev/core/components/resolver.go | Unions pattern-discovered components into FindAllComponents with caching and shadowing behavior. |
| internal/app/azldev/core/components/pattern_discovery.go | Implements filesystem globbing + capture of {component} segment into synthesized components. |
| internal/app/azldev/core/components/pattern_discovery_test.go | Tests discovery, inheritance/override semantics, and same-scope collision errors. |
| docs/user/reference/config/overlays.md | Documents {component} placeholder discovery/inheritance/shadowing behavior for users. |
4a56921 to
f2c8de7
Compare
| } | ||
|
|
||
| count := strings.Count(entry, OverlayFilesComponentPlaceholder) | ||
| if count == 0 { |
There was a problem hiding this comment.
I think this stops us having non-component placeholder entries, ie have overlay-files = ["static/overlays/*.toml"]. If this is expected, docs need to be updated (overlays.md:166) to match.
There was a problem hiding this comment.
This is intentional. Project-level entries without {component} are rejected because they would silently apply the same files to every component. Updated the doc.
| {name: "star before placeholder", entry: "comps*/{component}/*.overlay.toml", wantErr: true}, | ||
| {name: "whole segment at start", entry: "{component}/overlays/*.overlay.toml", wantErr: false}, | ||
| {name: "whole segment in middle", entry: "comps/{component}/overlays/*.overlay.toml", wantErr: false}, | ||
| {name: "whole segment at end", entry: "comps/overlays/{component}", wantErr: false}, |
There was a problem hiding this comment.
I think this might be an error, won't it glob files with the name of the component, not directories? I think it probably always needs to be afolder right?
There was a problem hiding this comment.
Great catch, updated the check and test.
| // declaration overrides pattern-discovered overlay files for the same name. | ||
| func logShadowedByDeclaration(shadowed *patternDiscoveredComponent) { | ||
| slog.Warn( | ||
| "project 'overlay-files' pattern match shadowed by explicit component declaration; overlay files will not be applied", |
There was a problem hiding this comment.
I think this might miss shadowing from project defaults by groups. Not sure if thats something worth an error though.
There was a problem hiding this comment.
This would be resolved by the scope restriction, now rejecting overlay-files in distro-version and component-group default-component-config entirely.
| // which suppresses inheritance (see [projectconfig.ComponentConfig.MergeUpdatesFrom]). | ||
| // A declaration with no overlay-files inherits the pattern and the | ||
| // discovered files still apply, so there's nothing to warn about. | ||
| if declared, ok := r.env.Config().Components[name]; ok && declared.OverlayFiles != nil { |
There was a problem hiding this comment.
Might want to use len(declared.OverlayFiles) > 0 rather than nil, if I recall from the previous PR the way to disable inheritance is to set overlay-files=[]. 50/50 on this, we could still warn because thats weird, but it is intentionally explicitly saying "no overlays" for a component.
There was a problem hiding this comment.
Fixed. Changed to len(declared.OverlayFiles) > 0.
| } | ||
|
|
||
| prefix := entry[:idx] | ||
| if globMetaIdx := strings.IndexAny(prefix, "*?["); globMetaIdx >= 0 { |
There was a problem hiding this comment.
Might be able to add https://pkg.go.dev/github.com/bmatcuk/doublestar/v4#ValidatePattern here as well, get early signal back on a bad glob pattern.
There was a problem hiding this comment.
Now using doublestar.ValidatePattern on the substituted glob validation.
| err error | ||
| } | ||
|
|
||
| func (c *discoveredComponentsCache) load(env *azldev.Env) (map[string]*patternDiscoveredComponent, error) { |
There was a problem hiding this comment.
Found this a while back, might do the trick here: https://pkg.go.dev/sync#OnceValues
There was a problem hiding this comment.
switched to using sync.OnceValues
| } | ||
|
|
||
| byName := make(map[string]*patternDiscoveredComponent) | ||
| // Deduplicate files per component before appending to preserve stable ordering. |
There was a problem hiding this comment.
Curious if this is needed, my understanding was that glob would deduplicate internally.
There was a problem hiding this comment.
doublestar.Glob returns unique matches per call so the seen map was dead code and has been removed.
f2c8de7 to
1f37e09
Compare
1f37e09 to
27e7ffb
Compare
27e7ffb to
594c51b
Compare
|
Thanks Daniel McIlvaney (@dmcilvaney) for the review! After thinking about the scope and usage of
I update the implementation and now the scope rule is enforced.
|
594c51b to
fc2fd2b
Compare
fc2fd2b to
0573473
Compare
Dan Streetman (ddstreet)
left a comment
There was a problem hiding this comment.
question(blocking): Sorry I don't really understand what this is doing or solving, can you please add an example or at least more detail?
The main goal is to get rid of empty per-component config files when the overlay directory already tells us the component name. In azurelinux repo, 155 of 540 For example, in azurelinux, [components.GraphicsMagick]The actual changes live under [default-component-config]
overlay-files = ["{component}/overlays/*.overlay.toml"]azldev would discover For the explicitly listed group members that still need a component declaration. For example, I'll rebase this PR against main and update the guidance in skills and instructions. |
…overed components
Add '{component}' placeholder in project-level overlay-files entries. The resolver globs matching paths, extracts each captured segment as a component name, and synthesizes components not otherwise declared. Explicit component declarations shadow pattern-discovered matches (warning emitted).
- projectconfig: validate placeholder syntax per scope; reject backslash separators and glob metachars before the placeholder
- components: pattern_discovery walks overlay-files entries; Resolver caches discovery results per instance
- schema/docs: user-facing overlays.md updated
Document shared placeholder patterns and explicit-group declarations in both skill modes and the overlay reference. Update the rebased discovery fixture for required upstream-status metadata. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
0573473 to
733fe58
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Pattern-discovered components lack repository provenance, causing default lock-file mode to skip synthetic history and automatic release bumps for overlay changes even when lock-file fingerprints are committed.
6 open issues
Discovered components lack project provenance for lock-file history · New Re-globbing discovered filenames can silently omit overlay files · New Unmatched inherited patterns fail component discovery · New Alternation in prefixes passes validation but prevents discovery · New PR description says non-project scopes (distro version, component group, per-component) "reject the…absolutizeOverlayFilesPlaceholderEntriesrelies onmakeAbsolute, which usespath.Join(POSIX…
Review effort: Balanced
There was a problem hiding this comment.
🟡 Changes recommended
Stale schema snapshots will fail tests, and component discovery still loses inherited overlays and Git/release history while silently omitting matches from brace-alternative patterns.
8 open issues
Regenerate schema snapshot baselines · New Discovered components lack project provenance for lock-file history Preserve inherited overlay files when synthesizing components · New Alternation in prefixes passes validation but prevents discovery Unmatched inherited patterns fail component discovery Re-globbing discovered filenames can silently omit overlay files PR description says non-project scopes (distro version, component group, per-component) "reject the…absolutizeOverlayFilesPlaceholderEntriesrelies onmakeAbsolute, which usespath.Join(POSIX…
Review effort: Balanced
31c8e8f to
6dadb7f
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Discovery patterns can silently lose their component placeholder during path normalization, and unescaped component names can cause overlay loading to miss intended files or load a sibling component’s files.
4 open issues
Escape component names before inserting them into overlay globs · New Preserve the component placeholder during path normalization · New PR description says non-project scopes (distro version, component group, per-component) "reject the…absolutizeOverlayFilesPlaceholderEntriesrelies onmakeAbsolute, which usespath.Join(POSIX…
6 resolved since last review
Regenerate schema snapshot baselines Discovered components lack project provenance for lock-file history Preserve inherited overlay files when synthesizing components Alternation in prefixes passes validation but prevents discovery Unmatched inherited patterns fail component discovery Re-globbing discovered filenames can silently omit overlay files
Review effort: Balanced
Preserve plain overlay-file patterns and empty-list overrides in project defaults and component settings, with opt-in {component} discovery. Reject overlay-files entirely in distro-version and component-group defaults, including empty lists.
Preserve complete inherited overlay lists and optional placeholder matches, and restore locked synthetic history without forcing Git lookups for undeclared components without resolved lock data.
Reject brace-alternative prefixes and parent-directory segments after {component}, while allowing parent-directory segments before it.
Update configuration references, both emitted skill modes, schema descriptions and snapshots, and regression coverage for scope validation, discovery, inheritance, filenames, optional matches, history and no-lock behavior.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
6dadb7f to
bf8910d
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The component resolver leaves the complete result set unsorted, and brace alternatives can bypass overlay parent-directory validation.
4 open issues
Sort the complete component set before returning it · New Reject parent segments in brace alternatives · New PR description says non-project scopes (distro version, component group, per-component) "reject the…absolutizeOverlayFilesPlaceholderEntriesrelies onmakeAbsolute, which usespath.Join(POSIX…
2 resolved since last review
Review effort: Balanced
Keep shared {component} path substitution for declared and spec-discovered components, while retaining central component declarations. Remove overlay-driven discovery, caching, shadow warnings, and the unused splitting helper.
Update emitted skills, configuration guidance, schema descriptions, and regression tests to describe declaration-based overlay loading.
9605959 to
aab618f
Compare
There was a problem hiding this comment.
🟡 Changes recommended
In --without-lockfile mode, path normalization can erase forbidden component placeholders before validation, allowing invalid entries to pass and load unintended shared overlays.
5 open issues
Validate component overlay paths before normalization · New Sort the complete component set before returning it PR description says non-project scopes (distro version, component group, per-component) "reject the…absolutizeOverlayFilesPlaceholderEntriesrelies onmakeAbsolute, which usespath.Join(POSIX… Declaration-free curl example cannot discover overlay directories · New
1 resolved since last review
Review effort: Balanced
| if err := validateOverlayFilesEntries( | ||
| component.OverlayFiles, validateComponentOverlayFilesEntry, | ||
| ); err != nil { | ||
| return fmt.Errorf("invalid 'overlay-files' for component %#q:\n%w", componentName, err) | ||
| } |



Summary
Add
{component}substitution to project-leveldefault-component-config.overlay-files, allowing centrally declared components to share a per-component overlay layout without empty dedicated.comp.tomlfiles.The pattern resolves relative to the file declaring the default. Each known component substitutes its own name and loads its matching overlays. Overlay directories do not create components: central declarations remain required.
Behavior and restrictions
{component}is permitted only in project-level defaults, exactly once as a complete directory segment followed by a file path or pattern...segments are forbidden after the placeholder;..remains allowed before it. Use separate list entries for multiple directories.overlay-files, including[].[]disables inheritance.Documentation and tests
Update emitted agent-skill templates, configuration references, JSON schema, and scenario snapshots. Consolidate resolver regressions into
resolver_test.goand placeholder validation/loading regressions intoprojectconfig/overlay_files_placeholder_test.go.Validation
mage fix all,mage docs,mage scenarioUpdate, andmage allpassed with network-dependent tests disabled.