You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add support for proper lockless rendering, and other related commands - #430
This adds support for lockless rendering, in a controlled per-component transition.
During the transition, the tool will look for a file named lockless-components.txt in the same dir as the azldev.toml file (i.e., the top dir of the repo). If a component name is found inside this file, the component is considered a 'lockless' component; otherwise, the component is a legacy 'locked' component. Since our repo currently has an empty lockless-components.txt file, all our components are currently considered 'locked', and none of these updates to azldev should have any effect at all on any 'locked' component.
Until all components have been transitioned over, this adds a new command azldev comp make-lockless, which adds the specified component name(s) to the lockless-components.txt file, and runs azldev comp refresh-upstream-commit for them, to generate the component's default upstream-commit toml file. After that, the component's 'lock' file will no longer be used for anything (unless the component is removed from the lockless-components.txt file).
After transitioning all components to lockless, we should remove support for 'locked' files from azldev, and remove the azldev comp make-lockless command; and then we can remove the lockless-components.txt file from our main repo.
Most operations are the same for lockless components, but notably:
render operations should now work correctly, and should update the release value and generate the changelog entry correctly.
render operations are now slightly different, instead of backwards-searching the git commits to try to find text for a changelog entry, azldev will now prompt the user to provide a changelog entry or autorelease git commit message. This may be avoided by providing the message using the new --changelog-comment parameter.
render operations now will git commit the rendered files, unless the --no-git-commit parameter is used.
render operations now require no uncommitted changes in the repo (although uncommitted changes in the specs/ dir other than the rendering component's dist-git dir are ignored). this can be forced with --allow-uncommitted
renders no longer use mock for anything.
renders for autorelease components do not pre-rpmautospec the rendered dist-git dir.
there is a new command azldev comp dist-git-repo which converts the committed dist-git dir for a component into an industry-standard dist-git repo (at the path specified to --output-dir).
Besides those listed above, I have probably forgot some other changes.
Note this is still undergoing copilot testing, and there may be additional changes required; but this should not impact any components that still use 'lock' files, so we can slowly migrate components to lockless one-at-a-time and address any transition issues as we find them.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Skipped local components can still have generated TOML files deleted or flagged as stale, and shared single-branch cloning can break locked renders/builds using valid commits pinned on other branches.
This pull request requires changes. The following issues remain unresolved:
Critical — internal/app/azldev/cmds/component/refreshupstreamcommit.go:173: The orphan pass can delete generated TOML files for configured locked local components, or report them as stale under --check-only. Exempt configured locked components while preserving cleanup for genuinely removed components.
Moderate — internal/app/azldev/cmds/component/query_lockless.go:62: Querying all components serially clones and rewrites the full project history for each lockless component. Reuse shared repository objects or bulk extraction while retaining separate rpmautospec repositories.
Moderate — internal/app/azldev/cmds/component/render_release_carryover.go:39: An autorelease-to-static transition can restore the old %autorelease macro and skip the required static bump. Resolve the previous release to a concrete baseline instead.
Moderate — internal/utils/git/git.go:211: Shared single-branch cloning breaks locked workflows pinned to commits unavailable on the selected branch or through tags. Make this behavior opt-in or fetch the pinned commit before checkout.
Lockless builds pass the raw With, Without, and Defines values to mock without applying Build.Undefines. For example, an inherited defines.dist = ".azl3" with undefines = ["dist"] is correctly omitted from the rendered macros file, but this path reintroduces it through --define dist .azl3 in both SRPM and RPM builds. Filter the options using the same undefine semantics as buildMacrosMap before forwarding them, so transitioning a component does not change its effective build macros.
Fetch pinned commits missing from the single-branch clone
Lockless source preparation fails when an explicit upstream pin is reachable only from another branch and is not fetched through a tag. The single-branch clone omits that commit, and checkoutTargetCommit never fetches it before checkout. Keep a multi-branch clone when effectiveCommit is set, or explicitly fetch the pinned commit.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Critical findings omitted from the published review involve incorrect macro resolution and source retrieval and pinned-commit checkout failures, requiring human assessment.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Unpublished findings identify a critical regression in cross-branch commit pinning and failures on ignored unreadable directories, warranting human assessment.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed to evaluate unpublished findings concerning cross-branch pinned-commit checkout failures and lost error diagnostics when the mock probe fails.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because unpublished findings identify pinned-commit checkout failures and ignored macro undefines that can disrupt source preparation and builds.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because lockless builds forward raw options that can reinstate macros excluded by build.undefines, causing build settings to diverge from the rendered spec.
Prepare lockless components directly in the requested output directory
using the locked source acquisition, macro, overlay and hash pipeline.
Read release/changelog history from the configured rendered-specs tree
without generating synthetic history or committing prepared changes.
Honor source-skipping and missing-hash flags, and remove Git metadata only
after successful preparation when --without-git is requested. Preserve
partial output on failure and use the shared --force output guard.
Update command help, generated docs, agent skills and regression tests.
Validation: mage fix all, mage unit and mage docs passed. Quality checks
in mage all passed; scenario tests and mage scenarioUpdate remain blocked
by the existing Docker-provider/$XDG_RUNTIME_DIR initialization failure.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dc8e9aa-35cd-42a2-9377-a73317ee4a51
Resolve the current upstream commit and distro from working-tree TOML
during lockless source preparation. Continue reading the previous render's
configuration and component dist-git snapshot from committed history.
Keep render's committed-input behavior unchanged. Cover modified and
untracked TOML and overlay inputs, and verify that dirty dist-git specs
and sidecars do not affect release/changelog baselines.
Update user documentation, generated help and emitted agent skills.
Validation: mage fix all, mage unit and mage docs passed. Quality checks
in mage all passed; scenario tests and mage scenarioUpdate remain blocked
by the existing Docker-provider/$XDG_RUNTIME_DIR initialization failure.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dc8e9aa-35cd-42a2-9377-a73317ee4a51
Limit the lockless render preflight to changes outside the configured
project.rendered-specs-dir and changes in selected component dist-git
directories. Ignore changes in unrelated rendered output while keeping
the --allow-uncommitted override and locked-only behavior unchanged.
Use literal Git pathspecs to preserve path boundaries, handle renamed
files and inspect the whole repository from nested project directories.
Update regression tests, generated help, user docs and agent skills.
Validation: mage fix all, mage unit and mage docs passed. Native Git
scope checks passed all 34 cases. Quality checks in mage all passed;
scenario tests and mage scenarioUpdate remain blocked by the existing
Docker-provider/$XDG_RUNTIME_DIR initialization failure.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dc8e9aa-35cd-42a2-9377-a73317ee4a51
Exempt configured locked components from generated upstream-commit orphan
checks and pruning, including local components. Keep cleanup for components
that have actually been removed.
Cover normal refresh and check-only preservation alongside genuine orphan
cleanup with in-memory regression tests.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dc8e9aa-35cd-42a2-9377-a73317ee4a51
Restore branch selection without excluding commits on other branches so
locked renders and builds retain support for their existing pins.
Opt into single-branch fetching explicitly for lockless preparation and
snapshot identity resolution. Cover both clone modes and option forwarding.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dc8e9aa-35cd-42a2-9377-a73317ee4a51
Calculate the previous complete release from committed component history
instead of restoring its autorelease macro into a new static spec. Resolve
its generated changelog before the normal static bump as well.
Preserve version-reset and no-dist behavior, propagate calculation errors,
and remove temporary repositories on both success and failure. Cover both
spec editors and document the extraction prerequisite.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dc8e9aa-35cd-42a2-9377-a73317ee4a51
Clone one immutable project snapshot for each bulk query, then extract
component histories concurrently using shared objects instead of repeating
full project clones serially.
Repack each filtered repository and remove its alternate object dependency
before passing it into Mock. Cover bulk and mixed queries, extraction
failures, independent object stores, and scratch cleanup.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dc8e9aa-35cd-42a2-9377-a73317ee4a51
The lockless prepare-sources path called CheckOutputDir before checking
for dry-run mode. With '--force', that helper removes a nonempty output
directory through the non-dry-run-aware filesystem, so a dry run could
delete existing output.
During a dry run, still reject a nonempty output directory when
'--force' is not set, but return before any cleanup or preparation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ae64263c-02b0-4e70-89b2-030ec0389a00
Restore non-symlink collision protection for locked render aliases while
retaining replacement of occupied alias paths for lockless components.
Cover file and directory collisions in both modes and retain the existing
relative and dangling-symlink regressions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dc8e9aa-35cd-42a2-9377-a73317ee4a51
Restore recursive copying with preserved file modes for locked components.
Keep rename-based publication and staged Git history for lockless renders.
Cover cross-device rename failures, staged-file retention, and permissions
with in-memory filesystem tests for both modes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dc8e9aa-35cd-42a2-9377-a73317ee4a51
Restrict root Git and rebuild-marker exclusions to lockless rendering.
Remove staged Git metadata before comparing locked output, including when
file filtering is disabled, without changing existing output in check-only.
Cover locked and lockless metadata drift, staging cleanup, and nested
markers using the render completion path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dc8e9aa-35cd-42a2-9377-a73317ee4a51
Restore normalized Release tag updates by default in both spec editors.
Opt into preserving casing and alignment for lockless preparation,
changelog carryover, autorelease transitions, and source overlays.
Test both editor modes and mutation APIs, plus source-preparer overlays
and generated release bumps in locked and lockless modes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dc8e9aa-35cd-42a2-9377-a73317ee4a51
diff-sources builds its source preparer without lockless history options,
so Release overlays on lockless components were rewritten as
'Release: <value>' even though render and build preserve the original
casing and spacing.
Add a preparer option that requests preserved Release formatting and set
it for lockless components in diff-sources. Locked components keep the
normalized rewrite.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 16503351-1bf3-4693-96f1-26283e4fc641
Add UpdateExistingTagFunc, which computes each matching tag's new value
from its current value, and GetTags, which returns every matching tag
value in spec order. Both are implemented for the legacy and structural
editors and exposed on the Spec facade.
UpdateExistingTag is now a wrapper that returns the same value for every
tag, so its output and errors are unchanged for existing callers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 16503351-1bf3-4693-96f1-26283e4fc641
Lockless rendering read only the last Release tag and wrote its updated
value to every Release tag. Specs with conditional Release tags, such as
yara's '%if %{with release}' / '%else' pair, had every branch replaced
by the last one, changing the default build's NEVR.
Compute the autorelease base once and apply it to each Release tag,
keeping each tag's own options. Static carryover now copies committed
Release values by position and fails when the upstream and committed
tag counts differ. Specs that mix '%autorelease' and static Release
tags, and committed specs with differing '%autorelease' tags converted
to a static release, now fail with a specific error.
Locked components do not use this code path. Lockless specs with a
single Release tag render exactly as before.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 16503351-1bf3-4693-96f1-26283e4fc641
Historical project snapshots are written against older config schemas.
Strictly decoding them fails once azldev drops a field, for example the
removed [test-suites] table in older images.toml files. That broke the
lockless render upstream history lookup for every component last
rendered before the removal, so render commits fell back to a generic
"Update <name> to version <ver>" message instead of the upstream log.
Always ignore unknown fields when loading historical configuration.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8b620ba1-9249-46b7-bfb4-78616d6f1ec1
Lockless render validation queried existing and new source NEVRs with plain
rpmspec, which expands an unprocessed `%autorelease -b N` to N. Builds
instead filter the rendered directory's history and let rpmautospec count
commits, so valid base bumps were rejected (e.g. SymCrypt, telegraf) and
the check did not reflect the release actually built.
For unprocessed %autorelease specs, reconstruct the filtered rendered
history and run `rpmautospec process-distgit` before querying. The existing
spec uses committed history; the new spec adds the prepared output as the
next commit unless its tree is unchanged. Merge-free histories are replayed
into a temporary repository that reads project objects via alternates and
never writes to the project; histories with merges fall back to cloning and
running `git filter-repo --subdirectory-filter`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8b620ba1-9249-46b7-bfb4-78616d6f1ec1
rpmdev-bumpspec bumps every line starting with 'Release:', including
text inside %changelog, such as a quoted spec snippet in a previous
entry. Lockless renders then rewrote that historical changelog text.
Before running rpmdev-bumpspec for a lockless component, temporarily
mask %changelog lines that start with 'Release:' and restore them
afterward, also restoring the original spec if the bump fails. Specs
without such lines are bumped exactly as before, and locked components
are unaffected because they use a separate bumpspec path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 16503351-1bf3-4693-96f1-26283e4fc641
Use the source manager to retrieve configured payloads before downloading
remaining dist-git manifest entries from lookaside. Preserve download-origin
fallback, custom generation, hash validation, and disable-origins policy
without replaying overlays or replacing the committed rendered spec.
Cover origin downloads with and without a manifest, disabled origins, and
custom-provider dispatch. Update build guidance for both skill catalogs.
Validation: mage fix all, mage unit, and mage all build/quality checks
passed. Container scenarios were blocked by Docker socket permissions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 83cf8df4-30c0-474c-8dca-e3b3a2e54873
Compare each component's committed rendered tree before filtering unchanged
inputs. Include spec, sidecar, manifest, and no-change-rebuild marker changes
so render-only commits enter all-component rebuild selection. Continue to
report sources-change only for manifest content changes.
Cover output-only changes, added/removed renders, unchanged component trees,
and unrelated changes using in-memory Git repositories. Document the
additional change-detection input.
Validation: mage fix all, mage unit, and mage all build/quality checks
passed. Container scenarios were blocked by Docker socket permissions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 83cf8df4-30c0-474c-8dca-e3b3a2e54873
Include captured stderr when the plugin prerequisite probe fails. Keep the
original error wrapped so callers can still inspect its exit status.
Add a regression test for a nonzero probe exit that reports diagnostics and
prevents the SRPM build from starting.
Validation: mage fix all, mage unit, and mage all build/quality checks
passed. Container scenarios were blocked by Docker socket permissions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 83cf8df4-30c0-474c-8dca-e3b3a2e54873
Do not limit clones to one branch when the component has an effective
upstream commit. Explicit pins and locked commits may exist only on other
branches and cannot be checked out from a single-branch clone. Keep the
single-branch optimization for floating sources.
Test generated clone arguments and checkout targets for floating, explicit,
and locked pins without invoking external commands. Document pin behavior.
Validation: mage fix all, mage unit, and mage all build/quality checks
passed. Container scenarios were blocked by Docker socket permissions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 83cf8df4-30c0-474c-8dca-e3b3a2e54873
Stop forwarding live build defines and with/without flags to Mock when
building committed dist-git. The rendered spec already loads the resolved
macro file, including TOML undefine semantics. Remove the redundant builder
option path for both SRPM and binary builds while retaining local repos,
rpmautospec, and no-check behavior.
Test that conflicting live macro settings do not reach Mock or overwrite
the committed macro file. Document the committed-spec source of macros.
Validation: mage fix all, mage check all, and the full unit suite passed.
The build and default quality checks passed; Docker-enabled container
scenarios are still running separately.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 83cf8df4-30c0-474c-8dca-e3b3a2e54873
Emit resolved build and provenance macros as sorted, lazy %define entries
before user overlays. Do not write an .azl.macros file or inject its load
directive and Source9999 tag for lockless components.
Keep file-based macro loading unchanged for locked components. Cover empty
values, deferred expansion, with/without flags, and skipped overlays. Update
CLI documentation and emitted agent skills for both workflows.
Unit, quality, and changed-line mutation checks pass. Scenario failures for
stale global-flag and schema snapshots also reproduce before this change.
Network-dependent scenarios were not completed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 83cf8df4-30c0-474c-8dca-e3b3a2e54873
Regenerate local and container help, MCP, and schema snapshots to match
removal of the global --without-lockfile flag and the updated lockless
file-filter description.
Verify with mage scenarioUpdate and mage scenario, both using
TESTING_DISABLE_NETWORK_TESTS=1. The full network-dependent suite remains
unverified after its earlier stall during Mock chroot cleanup.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 83cf8df4-30c0-474c-8dca-e3b3a2e54873
Return build results directly and wrap errors from the shared Git path helper.\n\nValidation: mage docs, mage unit, and mage check all passed.\nContainer scenarios are blocked by Docker socket permissions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7e9e3287-a989-4992-970c-595c177a8505
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
End-to-end validation with real RPM/rpmautospec, git-filter-repo, and Mock tooling is unavailable, leaving release/changelog correctness, SRPM history handling, and compatibility across locked and lockless components unverified.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds support for lockless rendering, in a controlled per-component transition.
During the transition, the tool will look for a file named
lockless-components.txtin the same dir as theazldev.tomlfile (i.e., the top dir of the repo). If a component name is found inside this file, the component is considered a 'lockless' component; otherwise, the component is a legacy 'locked' component. Since our repo currently has an emptylockless-components.txtfile, all our components are currently considered 'locked', and none of these updates to azldev should have any effect at all on any 'locked' component.Until all components have been transitioned over, this adds a new command
azldev comp make-lockless, which adds the specified component name(s) to thelockless-components.txtfile, and runsazldev comp refresh-upstream-commitfor them, to generate the component's default upstream-commit toml file. After that, the component's 'lock' file will no longer be used for anything (unless the component is removed from thelockless-components.txtfile).After transitioning all components to lockless, we should remove support for 'locked' files from azldev, and remove the
azldev comp make-locklesscommand; and then we can remove thelockless-components.txtfile from our main repo.Most operations are the same for lockless components, but notably:
--changelog-commentparameter.--no-git-commitparameter is used.--allow-uncommittedazldev comp dist-git-repowhich converts the committed dist-git dir for a component into an industry-standard dist-git repo (at the path specified to--output-dir).Besides those listed above, I have probably forgot some other changes.
Note this is still undergoing copilot testing, and there may be additional changes required; but this should not impact any components that still use 'lock' files, so we can slowly migrate components to lockless one-at-a-time and address any transition issues as we find them.