Skip to content

Keep agent credentials after UNENROLL for the uninstall flow - #8091

Draft
rdner wants to merge 1 commit into
elastic:mainfrom
rdner:keep-credentials-after-unenroll
Draft

rdner wants to merge 1 commit into
elastic:mainfrom
rdner:keep-credentials-after-unenroll

Conversation

@rdner

@rdner rdner commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

What is the problem this PR solves?

The UNENROLL action is now the first step of the remote-uninstall flow. This PR changes how Fleet Server handles it:

  • UNENROLL keeps the agent active and its access API key valid, and sets uninstall_pending instead of invalidating, so the agent can still receive the follow-up UNINSTALL/CANCEL. While pending, only UNINSTALL/CANCEL are delivered (policy changes and all other actions are withheld).
  • UNENROLL invalidates the agent's output API keys (e.g. Elasticsearch) so the unenrolled agent immediately stops shipping data.
  • UNINSTALL is only allowed once the agent is unenrolled (pending uninstall); otherwise it is rejected and a failed action result is written to .fleet-actions-results. Terminal invalidation (access key invalidated, agent marked inactive) happens on the UNINSTALL ack.
  • This behavior is gated for backward compatibility: only agents that advertise UNINSTALL support via the uninstallable flag in their local_metadata enter the pending-uninstall flow. Older agents without the flag keep the previous behavior and have their credentials invalidated immediately on UNENROLL.
  • Server-side force-unenroll (which deletes the agent without contacting it) is unchanged, and MIGRATE remains terminal.

How to test this PR locally

Tests have been added.

Design Checklist

  • I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
  • I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
    - [ ] I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

  • I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool

Related issues

This is a follow up to #7895
Prerequisite to elastic/elastic-agent#16540

@rdner rdner self-assigned this Oct 21, 2026
@rdner rdner added the enhancement New feature or request label Oct 21, 2026
Copilot AI balanced review requested due to automatic review settings October 21, 2026 15:15
@rdner rdner added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Oct 21, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Credential retention and state-transition issues remain, alongside action-delivery defects and an invalid integration-test fixture.

6 open issues
1 file not reviewed
  • internal/pkg/model/schema.go: Generated file
What changed in this PR

Adds a pending-uninstall state so Fleet Server can retain agent credentials between UNENROLL and a follow-up UNINSTALL.

Changes:

  • Defers credential invalidation until UNINSTALL acknowledgement.
  • Restricts pending agents to UNINSTALL/CANCEL and records premature UNINSTALL failures.
  • Adds schema support, tests, and a changelog entry.
File Description
model/​schema.json Defines pending-uninstall state.
internal/​pkg/​server/​uninstall_action_integration_test.go Tests the two-step uninstall flow.
internal/​pkg/​model/​schema.go Adds the generated agent field.
internal/​pkg/​dl/​constants.go Adds the persistence field constant.
internal/​pkg/​api/​handleCheckin.go Filters actions and records uninstall rejections.
internal/​pkg/​api/​handleCheckin_test.go Tests filtering and rejection results.
internal/​pkg/​api/​handleAck.go Separates pending and terminal acknowledgements.
internal/​pkg/​api/​handleAck_test.go Tests credential retention after UNENROLL.
changelog/​fragments/​1792579200-keep-credentials-after-unenroll-for-uninstall.yaml Documents the lifecycle changes.

Review effort: Balanced

Comment thread internal/pkg/api/handleAck.go Outdated
Comment on lines +51 to +53
Agents: []string{resp.Item.Id},
ActionID: unenrollActionID,
Type: string(api.UNENROLL),

now := time.Now().UTC().Format(time.RFC3339)
doc := bulk.UpdateFields{
dl.FieldUninstallPending: true,
Comment on lines +592 to 594
pendingActions, rejectedUninstalls := filterActions(zlog, agent.Id, pendingActions, agent.UninstallPending)
ct.rejectUninstallActions(r.Context(), zlog, agent.Id, rejectedUninstalls)
actions, ackToken = convertActions(zlog, agent.Id, pendingActions)
Comment on lines +617 to +618
acdocs, rejectedUninstalls := filterActions(zlog, agent.Id, acdocs, agent.UninstallPending)
ct.rejectUninstallActions(ctx, zlog, agent.Id, rejectedUninstalls)
Comment on lines +1023 to +1025
if action.Type != string(UNINSTALL) && action.Type != string(CANCEL) {
zlog.Info().Str(ecs.AgentID, agentID).Str(ecs.ActionID, action.ActionID).Str(ecs.ActionType, action.Type).Msg("Withholding non-uninstall action from pending-uninstall agent")
continue
@mergify

mergify Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

This pull request does not have a backport label. Could you fix it @rdner? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit
  • backport-active-all is the label that automatically backports to all active branches.
  • backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
  • backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

@rdner
rdner force-pushed the keep-credentials-after-unenroll branch from b456166 to ba0701b Compare October 21, 2026 16:36
Copilot AI balanced review requested due to automatic review settings October 21, 2026 16:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The integration fixture lacks the required uninstall capability, so it takes the legacy invalidation path rather than exercising the intended flow.

6 open issues
1 resolved since last review
1 file not reviewed
  • internal/pkg/model/schema.go: Generated file

Review effort: Balanced

Comment on lines +61 to +62
AgentAck(t, ctx, srv, gotUnenrollID, resp.Item.Id, resp.Item.AccessApiKey)
AssertAgentPendingUninstall(t, ctx, srv, resp.Item.Id)
@cmacknz

cmacknz commented Oct 26, 2026

Copy link
Copy Markdown
Member

I haven't read this yet since it's a draft, but if we are changing the behaviour of the existing UNENROLL action, make sure it and the code that interacts with it in fleet-server tolerates older agent versions that will be missing any of the agent side changes you've made.

- UNENROLL keeps the agent active and its access key valid, sets
  uninstall_pending instead of invalidating; only UNINSTALL/CANCEL are
  delivered afterwards.
- UNENROLL invalidates the agent's output keys (e.g. Elasticsearch) so the
  unenrolled agent immediately stops shipping data.
- UNINSTALL is rejected with a failed action result unless the agent is
  unenrolled; terminal invalidation happens on the UNINSTALL ack.

Co-authored-by: Cursor <cursoragent@cursor.com>
@rdner
rdner force-pushed the keep-credentials-after-unenroll branch from ba0701b to abd448c Compare October 27, 2026 10:27
Copilot AI balanced review requested due to automatic review settings October 27, 2026 10:27
@rdner

rdner commented Oct 27, 2026

Copy link
Copy Markdown
Member Author

@cmacknz it's already taken care of.

The plan is that new agents will report Uninstallable: true in their capabilities and Fleet Server will change its unenroll behavior based on that.

The recover from UNENROLL will come as a separate follow up PR, otherwise this one would be hard to review.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines +1060 to +1061
if err := dl.CreateActionResult(ctx, ct.bulker, acr); err != nil {
zlog.Error().Err(err).Str(ecs.AgentID, agentID).Str(ecs.ActionID, action.ActionID).Msg("failed to write rejected UNINSTALL action result")

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants