Skip to content

fix(tests): correct release E2E image tag and telemetry checks - #1836

Open
zanejohnson-azure wants to merge 1 commit into
ci_prodfrom
zane/fix-release-e2e-checks
Open

zanejohnson-azure wants to merge 1 commit into
ci_prodfrom
zane/fix-release-e2e-checks

Conversation

@zanejohnson-azure

@zanejohnson-azure zanejohnson-azure commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fix the two recurring false-positive E2E failures observed in both the 3.9.1 release run 130085 and the 3.10.0 release run 131881. Agent image behavior is unchanged.

  • Validate ContainerInventory.ImageTag against each Kubernetes container's original image reference, matched by container ID for regular and init containers. Permit empty tags only for confirmed digest-only references; retain explicit-tag and implicit-latest validation. An unknown container with an empty tag still fails.
  • Record a UTC timestamp immediately before Helm deployment and propagate it through the deployment-job output, test template, runner, and Testkube configuration. Query startup traces since deployment instead of a rolling 30-minute window; retain recent heartbeat/metric checks and exact cluster/version/node coverage.
  • Use the actual telemetry version emitted by the build, including TELEMETRY_TAG overrides.
  • Pin Testkube's Git revision to the checked-out commit and stop on failed workflow apply/launch operations rather than potentially inspecting a stale execution.
  • Add focused Go and Bash regression tests and update the test documentation.

Image-tag validation behavior

The test matches each inventory record to its Kubernetes container by ContainerID, then parses the original image reference:

Image reference Validation
app:1.2 ImageTag must equal 1.2
app:1.2@sha256:… Still must equal 1.2
app@sha256:… Empty ImageTag is allowed
app ImageTag must equal latest

The separate Image, ImageID, and Repository checks still run. If an empty tag cannot be matched to a Kubernetes image reference, it still fails.

Validation

  • go test ./... in test/ginkgo-e2e/utils: passed.
  • bash test/testkube/install-and-execute-testkube-tests_test.sh: passed.
  • All three Ginkgo suites compile with the updated shared dependencies.
  • YAML syntax and all five release-job input connections checked.
  • Updated Testkube manifests accepted by Kubernetes server-side dry run; no cluster configuration changed.

Ran the locally compiled querylogs and containerstatus suites read-only against the unchanged 3.10.0-ci-prod-11-10-2026-c75cc056 deployment:

Cluster Querylogs Containerstatus
WCUS FIPS 17 passed, 0 failed 18 passed, 0 failed
Geneva multi-tenancy 16 passed, 0 failed 18 passed, 0 failed
Network-flow logs 17 passed, 0 failed 14 passed, 0 failed
All-node matrix 18 passed, 0 failed 18 passed, 0 failed
Workload identity 15 passed; 2 unchanged data-presence checks failed because fixtures were absent 14 passed, 0 failed

Workload identity passed every changed assertion. Its successful earlier pipeline run had removed Testkube, and the standalone rerun found no persistent volumes or active application log producers. The unchanged ContainerLog and KubePVInventory checks were deliberately not weakened. Normal release testing installs Testkube and its fixtures before executing these checks.

All four previously failing clusters now pass both suites. Final all-five-cluster confirmation still requires the normal deploy/test pipeline with its fixtures. That stage is gated to non-PR ci_prod runs.

Validate inventory tags against Kubernetes image references, anchor startup traces to deployment time, and pin Testkube to the checked-out commit. Preserve recent heartbeat and metric checks and fail closed on workflow setup errors.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@zanejohnson-azure
zanejohnson-azure requested a review from a team as a code owner November 11, 2026 23:50
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant