Skip to content

feat(amazonq): ITGen test HITL handlers, any-depth diff apply, completeHitlWithTestResults (behind ATX_ITGEN_ENABLED) - #2976

Open
xueningl-az wants to merge 3 commits into
feature/ITGen-RIVfrom
xueningl/ITGen-RIV-lsp-test-hitl
Open

xueningl-az wants to merge 3 commits into
feature/ITGen-RIVfrom
xueningl/ITGen-RIV-lsp-test-hitl

Conversation

@xueningl-az

@xueningl-az xueningl-az commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Row 14 of the ITGen plan: the language-server side of the ITGen–IDE contract v0 (§2–§6), so the IDE can run integration tests for the Transform agent.

  • With integration tests on, the agent nests LBV under local-verification (CR-315347405). The LBV error-fix rounds move to depth 3, and the ITGen fix steps (app-start fixes, code fixes) sit at depth 4. Today the LS applies diffs only for depth-2 steps, and it reports the nested LBV step, which has no diff, as a failed apply on every poll.
  • The new integ-app-launch / integ-test-run HITLs have to be handed to the IDE with their request, and the IDE needs a command to send results back.

Solution

Everything is gated on the env var ATX_ITGEN_ENABLED=true (isItGenEnabled(), a stand-in for the row 18 feature flag). With the flag off, every path behaves exactly as before, and there are explicit regression tests for EXECUTING, PLANNING and AWAITING_HUMAN_INPUT.

1. Apply step diffs at any depth (d18caf4c0)

  • A step is selected if it's SUCCEEDED and has a step-tagged CUSTOMER_OUTPUT artifact, at any depth. Depth no longer matters.
  • One paginated job-wide ListArtifacts call finds them, and only when a SUCCEEDED step still needs a download.
  • A step without an artifact is skipped, not failed. A download or apply failure still sets DiffApplyFailed.
  • Steps are applied in tree post-order (children before their parent).

2. Test HITL handlers (dde3be967)

  • Selection order: LBV → missing-packages → integ-app-launch → integ-test-run → *-checkpoint, at all three selection sites. integ-* HITLs are returned with their StepId and are never treated as a plan-review ZIP.
  • OpenHitls[] on getTransformInfo (TaskId, Tag, StepId, ExpiryTimestampUTC), built from the ListHitls call the poll already makes.
    • For integ-* HITLs, the request is saved once per task to artifactWorkspace/<job>/hitl/<taskId>/request.json. A schemaVersion major other than 1 gives RequestError instead.
    • For integ-test-run, the repo artifact with the generated tests is downloaded again for every task into itgen/<runId>/tests (TestsPath), because the agent rewrites it under the same id.
    • Agent-supplied ids used as folder names are validated.
  • New command aws/atxTransform/completeHitlWithTestResults {WorkspaceId, TransformationJobId, TaskId, ResultZipPath, SolutionRootPath}:
    • checks the ZIP holds summary.json (test run) or launch-result.json (app launch) with a supported schemaVersion;
    • then uploads it as HITL_FROM_USER → CompleteArtifactUpload → SubmitCriticalHitlTask by task id.

Testing

  • 543 netTransform unit tests pass, including 8 any-depth tests and 26 ITGen HITL tests (3 of them flag-off regression tests). tsc is clean, and lint has no errors.
  • I also ran a local harness that drives the real getTransformInfo against a fake FES through a full scripted job: LBV rounds → original app launch → test run → ITGen test fix → rerun → transformed launch fails → app-start fix (depth 4) → relaunch → test run → code fix (depth 4) → rerun. The harness isn't part of this PR.

Not in this PR: the beam diff path (beamed jobs still skip completed-step diffs), and the IDE handlers that consume this (rows 5/6, toolkit). The matching C# wrapper is in the toolkit PR from the same branch name.

Update (10/27): contract v0 schema 1.1

Follows the 10/26 contract update with Mark (interactive app launch), commit c5289a7c1:

  • completeHitlWithTestResults adds the customer's changed files to every transformed-side result ZIP (launch and test run) as workspace/metadata.json + workspace/<relative path>. It uses the same scan and metadata as updateWorkspace (tracked files modified since job start). The IDE's ZIP is left as it is, and a copy is uploaded. Original-side results are uploaded unchanged.
  • AtxAppLaunchResult gains optional launchedBy ('ide' | 'customer').
  • NOT_LOCALHOST is removed from AtxTestRunStatus (published URLs are allowed on the original side).
  • ATX_TEST_RUN_SCHEMA_VERSION is 1.1 (1.0 payloads are still accepted).
  • The side comment now says the original runs from artifactWorkspace/before/.

Tests: 548 netTransform tests pass (5 new for workspace/).

License

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…GEN_ENABLED

With integration tests on, the agent nests LBV under local-verification
(CR-315347405), so LBV error-fix rounds move to depth 3 and are never
applied, while the nested LBV step (depth 2, no diff) is reported as a
failed apply on every poll.

Behind the new ATX_ITGEN_ENABLED flag (stand-in for the row 18 feature
flag), completed steps are selected by having a step-tagged
CUSTOMER_OUTPUT artifact rather than by depth: one paginated job-wide
ListArtifacts call, only when a SUCCEEDED step still needs a download.
Steps without an artifact are skipped, not failed; steps apply in tree
post-order. With the flag off, the depth-2 behavior is unchanged.
…esults behind ATX_ITGEN_ENABLED

Language-server side of the ITGen-IDE contract v0 (sections 3-6), all
gated on ATX_ITGEN_ENABLED; with the flag off every path is unchanged.

- HITL selection: integ-app-launch, then integ-test-run, after LBV and
  missing-packages and before *-checkpoint, at all three selection sites.
  integ-* HITLs are returned with their StepId on the EXECUTING, PLANNING
  and AWAITING_HUMAN_INPUT paths, never treated as a plan-review ZIP.
- getTransformInfo returns OpenHitls[] (TaskId, Tag, StepId, expiry),
  built from the ListHitls the poll already made. For integ-* HITLs the
  agent's request is saved once per task to
  artifactWorkspace/<job>/hitl/<taskId>/request.json and rejected
  (RequestError) when the schemaVersion major isn't 1. For integ-test-run
  the repo artifact with the generated tests is downloaded again for every
  task into itgen/<runId>/tests (TestsPath), since the agent rewrites it
  under the same id. Agent-supplied ids used as folder names are checked.
- New aws/atxTransform/completeHitlWithTestResults: checks the result ZIP
  holds summary.json (test run) or launch-result.json (app launch) with a
  supported schemaVersion, uploads it as HITL_FROM_USER, completes the
  upload and submits the task by id (SubmitCriticalHitlTask).
@xueningl-az
xueningl-az requested a review from a team as a code owner October 26, 2026 20:14
@codecov-commenter

codecov-commenter commented Oct 26, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 74.90909% with 138 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...anguage-server/netTransform/atxTransformHandler.ts 74.79% 119 Missing and 2 partials ⚠️
...guage-server/netTransform/atxNetTransformServer.ts 15.00% 17 Missing ⚠️

📢 Thoughts on this report? Let us know!

Follows the 10/26 contract update (interactive app launch):

- completeHitlWithTestResults adds the customer's changed files to every
  transformed-side result ZIP (launch and test run) as
  workspace/metadata.json + workspace/<relative path>, with the same scan
  and metadata as updateWorkspace (tracked files modified since job
  start). The IDE's ZIP is left as it is; a copy is uploaded. Original
  side results are uploaded unchanged.
- AtxAppLaunchResult gains optional launchedBy ('ide' | 'customer').
- NOT_LOCALHOST is removed from AtxTestRunStatus (published URLs are
  allowed on the original side).
- ATX_TEST_RUN_SCHEMA_VERSION is 1.1 (1.0 payloads are still accepted).
- The side comment now says the original runs from artifactWorkspace/before/.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants