Repository navigation
[3.16.2] Cherry picks from develop to release-3.16 - #747
Merged
gmarciani merged 9 commits intoOct 20, 2026
Merged
Conversation
…ported as ICE when using multi InstanceTypes or SubnetIds (aws#734) Since [aws#578](aws#578), EC2 throttling during a compute node scale-up is never retried, and every launch failure is instead reported as insufficient capacity. **Preconditions**: a compute resource with more than one instance type **or** more than one subnet, and a scale-up large enough to exhaust the `RunInstances` resource token bucket, which holds 1000 instances and refills at 2/s. See the number in ref link. **On `best-effort` (the default)**: a single throttled call disables the **whole compute resource**, not only the batch that failed. Measured with `MaxCount 3000` and `sbatch -N 1500`: 500 nodes went DOWN and the remaining 1500 got `Temporarily disabling node due to insufficient capacity` — 2000 of 3000 nodes unusable for 600s (`insufficient_capacity_timeout`), with the job stuck `PENDING`. **On `all-or-nothing`**: the compute resource cannot scale up reliably. Every round releases the instances already launched and starts over, so a large scale-up would fail. `CreateFleet` returns one entry in `Errors` per launch template override, so a compute resource with more than one instance type or more than one subnet gets an entry for **every** override that was not fulfilled, not only for the one that actually failed. Those extra entries all carry the same pair: ``` ("UnfulfillableCapacity", "Failed to fulfill capacity. Please review errors in the response.") ``` which points at the real error rather than being one. With 36 overrides a single failure therefore arrives as 36 entries, so the pre-existing `len(err_list) == 1` condition in `Ec2CreateFleetManager._launch_instances` never holds. No `LaunchInstancesError` is raised, and `InstanceManager._launch_ec2_instances` falls back to a hardcoded `InsufficientInstanceCapacity`. Three things follow: * **Throttling is never retried.** `launch_ec2_instances` already implements the exponential backoff [the EC2 documentation recommends](https://docs.aws.amazon.com/ec2/latest/devguide/ec2-api-throttling.html#api-backoff), sized to wait for the bucket to refill a full `launch_max_batch_size` batch, but no exception ever reaches it. The batch is abandoned after a single attempt even though retrying is all it needed. * **The compute resource is failed over for nothing.** `InsufficientInstanceCapacity` is in `SlurmNode.EC2_ICE_ERROR_CODES`, so `clustermgtd` treats the failure as a capacity shortage and disables the whole compute resource, not just the batch that failed. * **The real cause is lost.** A `VcpuLimitExceeded`, an `Unsupported` instance type or an `UnauthorizedOperation` is all flattened into insufficient capacity. 1. **Drop the entries that only point at the real error** before deciding what to report, so the existing `len(err_list) == 1` condition works again for multi-override compute resources. The response is left untouched when those entries are all it carries, so the worst case is today's behaviour. 2. **Prefer throttling over any other remaining cause**, as a safety net for a response carrying more than one real error. Throttling is the only cause that resolves on its own, and reporting it as insufficient capacity costs a fail-over of the whole compute resource.
gmarciani
changed the title
* Add python 3.14 to test matrix * Bump python minimum version to 3.14 --------- Co-authored-by: Helena Greebe <hgreebe@amazon>
…ped (aws#740) When a CreateFleet response that launched nothing carries RequestLimitExceeded next to other real errors, retrying the throttling only helps if the pools EC2 did not get to try can still serve the batch. That holds when every other error is confined to one instance type and subnet pool: a capacity shortage, an unsupported type in one AZ, a full or missing subnet, an AZ that lacks the requested volume type or is temporarily unavailable. Any other error, such as an account quota or a permission problem, would fail the retry on every pool alike, so it is now reported immediately instead of waiting for the rate limit to refill while the instances already launched for the job sit idle. The pool-scoped set reuses EC2_ICE_ERROR_CODES and adds six subnet and AZ level codes. Codes outside the set, including unknown ones, are not retried: the batch fails with that code, as it did before the throttling fix minus the invented insufficient capacity. Responses without RequestLimitExceeded are unchanged.
…m nodes and the Comment field of Slurm reservations could alter the node and reservation information read by ParallelCluster.
Slurm 26.05 adds two keys to each job in the `SLURM_RESUME_FILE`: - [exclusive](SchedMD/slurm@7f63a8c) - [priority](SchedMD/slurm@219d033) Changelog: [26.05.0rc1](https://github.com/SchedMD/slurm/blob/476de143d0fd4137fd4606882a68443965529ba3/CHANGELOG/slurm-26.05.md?plain=1#L332) ("Add job priority value to SLURM_RESUME_FILE"), [L465](https://github.com/SchedMD/slurm/blob/476de143d0fd4137fd4606882a68443965529ba3/CHANGELOG/slurm-26.05.md?plain=1#L465) ("Add exclusive to the SLURM_RESUME_FILE"). Code: [power_save.c](https://github.com/SchedMD/slurm/blob/slurm-26-05-5-1/src/slurmctld/power_save.c#L650-L671). `SlurmResumeJob(**job)` raised `TypeError: unexpected keyword argument 'exclusive'`, which failed the whole resume. As a result, ParallelCluster could not launch dynamic nodes with Slurm 26.05. This commit: - [Must have] Accepts `exclusive` and `priority`, and ignores any key a future Slurm adds, so new Resume File fields no longer break scaling. - [Cleanup BTW] Removes `SlurmResumeJob.is_exclusive()`. It had no caller since aws#574 (job-level scaling for all jobs). Backward compatible: Slurm 25.11 doesn't emit either key, so both default to `None` and parsing is unchanged. Note: this is an initial support for Slurm 26.05 to allow ParallelCluster scale correctly. This commit doesn't consider incorporating new Slurm features.
…6.05". In 3.16.2 we removed known blockers to adopt Slurm 26.05, but we did not upgrade. This entry may set the wrong expectations.
gmarciani
marked this pull request as ready for review
October 19, 2026 20:49
gmarciani
marked this pull request as draft
October 19, 2026 20:50
gmarciani
marked this pull request as ready for review
October 20, 2026 17:50
hanwen-cluster
approved these changes
Oct 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Cherry-picks:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.