[hanwen-cluster]

Source: github.com/aws/aws-parallelcluster#7671

Summary

A dynamic node that still has a job can get stuck as IDLE+CLOUD+COMPLETING+POWER_DOWN after clustermgtd terminates its EC2 instance. The job never finishes completing, the node is never powered down, and clustermgtd takes no further action.

Root cause

1. clustermgtd finds the node unhealthy, terminates its instance and runs scontrol update state=power_down_force, which also resets NodeAddr to the node name.
2. POWER_DOWN_FORCE kills the job but leaves the node's base state alone. Slurm still treats the node as reachable and waits for EPILOG_COMPLETE. SuspendProgram doesn't run while the node is COMPLETING.
3. Normally slurmd stops answering, and after SlurmdTimeout Slurm marks the node DOWN and recovers it. But if EC2 gives the old private IP to another compute node, that node's slurmd answers in its place. So the node keeps looking alive, and the epilog never completes.
4. clustermgtd never notices. It decides whether a node should have an instance from NodeAddr, and NodeAddr has already been reset. Slurm still holds the old InstanceId, because it only clears InstanceId at POWERED_DOWN.

Change

Add SlurmNode.is_instance_expected() and use it in is_backing_instance_valid(). A node is now expected to have an instance if NodeAddr is set, or if InstanceId is set and the node isn't powering down or powered down. Slurm keeps InstanceId until POWERED_DOWN, so a stuck node is now reported as missing its instance. clustermgtd then treats it as unhealthy and sends another power_down_force. For a job that is already completing, that clears the node's epilog wait (kill_running_job_by_node_ptr), so the jobs are requeued and the node powers down.

In other words, InstanceId is only cleared by Slurm, while NodeAddr can be cleared by both clustermgtd and Slurm. If clustermgtd clears NodeAddr early, we are ignoring nodes in COMPLETING state. By relying on InstanceId, we cover COMPLETING state too.

Scope

• Detection logic only; the scontrol commands clustermgtd sends are unchanged.
• We considered setting nodes DOWN before power_down_force, which releases jobs immediately. It's deferred because it changes behavior for every power-down caller.
• We also considered not letting clustermgtd reset NodeAddr, and only rely on Slurm resetting NodeAddr upon POWERED_DOWN, but that increases the risk of IP reuse, and may have uninvestigated changes to cluster scaling

Tests

• The following tests have passed:

    test_essential_features.py::test_essential_features:
    test_slurm.py::test_slurm:
    test_slurm.py::test_slurm_from_login_nodes_in_private_network:
    test_slurm.py::test_slurm_scaling:
    test_slurm.py::test_error_handling:
    test_slurm.py::test_clustermgtd_instance_id_matching:
    test_slurm.py::test_slurm_maintenance_reservation:
    test_slurm.py::test_slurm_protected_mode:
    test_slurm.py::test_slurm_protected_mode_on_cluster_create:
    test_slurm.py::test_fast_capacity_failover:
    test_slurm.py::test_expedited_requeue:
    test_slurm.py::test_slurm_config_update:
    test_slurm.py::test_slurm_memory_based_scheduling:
    test_slurm.py::test_scontrol_reboot:
    test_slurm.py::test_scontrol_reboot_ec2_health_checks:
    test_slurm.py::test_slurm_overrides:
    test_slurm_accounting.py::test_slurm_accounting:
    test_slurm_accounting.py::test_slurm_accounting_external_dbd:
    test_slurm.py::test_slurm_custom_config_parameters:
    test_slurm.py::test_slurm_ticket_17399:
    test_slurm.py::test_slurm_custom_partitions:
    test_custom_munge_key.py::test_custom_munge_key:
    test_slurm_rest_api.py::test_slurm_rest_api:

References

• Link to impacted open issues.
• Link to related PRs in other packages (i.e. cookbook, node).
• Link to documentation useful to understand the changes.

Checklist

• Make sure you are pointing to the right branch.
• If you're creating a patch for a branch other than develop add the branch name as prefix in the PR title (e.g. [release-3.6]).
• Check all commits' messages are clear, describing what and why vs how.
• Make sure to have added unit tests or integration tests to cover the new/modified code.
• Check if documentation is impacted by this change.

Please review the guidelines for contributing and Pull Request Instructions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.