Add Hot Aisle bare metal support

[peterschmidt85]

Bare metal servers

The hotaisle backend now provisions Hot Aisle bare metal servers (currently 8x MI300X) in addition to VMs. Instance types are renamed to vm-mi300x-1, vm-mi300x-2, vm-mi300x-4, vm-mi300x-8, and bm-mi300x-8.

$ dstack offer --backend hotaisle
 #  BACKEND                   RESOURCES                 INSTANCE TYPE  PRICE
 1  hotaisle (us-michigan-1)  cpu=x86:104 mem=2048GB    bm-mi300x-8    $27.12
                              disk=115335GB
                              gpu=MI300X:192GB:8

Bare metal servers are charged upfront for a minimum reservation period of 8 hours. To keep the server for the full period, set idle_duration in the fleet configuration:

type: fleet
name: hotaisle-bm

nodes: 0..1
backends: [hotaisle]
instance_types: [bm-mi300x-8]

idle_duration: 8h

Starting dstack-shim over SSH on Hot Aisle instances is now retried on every instance check until it succeeds, instead of a single unchecked attempt.

While this PR is open, DSTACK_FF_HOTAISLE_BARE_METAL_NO_FORCE_RELEASE is enabled by default (set it to 0 to disable), so dstack releases bare metal servers without force. If a fleet is deleted or dstack terminates the instance within the minimum reservation period, Hot Aisle refuses the release: the instance stays terminating for about 15 minutes while dstack retries, then dstack marks it terminated, but the server stays alive at Hot Aisle and must be released manually. The default will change to 0 before merging (see the TODO in settings.py).

Confirmed with Hot Aisle

• There is one bare metal class (8x MI300X). The CPU varies by server (Xeon Platinum 8470 or 8462Y+), but the price doesn't, so instance types don't include the CPU.
• Servers are pre-staged, so a reservation is ready immediately and the default provisioning timeout applies.
• Bare metal servers are set up like VMs: the hotaisle user with passwordless sudo, and Docker and AMD drivers preinstalled.
• On each SSH login, the server fetches the SSH keys of team members with the user role from the Hot Aisle API, so the key dstack uploads right before reserving works immediately.
• The API supports 8x MI300X VMs (currently out of stock).
• The OS is on the 2x 480GB M.2. The 8x 15.36TB NVMe drives are free to use and may be mounted by default (unconfirmed).

Before merging

• Test on a real bare metal server using a fleet with instance_types: [bm-mi300x-8] and idle_duration: 8h. Keep all testing within the paid 8-hour period, ideally combined with work that needs 8x MI300X. Check the disk layout (lsblk, df -h) and, if needed, report the boot disk as the offer's disk size in gpuhunt and document the NVMe mount path.
• Merge and release Add Hot Aisle bare metal offers gpuhunt#267, then pin the released gpuhunt instead of the branch.
• Change the default of DSTACK_FF_HOTAISLE_BARE_METAL_NO_FORCE_RELEASE to 0 (see the TODO in settings.py).

🤖 Generated with Claude Code