perf(serverless): skip torch fitness checks unless loaded

[deanq]

Summary

The worker-start _cuda_init_check and _benchmark_check imported torch themselves. On a cold worker that import is ~1.5s, about 90% of the check, and it ran before the first job-take. That happens even for handlers that never use torch, or that import it lazily inside the handler function (every Flash handler does, because Flash ships only the function body).

• Both checks now use torch or CuPy only if the handler already imported it (sys.modules); they never import either
• With neither loaded, both checks skip before calling nvidia-smi. The gpu_test binary, which runs at import time, still verifies device init (cudaSetDevice) and allocation (cudaMalloc) on every GPU
• With torch or CuPy loaded, behavior is unchanged

Why gate instead of drop: for handlers that import torch at module level, the check reuses the loaded framework at almost no cost and still catches torch-level problems (missing kernels for the GPU arch, cuBLAS failures). Dropping only _cuda_init_check would not help, because _benchmark_check would then pay the same torch import.

Impact

Breakdown on a real RTX 4090 serverless worker, torch 2.9.1+cu128:

Step	Time
import torch	1550 ms (1352 / 1397 ms in a fresh process)
CUDA context creation	~150 ms
zeros(1024) on device	12 ms
gpu_test binary	293 / 343 ms

Worker-start fitness pass on the same worker type, Flash handler:

	Time
2.0.0rc1	1339 / 2147 ms (_cuda_init_check 1211 / 2021 ms)
This branch	31 ms

For a handler that uses torch, the import moves to its first job (1591 ms measured); for one that doesn't, it is gone. Either way the worker starts taking jobs 1.3-2.1s sooner.

Test plan

• New: CUDA init check skips without importing torch/CuPy or calling nvidia-smi when the handler loaded neither (fails on main)
• New: GPU benchmark skips the same way (fails on main)
• New: CuPy benchmark still runs when only CuPy is loaded
• Existing torch/CuPy success and failure tests unchanged and passing
• make quality-check: 1482 passed, 91.70% coverage
• Real GPU worker with this branch bundled: worker-start fitness pass 31 ms, both torch checks logged as skipped

Linear: SLS-815

[Copilot]

🟢 Approval recommended

The focused changes match the intended startup optimization, include regression coverage, and have no identified blocking issues.

No open issues 🎉

What changed in this PR

Reduces worker startup latency by running framework-based GPU fitness checks only when PyTorch or CuPy is already loaded.

Changes:

• Skips CUDA initialization and benchmark checks before GPU detection when neither framework is loaded.
• Adds regression tests for skip behavior and the loaded-CuPy benchmark path.

File	Description
tests/​test_serverless/​test_modules/​test_fitness/​test_system_checks.py	Tests import-free skipping and CuPy benchmark execution.
runpod/​_health/​system.py	Gates GPU checks on loaded frameworks and documents the behavior.

_{Review effort: Balanced}