diff --git a/pods/configuration/use-ssh.mdx b/pods/configuration/use-ssh.mdx index 19d516544..b0298ae8c 100644 --- a/pods/configuration/use-ssh.mdx +++ b/pods/configuration/use-ssh.mdx @@ -127,6 +127,10 @@ Where: If you're asked for a password when connecting to your Pod via SSH, this means something is not set up correctly. Runpod does not require a password for SSH connections, as authentication is handled entirely through your SSH key pair. + +If your Pod is unreachable or stuck, run `runpodctl ssh info ` to see the specific reason. See [Get SSH connection info](/runpodctl/reference/runpodctl-ssh#get-ssh-connection-info). + + Here are some common reasons why this might happen: - If you copy and paste the key *fingerprint* (which starts with `SHA256:`) into your Runpod user settings instead of the actual public key (the contents of your `id_ed25519.pub` file), authentication will fail. diff --git a/runpodctl/reference/runpodctl-pod.mdx b/runpodctl/reference/runpodctl-pod.mdx index 5beea0d2e..e38a4569d 100644 --- a/runpodctl/reference/runpodctl-pod.mdx +++ b/runpodctl/reference/runpodctl-pod.mdx @@ -52,7 +52,7 @@ Show all Pods including exited ones. By default, only running Pods are shown. -Filter by Pod status (e.g., `RUNNING`, `EXITED`). Cannot be used with `--all`. +Filter by desired status (`desiredStatus`), such as `RUNNING` or `EXITED`. Matching is case-insensitive. Does not accept `runtimeStatus` values like `initializing`; passing one returns no results. Cannot be used with `--all`. @@ -267,6 +267,73 @@ Delete a Pod: runpodctl pod delete ``` +## Pod runtime status + +`runpodctl pod get` and `runpodctl pod list` report a Pod's real runtime state in their JSON and YAML output, alongside the existing `desiredStatus`. `desiredStatus` is the state you asked for (for example, `RUNNING`), while `runtimeStatus` is what the Pod is actually doing. For example, a Pod still pulling a large image and a Pod that has been serving traffic for an hour both show `desiredStatus: RUNNING`, but their `runtimeStatus` values differ. + + +Always present. What the Pod is actually doing (see the table below). + + + +Present only when there is more to say. A stable token you can safely branch on in scripts. + + +For example, `runpodctl pod list --output json` returns entries like this for a running Pod: + +```json +{ + "id": "abc123xyz", + "name": "my-training-pod", + "desiredStatus": "RUNNING", + "runtimeStatus": "running", + "imageName": "runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04", + "gpuCount": 1, + "costPerHr": 0.69, + "createdAt": "2025-01-15T09:30:00Z", + "uptimeSeconds": 3600 +} +``` + +When a Pod is initializing, `runtimeStatus` is `initializing`, `runtimeStatusReason` is `awaiting_container`, and `uptimeSeconds` is omitted because no container is reporting yet. + +### Runtime status values + +| Value | Meaning | What to do | +| --- | --- | --- | +| `running` | desiredStatus is `RUNNING` and the platform is reporting runtime telemetry, so the container is up. Does not imply that any port (such as SSH) is reachable. | Ready to use. | +| `initializing` | desiredStatus is `RUNNING` but no telemetry is being reported yet. Usually the container is not up yet (image pull, container creation, or boot, which the platform does not distinguish). Read this as "no container reported," not "the container is provably down." | Keep polling. | +| `stopped` | desiredStatus is `EXITED`. The container is gone, but the Pod's disk is kept. | Run `runpodctl pod start ` to bring it back. | +| `terminated` | desiredStatus is `TERMINATED`: the Pod is being destroyed. A terminated Pod drops out of `runpodctl pod list` shortly after, so this is a narrow window. | None; the Pod is gone. Because a terminated Pod quickly drops out of `runpodctl pod list`, a teardown script should treat "Pod not found" (not `runtimeStatus: terminated`) as the reliable signal that a Pod is gone. | +| `error` | desiredStatus is `ERROR`: the Pod is in a state it won't recover from on its own. It still exists and may still incur charges, so the default `runpodctl pod list` keeps showing it. No SSH command is offered. | Check `runpodctl pod logs `, then delete or recreate the Pod. | +| `unknown` | The runtime state cannot be derived (an uncommon `desiredStatus`, or the runtime lookup wasn't made). | Read `desiredStatus` from the same output. | + +### Runtime status reasons + +| Token | Paired status | Meaning | +| --- | --- | --- | +| `awaiting_container` | `initializing` | No container is being reported yet for a Pod that should be running. | +| `runtime_unavailable` | `unknown` | The runtime lookup could not be made, so `running` and `initializing` cannot be told apart. | + +The API doesn't record why a Pod stopped in a machine-readable form, so `stopped`, `terminated`, and `error` have no reason token. + + +To poll a Pod to readiness in a script, wait for `runtimeStatus: running`. Branch on the `runtimeStatus` and `runtimeStatusReason` tokens in scripts. If `runtimeStatus` becomes `error`, stop polling, because the Pod won't recover on its own. + +```bash +# Wait until the Pod's container is up +until [ "$(runpodctl pod get --output json | jq -r '.runtimeStatus')" = "running" ]; do + sleep 5 +done +``` + +A `runtimeStatus` of `unknown` with reason `runtime_unavailable` during polling usually means the runtime lookup momentarily failed, not that the Pod is down—keep polling rather than treating it as terminal. + + +`runtimeStatus` and `desiredStatus` come from the same API read, so they always agree on the Pod's state. + +The `uptimeSeconds` field reports the container's uptime and is omitted when no container is reporting. + ## Pod URLs Access exposed ports on your Pod using the following URL pattern: diff --git a/runpodctl/reference/runpodctl-ssh.mdx b/runpodctl/reference/runpodctl-ssh.mdx index 15f12810c..ba831da48 100644 --- a/runpodctl/reference/runpodctl-ssh.mdx +++ b/runpodctl/reference/runpodctl-ssh.mdx @@ -30,7 +30,7 @@ Include Pod ID and name in output. Shorthand: `-v`. -The `ssh info` command returns connection details that you can use to connect via SSH manually. It does not start an interactive SSH session. +The `ssh info` command returns connection details that you can use to connect via SSH manually. It does not start an interactive SSH session. When a Pod is not reachable, `runpodctl ssh info` reports the specific reason instead of a generic message. To connect to your Pod, use the SSH command provided in the output: ```bash @@ -39,6 +39,26 @@ ssh user@host -p -i +`runpodctl ssh info` explains which of the following cases applies. It also reports the Pod's `runtimeStatus`, so you can match a not-ready result to the runtime status vocabulary: + +| Reason | Meaning | What to do | +| --- | --- | --- | +| Pod is initializing | No container is reported yet (image pull, container creation, or boot). | Wait and try again. | +| Pod is stopped | The Pod's container is gone but its disk is kept. | Start it with `runpodctl pod start `. | +| Pod is terminated | The Pod has been destroyed. | None; deploy a new Pod. | +| Pod is in an error state | The Pod is in a state it won't recover from on its own. | Check `runpodctl pod logs `, then delete or recreate the Pod. | +| No SSH port published | The Pod is running but does not publish `22/tcp`. | Add it with the `runpodctl pod update` command the CLI provides (see the Warning below). | +| No public IP | Port 22 is mapped, but the machine has no publicly routable IP. This affects direct SSH over a public IP specifically; basic SSH (proxied through Runpod, no public IP required) may still work. | Use [Basic SSH](/pods/configuration/use-ssh#basic-ssh-with-key-authentication), which does not need a public IP. If you need direct SSH over a public IP, redeploy on a machine that provides one. | +| Mapping not ready yet | Port 22 is declared, but the host has not published the mapping yet. | Wait and try again. | + +For the "No SSH port published" case, `runpodctl ssh info` gives you a ready-to-run command of the form `runpodctl pod update --ports ,22/tcp` that preserves your existing ports. + + +`runpodctl pod update --ports` replaces the Pod's entire port list (unlike `--env`, which merges), so always include your current ports when you add `22/tcp`. Changing the port list also bumps the Pod's version, which can restart the container. Processes and container-local state outside the volume disk may not survive. If the Pod is running an active job, checkpoint your work to the volume disk before changing ports, since the container may restart. + + +For the full list of runtime status values, see [Pod runtime status](/runpodctl/reference/runpodctl-pod#pod-runtime-status). + ### List SSH keys List all SSH keys associated with your account: