docs: Document API rate limits and how to build a client that handles them

api-reference-v2/overview.mdx

@@ -30,6 +30,10 @@ The Runpod REST API v2 provides programmatic access to all Runpod compute resour
 
 All requests require a [Runpod API key](/get-started/credentials) in the request headers. The API uses standard HTTP methods and returns JSON responses.
 
+## Rate limits
+
+The API limits how many requests each user can send per minute, hour, and day. Responses include `RateLimit` headers that report your remaining quota, and requests over the limit return `429 Too Many Requests`. To learn how to read these headers and build a client that handles them, see [Rate limits](/api-reference-v2/rate-limits).
+
 ## OpenAPI schema
 
 Retrieve the complete OpenAPI specification for client generation, request validation, or tooling integration.

api-reference-v2/rate-limits.mdx

@@ -0,0 +1,136 @@
+---
+title: "Rate limits"
+sidebarTitle: "Rate limits"
+description: "Learn how the Runpod REST API v2 limits request rates, read your remaining quota from response headers, and build a client that handles 429 responses."
+---
+
+The Runpod REST API v2 limits how many requests each user can send so that one integration can't degrade the API for everyone else. Limits apply per user across several time windows at once, and every authenticated response tells your client how much quota it has left. A client that reads this information can slow down before it reaches a limit and recover cleanly when it does.
+
+This page covers the management API at `https://api.runpod.io/v2`. Requests to Serverless endpoints (`https://api.runpod.ai/v2/ENDPOINT_ID/...`) have separate per-endpoint limits, described in [Send API requests](/serverless/endpoints/send-requests#rate-limits).
+
+## How limits work
+
+Each user has a quota for three windows: `minute`, `hour`, and `day`. A request counts against all three. When any window runs out, the API rejects further requests with `429 Too Many Requests` until that window resets, even if the other windows still have quota left.
+
+## Rate limit headers
+
+Responses to authenticated requests include two headers that follow the [IETF RateLimit header fields draft](https://datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers/). They're present on successful responses as well as on 429 responses, so your client can track quota before it's rejected. If your responses don't include these headers, your requests aren't subject to these limits.
+
+`RateLimit-Policy` describes the quota for each window. Each entry carries the quota `q` (requests allowed) and the window length `w` in seconds:
+
+```text
+RateLimit-Policy: "minute";q=60;w=60, "hour";q=3000;w=3600, "day";q=50000;w=86400
+```
+
+`RateLimit` describes the live state of each window. Each entry carries the remaining request count `r` and the seconds `t` until the window resets:
+
+```text
+RateLimit: "minute";r=0;t=12, "hour";r=2800;t=1812, "day";r=49500;t=45012
+```
+
+The quota values on this page are illustrative, not the limits that apply to your account. Read your actual quotas from the `RateLimit-Policy` header on your own responses, and don't hardcode them in your client.
+
+## 429 responses
+
+When your client exceeds a window, the API returns `429 Too Many Requests` with a `Retry-After` header that gives the number of seconds to wait, plus the `RateLimit` and `RateLimit-Policy` headers. The body uses the same `application/problem+json` format as other v2 errors, and `detail` names the window you exceeded:
+
+```http
+HTTP/2 429
+Content-Type: application/problem+json
+Retry-After: 12
+RateLimit: "minute";r=0;t=12, "hour";r=2800;t=1812, "day";r=49500;t=45012
+
+{
+  "title": "Too Many Requests",
+  "status": 429,
+  "detail": "rate limit exceeded for the minute window"
+}
+```
+
+## Build a client that respects limits
+
+Design your integration to treat a 429 as an expected signal rather than a failure:
+
+- Wait for the number of seconds in `Retry-After` before retrying. Retrying sooner returns another 429.
+- Watch the `RateLimit` header and slow down when `r` for any window gets close to zero, instead of waiting for a 429.
+- Cap the number of retries, and surface an error once the cap is reached.
+- Add random jitter to retry delays so that multiple workers or processes that were rejected together don't retry at the same moment.
+- Reduce request volume where you can. For example, list resources once and reuse the result instead of fetching each resource in a loop. Cache data that changes rarely, such as the [catalog](/api-reference-v2/catalog/list-gpu-types).
+- Share one rate-limit-aware client across your code. Quotas are per user, so separate processes using the same API key draw from the same quota.
+
+For errors other than 429, retry only when a retry is safe. Server errors (`5xx`) and network timeouts are usually transient, so retrying `GET` requests with exponential backoff is safe. A `POST` that creates a resource, such as a Pod, may have succeeded even though your client didn't receive the response. Before retrying it, list your resources to check whether the first attempt went through, or you may create a duplicate. Don't retry other `4xx` errors without changing the request, because they won't succeed on their own.
+
+### Example
+
+The following Python client honors `Retry-After` on 429 responses and pauses when the minute window is almost empty. It also retries transient errors on `GET` requests with exponential backoff and jitter. It requires the `requests` package and a `RUNPOD_API_KEY` environment variable.
+
+```python
+import os
+import random
+import re
+import time
+
+import requests
+
+BASE_URL = "https://api.runpod.io/v2"
+MAX_RETRIES = 5
+RETRYABLE_STATUSES = {500, 502, 503, 504}
+
+session = requests.Session()
+session.headers["Authorization"] = f"Bearer {os.environ['RUNPOD_API_KEY']}"
+
+
+def seconds_until_reset(ratelimit_header, window="minute", threshold=1):
+    """Return how long to pause if the given window is almost exhausted."""
+    match = re.search(rf'"{window}";r=(\d+);t=(\d+)', ratelimit_header or "")
+    if match and int(match.group(1)) < threshold:
+        return int(match.group(2))
+    return 0
+
+
+def request(method, path, **kwargs):
+    for attempt in range(MAX_RETRIES):
+        try:
+            response = session.request(method, f"{BASE_URL}{path}", timeout=30, **kwargs)
+        except (requests.ConnectionError, requests.Timeout):
+            # The request may have reached the server. Only retry methods that are safe to repeat.
+            if method != "GET" or attempt == MAX_RETRIES - 1:
+                raise
+            time.sleep(2**attempt + random.uniform(0, 1))
+            continue
+
+        if response.status_code == 429:
+            # Wait as long as the API asks, plus jitter so parallel clients spread out.
+            time.sleep(int(response.headers.get("Retry-After", 2**attempt)) + random.uniform(0, 1))
+            continue
+
+        if response.status_code in RETRYABLE_STATUSES and method == "GET":
+            time.sleep(2**attempt + random.uniform(0, 1))
+            continue
+
+        response.raise_for_status()
+
+        # Slow down before the next request if this window is nearly used up.
+        time.sleep(seconds_until_reset(response.headers.get("RateLimit")))
+        return response.json() if response.content else None
+
+    raise RuntimeError(f"{method} {path} failed after {MAX_RETRIES} attempts")
+
+
+pods = request("GET", "/pods")
+print(pods)
+```
+
+The `request` function sends every call through one shared session and handles each outcome differently:
+
+- A 429 response waits for the number of seconds in `Retry-After`, plus up to one second of jitter, and then retries. If the header is missing, the client falls back to exponential backoff.
+- A network error or a `5xx` response is retried with exponential backoff only for `GET` requests. Other methods raise the error immediately, because a `POST` that timed out may already have created a resource.
+- A successful response is passed to `seconds_until_reset`, which reads the `minute` entry of the `RateLimit` header. When no requests remain in that window, the client sleeps until the window resets, so the next call doesn't receive a 429.
+
+After `MAX_RETRIES` attempts, the client raises an error instead of retrying indefinitely.
+
+## Next steps
+
+- [Send API requests](/serverless/endpoints/send-requests#rate-limits): Learn about the separate rate limits for Serverless endpoint requests.
+- [API v2 overview](/api-reference-v2/overview): Review the base URL, authentication, and available resources.
+- [Migrate from v1](/api-reference-v2/migrate-from-v1): Move an existing REST v1 or GraphQL integration to API v2.

docs.json

@@ -594,6 +594,7 @@
             "group": "API v2",
             "pages": [
               "api-reference-v2/overview",
+              "api-reference-v2/rate-limits",
               "api-reference-v2/migrate-from-v1"
             ]
           },

serverless/endpoints/send-requests.mdx

@@ -9,7 +9,7 @@ import { QueueBasedEndpointsTooltip, LoadBalancingEndpointTooltip } from "/snipp
 After creating a [Serverless endpoint](/serverless/endpoints/overview), you can start sending HTTP requests to submit jobs and retrieve results:
 
 ```sh
-curl -x POST https://api.runpod.ai/v2/ENDPOINT_ID/runsync \
+curl -X POST https://api.runpod.ai/v2/ENDPOINT_ID/runsync \
      -H "authorization: Bearer RUNPOD_API_KEY" \
      -H "content-type: application/json" \
      -d '{ "input": {  "prompt": "Hello, world!" }}'
@@ -184,7 +184,7 @@ Your worker must contain logic to use this information for storage operations. W
 
 Runpod enforces rate limits per endpoint and operation:
 
-| Operation      | Method | Rate Limit                   | Concurrent Limit |
+| Operation      | Method | Rate limit                   | Concurrent limit |
 |----------------|--------|------------------------------|------------------|
 | `/runsync`     | POST   | 2000 requests per 10 seconds | 400 concurrent   |
 | `/run`         | POST   | 1000 requests per 10 seconds | 200 concurrent   |
@@ -199,10 +199,62 @@ Runpod enforces rate limits per endpoint and operation:
 
 Rate limits scale with your endpoint's worker count. The system uses whichever is higher between:
 
-1. **Base limit**: Fixed rate limit per user per endpoint (shown above)
-2. **Worker-based limit**: `number_of_running_workers × requests_per_worker`
+1. **Base limit**: Fixed rate limit per user per endpoint (shown above).
+2. **Worker-based limit**: `number_of_running_workers × requests_per_worker`.
 
-Requests exceeding the effective limit return `429 (Too Many Requests)`. Implement retry logic with exponential backoff to handle rate limiting gracefully.
+Requests exceeding the effective limit return `429 (Too Many Requests)`.
+
+### Handle rate limits in your client
+
+Rate limits apply to each endpoint for each user. All of your processes that call the same endpoint with the same API key share one limit. Build your client to expect 429 responses and to keep its request rate below the limits in the table:
+
+- Retry 429 responses with exponential backoff and random jitter, and cap the number of retries.
+- Limit how many requests your client has open at once. Each `/runsync` request stays open until the job finishes or the wait time expires. A burst of slow jobs can reach the concurrent limit well before the request rate limit. Use a bounded worker pool or semaphore rather than sending every request at once.
+- Poll `/status` at a measured interval that grows the longer a job runs, rather than in a tight loop. For jobs that take more than a few seconds, use [webhook notifications](#webhook-notifications) instead of polling, or `/stream` for incremental results.
+- Avoid calling `/requests` and `/purge-queue` from automated loops. Their limits are much lower than the job submission limits.
+
+Retrying `/run` or `/runsync` after a network timeout or `5xx` error can create a duplicate job because the first request may have been queued before the connection failed. A duplicate job runs and is billed separately. When `/run` returns a job ID, track the job with `/status` instead of resubmitting it, and use [`/retry`](/serverless/endpoints/operation-reference) to requeue a job that failed or timed out.
+
+The following example submits a job with `/run`, retries 429 responses with backoff, and polls `/status` with a growing interval. It requires the `requests` package and the `RUNPOD_API_KEY` and `ENDPOINT_ID` environment variables.
+
+```python
+import os
+import random
+import time
+
+import requests
+
+BASE_URL = f"https://api.runpod.ai/v2/{os.environ['ENDPOINT_ID']}"
+HEADERS = {"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"}
+MAX_RETRIES = 5
+
+
+def call(method, path, **kwargs):
+    """Send a request, retrying only when the endpoint rate-limits it."""
+    for attempt in range(MAX_RETRIES):
+        response = requests.request(method, f"{BASE_URL}{path}", headers=HEADERS, timeout=30, **kwargs)
+        if response.status_code != 429:
+            response.raise_for_status()
+            return response.json()
+        # Back off exponentially. Jitter spreads out parallel clients.
+        time.sleep(2**attempt + random.uniform(0, 1))
+    raise RuntimeError(f"{path} was rate-limited {MAX_RETRIES} times")
+
+
+job = call("POST", "/run", json={"input": {"prompt": "Hello, world!"}})
+
+interval = 1
+while True:
+    status = call("GET", f"/status/{job['id']}")
+    if status["status"] in ("COMPLETED", "FAILED", "CANCELLED", "TIMED_OUT"):
+        break
+    time.sleep(interval)
+    interval = min(interval * 2, 30)  # Poll less often the longer the job runs.
+
+print(status)
+```
+
+To manage Runpod resources such as Pods and endpoint configurations, use the [REST API v2](/api-reference-v2/overview). It has its own per-user limits and reports your remaining quota in response headers. See [API v2 rate limits](/api-reference-v2/rate-limits).
 
 ## Error handling
 

serverless/vllm/vllm-requests.mdx

@@ -187,9 +187,10 @@ Add parameters to control how the model generates text. Include these in the `sa
 
 ## Error handling
 
-Implement retry logic with exponential backoff to handle network issues, rate limits, and cold starts.
+Implement retry logic with exponential backoff and jitter to handle network issues, rate limits, and cold starts. Retrying a request after a timeout or server error can run the same job twice, so cap your retries. For more guidance, see [Handle rate limits in your client](/serverless/endpoints/send-requests#handle-rate-limits-in-your-client).
 
 ```python
+import random
 import requests
 import time
 
@@ -200,13 +201,11 @@ def send_request(url, headers, payload, max_retries=3):
             response.raise_for_status()
             return response.json()
         except requests.exceptions.HTTPError as e:
-            if e.response.status_code == 429:  # Rate limit
-                time.sleep(5)
-            elif e.response.status_code >= 500:
-                time.sleep(2 ** attempt)
+            if e.response.status_code == 429 or e.response.status_code >= 500:  # Rate limit or server error
+                time.sleep(2 ** attempt + random.uniform(0, 1))
             else:
                 raise
         except requests.exceptions.RequestException:
-            time.sleep(2 ** attempt)
+            time.sleep(2 ** attempt + random.uniform(0, 1))
     raise Exception("Max retries exceeded")
 ```