From 925ded400afc0b5906a38a6edb4835afc34a60ad Mon Sep 17 00:00:00 2001 From: "promptless[bot]" Date: Tue, 13 Oct 2026 19:23:13 +0000 Subject: [PATCH 1/3] Document API rate limit headers and defensive client patterns --- api-reference-v2/rate-limits.mdx | 124 +++++++++++++++++++++++++ docs.json | 1 + serverless/endpoints/send-requests.mdx | 55 ++++++++++- serverless/vllm/vllm-requests.mdx | 12 ++- 4 files changed, 186 insertions(+), 6 deletions(-) create mode 100644 api-reference-v2/rate-limits.mdx diff --git a/api-reference-v2/rate-limits.mdx b/api-reference-v2/rate-limits.mdx new file mode 100644 index 000000000..61fc11e46 --- /dev/null +++ b/api-reference-v2/rate-limits.mdx @@ -0,0 +1,124 @@ +--- +title: "Rate limits" +sidebarTitle: "Rate limits" +description: "Learn how the Runpod REST API v2 limits request rates, read your remaining quota from response headers, and build a client that handles 429 responses." +--- + +The Runpod REST API v2 limits how many requests each user can send so that one integration can't degrade the API for everyone else. Limits apply per user across several time windows at once, and every authenticated response tells your client how much quota it has left. A client that reads this information can slow down before it reaches a limit and recover cleanly when it does. + +This page covers the management API at `https://api.runpod.io/v2`. Requests to Serverless endpoints (`https://api.runpod.ai/v2/ENDPOINT_ID/...`) have separate per-endpoint limits, described in [Send API requests](/serverless/endpoints/send-requests#rate-limits). + +## How limits work + +Each user has a quota for three windows: `minute`, `hour`, and `day`. A request counts against all three. When any window runs out, the API rejects further requests with `429 Too Many Requests` until that window resets, even if the other windows still have quota left. + +## Rate limit headers + +Responses to authenticated requests include two headers that follow the [IETF RateLimit header fields draft](https://datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers/). They're present on successful responses as well as on 429 responses, so your client can track quota before it's rejected. Callers that are exempt from rate limiting don't receive them. + +`RateLimit-Policy` describes the quota for each window. Each entry carries the quota `q` (requests allowed) and the window length `w` in seconds: + +```text +RateLimit-Policy: "minute";q=60;w=60, "hour";q=3000;w=3600, "day";q=50000;w=86400 +``` + +`RateLimit` describes the live state of each window. Each entry carries the remaining request count `r` and the seconds `t` until the window resets: + +```text +RateLimit: "minute";r=0;t=12, "hour";r=2800;t=1812, "day";r=49500;t=45012 +``` + +These values are examples. Read the headers from your own responses rather than hardcoding the quotas shown here. + +## 429 responses + +When your client exceeds a window, the API returns `429 Too Many Requests` with a `Retry-After` header that gives the number of seconds to wait, plus the `RateLimit` and `RateLimit-Policy` headers. The body uses the same `application/problem+json` format as other v2 errors, and `detail` names the window you exceeded: + +```http +HTTP/2 429 +Content-Type: application/problem+json +Retry-After: 12 +RateLimit: "minute";r=0;t=12, "hour";r=2800;t=1812, "day";r=49500;t=45012 + +{ + "title": "Too Many Requests", + "status": 429, + "detail": "rate limit exceeded for the minute window" +} +``` + +A rejected request has no effect, so you can send the same request again after waiting. + +## Build a client that respects limits + +Design your integration to treat a 429 as an expected signal rather than a failure: + +- Wait for the number of seconds in `Retry-After` before retrying. Retrying sooner returns another 429. +- Watch the `RateLimit` header and slow down when `r` for any window gets close to zero, instead of waiting for a 429. +- Cap the number of retries, and surface an error once the cap is reached. +- Add random jitter to retry delays so that multiple workers or processes that were rejected together don't retry at the same moment. +- Reduce request volume where you can. For example, list resources once and reuse the result instead of fetching each resource in a loop. Cache data that changes rarely, such as the [catalog](/api-reference-v2/catalog/list-gpu-types). +- Share one rate-limit-aware client across your code. Quotas are per user, so separate processes using the same API key draw from the same quota. + +For errors other than 429, retry only when a retry is safe. Server errors (`5xx`) and network timeouts are usually transient, so retrying `GET` requests with exponential backoff is safe. A `POST` that creates a resource, such as a Pod, may have succeeded even though your client didn't receive the response. Before retrying it, list your resources to check whether the first attempt went through, or you may create a duplicate. Don't retry other `4xx` errors without changing the request, because they won't succeed on their own. + +### Example + +The following Python client honors `Retry-After` on 429 responses and pauses when the minute window is almost empty. It also retries transient errors on `GET` requests with exponential backoff and jitter. It requires the `requests` package and a `RUNPOD_API_KEY` environment variable. + +```python +import os +import random +import re +import time + +import requests + +BASE_URL = "https://api.runpod.io/v2" +MAX_RETRIES = 5 +RETRYABLE_STATUSES = {500, 502, 503, 504} + +session = requests.Session() +session.headers["Authorization"] = f"Bearer {os.environ['RUNPOD_API_KEY']}" + + +def seconds_until_reset(ratelimit_header, window="minute", threshold=1): + """Return how long to pause if the given window is almost exhausted.""" + match = re.search(rf'"{window}";r=(\d+);t=(\d+)', ratelimit_header or "") + if match and int(match.group(1)) < threshold: + return int(match.group(2)) + return 0 + + +def request(method, path, **kwargs): + for attempt in range(MAX_RETRIES): + try: + response = session.request(method, f"{BASE_URL}{path}", timeout=30, **kwargs) + except (requests.ConnectionError, requests.Timeout): + # The request may have reached the server. Only retry methods that are safe to repeat. + if method != "GET" or attempt == MAX_RETRIES - 1: + raise + time.sleep(2**attempt + random.uniform(0, 1)) + continue + + if response.status_code == 429: + # Wait as long as the API asks, plus jitter so parallel clients spread out. + time.sleep(int(response.headers.get("Retry-After", 2**attempt)) + random.uniform(0, 1)) + continue + + if response.status_code in RETRYABLE_STATUSES and method == "GET": + time.sleep(2**attempt + random.uniform(0, 1)) + continue + + response.raise_for_status() + + # Slow down before the next request if this window is nearly used up. + time.sleep(seconds_until_reset(response.headers.get("RateLimit"))) + return response.json() if response.content else None + + raise RuntimeError(f"{method} {path} failed after {MAX_RETRIES} attempts") + + +pods = request("GET", "/pods") +print(pods) +``` diff --git a/docs.json b/docs.json index f9eccb0e6..b4b2b64c3 100644 --- a/docs.json +++ b/docs.json @@ -564,6 +564,7 @@ "group": "API v2", "pages": [ "api-reference-v2/overview", + "api-reference-v2/rate-limits", "api-reference-v2/migrate-from-v1" ] }, diff --git a/serverless/endpoints/send-requests.mdx b/serverless/endpoints/send-requests.mdx index 7ff04208b..1791308da 100644 --- a/serverless/endpoints/send-requests.mdx +++ b/serverless/endpoints/send-requests.mdx @@ -202,7 +202,60 @@ Rate limits scale with your endpoint's worker count. The system uses whichever i 1. **Base limit**: Fixed rate limit per user per endpoint (shown above) 2. **Worker-based limit**: `number_of_running_workers × requests_per_worker` -Requests exceeding the effective limit return `429 (Too Many Requests)`. Implement retry logic with exponential backoff to handle rate limiting gracefully. +Requests exceeding the effective limit return `429 (Too Many Requests)`. + +### Handle rate limits in your client + +Rate limits apply to each endpoint for each user. All of your processes that call the same endpoint with the same API key share one limit. Build your client to expect 429 responses and to keep its request rate below the limits in the table: + +- Retry 429 responses with exponential backoff and random jitter, and cap the number of retries. If the response includes a `Retry-After` header, wait at least that many seconds before retrying. +- Limit how many requests your client has open at once. Each `/runsync` request stays open until the job finishes or the wait time expires. A burst of slow jobs can reach the concurrent limit well before the request rate limit. Use a bounded worker pool or semaphore rather than sending every request at once. +- Poll `/status` at a measured interval that grows the longer a job runs, rather than in a tight loop. For jobs that take more than a few seconds, use [webhook notifications](#webhook-notifications) instead of polling, or `/stream` for incremental results. +- Avoid calling `/requests` and `/purge-queue` from automated loops. Their limits are much lower than the job submission limits. + +Retrying `/run` or `/runsync` after a network timeout or `5xx` error can create a duplicate job, because the first request may have been queued before the connection failed. A duplicate job runs, and is billed, separately. When `/run` returns a job ID, track the job with `/status` instead of resubmitting it, and use [`/retry`](/serverless/endpoints/operation-reference) to requeue a job that failed or timed out. + +The following example submits a job with `/run`, retries 429 responses with backoff, and polls `/status` with a growing interval. It requires the `requests` package and the `RUNPOD_API_KEY` and `ENDPOINT_ID` environment variables. + +```python +import os +import random +import time + +import requests + +BASE_URL = f"https://api.runpod.ai/v2/{os.environ['ENDPOINT_ID']}" +HEADERS = {"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"} +MAX_RETRIES = 5 + + +def call(method, path, **kwargs): + """Send a request, retrying only when the endpoint rate-limits it.""" + for attempt in range(MAX_RETRIES): + response = requests.request(method, f"{BASE_URL}{path}", headers=HEADERS, timeout=30, **kwargs) + if response.status_code != 429: + response.raise_for_status() + return response.json() + # Honor Retry-After when present; otherwise back off exponentially. Jitter spreads out parallel clients. + delay = float(response.headers.get("Retry-After", 2**attempt)) + time.sleep(delay + random.uniform(0, 1)) + raise RuntimeError(f"{path} was rate-limited {MAX_RETRIES} times") + + +job = call("POST", "/run", json={"input": {"prompt": "Hello, world!"}}) + +interval = 1 +while True: + status = call("GET", f"/status/{job['id']}") + if status["status"] in ("COMPLETED", "FAILED", "CANCELLED", "TIMED_OUT"): + break + time.sleep(interval) + interval = min(interval * 2, 30) # Poll less often the longer the job runs. + +print(status) +``` + +To manage Runpod resources such as Pods and endpoint configurations, use the [REST API v2](/api-reference-v2/overview). It has its own per-user limits and reports your remaining quota in response headers. See [API v2 rate limits](/api-reference-v2/rate-limits). ## Error handling diff --git a/serverless/vllm/vllm-requests.mdx b/serverless/vllm/vllm-requests.mdx index c62131ed5..fa9edf1d6 100644 --- a/serverless/vllm/vllm-requests.mdx +++ b/serverless/vllm/vllm-requests.mdx @@ -187,9 +187,10 @@ Add parameters to control how the model generates text. Include these in the `sa ## Error handling -Implement retry logic with exponential backoff to handle network issues, rate limits, and cold starts. +Implement retry logic with exponential backoff and jitter to handle network issues, rate limits, and cold starts. Retrying a request after a timeout or server error can run the same job twice, so cap your retries. For more guidance, see [Handle rate limits in your client](/serverless/endpoints/send-requests#handle-rate-limits-in-your-client). ```python +import random import requests import time @@ -200,13 +201,14 @@ def send_request(url, headers, payload, max_retries=3): response.raise_for_status() return response.json() except requests.exceptions.HTTPError as e: - if e.response.status_code == 429: # Rate limit - time.sleep(5) + if e.response.status_code == 429: # Rate limit: honor Retry-After when present + delay = float(e.response.headers.get("Retry-After", 2 ** attempt)) + time.sleep(delay + random.uniform(0, 1)) elif e.response.status_code >= 500: - time.sleep(2 ** attempt) + time.sleep(2 ** attempt + random.uniform(0, 1)) else: raise except requests.exceptions.RequestException: - time.sleep(2 ** attempt) + time.sleep(2 ** attempt + random.uniform(0, 1)) raise Exception("Max retries exceeded") ``` From 931d063e59ee6ad0bde10a4f4f465adf9c4350f5 Mon Sep 17 00:00:00 2001 From: "promptless[bot]" Date: Tue, 13 Oct 2026 21:29:39 +0000 Subject: [PATCH 2/3] Address review: next steps, code explanation, overview link, conservative rate-limit claims, send-requests fixes --- api-reference-v2/overview.mdx | 4 ++++ api-reference-v2/rate-limits.mdx | 20 ++++++++++++++++---- serverless/endpoints/send-requests.mdx | 15 +++++++-------- serverless/vllm/vllm-requests.mdx | 5 +---- 4 files changed, 28 insertions(+), 16 deletions(-) diff --git a/api-reference-v2/overview.mdx b/api-reference-v2/overview.mdx index 5f69a49c6..593fa8cfa 100644 --- a/api-reference-v2/overview.mdx +++ b/api-reference-v2/overview.mdx @@ -30,6 +30,10 @@ The Runpod REST API v2 provides programmatic access to all Runpod compute resour All requests require a [Runpod API key](/get-started/credentials) in the request headers. The API uses standard HTTP methods and returns JSON responses. +## Rate limits + +The API limits how many requests each user can send per minute, hour, and day. Responses include `RateLimit` headers that report your remaining quota, and requests over the limit return `429 Too Many Requests`. To learn how to read these headers and build a client that handles them, see [Rate limits](/api-reference-v2/rate-limits). + ## OpenAPI schema Retrieve the complete OpenAPI specification for client generation, request validation, or tooling integration. diff --git a/api-reference-v2/rate-limits.mdx b/api-reference-v2/rate-limits.mdx index 61fc11e46..d73a6d5bf 100644 --- a/api-reference-v2/rate-limits.mdx +++ b/api-reference-v2/rate-limits.mdx @@ -14,7 +14,7 @@ Each user has a quota for three windows: `minute`, `hour`, and `day`. A request ## Rate limit headers -Responses to authenticated requests include two headers that follow the [IETF RateLimit header fields draft](https://datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers/). They're present on successful responses as well as on 429 responses, so your client can track quota before it's rejected. Callers that are exempt from rate limiting don't receive them. +Responses to authenticated requests include two headers that follow the [IETF RateLimit header fields draft](https://datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers/). They're present on successful responses as well as on 429 responses, so your client can track quota before it's rejected. If your responses don't include these headers, your requests aren't subject to these limits. `RateLimit-Policy` describes the quota for each window. Each entry carries the quota `q` (requests allowed) and the window length `w` in seconds: @@ -28,7 +28,7 @@ RateLimit-Policy: "minute";q=60;w=60, "hour";q=3000;w=3600, "day";q=50000;w=8640 RateLimit: "minute";r=0;t=12, "hour";r=2800;t=1812, "day";r=49500;t=45012 ``` -These values are examples. Read the headers from your own responses rather than hardcoding the quotas shown here. +The quota values on this page are illustrative, not the limits that apply to your account. Read your actual quotas from the `RateLimit-Policy` header on your own responses, and don't hardcode them in your client. ## 429 responses @@ -47,8 +47,6 @@ RateLimit: "minute";r=0;t=12, "hour";r=2800;t=1812, "day";r=49500;t=45012 } ``` -A rejected request has no effect, so you can send the same request again after waiting. - ## Build a client that respects limits Design your integration to treat a 429 as an expected signal rather than a failure: @@ -122,3 +120,17 @@ def request(method, path, **kwargs): pods = request("GET", "/pods") print(pods) ``` + +The `request` function sends every call through one shared session and handles each outcome differently: + +- A 429 response waits for the number of seconds in `Retry-After`, plus up to one second of jitter, and then retries. If the header is missing, the client falls back to exponential backoff. +- A network error or a `5xx` response is retried with exponential backoff only for `GET` requests. Other methods raise the error immediately, because a `POST` that timed out may already have created a resource. +- A successful response is passed to `seconds_until_reset`, which reads the `minute` entry of the `RateLimit` header. When no requests remain in that window, the client sleeps until the window resets, so the next call doesn't receive a 429. + +After `MAX_RETRIES` attempts, the client raises an error instead of retrying indefinitely. + +## Next steps + +- [Send API requests](/serverless/endpoints/send-requests#rate-limits): Learn about the separate rate limits for Serverless endpoint requests. +- [API v2 overview](/api-reference-v2/overview): Review the base URL, authentication, and available resources. +- [Migrate from v1](/api-reference-v2/migrate-from-v1): Move an existing REST v1 or GraphQL integration to API v2. diff --git a/serverless/endpoints/send-requests.mdx b/serverless/endpoints/send-requests.mdx index 1791308da..eac83380e 100644 --- a/serverless/endpoints/send-requests.mdx +++ b/serverless/endpoints/send-requests.mdx @@ -9,7 +9,7 @@ import { QueueBasedEndpointsTooltip, LoadBalancingEndpointTooltip } from "/snipp After creating a [Serverless endpoint](/serverless/endpoints/overview), you can start sending HTTP requests to submit jobs and retrieve results: ```sh -curl -x POST https://api.runpod.ai/v2/ENDPOINT_ID/runsync \ +curl -X POST https://api.runpod.ai/v2/ENDPOINT_ID/runsync \ -H "authorization: Bearer RUNPOD_API_KEY" \ -H "content-type: application/json" \ -d '{ "input": { "prompt": "Hello, world!" }}' @@ -184,7 +184,7 @@ Your worker must contain logic to use this information for storage operations. W Runpod enforces rate limits per endpoint and operation: -| Operation | Method | Rate Limit | Concurrent Limit | +| Operation | Method | Rate limit | Concurrent limit | |----------------|--------|------------------------------|------------------| | `/runsync` | POST | 2000 requests per 10 seconds | 400 concurrent | | `/run` | POST | 1000 requests per 10 seconds | 200 concurrent | @@ -199,8 +199,8 @@ Runpod enforces rate limits per endpoint and operation: Rate limits scale with your endpoint's worker count. The system uses whichever is higher between: -1. **Base limit**: Fixed rate limit per user per endpoint (shown above) -2. **Worker-based limit**: `number_of_running_workers × requests_per_worker` +1. **Base limit**: Fixed rate limit per user per endpoint (shown above). +2. **Worker-based limit**: `number_of_running_workers × requests_per_worker`. Requests exceeding the effective limit return `429 (Too Many Requests)`. @@ -208,7 +208,7 @@ Requests exceeding the effective limit return `429 (Too Many Requests)`. Rate limits apply to each endpoint for each user. All of your processes that call the same endpoint with the same API key share one limit. Build your client to expect 429 responses and to keep its request rate below the limits in the table: -- Retry 429 responses with exponential backoff and random jitter, and cap the number of retries. If the response includes a `Retry-After` header, wait at least that many seconds before retrying. +- Retry 429 responses with exponential backoff and random jitter, and cap the number of retries. - Limit how many requests your client has open at once. Each `/runsync` request stays open until the job finishes or the wait time expires. A burst of slow jobs can reach the concurrent limit well before the request rate limit. Use a bounded worker pool or semaphore rather than sending every request at once. - Poll `/status` at a measured interval that grows the longer a job runs, rather than in a tight loop. For jobs that take more than a few seconds, use [webhook notifications](#webhook-notifications) instead of polling, or `/stream` for incremental results. - Avoid calling `/requests` and `/purge-queue` from automated loops. Their limits are much lower than the job submission limits. @@ -236,9 +236,8 @@ def call(method, path, **kwargs): if response.status_code != 429: response.raise_for_status() return response.json() - # Honor Retry-After when present; otherwise back off exponentially. Jitter spreads out parallel clients. - delay = float(response.headers.get("Retry-After", 2**attempt)) - time.sleep(delay + random.uniform(0, 1)) + # Back off exponentially. Jitter spreads out parallel clients. + time.sleep(2**attempt + random.uniform(0, 1)) raise RuntimeError(f"{path} was rate-limited {MAX_RETRIES} times") diff --git a/serverless/vllm/vllm-requests.mdx b/serverless/vllm/vllm-requests.mdx index fa9edf1d6..9e29eff2c 100644 --- a/serverless/vllm/vllm-requests.mdx +++ b/serverless/vllm/vllm-requests.mdx @@ -201,10 +201,7 @@ def send_request(url, headers, payload, max_retries=3): response.raise_for_status() return response.json() except requests.exceptions.HTTPError as e: - if e.response.status_code == 429: # Rate limit: honor Retry-After when present - delay = float(e.response.headers.get("Retry-After", 2 ** attempt)) - time.sleep(delay + random.uniform(0, 1)) - elif e.response.status_code >= 500: + if e.response.status_code == 429 or e.response.status_code >= 500: # Rate limit or server error time.sleep(2 ** attempt + random.uniform(0, 1)) else: raise From e3e1e97b7021c7091bfbda3eec14a651f6116aea Mon Sep 17 00:00:00 2001 From: "promptless[bot]" Date: Fri, 16 Oct 2026 18:15:22 +0000 Subject: [PATCH 3/3] Remove unnecessary commas in duplicate-job guidance --- serverless/endpoints/send-requests.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/serverless/endpoints/send-requests.mdx b/serverless/endpoints/send-requests.mdx index eac83380e..552227106 100644 --- a/serverless/endpoints/send-requests.mdx +++ b/serverless/endpoints/send-requests.mdx @@ -213,7 +213,7 @@ Rate limits apply to each endpoint for each user. All of your processes that cal - Poll `/status` at a measured interval that grows the longer a job runs, rather than in a tight loop. For jobs that take more than a few seconds, use [webhook notifications](#webhook-notifications) instead of polling, or `/stream` for incremental results. - Avoid calling `/requests` and `/purge-queue` from automated loops. Their limits are much lower than the job submission limits. -Retrying `/run` or `/runsync` after a network timeout or `5xx` error can create a duplicate job, because the first request may have been queued before the connection failed. A duplicate job runs, and is billed, separately. When `/run` returns a job ID, track the job with `/status` instead of resubmitting it, and use [`/retry`](/serverless/endpoints/operation-reference) to requeue a job that failed or timed out. +Retrying `/run` or `/runsync` after a network timeout or `5xx` error can create a duplicate job because the first request may have been queued before the connection failed. A duplicate job runs and is billed separately. When `/run` returns a job ID, track the job with `/status` instead of resubmitting it, and use [`/retry`](/serverless/endpoints/operation-reference) to requeue a job that failed or timed out. The following example submits a job with `/run`, retries 429 responses with backoff, and polls `/status` with a growing interval. It requires the `requests` package and the `RUNPOD_API_KEY` and `ENDPOINT_ID` environment variables.