docs: Document API rate limits and how to build a client that handles them

[promptless]

Open in Promptless

Greg Wester asked for more guidance on API limits, specifically how to program an HTTP client defensively against Runpod's APIs. This revision addresses Lavanya Gunreddi's style and accuracy review.

What changed

• New page: API v2 › Rate limits (api-reference-v2/rate-limits.mdx). Documents the per-user minute/hour/day quota windows, the RateLimit and RateLimit-Policy headers returned on authenticated responses, and the 429 response with Retry-After and a problem+json body. Adds client guidance, a Python example client, an explanation of how the example works, and a "Next steps" section. Quota values are labeled as illustrative, and readers are told to read their real quotas from RateLimit-Policy.
• API v2 › Overview: new "Rate limits" section that links to the new page.
• Serverless › Send API requests: new "Handle rate limits in your client" subsection covering backoff with jitter, bounding concurrency (especially for /runsync), polling /status with a growing interval or using webhooks//stream, and avoiding duplicate jobs when retrying /run//runsync, plus a Python submit-and-poll example. Also fixes the opening curl example (-x → -X), sentence-cases the rate-limit table headers, and adds final periods to the "Dynamic rate limiting" list.
• vLLM › Send requests: the retry example adds jitter and links to the new guidance.
• docs.json: adds the new page to the API v2 group after the Overview.

Review feedback

• Item 1 (quotas and claims): quota values are now explicitly illustrative, not account limits. Removed "A rejected request has no effect" because nothing in the spec supports it. Reworded the exempt-callers sentence to say what a reader sees: if the headers are missing, the requests aren't subject to these limits (the spec says the headers are omitted for exempt callers). Real quota values and who is exempt still need engineering confirmation.
• Item 2: added "Next steps" with links to Send API requests, the API v2 overview, and the migration guide.
• Item 3: added an explanation after the Python client covering the 429 path, the GET-only retry, and seconds_until_reset.
• Item 4: added a "Rate limits" section to the API v2 overview.
• Items 5 and 6: nothing confirms that Serverless 429s send Retry-After, so the Serverless bullet and the Serverless and vLLM examples no longer read it and use exponential backoff with jitter instead. Only the API v2 example reads Retry-After, as int(...) to match the spec's integer type, so the inconsistency is gone.
• Optional send-requests.mdx fixes: all three applied.
• Follow-up comma fixes in the duplicate-jobs paragraph of send-requests.mdx, both applied verbatim: removed the comma before "because", and changed "A duplicate job runs, and is billed, separately" to "A duplicate job runs and is billed separately".

Verification

• Checked claims against api-reference-v2/openapi.json.
• All Python snippets parse. Vale reports no new errors on the changed lines. It flags "Next steps" entries capitalized after a colon, which matches the corpus convention.

Trigger Events

• Message in Slack channel #topic-documentation: @Promptless can we have more information in our docs on API limits, like how to defensively program an HTTP cl...
• Message in Slack channel #topic-documentation: I reviewed the Promptless branch (promptless/api-rate-limit-client-guidance) against the Runpod docs style g...

---

Tip: Use Vale? Point your collection at your vale.ini in Configuration.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
runpod-docs	🟢 Ready	View Preview	Oct 16, 2026, 6:15 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

[lavanya-gunreddi]

@Promptless Apply these two fixes only:

Fix 1 — Remove unnecessary comma before “because” in serverless/endpoints/send-requests.mdx

Current:

Retrying /run or /runsync after a network timeout or 5xx error can create a duplicate job, because the first request may have been queued before the connection failed.

Fixed:

Retrying /run or /runsync after a network timeout or 5xx error can create a duplicate job because the first request may have been queued before the connection failed.

Fix 2 — Remove awkward commas in the same paragraph in serverless/endpoints/send-requests.mdx

Current:

A duplicate job runs, and is billed, separately.

Fixed:

A duplicate job runs and is billed separately.

[promptless]

In reply to this discussion

Got it: I'll apply only the two comma fixes in serverless/endpoints/send-requests.mdx. I'll reply here once the suggestion is updated.

[promptless]

In reply to this discussion

I applied both comma fixes exactly as written in serverless/endpoints/send-requests.mdx. The paragraph now reads: "…can create a duplicate job because the first request may have been queued before the connection failed. A duplicate job runs and is billed separately." Nothing else changed.

Docs PR: #911