Rate Limiting Your Own App Against LLM Provider Quotas: A Practical Pattern
By Abhishek Patoliya · 2026-08-17
A feature launches, gets a burst of usage, and within minutes your app is throwing errors — not because your code broke, but because your LLM provider started returning 429s and your retry logic, which just backs off and tries again, is now retrying into a limit that hasn't cleared yet, compounding the backlog. The fix isn't smarter retries. It's rate limiting LLM provider calls from your own side, proactively, before you ever hit the provider's ceiling.
Why Reactive Retries Aren't Enough
Retry-with-backoff handles occasional, isolated rate-limit hits well. It handles sustained overload badly: if your traffic consistently exceeds your rate limit rather than briefly spiking past it, every retry just adds to a growing queue of requests all competing for the same limited quota, and your effective latency degrades for every user, not just the ones who happened to trigger the original 429. The provider's limit is a hard ceiling; your app needs its own limiter that stays under it, not a recovery strategy for after you've already gone over.
Rate Limiting With the Token Bucket Pattern
Most provider rate limits are expressed as both requests-per-minute and tokens-per-minute. A token bucket limiter models this directly: a bucket holds a capacity of "tokens" (in the rate-limiting sense — permission units, distinct from LLM tokens) that refill at a steady rate, and every request must acquire enough bucket tokens before it's allowed to proceed.
class TokenBucket {
constructor(capacity, refillPerSecond) {
this.capacity = capacity;
this.tokens = capacity;
this.refillPerSecond = refillPerSecond;
this.lastRefill = Date.now();
}
refill() {
const now = Date.now();
const elapsed = (now - this.lastRefill) / 1000;
this.tokens = Math.min(this.capacity, this.tokens + elapsed * this.refillPerSecond);
this.lastRefill = now;
}
async acquire(cost = 1) {
this.refill();
while (this.tokens < cost) {
await sleep(100);
this.refill();
}
this.tokens -= cost;
}
}
// Sized just under your provider's actual limit, not at it
const limiter = new TokenBucket(/* capacity */ 45, /* refill/sec */ 0.75);
async function callLLM(prompt) {
await limiter.acquire();
return anthropic.messages.create({ model: "claude-sonnet-5", messages: [{ role: "user", content: prompt }], max_tokens: 1024 });
}
Sizing the bucket slightly under your actual provider limit (not exactly at it) leaves headroom for other traffic sharing the same API key — background jobs, a different feature, a teammate testing in the same environment — that your limiter doesn't directly control but still competes for the same quota.
Queueing, Not Just Blocking
The acquire method above blocks the calling code until capacity is available, which works for background jobs but produces a bad user experience for interactive requests — a user shouldn't sit on a spinner while your limiter silently waits behind other queued requests. For request paths a user is actively waiting on, give the limiter a short timeout and fail fast with a clear "high demand, try again shortly" message rather than making them wait indefinitely behind a queue they can't see.
Prioritization: Not All Calls Deserve Equal Access to the Limit
Once you have a shared limiter, you have a natural point to add priority. A user-facing chat response and a background bulk-reprocessing job shouldn't compete for capacity on equal footing — give interactive requests priority access to the bucket and let background jobs yield when the bucket is under pressure, checking in less aggressively and backing off further when interactive traffic is high.
Per-Key vs Per-App Limiting
If multiple services or environments share one provider API key, a limiter local to just one service doesn't protect the shared quota — a different service can still exhaust it. For a genuinely shared key, the limiter needs to live somewhere all consumers pass through (a shared Redis-backed token bucket, or a dedicated proxy service that all calls route through), not duplicated independently in each service with no shared state.
Handling Multiple Rate Limit Tiers From One Provider
Providers frequently enforce more than one limit simultaneously — a per-minute limit and a per-day limit, or separate limits for different model tiers under the same account. A single token bucket only protects against one of these. The practical fix is running a limiter per tier and requiring every request to acquire capacity from all applicable buckets before proceeding, not just the first one you happened to implement.
class MultiTierLimiter {
constructor(limiters) {
this.limiters = limiters; // e.g. { perMinute: bucket1, perDay: bucket2 }
}
async acquire(cost = 1) {
// Must clear every tier, not just one — a per-day limit hit still
// blocks the request even if the per-minute bucket has room
await Promise.all(Object.values(this.limiters).map((l) => l.acquire(cost)));
}
}
Missing the daily or monthly tier is a common gap — teams build a solid per-minute limiter, confirm it prevents 429s under normal load, and then get surprised weeks later when a legitimate traffic increase trips a daily cap the limiter was never tracking.
What to Do When You're Rate Limiting Correctly and Still Need More Capacity
A well-implemented limiter doesn't create more capacity — it just prevents your app from exceeding what you already have and failing ungracefully. If the limiter is regularly the bottleneck (requests queueing for a noticeable, user-visible amount of time under normal load, not just rare spikes), that's a signal to act on the underlying capacity, not just tune the limiter further: request a quota increase from your provider, distribute load across multiple API keys or provider accounts where the provider's terms allow it, or reduce the number of calls you're actually making per task (batching multiple extractions into one call, or caching results for identical or near-identical inputs) rather than just widening the bucket and hoping the provider's real limit accommodates it.
Observability: Knowing How Close You Are to the Limit
A rate limiter that works silently gives you no visibility into how much headroom you actually have left, which makes capacity planning reactive instead of proactive. Most providers return current usage information in response headers (remaining requests, remaining tokens, and when the window resets) — log these on every call rather than discarding them, and graph utilization as a percentage of your limit over time. A steady climb toward 100% utilization during business hours is a much more useful signal to act on early than the first 429 error, which by definition means you've already exceeded what your limiter was supposed to prevent.
This data also answers the question every team eventually asks: "are we actually close to needing a quota increase, or does it just feel that way during our busiest hour?" Without utilization logged over time, that's a guess. With it, it's a chart you can point to in a support request to your provider, and a number you can use to size the next quota tier realistically instead of over- or under-requesting.
None of this — the bucket, the multi-tier check, the utilization graph — needs to be built before your first production traffic. It's reasonable to launch with a simple single-tier limiter and add the rest as your actual traffic patterns reveal which gaps matter. What isn't reasonable is launching with no limiter at all and finding out about the gaps from a wave of 429s during your first real spike in usage. The cost of adding a basic token bucket is a few dozen lines of code and one afternoon; the cost of not having one is an outage during exactly the traffic spike you were hoping would go well. Build the simple version now; the multi-tier and prioritized versions can wait until your traffic actually justifies the added complexity of maintaining and monitoring them over time. Start simple, measure what actually breaks in production, and add the next layer of sophistication only once real traffic data tells you it's genuinely needed rather than guessing upfront.
Building this specifically in n8n rather than application code? See n8n Rate Limiting and API Throttling for the same pattern implemented with native nodes.
Written by Abhishek Patoliya, Senior n8n Developer & Automation Architect, Surat, India. Verified n8n Creator.