Most API rate limit implementations are trivially bypassed by naive proxy rotation — but that same rotation triggers anti-abuse heuristics within minutes. The real challenge is not just staying under the limit, but doing so without looking like a bot. This requires understanding the two distinct rate limit scopes (per-IP and per-key), applying jittered backoff, randomizing pool rotation, and shaping request timing to mimic organic traffic. The token bucket pattern, layered on a proxy pool, provides a robust foundation.
Per-IP vs Per-Key: The Two Axes of Rate Limiting
Rate limits operate on at least two independent dimensions: the source IP and the API key (or token). A single key may allow 5,000 requests per hour (GitHub’s authenticated limit), but the same key from a single IP may be throttled at a lower burst ceiling. Unauthenticated requests are even more constrained — typically 60 requests per hour per IP. Ignoring the IP dimension is the fastest way to trigger a 429 Too Many Requests (RFC 6585) response. A proxy pool must distribute requests across multiple IPs to avoid saturating any single origin, but each IP still shares the same key. If the key’s global limit is 5,000/hour and you have 50 proxies, each proxy can only make 100 requests per hour before the key-level counter fires. Map both limits before writing a single line of code.
Backoff with Jitter: The Difference Between Polite and Predictable
Exponential backoff without jitter is a fingerprint. Servers see requests arriving at perfectly doubling intervals and flag them as automated. The fix is full jitter: sleep(random.uniform(0, min(cap, base * 2 ** attempt))). This spreads retries across the time window, making the pattern indistinguishable from a burst of real users. AWS’s own documentation on exponential backoff recommends jitter for exactly this reason. The following Python snippet implements a token bucket that refills at a fixed rate, sleeps with jitter when empty, and rotates the proxy on each successful acquire:
import time
import random
from collections import deque
class TokenBucketProxyPool:
def __init__(self, proxies, rate, burst):
self.proxies = deque(proxies)
self.rate = rate # tokens per second
self.burst = burst
self.tokens = burst
self.last_refill = time.monotonic()
def refill(self):
now = time.monotonic()
elapsed = now - self.last_refill
self.tokens = min(self.burst, self.tokens + elapsed * self.rate)
self.last_refill = now
def acquire(self):
while True:
self.refill()
if self.tokens >= 1:
self.tokens -= 1
self.proxies.rotate(1) # simple rotation, but see section below
return self.proxies[0]
else:
sleep_time = (1 - self.tokens) / self.rate
jitter = random.uniform(0, sleep_time * 0.5)
time.sleep(sleep_time + jitter)
# Usage: pool = TokenBucketProxyPool(proxies, rate=2.0, burst=10)
# proxy = pool.acquire() # blocks until token available
Proxy Pool Rotation: Randomize, Don’t Round-Robin
Sequential round-robin through a proxy list is detectable. A server that sees requests from 1.2.3.4, then 5.6.7.8, then 9.10.11.12 in lockstep will correlate the pattern across IPs. Instead, pick proxies randomly with replacement, and add a per-proxy cooldown. Public proxy directories report 60–80% failure rates — a proxy that returns a 429 or a connection error should be demoted to a dead queue for at least 60 seconds. Implement a weighted random selection where recently successful proxies are more likely to be chosen. The token bucket above uses a simple rotate(1) for clarity; in production replace that with random.choice() and track per-proxy failure counts.
Request Shaping: Mimic the Mean Without Crossing the Line
Even with a perfect token bucket and proxy pool, a constant request rate of exactly 2.0 per second is unnatural. Real users pause, batch, and vary inter-request intervals. Add a small random delay (e.g., time.sleep(random.uniform(0.1, 0.5))) before each acquire() call, and vary the token bucket’s rate by ±10% every few minutes. Do not, however, add delays that exceed the API’s documented rate limit window — that defeats the purpose. The goal is to stay within the limit while appearing to be a cluster of legitimate clients. Over-shaping (e.g., inserting human-like typing delays) is wasted effort; the API cares about request frequency, not inter-keystroke timing.
Ethical Mitigation: Know Where the Line Is
Every technique described here is a legitimate optimization — until it violates the API’s Terms of Service. Most ToS explicitly prohibit “circumventing rate limits” or “using automated means to access the service.” Reading the ToS is not optional. If the API allows 100 requests per minute per key, using 50 proxies each making 2 requests per minute is compliant. Using 100 proxies each making 100 requests per minute is not — it is abuse, regardless of how clever your jitter is. The distinction is intent and volume. A token bucket layered on a proxy pool is a tool; the same tool that scrapes a public directory ethically can also be used to DDoS a small API. Document your rate limit targets, audit your logs for 429s, and never exceed the documented per-key ceiling. If you need more throughput, ask for a higher limit or pay for a dedicated plan.