Rate Limiting in the Real World

Background

This is the final part of the interview conversation we have been walking through in this series. The scheduled time was one hour, but the conversation touched upon so many interesting corners that neither of us realized we were about to touch the two hour mark.

In the previous parts we discussed choosing the algorithm, then how to shape it based on the requirement and manage the problems of scalability like concurrency, atomicity, hot keys, latency and failures. In this part we will see how reliability is designed and what can go wrong.

The questions in this part are the ones I parked at the end of the last one. We will work through five of them and bring the series to a close.

Placement of Rate Limiter

A request travels via Browser → CDN → Load Balancer → API Gateway → Backend Service (maybe several microservices). If you had to pick one layer to place a rate limiter, where would you put it, and why? Is one layer actually enough for a real production system?

My answer - “Ideally it should be at multiple places: at the CDN or LB level for abusive or unknown requests, and at the API Gateway where rate limiting applies to registered users and can be decided based on the user tier”.

I was confident about my answer and it was right, but I missed a subtle corner.

Level 1: CDN

The CDN is the right place to eliminate unknown or abusive bursts and DDoS style traffic before it costs us any compute. Tools like Cloudflare do this easily.

But this cannot be the only layer, because we can have several types of users. More importantly, paid users on different plans can have different API keys, which define the QPS allowed for them.

Level 2: API Gateway

Here we carry out the auth checks and take user aware rate limiting decisions.

Level 3: Inter Service Rate Limiting

This is what I missed, and honestly, I had never thought about it even in my past experience. But the interviewer gave me cases where I found this to be an important gate we should have at peak scale.

Rate limiting at inter-service communication is important to handle cases of cascading failure or retry storms.

One example is a Producer-Consumer scenario where the consumer is not able to match the producer’s pace and eventually gives up. Imagine the Black Friday sale where orders are placed at the pace of 100k req/hour but the invoicing service can only handle 50k req/hour.

External rate limiting protects the system from malicious traffic. Internal rate limiting protects services from each other; without it, a slowdown in one service can cascade into a full outage across the whole system.

Key Strategy

If you rate-limit purely by IP address, what’s a scenario where that badly misfires? Either being too strict on innocent users, or too lenient on an actual abuser?

My answer - “IP can be misleading here as a paid client may be allowed 100 req/s, but if we rely only on IP, each IP gets its own limit. Instead, we can use the API key for more accurate measurement. If we have a device limit, we can track live sessions so they cannot be active on more than X devices, such as 2, 3, or 4.”

I was sure of a correct answer this time and had a smile on my face. But the interviewer asked me again, “Ashok can you think of a case where the IP based rate limiting can really prove fatal?” And yet, it did not click!

The bigger picture was that multiple legitimate users behind the same NAT or corporate proxy could get throttled together. So if we track only the IP and a single user is super active, the other users will not be able to access the service. This is why we need different parameters to rate limit on.

In the real world we generally prefer a combination of IP + API key.

Response Structure

Another aspect of rate limiting is to make the client aware of the limit expiration or refills. This one was easy for me to navigate.

If a request is correctly rate-limited at the edge and rejected, versus rejected at the gateway, versus rejected internally between two of your services, does the client experience these differently? Should they?

My answer - “Ideally, the client should not know about the internal issues, and the behavior should be the same. But the catch is that the customer is not at fault, and hence we make it look different by showing ‘Service not available’ if it is an internal limit breach. For a user’s limit breach we can respond with HTTP status 429”

The follow up to this was:

Should you tell clients about their rate limit status even on successful requests, before they hit the wall?

My answer - “Yes, we should tell them so that the client can be aware and save the requests. That will also save our compute and resources. We can return the header of the limit left in the response.”

The industry standard headers for making the user aware of the rate limiting look like:

1
2
3

X-RateLimit-Limit: 1000        // total requests allowed in this window
X-RateLimit-Remaining: 743     // requests left in current window
X-RateLimit-Reset: 1679419665  // timestamp (or seconds) when the window resets

And in case of 429:

1

Retry-After: 42                // seconds until they should retry

This went well and the answers were satisfactory, but the follow-up was not as straightforward to answer.

Burst at Refill

There was also a question regarding the burst that can happen during the refill. We did not discuss it in too much depth, but I gave a high level answer.

Imagine a partner integration, a company like Amazon has an allowed limit of 10,000 req/min. They queue up requests locally and fire them the instant they’re allowed to, because they want to be fast for their own users. Does your backend care about that? What if 10 such partners do the same simultaneously?

My answer - We have to allow the requests as these are legitimate cases. The problem that can hit the backend systems here is the Thundering Herd problem. We should have proper measures to tackle these kinds of situations, and we can also carry out load testing for validation.

There are multiple measures that can be deployed to mitigate and manage the Thundering Herd problem. A few of them can be:

• Queueing / buffering requests to absorb bursts
• Concurrency with idempotency to protect downstream services
• Graceful degradation as and when necessary
• Autoscaling, where appropriate
• Load/stress testing to determine safe capacity

Summary

Across three parts we covered a lot of ground and some deep technical challenges. Now let’s put together a condensed summary of all these aspects, set against the initial question.

We need to design a rate limiter. Before we talk about where it lives in the system or how it scales, let’s start simple. What algorithms would you consider for rate limiting, and what are the trade-offs between them? Walk me through the ones you know.

And the quick answer will look like:

• Algorithm: We’d default to token bucket because it allows legitimate bursts while controlling sustained traffic. If we need smoother enforcement, we’d use a sliding window counter. Both use O(1) memory per client, unlike a sliding window log.

• Distributed State: Since we’re running across multiple instances, we’d centralize the state in REDIS and use a Lua script for atomic check-and-update operations, avoiding race conditions and distributed locks.

• High QPS & Resilience: For high QPS, we wouldn’t necessarily hit REDIS on every request. We’d keep a local approximate cache per instance and accept a small, bounded amount of over-admission to reduce network calls. REDIS can also have a graceful fallback using a circuit breaker rather than failing completely open or closed.

• Rate Limiting Layers: We’d put rate limiting at three layers: the edge/CDN for coarse IP-based protection against volumetric abuse, the API gateway for per-user or API-key limits based on pricing tiers, and internally between services to prevent cascading failures. That internal layer is more about resilience than abuse prevention.

• Client Contract: We’d return 429 when they’ve exceeded their limit and 503 when we’re degraded on our side. We’d also include rate-limit headers so clients can self-throttle before hitting the limit.

• Edge Cases: Finally, we’d key limits on the user/API key rather than IP when authentication is available, use REDIS’s clock to avoid cross-node clock skew, and add jitter to refill/reset timing to prevent a thundering herd when limits refresh.

I also got feedback on the same call, and I really appreciate that he was honest and kind enough to share it with me directly.

Ashok, you have good instincts and you connected the dots well across the conversation. Where you were shakier was the exact mechanics of the state management details and how things hold up under peak QPS.

Reflections

If Part 1 was about picking the right algorithm and Part 2 was about making that algorithm survive contact with a distributed system, this part was about something quieter: making sure the rate limiter is honest with the people it is limiting. Where you place it, what you tell the client, and how you fail. None of that shows up in an algorithm’s pseudocode, but all of it is what a client actually experiences.

The IP-based question was the one that stuck with me longest. Not because the answer was hard, but because my first instinct — “use the API key, that’s more accurate” — was correct and still incomplete. It took a direct nudge to get to the real edge case. That’s a good reminder that “correct” and “complete” aren’t the same thing, and an interviewer’s follow-up question is usually pointing at that gap.

Looking back across all three parts, the pattern repeats at every layer: there is rarely one right answer, only a right answer for a given set of constraints.

That’s ultimately what this conversation gave me back. Two hours in, after having barely thought about rate limiting since I last operated one in production, the algorithms turned out to be the easy part to relearn. The judgment about trade-offs is the part that only comes back with practice.