I have not run a million requests through an agent. I'll say that first, so you can weigh the rest. I have run document jobs of thousands of pages through queues and workers, and that showed me where the load goes. The rest of this piece is design reasoning built on that, and I'll mark which part is which.
The first thing that broke was not our servers
On a production AI platform for U.S. legal documents, our page workers ran 120 threads against a cloud OCR service with a fixed quota of 25 calls per second. Most of those threads were just waiting, and the service kept answering "slow down". We cut the threads to 32, and the throttling errors stopped.
That is the whole lesson in small. Past the provider's quota, more workers add only waiting and errors. Your own web servers are rarely the problem.
Where the bottleneck goes next
From what we saw, the limit moves in this order:
- The model provider. Rate limits and quotas, per account, per model.
- The queue. Work arrives faster than the provider lets you drain it, so the backlog grows.
- The database. Every step writes its state, and every retry writes again. On the same platform, a burst of uploads once used up the database connection pool, and an upload returned "OK" while nothing had been saved to run.
The queue is the quiet one. AWS's engineers put it this way in their Builders' Library: "The hidden risk with asynchronous systems is dealing with large backlogs." A queue that is ten minutes behind looks healthy on every CPU graph. Alarm on the age of the oldest message, not only on errors.
Think in tokens per minute
At this point you stop thinking in requests per second. Providers count tokens. OpenAI's docs list limits in requests and tokens per minute and per day, and note two things that catch people out: failed requests still count against the limit, and your max_tokens setting can count before the model writes a word. Anthropic splits the limit into input tokens and output tokens per minute, refilled continuously rather than reset each minute.
Here is the sum for a million requests. This part is arithmetic.
- A million requests a day is about 700 a minute, on average. Peaks are higher.
- In the multi-step features we have built, one user action meant 10 to 15 model calls. That is 7,000 to 10,400 calls a minute.
- Say each call sends 5,000 input tokens. I picked that number for the sum; measure your own. It gives 35 to 52 million input tokens a minute.
Anthropic's published Scale tier allows 10 million input tokens a minute for each of its main model classes, as of October 2026. So an average minute, not even a peak one, needs several times what one account's standard limit allows. You change the design, or you negotiate, or both.
Three changes help most:
- Fewer tokens per call. Anthropic's docs say that for most Claude models, cached input tokens don't count toward the input limit. A long system prompt that is the same on every call is the first thing to cache.
- Fewer calls per request. A small model or plain code for the easy steps. Not every step needs the biggest model.
- Less than real time where you can. Work that can wait an hour can go through a batch API. Anthropic's Message Batches API, for example, has its own rate limits, separate from live calls.
Design for the limit
One shared rate limiter. If each worker has its own limit, the limits add up to more than the provider allows the moment you scale out. Keep one budget for all workers, in Redis or the database, and make every worker ask it before a call.
Back off when the provider says slow down. Use the Retry-After header when it is there. When it isn't, OpenAI's docs recommend exponential backoff with random jitter, so your workers don't all retry in the same second. Retry in one layer only. AWS shows how retries stack: five layers deep with three retries each, the bottom of the stack sees 243 times the load. If the SDK retries, your code retries and the queue redelivers, you have built the same thing.
Separate queues, so one big customer doesn't starve everyone. AWS's advice is that no customer's workload should affect another customer's. They use per-customer limits and spread each customer over a small set of queues. A queue per large customer, plus one for everyone else, is a fine start.
Sometimes a second provider or region. When one region has a bad hour, every job in the queue fails at the same step. A second route costs you testing on two models. Decide if your users need it before you need it.
Each of these assumes the step is safe to run twice, because at this volume retries happen every minute. Why AI features need background jobs covers that part, and security for agentic workflows covers what a repeated agent step can do to your data.
Know what one request costs
Log the tokens of every call: input, cached input, output, with the request id, the step and the model. Apply prices later, from a table. Then cost per request is a query, not a guess. Store tokens, not dollars, because prices change and you will want to re-price history.
We didn't do this early. For a long time nobody on the team could say what one document cost us. At a million requests a day you can't carry that gap: a prompt change that adds 2,000 tokens is a line on next month's bill, and you want to see it the same week.
What I would test before believing any of this
Build a fake provider: a small server that enforces a tokens-per-minute limit and returns 429 with Retry-After when you go over. Point your workers at it, push ten times your expected load, and watch three numbers: the age of the oldest queue message, the 429 rate, and tokens per request. If you plan to take an agent from a pilot to real traffic, this is the test we run first in prototype to production work.
Do this week
Take one AI feature you have built. Read the token counts from the API's usage field for ten real requests. Multiply by the requests per minute you hope to get. Put that next to the limits page in your provider's console.
Scale for an agent is mostly someone else's limit. Find yours on paper before your users find it for you.