The Economics and Physical Limits of Running AI Agents at Scale

A single call to a frontier model has a price that anyone can read off a rate card: so many dollars per million input tokens, so many per million output tokens, published, dated, and easy to multiply. An agent is not a single call. It is a loop — propose an action, call a tool, read the result, decide again — and the number that actually determines whether a long-horizon agent deployment is affordable is not the price of one step but the shape of the curve traced by many of them strung together.

That curve has a specific, documented shape, and it is worse than most single-request cost estimates assume. Because a model API is stateless between calls, an agent that wants to keep acting on what it has already done must resend its own history on every single turn — not as a courtesy, but as a structural requirement of how the interface works. Tool definitions get re-declared. Prior reasoning gets re-read. The result of a search performed ten steps ago rides along into step eleven, step twelve, and every step after that, until something removes it. Prompt caching softens this substantially, and this article works through by how much, using the multipliers Anthropic and OpenAI actually publish. It does not remove the underlying shape.

This article treats a long agentic loop as an economics-and-engineering problem with three coupled parts: what a single step actually costs once its fixed tax and its retransmitted history are counted; how that cost compounds as the number of steps grows, with and without caching; and where the physical ceilings sit — a context window that degrades in quality well before it errors out, a rate limit that a single trajectory can approach on its own, and a latency budget that decides whether a step can run synchronously in front of a waiting person at all. None of the mechanics below are hypothetical. They are drawn from Anthropic’s and OpenAI’s own pricing, caching, and rate-limit documentation, plus independent research on context degradation and long-horizon agent cost, each verified against its own page on 12 August 2026.

A step is not a request

Start with what one step in an agentic loop actually contains, because “a request” undersells it. A step is a Messages API call, or its equivalent on another provider, that carries at minimum: a system prompt, a set of tool definitions, the entire prior conversation up to this point, whatever new material this step introduces — a tool result, an instruction, an observation — and it returns whatever the model generates, some of which may itself be a tool call feeding the next step.

The first thing added to every one of those requests, regardless of size, is a fixed tax for the tools themselves. Anthropic’s pricing documentation states this precisely rather than leaving it implicit: enabling tools at all adds a system-prompt overhead published per model and per tool-choice setting — 354 tokens for Claude Sonnet 5 with tool choice left to the model, 474 tokens if a specific tool is forced, before a single token of any individual tool’s own name, description, or schema is counted ^[1]. Individual tools add more on top: the bash tool adds 244 to 325 tokens depending on model generation, the text editor tool adds 700, and the computer-use beta adds several hundred to the system prompt plus 735 per tool definition ^[1]. None of this varies with what the step actually does. A step that calls no tool at all still carries the entire tool-availability tax if tools are attached to the request, because the model has to be told what it could call before it can decide not to call anything.

Call this fixed component $\tau$. It is paid once per step, in full, every step, for as long as the loop runs — the toll is the same on step one hundred as on step one, whether or not that step’s own contribution to the task is large or trivial. Over many steps, a workload that defines five tools and uses one on a given turn is still paying the fixed tax for all five, every turn; narrowing the tool set narrows $\tau$, but nothing the model does at inference time avoids it.

The second component is genuinely new content: whatever this step adds that the loop has not seen before — a fresh tool result, a new instruction, an observation from the environment. Call this $\delta$. The third is the step’s own output, call it $\bar{o}$, priced at the output rate, which is five times the input rate on every currently supported Claude tier, from Haiku 4.5 through Opus 5, and roughly six times on OpenAI’s GPT-5.6 family ^{[1, 7]}.

Add $\tau$, $\delta$, and $\bar{o}$ together and a single step is priced — which is exactly what a per-request cost estimate does, and exactly why it understates a loop. The fourth component, the one a single-step estimate cannot see, is everything the step has to carry forward from every step before it.

The shape of the retransmission tax

Because the API is stateless between calls, step $i$ in the loop must include, as ordinary input, everything that made it into the transcript through step $i-1$: the tool results, the intermediate reasoning, the earlier outputs. Call the amount a single step permanently adds to that growing transcript $\sigma$ — roughly $\delta$ plus $\bar{o}$, since both the new material a step introduces and the output it produces become part of what every later step has to resend. Without caching, step $i$’s input is not just $\tau + \delta$; it is $\tau + \delta$ plus the accumulated weight of every prior step, $(i-1)\sigma$:

Summed over an $n$-step trajectory, that gives

The middle term is the one that matters. It grows as $n(n-1)/2$ — quadratically in step count — while the other two terms grow only linearly. For a short loop, the quadratic term is negligible next to the fixed tax and the output cost. For a long one, it dominates completely, and no amount of shrinking $\tau$ or $\bar{o}$ changes that; the loop’s own length has become the largest line item, all by itself.

Prompt caching is the documented fix for exactly this, and it is worth being precise about what it fixes and what it does not. Anthropic’s automatic caching, recommended for multi-turn use, places a single cache marker at the end of a request; the system caches everything up to that marker, and the next request’s identical prefix is read from cache rather than reprocessed ^[2]. Anthropic’s documentation walks through exactly this pattern for a growing conversation: at each new turn, the system prompt and all prior turns are read from cache, and only the newest exchange is freshly written to cache for next time ^[2]. A cache read costs a tenth of the base input price; a five-minute cache write costs 1.25 times it ^{[1, 2]}. Substituting those multipliers into the model — reading the accumulated history at the discounted rate $\gamma \approx 0.1$ instead of the full rate, and writing only the newest increment at the write rate $w \approx 1.25$ — gives

The quadratic term survives. Its coefficient shrinks by the cache-read discount, roughly tenfold, but the shape — cost rising with the square of step count — does not go away, because caching discounts the price of resending the growing history; it does not stop the history from having to be resent. OpenAI’s own cached-input pricing lands at the same roughly tenfold discount on every current GPT-5.6 tier ^[7], so the same qualified conclusion holds on that stack too: this is a property of a stateless, replay-everything request format, not a pricing quirk of one vendor.

The following arithmetic is illustrative, not a measurement of a real deployment. It applies Claude Sonnet 5’s published rates — 2 dollars per million input tokens, 10 dollars per million output tokens, a 1.25x five-minute cache write, a 0.1x cache read ^{[1, 2]} — to a stylized loop with a 400-token tool tax, a 600-token new-content increment, and a 300-token output per step.

Caching buys roughly a two-times reduction at ten steps, five-times at fifty, and close to an eight-times reduction at two hundred — climbing toward, but never reaching, the tenfold ceiling set by the cache-read discount itself. The last column is the more important fact: even with caching applied on every single turn, the discounted retransmission term goes from a rounding error at ten steps to three-quarters of the entire bill at two hundred. A longer loop does not just cost more. It spends a growing share of itself paying, again and again at a discount, to remind the model what already happened.

What a retry costs, and where it lands

A retry is not a second, independent request. It is the same step, re-attempted, and because a step’s cost is $c_i$ — a function of how much history has already accumulated by step $i$ — a retry at step one hundred costs roughly as much as step one hundred did the first time, not as much as step one. This is easy to miss if retries are budgeted as a flat percentage overhead on an average request cost, because the average request in a long loop is far cheaper than a late one.

Anthropic’s SDKs retry connection errors, rate-limit errors, and server errors twice by default with exponential backoff, honouring a retry-after header when the API supplies one, and the retry count is configurable ^[3]. Anthropic’s own error documentation also distinguishes two causes worth separating operationally: a 429 rate-limit error means the calling organization exceeded its own configured limit, while a 529 overload error means the API is under high traffic across all users, a condition a caller’s own backoff schedule cannot fix ^[4]. Both are retried the same way by default, but only one of them is something the caller’s own request pattern controls.

If a step fails and is retried with probability $q_i$, independent of other steps, the expected cost of the trajectory becomes

Holding $q_i = q$ constant across the trajectory — a simplification, since real failure rates are neither constant nor independent — the expected extra cost from retries is $\frac{q}{1-q}\sum_i c_i$, and because $c_i$ increases with $i$, that extra cost is weighted toward the trajectory’s most expensive, latest steps rather than distributed evenly across it. A one-percent per-step retry rate is a rounding error at step five. Applied at step two hundred of an uncached or lightly cached loop, it is a rounding error on a much larger number.

Two further mechanisms turn a retry into something more expensive than a clean re-attempt. A response cut off by the max_tokens cap bills every token generated up to the cap at the ordinary output rate — Anthropic states the billed count is the full, authoritative total regardless of truncation — and the documented remedy is to raise the cap or lower the effort level and try again, which does not refund the truncated attempt ^[1]. And on models that preserve reasoning across turns, thinking tokens generated on one step are billed once as output when produced, then again as input on every later step that carries the conversation forward ^[1] — a structural, intended cost of maintaining reasoning continuity across a long loop, layered on top of the retransmission tax derived above.

The governor: rate limits as a documented throughput ceiling

Cost is not the only ceiling a long loop runs into. Anthropic’s Messages API enforces separate limits on requests per minute, input tokens per minute, and output tokens per minute, and exceeding any one of them returns a 429 error regardless of headroom on the others ^[3]. OpenAI’s API enforces an analogous but structurally different set: requests per minute, tokens per minute — a single combined count of input and output together in one rolling sixty-second window — requests per day, and tokens per day, with whichever ceiling is reached first triggering the limit ^[8]. Anthropic splits input and output into two separately governed pools; OpenAI’s tokens-per-minute figure merges them into one. Neither structure is more correct than the other; they are two different places to put the same governor, and a workload’s shape, output-heavy or input-heavy, will hit one harder than the other depending on which API it runs against.

The mechanism worth understanding in detail is what counts toward the ceiling, because it interacts directly with everything derived above. Anthropic’s rate-limit documentation states that for most Claude models, only uncached input tokens count toward the input-tokens-per-minute limit — tokens read from cache do not, with Claude Haiku 3.5 the one marked exception ^[3]. Anthropic’s own worked example: an organization with a 2,000,000 input-tokens-per-minute limit running an 80 percent cache hit rate can process roughly 10,000,000 total input tokens per minute, because eight of every ten tokens never touch the metered ceiling ^[3].

That exemption changes the shape of the rate-limit exposure the same way it changed the shape of the dollar cost. In the uncached model above, step $i$ sends $(i-1)\sigma$ tokens of retransmitted history as ordinary, fully metered input — every one of which counts against the per-minute ceiling, every single step, growing linearly with $i$ within a single trajectory. A long, rapid, uncached loop can approach its own organization’s input-tokens-per-minute ceiling from its own growth alone, with no other caller involved: the trajectory throttles itself. With per-turn caching applied, the accumulated history is read from cache and, per the documentation above, does not count toward the limit at all; only the newest increment written each turn — a roughly constant $w\sigma$ tokens per step, not a growing one — consumes ceiling headroom. Caching is therefore doing two separate jobs that are easy to conflate: it discounts the price of the growing history, and independently, it exempts that same growing history from the throughput governor. A workload that caches for the price discount and does not realize it is also buying rate-limit headroom is undercounting what caching is worth.

None of this addresses latency, which is governed by a separate physical constraint. Generating a token requires reading the model’s weights and the accumulated key-value state for the full preceding context out of accelerator memory on every step, which is why decoding is bound by memory bandwidth rather than arithmetic, and why that cost scales with how much context is resident regardless of what a caller was billed for producing it ^[11]. For an interactive loop where a person is waiting on each step, a per-step latency that grows with the transcript’s size eventually crosses whatever budget makes synchronous use tolerable. Anthropic’s own guidance is blunt about where that budget sits for its API: streaming or the Batch API are recommended for any request running past roughly ten minutes, and the SDKs validate that a non-streaming request is not expected to exceed a ten-minute timeout at all ^[4]. A loop long enough to need summarizing for cost or rate-limit reasons is frequently also a loop growing too slow, per step, to keep a person waiting on it synchronously — the same growth in accumulated context pushes on cost, on the rate-limit ceiling, and on the interactive latency budget simultaneously, because all three are downstream of the same growing history.

A floor below the ceiling: context rot and compaction

A context window has a stated maximum, and it is tempting to treat that number as the only ceiling that matters — the point past which a request simply fails. Independent research suggests the more binding constraint sits well below it. Chroma’s July 2025 technical report, testing eighteen models across four model families including Claude, GPT, Gemini, and Qwen variants, found that model reliability on simple tasks — retrieval, exact replication — degrades measurably as input length grows, and termed the effect context rot to distinguish it from context window overflow: for the 1M-token-class models tested, a clearly observable degradation typically appeared somewhere around 300,000 to 400,000 tokens, well short of the stated ceiling ^[9]. The degradation was uneven — it depended on how similar a target was to distracting surrounding content, not on length alone — but its existence at all means a hard token ceiling is not the number a long-running loop should be designed against.

Anthropic’s own engineering guidance names the same phenomenon directly, stating that models “lose focus or experience confusion” as token count increases, and treats context as a finite resource with diminishing returns rather than a simple capacity to fill ^[6]. Its documented response is not one technique but three, used together: compaction, which summarizes a conversation nearing its limit and reinitializes from the compressed summary while explicitly preserving architectural decisions and unresolved issues; structured external note-taking, so an agent can externalize state rather than carry all of it in-context; and sub-agent delegation, where a specialized agent returns only a condensed result, typically one to two thousand tokens, rather than its full working transcript ^[6].

A separate, narrower mechanism, context editing, targets the retransmission tax specifically rather than quality degradation: it automatically strips stale tool calls and results from the window as a session approaches its limit, preserving the surrounding conversation. Anthropic reports a concrete figure from a 100-turn web-search evaluation: context editing alone reduced token consumption by 84 percent relative to the same evaluation without it, and combined with a persistent memory tool it produced a 39 percent improvement in task outcomes ^[5]. An 84 percent reduction across a hundred-turn trajectory is not a rounding adjustment to the model derived above — it is close to the entire quadratic term, since that term is exactly the cost of carrying stale history forward turn after turn.

Compaction and editing are not free, and the model above shows why they earn their keep anyway. Reset the accumulated history to a small fixed-size summary of $s$ tokens every $m$ steps, and the trajectory becomes a sequence of $n/m$ short quadratic segments instead of one long one: each segment’s own retransmission cost scales with $m^2$, but the number of segments scales with $n/m$, so the total across the whole trajectory scales with $m \cdot n$ — linear in $n$ for a fixed segment length $m$, not quadratic. The summarization step itself is not free — reading through a full segment once to compress it costs roughly what one ordinary step near the end of that segment costs — so an interval $m$ chosen too short pays that overhead too often, and one chosen too long lets the quadratic term inside each segment grow large again before it is cut. Anthropic’s own reasoning behind sub-agent summaries, returning one to two thousand tokens rather than a full working transcript ^[6], is exactly a statement about keeping $s$ small relative to $m$, which is what makes the reset worth performing at all rather than merely deferring the same bill.

Task horizon is a cost multiplier

Step count is not an arbitrary variable in the model above; it is the operational meaning of “task horizon,” the metric a growing body of capability research uses to describe what agents can now do. METR’s 50-percent task-completion time horizon measures the length of task, expressed in the time a skilled human would need, that a frontier model can complete with 50 percent reliability, and the paper’s central empirical finding is that this horizon has been doubling roughly every seven months since 2019, with the frontier models tested as of March 2025, including Claude 3.7 Sonnet, reaching a horizon of around 50 minutes of human-equivalent task time ^[10]. The paper is explicit that this is a capability metric, extrapolated cautiously: its authors state that if the trend generalizes to real-world software tasks, systems capable of month-long tasks are roughly five years out from that 2025 baseline, a forecast attributed to the paper, not a settled fact ^[10].

A task that takes a skilled human longer is not, mechanically, a task with a bigger single step. It is almost always a task with more steps — more tool calls, more intermediate observations, more turns of the loop derived above. That connection is inference, not something METR’s paper measures directly, but it is the natural reading of what a “longer task” means operationally for a tool-using agent, and if it holds even approximately, it has a specific consequence given the cost model above: a task horizon that doubles does not make an agent’s operating cost merely double. Because the dominant term in that cost is quadratic in step count, a doubling of steps roughly quadruples the retransmission component of the bill, discounted but not eliminated by caching. Capability measured in task horizon and the cost of exercising that capability are coupled, and they are coupled at different rates — which is precisely the kind of practical ceiling this article set out to locate: not a wall a model hits, but a cost curve that steepens faster than the capability it is paying for.

Independent systems research is converging on the same diagnosis from the engineering side rather than the economics side. A 2026 study proposing a cache-efficient context-management framework for LLM agents states its motivating problem plainly: context accumulation drives up inference costs in long-horizon agent sessions, and reports that its combined compaction-and-eviction approach cut cost by roughly 56 to 87 percent across the evaluation modes it tested, relative to running the same sessions without it ^[12]. That range brackets the roughly 46 to 84 percent reductions implied by the illustrative model and the context-editing figure discussed above, arrived at by a completely independent methodology — the kind of convergence that suggests the quadratic-retransmission diagnosis is a structural property of the loop, not an artifact of one stylized calculation.

Predictions, with the observations that would falsify them

These are forecasts, clearly separated from the sourced analysis above. Horizon: 12 August 2028.

One. Automatic context compaction will become a default-on behavior in mainstream agent frameworks and SDKs, rather than an opt-in feature a developer must discover and enable, because the quadratic cost and rate-limit dynamics shown here make an unmanaged long loop expensive enough that shipping the safer default costs a vendor little and saves callers who never read the documentation. Disconfirmed if the leading agent frameworks at the horizon date still ship with automatic compaction or context editing off by default.

Two. At least one major provider will ship a non-preview, production server-side session mechanism that lets a multi-turn tool-using loop continue without the client resending the full prior transcript on every call, rather than only discounting that resend through caching. Disconfirmed if, by the horizon date, every major provider’s primary agent-facing API still requires the calling client to include the full conversation history in every request.

Three. Capability papers that report a task-horizon or long-horizon success-rate number will increasingly report the token or dollar cost of achieving it in the same publication, because this article’s own derivation shows the two are coupled at different rates and a horizon number without a cost number is an incomplete description of what was actually demonstrated. Disconfirmed if influential long-horizon capability papers at the horizon date still report success rate or time horizon with no accompanying cost or token accounting for the trajectories evaluated.

Four. At least one major provider will expose per-trajectory or per-session cost telemetry in its agent console or SDK — a running total tied to a specific multi-step session, distinct from aggregate account-level usage — because the analysis above shows trajectory-level cost behaves qualitatively differently from request-level cost and is worth its own metric. Disconfirmed if, by the horizon date, no major provider’s agent tooling exposes cost broken out per session or trajectory as a first-class figure distinct from raw aggregate token usage.

What to take away

None of the mechanisms in this article are exotic. A stateless API has to be sent everything it needs to know on every call; a fixed tax for tool definitions gets paid whether or not a tool is used; a retry resends what the original attempt already sent; a rate limit is enforced per minute regardless of why the tokens were sent; a context window degrades before it errors. Individually, each of these is a documented, unremarkable fact about how current agent-facing APIs are built. Strung together across a loop of dozens or hundreds of steps, they compose into a cost curve that grows faster than the task it is serving, discounted substantially by caching but not converted into a flat or even linear line. The practical implication is not that long-horizon agents are uneconomical — caching, compaction, and cache-aware rate limits are documented, working mitigations, not proposals — but that a cost estimate built by pricing one step and multiplying by an expected step count will be wrong in a specific, predictable direction, and by an amount that grows with exactly the thing everyone is trying to build more of: how long an agent can usefully keep going on its own.