The Hardest Unsolved Problems in AI Agent Architecture

The difference between “hard” and “still open”

Every few months a new agent framework claims to have solved planning, memory, or coordination — and every few months a paper’s own limitations section, or a piece of official SDK documentation, quietly admits the opposite. This article is built from the second kind of evidence only. It does not argue from first principles that some property of agents should be difficult; it collects the specific place, in each case, where the people who built the system in question wrote down that the problem is not solved, and describes what they did instead.

Five problems recur across that literature: assigning credit for a good or bad outcome to the right decision, when many decisions separated the outcome from its cause; keeping a working record of a long task without either truncating it or drowning in it; coordinating several agents on a shared goal without the coordination overhead itself becoming the bottleneck; checking that a proposed action is safe before it becomes unrecoverable; and detecting that a sub-task has failed when nothing in the system is designed to say so. None of these are new observations dressed up as agent-era problems — planning under partial observability has been formally difficult since long before large language models existed ^[4] — but each has a distinct, current, and separately documented failure mode inside agent architectures specifically, and that is the level at which this article stays.

The scale at which the question now matters is itself measured, not assumed. METR’s benchmark of long software tasks defines a 50%-success time horizon — the length of task, measured in the time a skilled human would need, that a model can complete with even odds — and reports that frontier models such as Claude 3.7 Sonnet stood at roughly fifty minutes on that measure, a figure the study’s authors say has been doubling approximately every seven months since 2019 ^[1]. The researchers are explicit that this is a measured trend with real limits on how far it can be trusted to generalize; the paper states its own concerns about external validity, since the tasks tested are a specific, curated set of software problems rather than the full range of things an agent might be asked to do. Read narrowly, the result says only that reliability over longer horizons is improving. It does not say why long horizons are hard in the first place, or what specifically breaks when a task outruns a system’s current time horizon. The five sections below are about that second question.

Credit assignment breaks down when the horizon gets long

Reinforcement learning’s standard tool for turning a sequence of actions and a single delayed reward into a training signal is the policy gradient, and its textbook reward-to-go form is

Read the inner sum literally: every action in the trajectory is credited with everything that happens from that point on, not with its own specific contribution. When $T$ is small and rewards are dense, that crude attribution washes out quickly. When $T$ is large and the reward is a single terminal signal — a multi-step coding task that either compiles and passes its tests or does not, a multi-turn support conversation that either resolves the case or does not — the same sum assigns identical credit to a decisive action, a neutral one, and a mistake that happened to be recovered from later. The variance of the resulting gradient estimate grows with the length of the trajectory, and no architectural trick changes that arithmetic; it only changes how the trajectory is chunked before credit is assigned.

This is not a hypothetical concern read into recent work; it is the stated motivation for it. Zhou and colleagues introduce ArCHer specifically because, in their account, single-turn reinforcement-learning methods “cannot endow LLMs with the ability to intelligently seek information over multiple turns, perform credit assignment, or reason about their past actions” ^[2]. Their fix is architectural rather than a claim that the underlying problem is closed: a hierarchical value function aggregates reward at the level of a full conversational turn and only then distributes gradient signal to the tokens inside that turn, which the paper reports as roughly a hundredfold improvement in sample efficiency over on-policy alternatives — a substantial practical gain, and, by construction, still an approximation that assigns credit at the turn boundary rather than at the level of the specific decision inside it.

A separate line of work avoids gradient-based training and instead asks the model to reflect on its own trajectory in natural language. Reflexion converts environment feedback into a verbal self-critique that is stored and consulted on the next attempt at the same or a similar task ^[3]. That sidesteps the variance problem above, since there is no gradient to have high variance, but it introduces a different one: the paper’s own account restricts long-term memory to a bounded sliding window of past reflections, and it flags that the underlying optimization can still settle into non-optimal local minima. Verbal reinforcement has not eliminated the credit-assignment problem; it has moved it from a numerical gradient into a piece of text a language model has to write well, with no architectural guarantee that it does.

State management: memory is not the same as context

A useful decomposition for what an agent needs to track is the belief it holds over the true state of its task, updated as new observations arrive. In the classical formalism, if $b(s)$ is the agent’s probability distribution over possible states before an observation, $a$ is the action just taken, $o$ is the observation that followed, $T$ is the environment’s transition model and $O$ its observation model, the updated belief is

with $\eta$ a normalizing constant. Kaelbling, Littman and Cassandra’s original treatment of this update is also where the difficulty is made explicit: they show that the amount of memory an optimal policy needs is not bounded in advance by the size of the problem, and that reducing the reliability of a single observation channel in their own worked example forces a much larger plan graph, and a correspondingly larger memory requirement, just to stay confident ^[4]. Applied to a long-running LLM agent, $s$ is everything true about the task — every file changed, every external call made, every fact established — and $b$ is whatever the agent’s context window currently represents. The update above is exact only if the full state and its full history are kept; an agent cannot keep them, because the context window is finite and every token inside it costs money and attention.

The two documented responses to this are both compressions, not solutions. Packer and colleagues’ MemGPT frames the problem as a systems one and borrows directly from operating-system virtual memory: content is moved between a small fast context and a larger slow store, with the model itself issuing something like a page fault to pull older material back in when it is needed ^[5]. That is a genuinely useful mechanism, and it is explicitly a management layer around a limit that has not gone away — the underlying context window is exactly as small as it was before. Anthropic’s own current documentation for context compaction describes the same trade-off from the vendor side: when a long-running conversation nears its context limit, older content is replaced with a generated summary, and the platform’s reference material recommends the mechanism specifically because it “handles context management automatically, without client-side summarization code” — not because the summary is lossless. The same documentation set separately walks through how to preserve specific recent messages verbatim precisely because the default summarization step does not preserve them by default ^[6]. A summary is a bet about which details will matter later, made before it is known which details those are. Lose that bet on a long task and the agent does not fail loudly; it proceeds confidently from a belief state that has quietly diverged from the true one.

Multi-agent coordination cost grows faster than the group

For $n$ agents that each need a channel to every other agent, the number of pairwise links a fully connected coordination topology requires is

which grows quadratically in $n$ even before accounting for the content sent over each link. That is a modeling assumption, not a law of nature — a system that routes all communication through a single orchestrator instead of a full mesh pays a linear cost in links, at the price of a central bottleneck and a single point of failure — but every real multi-agent framework has to choose a point on that trade-off, and reasonable designers disagree about where: full-mesh topologies keep no single node load-bearing but scale their wiring badly; orchestrated topologies scale their wiring well but concentrate risk in the orchestrator. Neither choice dominates the other in general, which is itself evidence that this is an open design question rather than a solved one.

The empirical picture is more discouraging than the combinatorics alone would suggest, because added agents do not only cost bandwidth; they cost correctness. Cemri and colleagues built the first systematically annotated failure taxonomy for multi-agent LLM systems from more than 1,600 traces across seven popular frameworks, reaching a Cohen’s kappa of 0.88 between six expert human annotators — a level of agreement that makes the resulting numbers hard to dismiss as noisy labeling ^[7]. Fourteen distinct failure modes fall into three categories: specification issues, inter-agent misalignment, and task verification. The paper’s framing of the second category is the coordination-specific one: agents talk past each other, restate rather than build on a peer’s partial results, or act on information a peer had already superseded. Despite the growing enthusiasm for multi-agent systems, the paper’s headline observation is that their measured performance gains over comparable single-agent baselines remain minimal on the benchmarks studied. That is a striking asymmetry: the wiring cost of adding agents is easy to state formally, and it is quadratic; the benefit of adding them is empirically small and inconsistent across the frameworks tested. The paper states plainly that the failures it identifies require more sophisticated solutions than currently exist — a stronger claim than saying the field simply has not gotten around to it yet.

Verifying an action before it cannot be undone

Somewhere in almost every serious agent deployment there is a moment where the agent proposes an action that cannot be cleanly reversed — a financial transfer, a production deployment, a message sent to a real person, a file deleted from the only copy — and the documented response, across both a frontier lab’s policy research and a shipping SDK’s engineering documentation, is the same one: pause and ask a human. That convergence is worth taking seriously as evidence, but it is evidence for a workaround, not for a solved verification problem.

OpenAI’s white paper on governing agentic systems states the reasoning plainly: “Some decisions may be too important for users to delegate to agents, if there is even a small chance that they’re done wrong (such as independently initiating an irreversible large financial transaction),” and it names requiring proactive user authorization — keeping a human in the loop — “a standard way to limit egregious failures of agentic AI systems” ^[9]. The same document immediately qualifies its own recommendation rather than resting on it: a system deployer has no reliable way to ensure the approving human actually has enough context to understand what they are approving, and the problem gets worse, not better, as the number of approvals grows, because a user forced to approve many decisions quickly has correspondingly less attention to give any single one. A safeguard whose reliability degrades with the volume of decisions it is asked to gate is not a general solution to verifying irreversible actions; it is a rate-limited one.

The engineering side of the same convergence shows up in OpenAI’s Agents SDK documentation, which draws a working distinction between guardrails that validate input, output, or a tool call automatically, and human review that pauses a run so a person or a policy can approve or reject a specific sensitive action before it executes ^[10]. The documentation is explicit that this protection is not ambient: guardrail coverage attaches only at specific points in an agent chain by default, and the stated guidance for complex workflows is to place validation next to the specific tool that produces the side effect, rather than relying on a chain-level check, because an application does not automatically inherit review coverage — it has to be added, deliberately, tool by tool, to the harness being built. That is a documented engineering practice, not a mechanism that verifies an action’s safety on its own terms; it verifies that a human or a policy function was consulted, which is a weaker and different guarantee than proving the action itself is safe.

When a sub-task fails without saying so

The failure mode that should worry an operator most is not the one that stops the agent — a crashed process or a raised exception is at least visible — it is the one that lets the agent continue, confidently, on top of a sub-task that quietly did not do what it claimed. Cemri and colleagues’ taxonomy gives this its own category, task verification, with its own numbers rather than a general impression: across the annotated traces, 8.2% of all recorded failures were coded specifically as “no or incomplete verification,” described in the paper as a failure mode that allows “errors or inconsistencies to propagate undetected,” with a further 9.1% coded as verification that was attempted but wrong, and 6.2% as a task or exchange ended before it was actually finished. Together, the three verification-related failure modes the paper defines account for close to a quarter of every failure observed in the dataset ^[7]. That is not a tail case; it is one of the largest single categories the taxonomy identifies, larger than most of the specification-level failures that dominate casual discussion of prompt design.

Kapoor and colleagues’ separate critique of agent benchmarking practice explains why this category is so persistent and so hard to price into a deployment decision: they document that many widely used agent benchmarks have inadequate holdout sets, sometimes none at all, which lets an agent that has learned to look successful against a specific benchmark’s checking logic pass evaluation while still failing the underlying task in ways that logic does not catch ^[8]. A verification step that is itself gameable is not a verification step in the sense a real deployment needs; it is a second thing that can fail silently alongside the first. The practical consequence for long-running or multi-stage agent work is that a silently wrong sub-task result does not stay contained. It becomes an input to whatever runs next, and every downstream step inherits a false premise with no signal that it should not.

The shared root: a belief state nobody can fully check

Read back across the five problems and a single structural fact keeps reappearing under different names: an agent acts on a belief about the state of its task, that belief is necessarily an approximation of a much larger true state, and nothing in current architectures gives a cheap, general way to check the approximation against the truth before acting on it. Credit assignment fails because the belief about which past action caused the present outcome is itself inferred rather than observed, and the inference gets noisier the more steps separate cause from effect. State management fails because keeping the belief exactly current would require keeping the full history, which a bounded context window cannot do. Coordination overhead is what it costs to keep several agents’ separate beliefs about a shared task close enough to each other to stay useful. Irreversible-action verification is an attempt to insert a check on the belief at exactly the one moment it matters most, through the one channel — a human — that is not itself part of the same possibly mistaken system. And silent sub-task failure is what happens once the belief and the truth have already diverged and nothing in the architecture is positioned to notice.

None of this is a discovery specific to language-model agents. Kaelbling, Littman and Cassandra were writing about a robot navigating a building with an unreliable sensor, not a coding agent, when they showed that maintaining an exact belief state under partial observability has no general shortcut, and that the memory an optimal policy needs is not bounded by the size of the problem in any way that can be computed in advance ^[4]. What has changed is the scale at which the problem is now being deployed commercially, faster than the underlying theory has produced a general answer for it.

What is actually deployed today is mitigation, not resolution

It is worth being precise about what the industry’s current responses actually are, because none of the sections above describes a technique that removes its problem; each describes a technique that narrows the conditions under which the problem causes visible damage.

Hierarchical value functions reduce the variance of credit assignment by aggregating reward at a coarser grain, at the cost of assigning that coarser credit less precisely inside each aggregate ^[2]. Context compaction and virtual memory management extend the effective horizon over which an agent can operate coherently, at the cost of a summarization step that is a genuine, acknowledged loss of information rather than a lossless compression ^{[5, 6]}. Constraining which agents talk to which, or routing communication through a central orchestrator, reduces coordination cost at the price of the bottleneck and single point of failure that centralization always creates ^[7]. Human approval gates reduce the rate of catastrophic irreversible mistakes at the price of approval fatigue that grows with exactly the volume of decisions the system is meant to handle ^{[9, 10]}. And task-level verification checks catch some silent failures at the price of needing their own, separately trustworthy verification logic, itself subject to the same benchmark-gaming and overfitting problems documented in agent evaluation generally ^[8].

This pattern is worth naming because it changes what a reasonable buyer or builder of agent systems should ask for. The question is never whether a given problem has been solved, because by the field’s own account none of them have. The useful question is narrower: at what point does this particular mitigation’s known failure mode become likely enough, in this specific deployment, to matter, and what observable signal would say it had happened.

Predictions, with the observations that would falsify them

These are forecasts, kept explicitly separate from the sourced analysis above. Horizon: 12 August 2029.

One. Hierarchical or multi-level credit assignment — aggregating reward at the level of a turn, a sub-task, or a tool call rather than a single token — becomes the default training approach for agentic reinforcement learning rather than one option among several. Assumption: the sample-efficiency gains reported for turn-level methods generalize beyond the tasks in which they were first demonstrated ^[2]. Observable indicator: training methodology sections of published agent system cards describe turn- or task-level reward aggregation as standard practice. Disconfirmed if leading agent labs are still training primarily against flat, single-level trajectory returns in 2029.

Two. Context management consolidates around vendor-provided, server-side mechanisms — compaction, external memory stores, retrieval — rather than around bespoke summarization logic built separately by individual application teams, because the failure modes of ad hoc summarization are already visible enough that platforms are absorbing the responsibility ^{[6, 5]}. Observable indicator: major agent SDKs ship context management as an on-by-default platform feature rather than a documented pattern applications must implement themselves. Disconfirmed if the leading agent SDKs still expect most application teams to write their own context-truncation logic by default in 2029.

Three. Approval fatigue, not model capability, is identified as the binding constraint on how much irreversible authority organizations grant to agents, and drives investment toward risk-tiered approval — fast, cheap checks on low-stakes actions and slower, better-resourced human review reserved for genuinely high-stakes ones — rather than uniform approval-for-everything policies ^[9]. Observable indicator: published enterprise agent-governance guidance recommends differentiated approval thresholds keyed to action reversibility rather than a single uniform gate. Disconfirmed if published guidance in 2029 still recommends one uniform approval threshold regardless of an action’s reversibility or stakes.

What to take away

Take the five problems as a checklist for evaluating a claim, not as a list of things to wait for someone else to fix. An architecture that proposes to handle long-horizon tasks should be asked, specifically, how it assigns credit across steps, not whether it is described as agentic. A memory system should be asked what it discards under pressure and how that discard is decided, not whether it advertises a large context window. A multi-agent design should be asked what its coordination topology costs in messages and failure modes as agents are added, not how many agents appear in its diagram. Any system with access to an irreversible action should be asked what specifically stands between a proposal and its execution, and what happens when that check fires many times a minute. And any long-running pipeline should be asked how a silently wrong intermediate result would be noticed — because by the field’s own evidence, in roughly one failure in five, it currently would not be.