Why Coding Agents Fail: Long-Horizon Reliability in OpenAI Codex

Long tasks fail through state corruption, weak observations, compounding decisions, and false completion—not merely because the model writes a bad line of code.

Abstract

Coding-agent reliability is often inferred from the duration of successful demonstrations or aggregate benchmark resolution rates. Neither identifies the mechanisms governing a particular long trajectory. This article decomposes long-horizon failure into specification error, observation failure, state corruption, action error, environmental drift, evaluation weakness, and unauthorized transition. It develops conditional and recovery-aware reliability models, distinguishes pass-at-k exploration from pass-to-the-k consistency, and interprets time-horizon measurements without turning trend extrapolation into certainty. OpenAI Codex is used as the primary product case, but the analysis applies to repository agents generally. The engineering conclusion is that dependable horizon comes from state management, discriminating observations, checkpoint placement, independent verification, and bounded authority—not from uninterrupted model persistence alone.

Duration is not reliability

A coding agent that works for seven hours has demonstrated persistence. It has not, by duration alone, demonstrated correctness, recovery, or safe autonomy. A long transcript can represent productive iteration, repeated confusion, or an early mistaken premise carried through hundreds of locally coherent actions.

OpenAI has reported long autonomous Codex runs and multi-million-token demonstrations in its product materials ^[2]. Such cases are evidence that the harness can maintain and execute extended interaction. They are not random samples from a declared task distribution, and they do not estimate the probability that a new repository task will be completed correctly. The distinction between possible horizon and reliable horizon is the starting point for serious evaluation.

Let a task generate a trajectory

\tau=(s_0,o_0,a_0,s_1,\ldots,a_{T-1},s_T),

where $s_t$ is environmental state, $o_t$ the agent’s observation, and $a_t$ its action. Task success is not a property of the final message. It is an externally evaluated predicate $V(s_T,g)$ over final state and goal $g$ . System reliability under task distribution $D$ is

R=\E_{g\sim D}\E_{\tau\sim P(\tau\mid g)} \left[V(s_T,g)\right].

Every term matters. Change the task distribution, model, harness, tools, retry budget, permissions, environment, or evaluator and the reliability estimate changes.

Seven failure families

Long-horizon coding failures can be organized by where the trajectory becomes invalid.

1. Specification failure

The requested goal may be incomplete, internally inconsistent, or phrased at the wrong abstraction. An agent can implement exactly what was written while violating the user’s actual requirement. Longer execution increases the cost of discovering that the target was wrong.

Executable acceptance conditions reduce ambiguity but do not eliminate it. A test can encode the same mistaken specification. Human checkpoints are most valuable before architecture hardens, not only after a polished diff appears.

2. Observation failure

The decisive fact may exist but remain unobserved: a nested instruction file, generated schema, feature flag, hidden call site, platform-specific branch, or production invariant absent from tests. The agent’s belief state then diverges from the repository.

SWE-agent’s results show that the agent-computer interface changes how effectively a model navigates, edits, and executes within repositories ^[6]. That does not make interface design the only bottleneck. It establishes that “the model had access to the repository” is too coarse; access must become an informative observation.

3. State corruption

Long runs accumulate plans, summaries, edits, failed hypotheses, terminal state, and external artifacts. Context compaction or handoff can omit a constraint. A dirty working tree can invalidate an earlier test result. Parallel workers can make individually correct changes against incompatible assumptions.

State corruption is dangerous because the next action can look rational relative to the corrupted state. The model is not necessarily forgetting tokens; the system has lost a sufficient statistic for the task.

Figure 1. State corruption is not a forgotten fact but a moved reference: the patch is still seated and still trusted, so every link routed through it is locally right and globally wrong.Image prompt and art direction by Brecht Corbeel; image generated to that direction.

4. Action failure

The agent can choose the wrong file, form invalid tool arguments, apply an overbroad patch, execute a destructive command, or misunderstand a partial failure. Tool calls introduce a distributed-systems problem: retries are safe only when the operation is idempotent or when prior effects can be observed.

5. Environmental drift

Dependencies, services, branches, generated files, network resources, and tests can change during execution. Local and cloud environments may differ. A test that passes against cached state may fail from a clean build. Reproducibility requires pinning or recording relevant environmental variables rather than assuming a repository commit determines the entire world.

6. Evaluation failure

The system can reach a state that satisfies available checks but not the real goal. Weak tests admit false positives; flaky tests create ambiguous signals; generated tests may reproduce the implementation’s misunderstanding. “All tests pass” means all executed tests passed under one observed environment.

SWE-bench improved realism by applying patches to real repositories and running tests, while its original corpus and later subsets also exposed infrastructure and validation challenges ^[5]. SWE-Lancer added freelance engineering and managerial tasks with real payout values and expert-verified evaluation, yet its authors still reported frontier models failing a majority of the studied tasks at publication ^[7]. Each benchmark samples a different slice of work.

7. Authority failure

An action can be technically correct and unauthorized: reading unrelated data, enabling network access, changing deployment state, or merging without accountable review. Current Codex documentation separates technical sandbox boundaries from approval policy, an important distinction for preventing capability from silently becoming authority ^[3].

Security failures cross categories. AgentDojo evaluates agents using tools over untrusted data and includes hundreds of prompt-injection security cases ^[10]. A malicious observation can induce an action failure and an authority failure simultaneously.

Why stage reliability compounds

Suppose a task requires $n$ materially dependent stages. If $p_i$ is the probability stage $i$ succeeds conditional on all prior required state being correct, then

R_{\mathrm{no\ recovery}}=\prod_{i=1}^{n}p_i.

The equal- $p$ simplification illustrates scale: $0.99^{100}\approx0.366$ . This is not a model of real coding trajectories—the stages are correlated, checkpoints exist, and some errors are reversible—but it defeats one intuition. “Usually correct” local behavior does not imply dependable long execution.

Correlations can make the outcome better or worse. A strong architectural choice can simplify many later stages. A false initial assumption can poison them all. Benchmark averages conceal this path dependence because they aggregate final outcomes across tasks.

Recovery is a chain, not a retry button

An agent can recover only if the system generates a discriminating observation. Let stage $i$ have direct success probability $p_i$ . If failure occurs, let $d_i$ be the probability it is detected, $c_i$ the probability the cause is diagnosed correctly, and $r_i$ the probability repair succeeds without creating another material defect. A simplified effective stage reliability is

p_i^*=p_i+(1-p_i)d_i c_i r_i.

The formula is intentionally conservative and incomplete. Multiple retries, repair costs, correlated defects, and false alarms require richer models. Its value is conceptual: “the agent can iterate” improves reliability only through detection, diagnosis, and repair.

One CI runner node drawn part-way out of its rack on open rails with the lid off and a replacement board held tilted above the empty standoffs, while a node further down the rack shows a front-panel status differing from its neighbours and sits untouched — **Figure 2.** Recovery changes the reliability equation only when an error is observed, diagnosed correctly, and repaired without introducing a new defect.Image prompt and art direction by Brecht Corbeel; image generated to that direction.

This is why ground-truth feedback is central in production-agent guidance. Anthropic describes agents as models acting in a loop with environmental feedback and recommends checkpoints, stopping conditions, sandboxing, and extensive tests for autonomous systems ^[4]. OpenAI’s original Codex launch likewise foregrounded test logs, terminal evidence, and manual validation ^[1].

Retries can also degrade reliability. If the evaluator is noisy, repeated attempts may overfit to the test harness. If a command has side effects, retry can duplicate state. If the agent changes tests to accommodate its patch, it may erase the evidence that exposed the defect. A recovery policy must define which artifacts are mutable and which form the measurement apparatus.

Common-mode failure defeats naive parallelism

The product formula is most misleading when failures share a cause. Let $Z$ represent a latent condition such as an ambiguous requirement, poisoned dependency, missing platform constraint, or flawed test oracle. Conditional on $Z$ , stages or parallel agents may appear independent; marginally, their errors are correlated:

R=\E_Z\left[\prod_{i=1}^{n}p_i(Z)\right].

Running five agents against the same incomplete specification does not create five independent opinions. They can converge on the same wrong interpretation because they share context, model family, tools, and evaluator. Even model diversity may leave the common-mode cause untouched if every run receives the same poisoned observation or passes the same weak test.

This distinction matters for parallel Codex workflows. Worktrees isolate file mutations, not assumptions. Parallel attempts improve exploratory coverage when their prompts, strategies, or evidence channels differ and when a selector can distinguish quality. Parallel workers improve throughput when subtasks are separable and global invariants are explicit. They do not automatically improve epistemic confidence.

A defensible ensemble records sources of diversity. Did agents inspect independent call paths? Did one derive tests from the specification while another implemented? Did a reviewer receive the proposed patch without the generator’s rationale, reducing anchoring? Did validation run in a clean environment? Agreement is strongest when it survives genuinely different observations and methods.

Common-mode analysis also explains why one spectacular demonstration should update beliefs less than many varied deployments. A demonstration conditions on a compatible task, environment, and review path. Reliability engineering seeks performance under perturbation: missing dependency caches, ambiguous reports, unfamiliar repositories, conflicting instructions, flaky services, and adversarial content.

False completion is the characteristic agent failure

A compiler emits an error or returns an artifact. A language agent also emits a narrative about completion. Those channels can disagree.

Define $C$ as the agent’s completion claim and $V$ as external task success. Four states follow:

	$V=1$	$V=0$
$C=1$	correct completion	false completion
$C=0$	unrecognized success	recognized incompletion

False completion is operationally expensive because it transfers discovery to the reviewer. It can arise from missing tests, misread output, stale state, premature stopping, or incentives in the prompt to present a polished result. A reliable harness should make “blocked,” “partially complete,” and “evidence inconclusive” legitimate terminal states.

The completion decision should be computed from an evidence contract where possible. For example: clean checkout, specified command set, zero unexpected diff, required artifact, and reviewer approval. Natural-language confidence can accompany that contract but should not replace it.

Pass@k and pass-to-k point in opposite directions

Code-generation research often reports $\mathrm{pass@}k$ : the probability that at least one of $k$ sampled solutions passes. If independent attempts each succeed with probability $q$ , then

\Pr(\text{at least one success})=1-(1-q)^k.

This is valuable for search: more attempts increase the chance of finding one acceptable candidate. Production consistency asks a different question. If $k$ required deployments must all succeed, the same independence simplification gives

\Pr(\text{all }k\text{ succeed})=q^k.

The first rises with $k$ ; the second falls. $\tau$ -bench introduced a repeated-trial metric in this spirit and reported substantial deterioration across repeated tool-agent-user runs in its evaluated setting ^[9]. Coding-agent reports should state whether multiple trajectories were sampled and selected, or whether one configured system succeeds repeatedly.

A tray of identical drive carriers lying at random angles on a bench with one just tipped into a dock and still rocking against the connector, and behind it a single monitor showing one configuration's runs repeated in a row with the last still executing — **Figure 3.** Exploration and consistency answer opposite questions: pass@k rewards finding at least one success, while repeated deployment asks whether success survives every required run.Image prompt and art direction by Brecht Corbeel; image generated to that direction.

Best-of- $k$ also has a selection problem: an oracle evaluator is unavailable in real work. The system needs a ranking function capable of recognizing the good candidate without hidden benchmark tests. If selection correlates poorly with true quality, additional samples generate review load rather than reliability.

Interpreting the time-horizon result

METR’s time-horizon framework asks a more interpretable question than “how many tasks passed?” It estimates the human completion time associated with tasks a model-agent system completes at 50% probability. The July 2026 revision reports an approximately seven-month historical doubling trend on the studied software tasks and discusses why extrapolation may not generalize to real-world work ^[8].

The result is important and easy to overread.

First, the horizon is a median-probability frontier, not a guarantee that half of arbitrary tasks below a duration threshold are automated. Task selection and scoring matter. Second, human time is a proxy for task difficulty with large variation: a two-hour unfamiliar puzzle and two-hour change in a mature system can demand different context. Third, trends can bend as bottlenecks shift from reasoning to environment access, tacit requirements, or costly verification.

Time-horizon evidence supports a claim that agent systems have improved their ability to sustain useful trajectories on evaluated tasks. It does not establish a date for autonomous software organizations.

Checkpoints should maximize information, not ceremony

Checkpoint placement is a value-of-information problem. Let an unchecked latent error at stage $i$ create expected downstream loss $L_i$ , a check cost $K_i$ , detection probability $d_i$ , and false-alarm cost $F_i$ . A check is favorable when, approximately,

d_i L_i > K_i+F_i.

High-value checkpoints occur before expensive or irreversible transitions: after specification, before schema migration, after dependency changes, before broad refactors, before network or credential escalation, and before merge or deployment. Cheap deterministic checks should run earlier and more often than expensive human review.

Not every step needs approval. Excess prompts train users to approve mechanically. The sandbox should permit routine bounded work; evidence gates should examine properties; human approval should focus on ambiguous intent and consequential authority.

Allocate a reliability budget before execution

For consequential work, “be careful” is not a stopping rule. Let the maximum tolerable probability of material failure be $\epsilon$ . A conservative decomposition assigns budgets $\epsilon_i$ to failure classes or transitions such that

\sum_i \epsilon_i \le \epsilon.

This union-bound allocation can be loose, but it forces explicit questions. How much risk comes from specification ambiguity? How much from an untested platform? What is the permitted probability of credential exposure or irreversible data migration? Which observation would reduce the dominant term?

The budget should be asymmetric. A reversible formatting error and an unauthorized production deletion cannot receive the same threshold. Low-consequence edits can rely on automated checks and rollback. High-consequence transitions require stronger preconditions, independent review, backups, staged rollout, or prohibition.

A git checkout on a small external drive sitting quiet on the bench beside a switch where a braided uplink lead is held a fraction short of its port, the latch not yet sprung, with already-latched ports settled below — **Figure 4.** A reliability budget has to be asymmetric, because the same mistake costs a revert while the change is still local and costs the deployment once it has been pushed.Image prompt and art direction by Brecht Corbeel; image generated to that direction.

A rational agent can stop for at least four reasons: success evidence meets the contract; the next action exceeds authority; expected information gain is below cost; or residual risk remains above budget with no available discriminating check. The last two prevent “keep trying” from becoming an infinite policy. More inference can add value, but it can also create additional edits, test overfitting, and review surface.

Residual uncertainty should be an output. A useful final report states which risk terms were reduced, which remain, and which assumptions were never tested. That report gives a human reviewer a decision problem. A generic confidence statement gives only rhetoric.

A reliability record for every agent run

An auditable long task should leave a compact reliability record:

immutable task statement and acceptance criteria;
model, harness, configuration, reasoning budget, and date;
starting commit and dirty-state hash;
applicable instruction files and environment definition;
granted filesystem, network, credential, and service capabilities;
tool calls with exit status and retained evidence;
checkpoints, failed hypotheses, retries, and rollbacks;
exact tests and independent review performed;
known gaps, skipped checks, and unresolved uncertainty;
final diff or artifact plus accountable acceptance.

This record does not make a result correct. It makes claims about the result inspectable and allows failure analysis to improve the system rather than becoming anecdote.

Reliable horizon is engineered

Better models raise many $p_i$ terms. Longer context can reduce some state loss. More tokens permit wider search. None substitutes for an environment that reveals errors and constrains consequences.

The durable path to longer reliable work combines decomposition without losing global invariants, state that survives compaction and handoff, tools with explicit semantics, clean and reproducible environments, independent checks, recovery-aware control, and authority proportional to evidence. A coding agent should be allowed to run longer when its observations become more discriminating and its rollback remains credible—not merely when it can continue generating actions.

Long-horizon autonomy is therefore not one capability that arrives at a model threshold. It is a reliability architecture. Codex supplies a changing policy and harness within that architecture; the repository, evaluators, permissions, and humans decide whether persistence becomes dependable engineering.