Reliable AI Agents Are Control Systems, Not Chatbots

The category error behind most agent debates

An AI agent is often described as a language model that can use tools. That definition is directionally correct and operationally inadequate. It places the model at the center and relegates everything that determines real-world reliability—state estimation, permissions, feedback, termination, and recovery—to implementation detail. A better abstraction is a stochastic control system with a linguistic policy.

Let the environment have a latent state $s_t$, expose an observation $o_t$, and accept an action $a_t$. The model and its surrounding prompt produce a policy

where $h_t$ is retained working state and $g$ is the task objective. A tool translates the proposed action into an environmental transition; the environment returns evidence; and an evaluator determines whether the evidence supports continuing, replanning, escalating, or stopping. The policy may be a frontier model, but the agent is the entire closed loop.

This framing clarifies Anthropic’s useful distinction between workflows and agents. In a workflow, code predetermines the path through model calls and tools. In an agent, the model dynamically selects its path and tool use. Anthropic’s own engineering guidance recommends beginning with the simplest architecture that works because autonomy commonly exchanges latency and cost for flexibility, while also creating opportunities for compounding error ^[3]. The important axis is therefore not “agentic versus non-agentic” as a product identity. It is how much control authority the stochastic policy receives, under what observations and constraints.

ReAct supplied an early, influential version of the loop by interleaving reasoning traces with actions and environmental observations. Its experiments showed why acting can improve reasoning: external interaction can replace an unsupported internal guess with an observation from a knowledge base or environment ^[7]. Toolformer attacked a neighboring problem at training time, teaching a model when to call simple APIs and how to incorporate their outputs ^[8]. Neither result implies that tool access automatically produces trustworthy agency. Both indicate that model performance changes when computation is coupled to external state.

Codex and Claude Code are harnesses, not merely models

OpenAI Codex and Anthropic Claude Code are best compared at the level of their documented harnesses. A model-only comparison would age quickly and conceal the architectural choices that users actually experience.

OpenAI’s 2025 Codex launch described asynchronous tasks running independently in isolated cloud environments preloaded with a repository. The agent could edit files, run tests, linters, and type checkers, and return terminal evidence for human review ^[1]. Later Codex documentation expanded the surfaces—CLI, IDE, cloud, and code review—and described configurable sandbox and network controls, including network-disabled defaults in relevant modes ^[2]. Those are not cosmetic product features. Isolation limits the transition function available to the agent; logs increase observability; tests instantiate an external evaluator; and review preserves a human authorization boundary.

Claude Code’s documentation describes a related but differently distributed harness operating across terminal, IDE, desktop, and web. It reads a codebase, edits files, executes commands, works with version control, consumes project instructions, and can connect to external systems through MCP ^[4]. Anthropic also emphasizes composable workflows and tool-interface design: prompt chains, routers, parallel workers, orchestrator–worker patterns, and evaluator–optimizer loops are different control topologies, not magical capabilities of a single model call ^[3].

The defensible conclusion is not that one product is universally “more agentic.” Both expose local and remote execution, project context, tools, and verification pathways, with details that continue to change. A serious comparison asks narrower questions:

1. Which resources can the process read, mutate, transmit, or destroy?
2. Which actions require approval, and can approval be scoped by capability rather than session?
3. What evidence accompanies a claimed result?
4. Can execution be reproduced from a clean environment?
5. How are partial failure, retries, and rollback represented?
6. Does the harness preserve provenance when work moves between local and cloud contexts?

These questions survive model releases. A benchmark percentage usually does not.

RAG is an evidence channel, not a memory solution

Retrieval-augmented generation is routinely marketed as “giving the model your data.” The original RAG formulation was more precise: combine parametric memory in a sequence model with non-parametric memory in an external index. A simplified sequence-level expression is

where the retriever assigns probability to documents $z$ and the generator conditions on a selected set $\mathcal{Z}_k(x)$ ^[6]. The decomposition matters because retrieval and generation fail differently.

The retriever can miss the decisive document, return an obsolete version, over-rank lexical resemblance, or collapse distinct entities. The generator can ignore a retrieved passage, combine incompatible passages, or state a conclusion more strongly than the evidence permits. An agent adds a third failure family: it can form the wrong query, stop searching too soon, or act before resolving conflicting evidence. “RAG accuracy” is therefore not one number. At minimum it factors into retrieval coverage, ranking quality, evidence use, citation fidelity, and downstream decision loss.

A production agent needs several forms of state that should not be conflated:

• Working context contains observations needed for the current decision.
• Episodic state records prior actions, outputs, and unresolved commitments.
• Semantic retrieval locates external claims or artifacts relevant to the task.
• Procedural state encodes policies, tool contracts, and organizational rules.
• Audit state is an append-only account of what was attempted and authorized.

Stuffing all five into a vector database does not create memory. It creates an index with unclear semantics. Durable systems assign each state type an owner, retention policy, update rule, and evidentiary status.

Why long tasks fail multiplicatively

Single-turn model quality can hide long-horizon fragility. Suppose a task has $n$ dependent stages and each stage succeeds with conditional probability $p_i$ given that all previous stages succeeded. Then

If one makes the deliberately crude assumption $p_i=p$, a 98% reliable stage repeated 50 times yields $0.98^{50}\approx0.364$. Real agent steps are neither independent nor identically distributed: an early mistake can corrupt later observations, while a test can expose and reverse it. The simple product is useful because it reveals the architecture’s burden. Long tasks require mechanisms that change conditional probabilities after observing evidence, not merely a model with a flattering average.

The empirical warning appears in agent benchmarks. SWE-bench constructs software tasks from real GitHub issues and associated code changes, forcing systems to coordinate edits across repositories rather than complete isolated snippets ^[9]. Its scores are informative, but they remain conditional on a particular harness, task filtering, test infrastructure, and contamination controls. The $\tau$-bench work goes further by evaluating tool-using agents against domain rules and final database states. In its reported experiments, leading function-calling agents completed fewer than half of tasks, and consistency over repeated trials deteriorated sharply; the proposed $\mathrm{pass}^k$ perspective makes repeated reliability visible rather than celebrating a lucky run ^[10].

For a simplified independent model with one-run success probability $q$, the probability of succeeding in every one of $k$ deployments is $q^k$. At $q=0.9$, eight consecutive successes occur with probability only $0.9^8\approx0.43$. Independence is again unrealistic, but the calculation expresses the right operational demand: a bank, compiler, or datacenter does not consume an average demo. It consumes a sequence of particular decisions.

Tool protocols enlarge both capability and attack surface

MCP standardizes communication among hosts, clients, and servers using JSON-RPC, with servers exposing resources, prompts, and tools ^[5]. Standardization can reduce bespoke connector work and make capabilities discoverable. It does not make those capabilities benign.

The specification explicitly treats tool descriptions as potentially untrusted, calls for user consent before data exposure or tool invocation, and emphasizes access controls. That is a profound point. A protocol can describe how an agent calls a tool without deciding whether this agent, for this user, in this task state, should possess that authority. Authentication identifies a principal; authorization constrains an action; policy evaluates context; and sandboxing limits consequences when every preceding layer is wrong.

The correct unit of permission is a capability narrow enough to reason about. “Access GitHub” is too broad. “Read these repositories, create a branch in this repository, and open a draft pull request, but do not merge or alter secrets” is closer. A high-risk action should carry an idempotency key, a precondition, an expected postcondition, and a compensating action where reversal is possible. Tool output must be treated as data, not as trusted instructions merely because it arrived through a registered server.

This is where coding agents offer a favorable proving ground. Compilers, test suites, static analyzers, type systems, and version control provide relatively cheap external feedback. Yet even here, passing tests establish only the properties encoded by those tests. OpenAI’s Codex materials and Anthropic’s agent guidance both preserve human review as a meaningful control rather than claiming that execution evidence eliminates it ^{[1, 3]}.

Ten design rules that survive model turnover

1. Define the terminal condition before the prompt

A task is not “done” when the model says it is done. Define observable postconditions: tests pass from a clean checkout, a database invariant holds, a document contains cited support, or a human approves a diff. If completion cannot be measured, autonomy cannot be bounded.

2. Separate proposing from committing

Let the model propose an action in a typed representation. A policy layer should validate schema, identity, scope, cost, and preconditions before an executor commits it. This creates a place for deterministic controls that cannot be argued away in natural language.

3. Make observations falsifiable

“The change works” is a claim. A test log, rendered screenshot, query result, or reproducible trace is evidence. Prefer observations another process can independently regenerate. Preserve negative evidence and failed attempts; deleting them makes the audit trail narratively clean and epistemically weak.

4. Budget the horizon

Set maximum steps, wall time, tokens, tool cost, and irreversible actions. A budget is not only financial. It bounds how far a mistaken internal model can push the environment before control returns to an evaluator.

5. Retrieve for decisions, not for decoration

Every retrieval should answer a decision-relevant question. Record the query, corpus version, selected passages, and the action they support. Measure whether decisive evidence was retrieved, not merely whether retrieved text sounded related.

6. Give tools narrow, typed semantics

Tools should expose explicit parameters, defaults, side effects, and errors. Distinguish read, propose, write, publish, and delete capabilities. Anthropic reports that tool-interface work can dominate prompt work because seemingly minor interface choices alter model error rates ^[3].

7. Place verification before expensive or irreversible transitions

Run cheap checks early: syntax before integration tests, a query plan before a migration, a dry run before a deployment. The evaluator should be organizationally and technically distinct enough that it does not merely repeat the generator’s assumptions.

8. Design rollback as part of the action

Record prior state, transaction boundaries, artifact hashes, and compensating operations. Some actions—sending confidential data or publishing a defamatory claim—are not meaningfully reversible, so the architecture must prevent rather than compensate for them.

9. Evaluate distributions and slices

Report repeated trials, confidence intervals, cost, latency, and failure categories. Slice by task length, repository familiarity, tool count, permission level, and adversarial content. A global mean can improve while the high-consequence tail gets worse.

10. Keep the human boundary explicit

Human oversight is not a ceremonial click after the agent has made reversal impractical. Specify what information the reviewer receives, which decisions remain open, and what happens when the reviewer is absent. Escalation is a first-class terminal state, not a failure to be hidden.

A risk-adjusted objective for agents

Optimizing task success alone invites systems to spend unlimited resources or take unacceptable risks. A more useful objective is

where $R$ is task reward, $C$ is computational and human-review cost, $L$ is realized loss from harmful actions, and $U$ is residual uncertainty at commitment. The coefficients are governance choices, not model parameters. Hospitals, game studios, semiconductor fabs, and personal coding projects should not assign them equally.

This formulation also explains why “use the strongest model everywhere” is rarely optimal. A deterministic parser may dominate a model for schema validation. A small model may route obvious requests. A frontier model may plan an ambiguous change. A test runner adjudicates executable behavior. A person authorizes a production migration. Reliability emerges from heterogeneous checks, not model monoculture.

What is likely to change—and what is not

Three forecasts appear robust enough to state conditionally.

First, coding agents will become less identifiable with a single interface. Codex already spans cloud, terminal, IDE, and review surfaces, while Claude Code documents a similar spread ^{[2, 4]}. Work will move among synchronous pairing, asynchronous delegation, and automated review. The hard problem will be preserving identity, state, permissions, and provenance across those transitions.

Second, model capability will commoditize portions of orchestration but increase the value of verified context. Better models can decide when to search or which tool to call, yet they cannot infer an unavailable contract, recover a deleted audit log, or know that a stale document was superseded unless the surrounding information system represents those facts.

Third, agent evaluation will move toward stateful, adversarial, repeated deployment. Static question sets are too easy to overfit and too distant from consequences. Benchmarks resembling SWE-bench and $\tau$-bench point toward executable tasks, policy constraints, final-state checks, and repeated-trial consistency ^{[9, 10]}. The strongest future result will not be “the agent sounded expert.” It will be “the system reached the correct state, within budget, under perturbation, with an audit trail and no unauthorized transition.”

That is a less anthropomorphic vision of AI agency, and a more ambitious engineering one. The model supplies adaptive inference. The system supplies evidence, authority, memory, and restraint. Only the combination deserves to act.