AI Feeds on the Distance Between an Intention and an Outcome

A request can be incomplete without being useless

“Fix the import regression” is not a small program. It is an instruction that leaves a live cavity between a person’s intention and a state of the world that can be checked. The cavity contains decisions: which compatibility promise matters, whether the nearby warning is part of the defect, which code path can change, which test is evidence, which collateral behavior must remain untouched, and when a patch is finished rather than merely plausible. A good engineer moves through that cavity. An agent moves through it too, but only if its tools, budget, and feedback let it convert an underspecified sentence into a verifiable terminal state.

That is the resource I mean by an intention gap. It is not consciousness, creativity, or a claim that a language model has needs. It is a feature of a task relation: an initial request states less than the completion condition requires, and someone—or something—must supply the missing decisions. The question is whether that feature explains agent performance better than the usual pile of task properties. If it does, an agent has something like a habitat. If it does not, “niche” is decorative ecology draped over a leaderboard.

Repository repair is unusually good terrain for this question because it can separate the two ends of the relation. SWE-bench pairs a repository and issue description with historically corresponding pull-request evidence; its original release contains 2,294 software-engineering problems from real GitHub issues across twelve Python repositories ^[1]. Defects4J likewise made controlled testing possible by curating real faults with reproducible outcomes ^[14]. More recent work has made the endpoint more operational: SWE-Lancer describes containerized, end-to-end checked freelance tasks, while Terminal-Bench describes terminal tasks with an instruction, test script, oracle, and sandboxed harness ^{[2, 3]}. Those artifacts do not solve the intention problem. They make it possible to hold the outcome still while changing what is said at the start.

The proposal in this article is deliberately narrower than a new benchmark announcement. It is a fifty-task experiment that has not been assembled or run. No repository has been selected, no hidden test has been executed, no human or model has made an attempt, and there is no success curve behind any of the figures. Fifty is a planned sample size, not a result dressed in future tense. The argument is that a result would be worth believing only if the missing-intention variable is defined before anyone sees it and can lose a contest against ordinary difficulty.

There is a practical reason to care. Evaluations already distinguish several things that an average score collapses. RepoBench separates repository retrieval, completion, and a combined pipeline ^[13]. AgentBench spreads multi-turn behavior across eight environments rather than calling one response a general agent ^[7]. WebArena and OSWorld make final computer state, not eloquence, part of the score ^{[8, 9]}. AppWorld explicitly allows different valid paths to a tested state while checking for collateral changes ^[10]. The missing variable is not another kind of accuracy. It asks why two tasks with an equally hard final test can place radically different burdens on the interpreter of the first sentence.

That distinction also prevents an easy category error. A hidden test is an evaluator’s compact representation of a desired terminal state; it is not a transcript of every reason a maintainer had for wanting that state. A ticket is an imperfect public representation of that intention; it is not automatically a permission slip for every patch that happens to satisfy one assertion. The proposed experiment uses the hidden predicate because it needs an outcome that can be applied identically to every attempt. It uses blinded human completion-class judgments because it needs a representation of what the request made reasonable before the evaluator revealed its private criterion. Neither side is treated as the complete truth. Their distance is the object under investigation.

This makes false positives expensive in the right way. Suppose a redacted request causes every annotator to invent ten fanciful implementation routes, but each route is plainly inconsistent with the repository’s public interfaces. They do not enter $\mathcal{C}_{tr}$. Suppose another request has only two plausible repair stories, but choosing between them changes a promise on which distant callers depend. They remain separate classes even if their patches share most lines. The annotation manual therefore needs positive examples, counterexamples, and a rule that every class name records the observable behavior and scope commitment that distinguish it. It must be frozen before an agent or a human is scored. Otherwise the panel will learn which ambiguity was useful to a particular system and retroactively call that the intention gap.

The study should publish the annotation disagreements, class merges, excluded tasks, and the redaction audit alongside any result. A confidence interval around a success percentage cannot repair an unstable upstream construct. In this design, an unmeasurable gap is a legitimate negative outcome: some requests may be vague in ordinary language yet too context-bound for a reliable finite partition. That failure is preferable to pretending that every unspoken norm can be converted into a number.

Do not count source code; count permissible stories about the request

The phrase “entropy of acceptable completions” can go wrong in three opposite ways. One can count all strings that compile, which makes a typing accident look vast. One can count only the maintainer’s historical patch, which pretends history wrote the only legitimate repair. Or one can infer the hidden tests and call their acceptance set the requester’s intention, which reverses the point: the agent does not see the hidden test.

The relevant set is smaller and stranger. For task $t$ at redaction level $r$, let $\mathcal{C}_{tr}$ be a finite partition of request-compatible completion classes. A class is a materially different, reasonable story a competent reader could tell about what the request permits: repair only the visible failure; repair the visible failure and preserve a compatibility convention; change a local abstraction; decline an unsafe change because the request is too ambiguous. Classes are defined by externally reviewable behavioral and scope differences, not by textual patch distance. Two patches that implement the same behavioral and scope story belong to one class. Two patches that both pass a visible test but make different compatibility promises do not.

This is an epistemic object, not the true set of hidden-test-passing patches. An annotation panel would see the frozen repository snapshot and the redacted request, but not the hidden tests or the historical patch. Before any model run, independent experienced maintainers would propose completion classes, reconcile synonymous proposals through a blinded adjudication protocol, and then estimate the distribution $\hat p(c\mid t,r)$ by having fresh annotators assign each request to the classes they judge compatible. Their uncertainty is part of the measure. If they cannot create a stable class partition or agree on membership, the task does not produce a trustworthy gap estimate; it does not become a convenient hard example.

The first factor is Shannon entropy:

Its unit is bits. A request that commits competent readers to one completion class has $H_{tr}=0$ bits. A request that leaves two equally plausible classes alive has one bit. This does not imply that either task takes one unit of labor. It says only how much uncertainty about the reasonable completion story remains after the initial request is read. The estimator should carry a resampling interval across annotators and adjudication choices; a point estimate without its classification fragility would make the most subjective part of the design look falsely exact.

Then comes the factor most benchmark descriptions hide inside “multi-step.” Let $K_t$ be the number of consequential state transitions required to go from the fresh container to the terminal predicate. It is not clicks, tokens, shell commands, or internal thoughts. A transition counts only when it changes a repository-relevant state that must be deliberately repaired, preserved, or validated before the hidden predicate can be attempted: for example, locating the affected behavior, altering a coupled implementation boundary, updating a declared behavior, or establishing a verification state. A pre-registered task graph, built from the frozen task package and reviewed by raters blind to outcomes, collapses mechanical command sequences into one transition and splits semantically independent commitments into more than one. Disagreement about $K_t$ is reported as measurement uncertainty, not settled by the most impatient rater.

I call the product intention-gap volume:

The word “volume” is deliberately qualified. These are constructed units, not cubic meters and not an intrinsic property of a repository. They make two axes do work at once: how many plausible completion stories the request supports, and how many consequential points at which the agent must turn a story into an irreversible or checkable change. Multiplying is appropriate only if the transition count is fixed across redactions and the entropy is measured from the request, not from the model’s successful patch. That is why the endpoint stays fixed and annotation happens before attempts begin.

The limiting cases are a useful sanity check. If a detailed request uniquely identifies a class, $H_{tr}=0$, then $G_{tr}=0$ even if the repair has many transitions. That task can be long yet not intention-wide. If a terse request admits many reasonable classes but only one operational change, $K_t=1$, then the gap is broad but shallow. If there is no identifiable path to an externally checkable state, $K_t$ is undefined rather than enormous; the task is unsuitable for this study. And if redaction merely removes irrelevant prose, the annotation distribution should not change much. A metric that rises whenever words disappear has measured prompt length, not intention.

The analogy breaks here. Biological niches have energy flow, reproduction, competitors, and history. This construction has task conditions, budgets, and a performance boundary. It borrows the word “niche” only after declaring its coordinates. It does not make an agent an organism; it asks whether agent competence occupies a region rather than a single ranking.

Fifty sealed rooms and one door in each

The proposed object is fifty containerized repository-repair tasks. Each task would have a frozen repository commit, a reproducible execution image, a fixed hidden-test predicate $A_t(x)\in\{0,1\}$, and a retained canonical historical resolution kept outside the agent and annotation contexts. The canonical resolution is evidence for task construction, not the grading oracle: an alternative patch may be valid if it reaches the fixed test predicate without violating predeclared collateral checks. The study would never equate source-code similarity with success.

The sampling frame has five predeclared ten-task strata spanning modest and extended repair horizons, single- and multi-file repositories, and cases where public issue language is respectively specific or sparse. These are design constraints, not claims that fifty examples represent all software work. Each candidate must first survive an oracle reproducibility check across independent container resets. It must also admit a clear collateral-damage test, so a patch cannot win by deleting an inconvenient behavior. A task whose fixed endpoint flakes, whose public history reveals the hidden test, or whose redaction removes all actionable meaning is rejected and its replacement rule is recorded before scoring begins.

Each retained task receives five request conditions. At $r=0$, the evaluator supplies the historical issue description cleaned only of direct links to the canonical patch and grading artifacts. At $r=1$, implementation hints such as named files or a quoted diagnostic are removed. At $r=2$, illustrative examples and narrow scope clues are removed. At $r=3$, local symptom detail is replaced by the more general duty it expresses. At $r=4$, only a minimally actionable problem statement remains. The exact transformation rules are written before selection, applied by a different editor from the task selector, and audited by maintainers who do not see system outcomes. The same snapshot, visible tools, time ceiling, cost ceiling, and hidden tests apply at all five levels.

The sequence matters. A word-count ablation is not a redaction ladder. It would reward accidental brevity and leave the causal object undefined. Each removal must correspond to an auditable cue class—implementation location, diagnosis, behavioral example, scope boundary, or priority rule—and the article’s hypothesis concerns the completion alternatives that remain plausible after that cue is gone. A removed clue can reduce token length while leaving $H_{tr}$ unchanged. Conversely, one short phrase such as “without breaking legacy callers” can collapse several completion classes while adding very few tokens.

The public precedent for keeping endpoint and action path separate is stronger than it first appears. OSWorld packages environment setup and execution-based evaluation rather than requiring one demonstration path ^[9]. AppWorld’s unit tests allow different completions but inspect unexpected state changes ^[10]. SWE-Lancer uses end-to-end tests for independent engineering tasks, and RE-Bench releases environments, human data, and agent trajectories rather than treating a final scalar as self-explanatory ^{[2, 6]}. None of those projects estimates request-compatible alternatives. They establish that fixed, inspectable terminal conditions are a practical premise rather than a fantasy.

The design does not put all pressure on a model. For every task condition, a preregistered cohort of novice programmers, expert maintainers, and agent configurations would work under matched access to the container and the same visible documentation. “Novice” and “expert” must be operationally defined by experience criteria established before recruitment, not by who happens to pass. Their active completion time, abstentions, collateral changes, and final predicate are logged. This comparison is not a claim that people and agents reason identically. It supplies a control against the easy story that redaction only changes the benchmark author’s expectations.

There are hard boundaries. Agents do not receive network search, issue history, the canonical pull request, or hidden-test output unless that channel is intentionally added as an experimental factor. Humans receive no privileged maintainer conversation. Any copied training contamination cannot be eliminated by prose alone, so task provenance, repository age, and public-patch exposure must be recorded and sensitivity analyses must remove suspected contaminated tasks. A positive result after such exclusions would still describe this test suite, not all labor a programmer might do.

A volume becomes a niche only after it draws a boundary

An intention gap is a task measurement. An agent niche is a performance region conditioned on a budget. Let $h_t$ denote the independently measured human task horizon in active minutes, not the number of agent tool calls. This follows the useful discipline in HCAST and the time-horizon work: human duration can calibrate a task scale, although it is not a complete measure of what the task means ^{[5, 4]}. Keep $h_t$ separate from $K_t$. A task can have many consequential transitions and still take little expert time; a task can take hours because of a slow feedback loop while making few intention choices.

For system $m$, define a budget

where the components are maximum direct monetary expenditure in U.S. dollars, maximum supplied input-token context, and maximum elapsed minutes. These are intentionally not converted into one fictional currency. A system that buys more attempts, receives more context, or waits longer has changed its environment. Let $Y=1$ mean that an attempt reaches $A_t(x)=1$ without a predeclared collateral violation. The niche is the set

where $p^*$ is a preregistered practical reliability threshold and the lower confidence bound is clustered by task, repository, and repeated attempt. The region lives in intention-gap/horizon space: $g$ is in bit-transitions and $h$ in active human minutes. It is not an ontological home for a machine. It is a reproducible statement of where that machine clears a reliability bar under one budget.

To compare systems without pretending that this irregular region is a Euclidean animal range, bin the preregistered plane into cells and define width as

The unit is weighted grid cells, stated with the binning rule. Report its depth too: the greatest gap or horizon cell satisfying the reliability rule. A broad, shallow system has a large $W_m$ but low maximum depth; a narrow, deep specialist may have small width yet persist at large $g$ or $h$. Neither is “more intelligent” by definition. The map makes a tradeoff inspectable.

That prediction is proposed, not observed. A code agent built around repository search and rapid test–patch iteration might occupy a narrow but deep strip. A general desktop agent might touch many small-gap tasks but lose reliability rapidly as both gap and horizon rise. An expert human may show a different contour because partial intention is reconstructed from experience, organizational norms, and an ability to ask questions that the sealed test intentionally removes. The point of plotting all three is to make those possibilities vulnerable to the same threshold.

The tool factor gives the map a second chance to be meaningful. Add one predeclared tool channel—a repository search index, a test-output view, or a static dependency browser—without changing the request or endpoint. The prediction is not “tools improve agents.” It is that the boundary of $\mathcal{N}_m(B)$ should move in a pattern tied to a named bottleneck: a search tool may expand deep multi-file conditions; a richer diagnostic may compress the apparent gap by eliminating false completion classes. If every tool merely shifts all cells uniformly, the language of niches adds very little to a simple resource-performance curve.

This also blocks an institutional sleight of hand. Ticketing systems, service-level agreements, templates, runbooks, and acceptance tests can make work more or less machine-legible. It is tempting to say that organizations “manufacture gaps for machines.” The study cannot establish that broad social claim. At most, it can test a sharper prediction: when a template removes the same cue class that annotation says collapses completion entropy, a system’s estimated niche boundary moves. If it does, a future field study can ask who benefits from writing work in that format and who bears the cost of intentions that were never represented.

The metric has to beat difficulty in a hostile tournament

Every component of the new measure can be accused of being ordinary difficulty with a dramatic name. That is the strongest objection, and it deserves to be the primary statistical comparison rather than a final paragraph of caution.

Start with the conventional mixed-effects baseline specified before task results are opened:

Here $q_{tr}$ is supplied input-context length in tokens, not tokens consumed after the agent has begun acting; using post-attempt expenditure would contaminate the predictor with behavior. The term $d_t$ is a predeclared ordinary benchmark-difficulty score based only on endpoint-side, redaction-invariant features such as repository size band, static dependency reach, test-suite scope, and selected task-family label. It is deliberately conventional and admittedly imperfect. The terms $u_{\mathrm{repo}(t)}$ and $u_t$ are repository and task random effects, while $\alpha_m$ represents model or agent-configuration identity. The baseline includes model identity, token length, ordinary difficulty, and task horizon exactly so that the new variable cannot win by repeating any one of them.

The challenger adds $G_{tr}$, its uncertainty-aware estimate, and a preregistered interaction with task horizon:

The primary scoreboard is not an in-sample coefficient. It is leave-one-repository-out predictive log loss, calibration slope, and Brier score, each computed before inspecting a graph of agents. The transfer scoreboard withholds whole task families or repositories, then asks which model better predicts the outcome from the descriptions and annotations already frozen. A positive in-sample $\gamma_1$ is not a result; a redaction variable is almost designed to correlate with difficulty. Only lower held-out error and better calibration earn the claim that it carries an independent signal.

This is where existing benchmarks instruct restraint. GAIA uses tasks that are conceptually simple for people yet require tool use, reasoning, and multimodal handling, showing that a hard result is not a one-dimensional label ^[11]. PaperBench decomposes research replication into thousands of gradable subproblems, illustrating both the value and the incompleteness of a rubric ^[12]. The RE-Bench authors release human trajectories and discuss budgeted human–agent comparisons, which is exactly the sort of design pressure a new variable should survive ^[6]. These are not evidence that intention gaps work. They are evidence that evaluation gains credibility by exposing the thing it chooses to hold constant.

There is a second source of self-deception: a seemingly elegant map can be fit after the fact. The task strata, cue-removal rules, completion-class rubric, definition of a consequential transition, budget, reliability threshold, model families, number of seeds, clustering strategy, replacement policy, and primary metrics must be preregistered. Exploratory alternatives—different entropy smoothing, a different class granularity, a nonlinear $G$ term—may be reported, but not promoted to the title result. The temptation will be strong because every flexible coding decision can make a curve look ecological.

The sample size is a limitation, not a badge. Fifty base tasks times five redaction conditions gives 250 task-condition cells, but those cells are nested within the same fifty endpoints. Repeated seeds reduce execution noise; they do not create new independent tasks. The intended use is an effect-size and falsification study, with intervals wide enough to admit that the planned suite is too small for a stable niche boundary. It must not treat a thousand tool traces as a thousand ecosystems.

The rival account may be enough

The conventional account says everything important has already been named. Redaction makes prompts shorter, removes diagnostic clues, and raises ordinary task difficulty. Agents fail because their retrieval, planning, code synthesis, or verification is weak. Human experts do better because they have better priors. The entropy panel merely translates that familiar story into bits after looking at the same task. On this account, a “niche” is a benchmark slice selected for flattering one scaffold; its boundary will not transfer when the repository or agent changes.

That rival predicts several concrete failures. First, $G_{tr}$ will add no held-out predictive value once $q_{tr}$, $d_t$, and $h_t$ are present. Second, different reasonable annotation panels will produce unstable rankings of tasks, so any apparent effect will depend on adjudicator taste. Third, the agent contours will be smooth functions of conventional difficulty rather than showing a stable gap-specific shape. Fourth, adding a tool will either help everywhere or merely duplicate one baseline feature. Fifth, tasks that move in $G$ after redaction will not show a matching change in the proposed completion-class distribution when humans are asked what the request permits.

The test should be designed to let all five happen. If the completion-class annotation cannot meet a preregistered reliability floor, do not fit the challenger. If the fixed endpoint cannot be kept stable across redactions, do not call the comparison causal. If $G$ lowers neither repository-held out nor family-held-out error, report the failure even if a particular agent’s curve looks dramatic. If the static baseline predicts transfer as well as the challenger, stop calling its residual region a niche. And if rotation through prompt conditions selects an agent that only looks good because it has memorized a public patch, contamination sensitivity must erase rather than excuse the claim.

There is an embarrassing possibility even after the measure wins. The design could discover that experts disagree about completion classes because the project’s norms are genuinely contestable, not because the task contains a measurable quantity. That would be a finding about the limits of operationalizing intention, not a license to harden the rubric until it produces an effect. Another embarrassing possibility is that $G$ predicts humans but not agents. Then the phrase “AI feeds on the distance” was too strong: the distance may be a human coordination problem that current systems do not navigate in a coherent way.

The opposite outcome has limits too. A robust positive interaction would not show that agents want ambiguous work, that software is alive, or that a company can safely hand it the vague work humans dislike. It would show only that, within a frozen set of repository repairs and stated budgets, a predefined description of missing completion choices improves prediction of externally verified outcomes. The value would be diagnostic. It could tell an evaluator why a score fell, tell an engineer whether a tool expanded a specific boundary, and tell a manager that “make it work” is not a stable unit of delegation.

The useful verdict is allowed to be no

The kill criterion is deliberately conjunctive. Intention-gap volume must improve held-out prediction over the baseline and improve transfer to withheld task families or repositories. The first guard stops an attractive re-description of ordinary difficulty. The second stops an attractive re-description of one benchmark’s phrasing conventions. Neither a negative coefficient nor a steep redaction curve matters by itself. Neither does a human–agent gap. The measure earns its name only by changing a prediction outside the subset from which it was designed.

If it fails, several things remain. Fixed-endpoint redaction is still a way to audit whether a task description has made the desired behavior discoverable. Completion-class disagreement can reveal where a project has not actually specified its own acceptance criteria. A tool-ablation record can show whether an agent needs context, diagnostics, or a more reliable verifier. Those are useful engineering results, but none requires the biological language.

If it succeeds, the consequence is narrower and more interesting than an AI life metaphor. Agent evaluation would gain a coordinate for delegation: not merely how long a task takes or how many files it touches, but how much intention must be reconstructed before an outcome can be checked. System builders could show a niche boundary with its budget and uncertainty instead of announcing a general agent. Institutions could decide whether to close gaps with better specifications, reserve them for people who can negotiate meaning, or deliberately test whether an agent can cross them. The result would still be a map of one designed environment, not a declaration that a model has become a new species.

That restraint is the residue. A machine does not need to be alive for partially stated intentions to become its resource constraint. But a metaphor earns no credit for naming the constraint. The fifty-room test must show that the distance between request and outcome predicts something that ordinary difficulty does not. Until then, the blank card is a proposal, not a habitat.