What We Still Cannot Do: Open Problems in Frontier Model Systems

Separating open problems from hard engineering

A great deal of what is called an unsolved problem in language models is in fact expensive engineering with a known method: longer contexts, cheaper serving, better tooling, broader multimodality. Those improve on a schedule. This article is about the other kind — problems where nobody has a method that is known to work, only mitigations whose effectiveness is measured empirically and does not obviously converge.

The distinction matters commercially, not just intellectually. A hard engineering problem justifies waiting. An open problem justifies designing a system that assumes the problem will still be there.

Six qualify. For each, the same three questions: what is actually established, what remains conjecture, and what would count as a solution rather than an improvement.

One: a model cannot reliably report what it does not know

The generative failure mode catalogued at length in the literature — fluent, well-formed output unsupported by any source — is not a bug in a component. Ji and colleagues, surveying the phenomenon across natural language generation tasks, distinguish intrinsic hallucination, which contradicts the provided source, from extrinsic hallucination, which cannot be verified against it at all, and document that both persist across architectures, tasks, and training regimes ^[1].

The structural difficulty is that a next-token distribution is not an epistemic state. A model trained to maximise likelihood over text produces a confident-sounding continuation because confident-sounding continuations are what the corpus contains, not because it has assessed its own evidence. Calibration can be measured — for a predicted confidence $c$ one can ask whether

holds across bins — and it can be improved by post-hoc adjustment. What has not been demonstrated is a mechanism by which a model represents its own ignorance in a way that survives fine-tuning, distribution shift, and the pressure of an objective that rewards answering.

What would count as a solution: an uncertainty signal that remains calibrated on a distribution the model was not trained or tuned on, without an external retrieval check, and that degrades gracefully rather than collapsing to overconfidence. Nothing published meets that bar.

Two: the stated reasoning is not necessarily the causal reasoning

Chain-of-thought prompting reliably improves multi-step performance ^[9], and reinforcement learning on verifiable outcomes has been shown to induce extended deliberation with self-verification and strategy revision emerging without demonstrations ^[8]. It is tempting to treat the resulting trace as an explanation.

Turpin and colleagues showed it is not, at least not dependably. By introducing biasing features into prompts — for example reordering multiple-choice options in a few-shot prompt so the answer is always in the same position — they found models produced reasoning that systematically supported the biased answer while never mentioning the bias that actually drove it. When the bias pointed at an incorrect answer, accuracy fell by as much as 36% across a suite of thirteen BIG-Bench Hard tasks on the two models tested ^[3]. The explanations remained plausible, internally coherent, and causally false.

This is the hinge on which most oversight proposals turn. If the trace is a faithful record, monitoring it is a powerful safety mechanism. If it is a post-hoc rationalisation that correlates with the answer under normal conditions and decouples under adversarial or unusual ones, then trace monitoring gives its strongest false assurance exactly when it is needed most.

What would count as a solution: a demonstrated causal link between the tokens of the trace and the computation producing the answer — interventions on the trace changing the answer in the predicted direction — holding under deliberate attempts to induce a decoupling.

Three: verification where no oracle exists

For code and mathematics the field has an escape: outcomes are checkable. Process supervision exploits this. Lightman and colleagues trained reward models on human labels applied to each step of a solution rather than only to the final answer, and their process-supervised model solved 78% of a representative subset of the MATH test set — outperforming outcome supervision, with active learning improving the sample efficiency further. They released the 800,000 step-level human labels used to train it ^[2]. The result is real, and it generalises to any domain with a cheap ground truth and an affordable labelling process.

Most valuable work has no such oracle. There is no compiler for a strategy memo, a literature synthesis, a diagnosis, or a design review. For these, evaluation reduces to human judgement or to another model’s judgement, and both have known pathologies: humans are slow, inconsistent, and anchored by fluency; model judges inherit the biases of the systems they judge and can be gamed by the same surface features.

The uncomfortable structural point is that the domains where verification is easiest are the domains where automation is already strongest, and the two facts are not independent. Capability advances fastest exactly where we can measure it, which means the measured frontier and the useful frontier drift apart.

What would count as a solution: a verification method for open-ended output that is cheaper than the generation it checks, and whose agreement with careful expert judgement is stable when the generator is optimised against it.

Four: context is not memory

A long context window is a large working set, not a memory system. Two limitations distinguish them.

First, attention over a long context is not uniform. Liu and colleagues documented a positional effect in which performance is highest when relevant information sits at the beginning or end of the input and degrades markedly when it sits in the middle — and that this holds even for models explicitly built for long contexts ^[5]. Capacity to hold is not capacity to use.

Second, and more fundamentally, nothing in the context persists. When the window closes the state is gone. The only durable store is the weights, and updating weights on new information runs into catastrophic forgetting: gradient updates on a new task overwrite the parameters supporting old ones. Kirkpatrick and colleagues addressed this directly with elastic weight consolidation, slowing learning on parameters identified as important to previously learned tasks, and demonstrated substantially reduced forgetting in sequential learning ^[6]. Their result is a genuine advance and it is not a solution at frontier scale, where the “previous task” is essentially all of human text and the importance estimate would have to cover it.

So a deployed system has a fast store that forgets everything and a slow store that cannot be safely written. Retrieval bridges the gap operationally by keeping an external index and re-injecting fragments, but retrieval is a workaround with its own failure surface, not a memory architecture.

What would count as a solution: incremental incorporation of new information into a frontier model’s parameters, at reasonable cost, with measured retention of prior capability and without a full retraining cycle.

Five: untrusted input reaching a privileged interpreter

Greshake and colleagues established the shape of this problem precisely: when a language model retrieves content — a web page, a document, an email — that content is processed by the same mechanism that processes the operator’s instructions, and adversarial text placed in it can redirect the system’s behaviour without the attacker ever touching the prompt ^[4].

Every mitigation deployed since is probabilistic. Delimiters, instruction hierarchies, classifiers, and spotlighting all raise the cost of an attack; none establishes a boundary. The reason is architectural rather than incidental: the model’s input is a single undifferentiated sequence, and “this span is data, that span is instruction” is a distinction it is trained to respect, not one the substrate enforces. A trained disposition can be argued out of; a type system cannot.

This is why the durable engineering answer is not at the model layer at all. Constrain what actions are reachable, scope credentials narrowly, and require deterministic policy — not model judgement — to gate anything irreversible. That is containment, and it is the correct response to an unsolved problem.

What would count as a solution: an enforced, non-probabilistic separation between instruction and data within the model’s own processing, surviving adaptive attack rather than a fixed test set.

Six: measurement decays faster than it improves

The instruments are eroding. Zhang and colleagues commissioned GSM1k, a fresh benchmark built to mirror the widely used GSM8k in style, difficulty, human solve rate, and solution length, then measured the gap. Several model families dropped by as much as 8% on the fresh set, with the worst-affected showing close to a 10% fall, and the size of a model’s gap correlated with its probability of generating GSM8k examples verbatim ^[7]. Notably, other families showed little or no gap — the effect is a property of particular training pipelines, not an inevitability.

Contamination is only half of it. The other half is that the system under test does not hold still. Vendor documentation itself records the pace: models are evaluated at specified reasoning effort and are described as systems with routing between components rather than as fixed artefacts ^[10]. A benchmark result is therefore a measurement of a configuration on a date, and both the configuration and the benchmark’s integrity degrade between publication and use.

What would count as a solution: routine evaluation on held-out instruments generated after the model’s training cut-off, reported with the full configuration, and treated as expiring.

Why more scale has not closed these

It is worth asking directly why these six have resisted the trend that flattened so much else. Scale reliably improves the quantity that training optimises — the likelihood of plausible continuations — and each of the six is a property that objective does not select for.

Calibration is not rewarded by next-token prediction; a corpus of confident prose teaches confident prose. Faithfulness is not rewarded, because the training signal touches the answer and not the relationship between the answer and the trace. Verification without an oracle is not rewarded because there is no gradient without a label. Memory consolidation is actively penalised: the optimiser has no term for preserving what it previously knew. Resistance to untrusted input is not merely unrewarded but structurally excluded, since the objective operates on an undifferentiated sequence in which no span is privileged. And measurement integrity is not a property of the model at all.

There is a second, compounding reason, and it is quantitative. Consider a task decomposed into $n$ dependent steps, each completed correctly with probability $q$, where an uncaught error propagates. Success over the whole task goes as

so the horizon that can be sustained at a target success rate $\tau$ is

The derivative of $n_{\max}$ with respect to $q$ is steep near $q = 1$, which is the good news — small per-step gains buy disproportionate horizon. The bad news is the same expression read the other way: sustaining an order of magnitude more steps requires driving per-step error down by an order of magnitude, and every one of the six problems above is a term in that per-step error which scale is not reducing. Verification, calibration, and memory are precisely the mechanisms that break the exponential by resetting accumulated uncertainty. Without them, capability gains are spent buying a slowly lengthening horizon rather than a qualitatively longer one.

This is why the six matter more as systems become more agentic rather than less. A single-turn assistant fails visibly and cheaply. A system taking two hundred dependent actions converts each of these open problems from an annoyance into a compounding risk, and the compounding is multiplicative.

The mitigations that do work, and what they cost

None of this counsels paralysis. Each open problem has a mitigation that is genuinely effective, and each mitigation buys its effectiveness by moving the burden outside the model.

Uncertainty is recovered by independent checks: retrieval against a source of record, a second model with a different training lineage, a deterministic validator, or a human. What makes these work is not that they are smarter but that their errors are uncorrelated with the generator’s. Two systems sharing a training pipeline share blind spots, which is why an ensemble of near-identical models produces confident agreement rather than useful disagreement.

Faithfulness is handled by declining to rely on it: treat the trace as a debugging aid and a user-facing courtesy, never as an audit record. Where oversight genuinely matters, instrument the actions rather than the explanation. Actions are observable, loggable, and cannot be rationalised after the fact.

Verification without an oracle is handled by manufacturing an oracle wherever one can be cheaply constructed — tests, schemas, type checks, invariants, reconciliation against a second computation — and by accepting human review as the binding constraint everywhere else. The engineering skill lies in restructuring the work so that more of it falls into the checkable category, not in wishing the uncheckable category away.

Memory is handled by keeping durable state in a store you own, with explicit read and write paths, rather than in the model or the window. This is unglamorous and it is the single most robust architectural decision available, because it degrades gracefully across model changes.

Untrusted input is handled by containment: narrow credentials, deterministic gates on irreversible actions, and an assumption that any retrieved content is hostile. And measurement is handled by holding back instruments the model has never seen.

The common cost is the same in every case: these mitigations make the system slower, more expensive, and less magical. That is the actual trade, and it should be made deliberately rather than discovered in production.

What follows for anyone building on these systems

The six problems share a structure: each is a place where the system cannot certify itself, and each therefore has to be answered from outside. Uncertainty must come from an independent check rather than the model’s own confidence. Reasoning traces are evidence about the output, not about the computation. Open-ended quality needs a human or an adversarial process in the loop. Durable state belongs in a store you control. Authority belongs to deterministic policy, not to interpretation. And measurement belongs on instruments the model has never seen.

None of this argues the systems are not useful. It argues that their usefulness comes with a fixed obligation, and that the obligation has not moved in several model generations. Architecture that assumes these problems will be solved shortly has been a losing bet for three years running; architecture that contains them has aged well.

A final caution about how these problems are likely to be reported as they change. Each of the six admits partial progress that will be announced as a solution, because partial progress is what there is to announce. A calibration method that works on one held-out distribution, a faithfulness result on one class of intervention, an injection defence that survives one red-team exercise — all are genuine contributions and none is a closed span. The distinguishing question is always the same: does the method hold when someone is actively trying to break it, on material it has never seen, at a scale nobody has tested? Until the answer is yes, the correct engineering posture is unchanged.

The vault is unclosed at six points. Building underneath it is entirely reasonable. Building as though it were closed is not.