The situation you are actually in
You are not building with a model. You are building with a supplier that ships a new version of its product on its own schedule, does not tell you what changed inside, prices it differently every few months, and exposes an interface that is stable while the thing behind it is not.
That framing is not cynicism; it is the documented state of affairs. OpenAI’s own change log records GPT-5.4 in March 2026, GPT-5.5 in April, and GPT-5.6 in July, with substantial mid-generation price movement in late July [2]. The model guidance verified on 8 August 2026 documents a family of variants, an unsuffixed alias that routes to the flagship, a reasoning_effort control spanning none through max, and a separate text.verbosity setting [1].
Everything below follows from taking that seriously. The five decisions that matter are binding, context, caching, failure handling, and detection.
Decision one: what you bind to
Pin the identifier. An alias is a routing decision made by someone else on a schedule you do not control. Binding to gpt-5.6 rather than gpt-5.6-sol means your application’s model can change with no deploy, no diff, and no notification. Pin explicitly, and treat a model upgrade as a code change that goes through your normal review.
Pin the effort. Reasoning effort is not a performance tuning knob; it is a capability setting. The GPT-5 system card states that its published evaluations were run at high reasoning effort [3], which means every headline number describes a configuration you have to opt into. A team that quietly lowers effort to reduce cost has shipped a capability regression, and it will not appear as one in any diff.
Choose effort per task class, not globally. Snell and colleagues found that allocating test-time compute adaptively to prompt difficulty substantially outperforms uniform allocation, and that in some regimes additional inference compute beats additional parameters [7]. The practical translation is that a single global effort setting is always wrong in one of two directions: overspending on easy work or underspending on hard work. Classify your traffic into two or three bands and set effort per band.
A workable default: low for extraction, classification, and reformatting; medium for drafting and summarisation; high or above only for tasks where a wrong answer costs more than several times the extra tokens. Measure before believing any of these.
Decision two: assembling context
Context assembly is where most application quality is won or lost, and the governing empirical fact is positional. Liu and colleagues showed that performance is highest when relevant information sits at the beginning or end of the input and degrades markedly in the middle, holding even for models explicitly built for long contexts [4].
Four consequences follow directly.
Order deliberately. If you retrieve ten passages and concatenate them by relevance score, your best passage is at position one and your second best is in the worst position in the window. Put the highest-value material at the start and the second-highest at the end.
Prefer fewer, better spans. More context is not free even within the window. Every additional token dilutes attention, enlarges the key–value cache, reduces achievable concurrency, and costs money. Retrieval that returns twenty passages to be safe is usually worse than retrieval that returns five and is right.
Keep the instruction adjacent to the work. Instructions placed at the top of a long context are separated from the task by everything in between. Restating the operative constraint immediately before the request is cheap and measurably effective.
Count tokens yourself. Provider-side truncation is silent by design: from the provider’s perspective nothing went wrong. Compute the input length locally, assert it against the window, and fail loudly rather than letting the oldest content fall off the front.
Decision three: what to cache
Caching is the single largest cost lever available to an application, and it is available because of how serving works. Prefill is compute-bound and repeated prefixes are common, so providers cache the key–value state of a prefix and charge less to reuse it — the same insight that motivates prefix sharing in serving systems generally [6].
Structure prompts so the stable part comes first. Put the system instruction, the tool definitions, and any fixed reference material at the very front, in a byte-identical form. Put the variable part — the user’s request, the retrieved passages, the conversation — after it. A single changed character near the front invalidates everything downstream of it.
Do not interleave. Alternating stable and variable content is the most common way applications lose cache benefit without realising it. One stable block, then one variable block.
Never cache what must be fresh. Anything with a validity period — prices, availability, permissions, the current state of a record — belongs after the cache boundary or outside the prompt entirely, fetched at call time. A cached stale permission is a security bug, not a performance optimisation.
Cache your own outputs too, keyed on the full configuration. The key must include the resolved model identifier, the effort setting, and the prompt template version. A response cache keyed on user input alone will serve results from a model that no longer exists.
Decision four: handling failures that return 200
The failures that matter do not raise exceptions. Four patterns cover most of them.
Verify the world, never the claim. If the model reports that it wrote a file, read the file. If it reports that a test passed, run the test. This is the same discipline that makes repository-level benchmarks meaningful — scoring against the project’s own tests rather than the agent’s account of its work [9] — and it is the highest-value single practice in this article.
Do not audit the explanation. Turpin and colleagues showed that biasing features in a prompt produce reasoning which systematically supports the biased answer without mentioning the bias, with accuracy dropping by as much as 36% across thirteen BIG-Bench Hard tasks [8]. A confident, coherent rationale is not evidence about the computation that produced the answer. Log actions and evidence; treat the trace as a courtesy.
Retry on the right axis. A retry with the identical prompt at the identical temperature samples from the same distribution and usually reproduces the same failure. A useful retry changes something: raise effort, restate the constraint, reduce the context to the essential span, or decompose the task. Cap retries by total token budget rather than by attempt count, because attempts are not equal in cost.
Validate structure mechanically. Ask for a schema, validate against it, and treat a validation failure as a first-class error path rather than something to patch with string repair. Repairing malformed output in application code hides the frequency of the underlying problem.
Decision five: detecting the change you were not told about
Everything above is undone if the supplier changes the tooling and you absorb it. Three mechanisms, in increasing order of effort.
Log the resolved configuration on every request. Resolved model identifier as returned, effort as sent, prompt and reasoning and output token counts separately, finish reason, and a hash of the prompt template version. Without these, a regression cannot be localised to a cause. With them, most regressions localise in minutes.
Hold a private evaluation set. Thirty to fifty items drawn from your own traffic with known-good outputs, never published, never pasted into a hosted context, never committed to a public repository. Run it five times and keep the distribution, not the mean. This is your only instrument that the supplier has not seen — and the reason it matters is exactly the reason public benchmarks decay: Zhang and colleagues, building a fresh set matched to a widely used one, measured drops of as much as 8% with the worst-affected families near 10%, correlated with verbatim reproduction of the original items, while other families showed essentially no gap [10].
Re-run on a calendar, not on announcements. Provider notes describe what the provider considers noteworthy. Serving-stack changes that alter behaviour are frequently not noteworthy from their side. A weekly automated run of the private set costs almost nothing and is the difference between finding a regression yourself and hearing about it from a customer.
Refresh a fraction of the private set periodically. It contaminates too — not through training, but through your own iterative tuning against it.
The security decision underneath all of them
One constraint cuts across the rest and cannot be handled at the prompt layer. Greshake and colleagues established that content retrieved by a model — a page, a document, a message — is processed by the same mechanism that processes your instructions, so text placed in material the system is likely to fetch can redirect its behaviour without the attacker ever touching your prompt [5].
There is no prompt-level fix. Delimiters, instruction hierarchies, and classifiers raise the cost of an attack; none establishes a boundary, because the model’s input is one undifferentiated sequence and the data-versus-instruction distinction is a trained disposition rather than an enforced type.
The engineering answer is to bound the consequences. Give the system the narrowest credentials that let it do the current task and no more. Route every irreversible action — writes, sends, payments, deletions — through deterministic policy that does not consult the model. Prefer reversible operations, and keep an audit trail of actions rather than of intentions. Assume every retrieved span is hostile and design so that a hostile span accomplishes nothing.
Costing a design before you build it
One arithmetic habit prevents most unpleasant surprises at the first invoice, and it takes ten minutes.
Write the per-request cost as three separate terms rather than one blended figure:
where
The cached term is usually the largest by token count and the smallest by cost, which is why prompt structure repays attention far more than prompt wording. The reasoning term is invisible in the output and can dominate everything else at high effort, which is why effort must be measured rather than assumed. And the output term is the one that scales with how verbose the model has been asked to be, which is what the separate verbosity control exists to manage [1].
Then multiply by request volume and by the retry rate you actually observe rather than the one you intend. A design with a 15% retry rate at high effort can cost several times a design with a 2% retry rate at medium effort while producing indistinguishable output, and neither the difference nor its cause is visible without the terms broken out.
Three anti-patterns worth naming
Prompt-tuning your way around a structural problem. If output is wrong because the necessary information was truncated, or sat in the middle of a long context, or was never retrieved, no amount of instruction rewriting fixes it. Diagnose which of the three it is before touching wording; the wording is almost never the cause and is always the first thing changed.
Treating the conversation as the state. Appending everything to a growing message list is the default and it fails twice: the cache is invalidated by every turn, and the earliest content — usually the operative constraints — drifts into the positional dead zone. Keep durable state in your own store, and assemble a fresh, ordered context per request from it.
Building the harness around one model’s quirks. Every application accumulates workarounds for behaviours of the specific model it was developed against. These become invisible constraints that break on the next version, and because they are spread through prompt text rather than collected in code, nobody can find them. Keep model-specific accommodations in one clearly marked place, and re-examine that place on every upgrade.
What a durable application looks like
Pinned identifiers and explicit effort per task class. Prompts structured stable-first for cache reuse, with the most valuable context at the beginning and the end. Local token accounting that fails loudly rather than truncating silently. Verification of world state rather than acceptance of self-report. Retries that change something. Structured output validated mechanically. Full configuration logged on every request. A private evaluation set run on a schedule. Credentials scoped to the task and irreversible actions gated deterministically.
None of it is difficult. What makes it uncommon is that every item costs something up front and pays off only when the supplier changes something without telling you — which, on the current cadence, will be in a few weeks.
