Tagged prompt injection · show all articles
Inside a language model's context window, a tool description, a tool result and a resource all read the same as an instruction — which means a compromised or malicious MCP server can write the agent's next action itself.
The failure is rarely the model. It is the loop with no edge, the handoff that drops what mattered, the fan-out that multiplies exposure, and the checkpoint nobody wired in before an irreversible action.
Claude Code ships its own permission system, hooks, subagents, and memory files. Each is documented, each has already failed in a specific and traceable way, and each failure is distinct from a generic "AI agent risk."
Claude carries more defensive machinery than most deployed models — classifiers, citations, constitutional training. Its failures surface at the seams of that machinery, and Anthropic's own records are usually the first place they are documented.
Retrieval-augmented generation fails quietly. Ten distinct, documented mechanisms, from a stale index to a fabricated citation, account for nearly every expensive production RAG incident, and each has its own detection and its own fix.
An agent can do exactly what its tool interfaces let it propose, no more. The schema, the granularity, the retry semantics and the error surface are not plumbing around the intelligence — they are the boundary of it.
A language model has one input channel and no way to mark part of it inert. Everything it reads is a candidate instruction, which moves the defensible boundary off the prompt entirely and onto the authority behind every action the system is allowed to take.
A retrieval layer does not let a model remember. It submits evidence in support of an answer not yet written. Almost every expensive failure in production RAG follows from confusing the two.
The dangerous failures are not the ones that throw an exception. They are the ones where the system keeps returning well-formed output that is quietly wrong, and nothing upstream notices.
Six problems that more scale has not solved, separated carefully from the many that are merely hard engineering, and the observations that would tell us any of them had been cracked.