A technical breakdown of the architecture behind every AI desktop agent, coding agent, and browser agent you've used
Most people describe AI agents the way they'd describe a person: "it read my file," "it decided to ask me first," "it figured out I meant the other folder." That language is useful for talking about agents, but it's misleading about how they're built. There is no single "it." There are two systems wearing one costume: a lightweight local process that can see and touch your machine, and a stateless remote model that does all of the thinking. Understanding where the line between them falls is the difference between reasoning clearly about what these systems can do and just anthropomorphizing a product.
This is our attempt at laying out that architecture plainly: the local agent, the brain, the protocol that connects them, and where the actual decisions get made.
The illusion of a single agent
When you use a coding agent, a desktop agent, or a browser agent, the interface feels continuous. You type a request, a spinner runs, files change, a message comes back. It looks like one entity is perceiving, thinking, and acting in a loop.
Architecturally, it's closer to two separate programs strapped together across a network boundary:
- A local agent: a small, largely dumb client running on your machine (or in a sandboxed container next to it). It can read files, run shell commands, click buttons in a browser, or call an API. It has no model weights and, on its own, no judgment.
- A brain: a large language model running on inference infrastructure somewhere else entirely, invoked fresh on every single turn. It has no persistent memory of its own, no filesystem, no hands. All it can do is read text and produce text.
Every "decision" you attribute to the agent (read this file, don't read that one, ask before deleting, summarize instead of forwarding) is a token sequence the brain generated in response to whatever the local agent handed it. The local agent's only real job is to shuttle information up to the brain and carry out whatever the brain says to do next.
The core loop
Anthropic's own engineering writeup on agent design puts it plainly: strip away the tooling and frameworks, and an agent is just "an environment, a set of tools, a system prompt... and the model is called in a loop." That's the entire architecture in its simplest form; everything else is scaffolding around that loop (Anthropic, Building Effective Agents).
Concretely, one iteration of the loop looks like this:
1. PERCEIVE: local agent gathers state (user input, file contents, directory listings, previous tool output, screenshots).
2. SERIALIZE: all of that gets packed into a request (system prompt + tool schemas + full conversation history, as text/JSON).
3. THINK: the brain (the model) receives this payload fresh, with no memory of anything not included in it, and produces either a text reply or a structured "tool_use" request.
4. ACT: the local agent parses that structured output and executes it: opens a file, hits an API, clicks a button, or literally does nothing but display text.
5. OBSERVE: the result of that action gets appended to the conversation, and the loop repeats from step 2.
Note what's missing from the local agent's side: reasoning. It doesn't decide whether to read a file: the brain says "call the read-file tool with this path," and the local agent complies (or refuses, if a permission gate blocks it; more on that below). The local agent is muscle. The brain is the only place anything resembling thought happens.
The local agent: muscle, not mind
The local agent's responsibilities are almost entirely mechanical:
Environment access. It holds whatever credentials and handles are needed to actually do things: filesystem access, a shell, a browser session, API keys for connected tools. This is the part of the system that touches the real world.
Tool execution. When the brain emits a structured request like {"name": "read_file", "input": {"path": "/Users/x/notes.txt"}}, the local agent is the thing that actually opens the file and reads the bytes. It has no opinion about whether that was a good idea.
Permission gating. This is the one place the local agent is allowed to have something like a rule of its own: a hardcoded boundary, not a judgment call. "Don't touch files outside this directory," "ask the human before running anything destructive," "this tool requires explicit allow-listing." These are enforced in code, independent of whatever the brain "wants," precisely because the brain's behavior is probabilistic and the local agent's constraints need to be absolute.
Packaging results. Whatever comes back from a tool call (file contents, an API response, a screenshot, an error) gets serialized back into the conversation format the brain expects, and sent up on the next turn.
Rendering. The chat bubbles, the "Claude is thinking" spinners, the diff views: all UI, all local, all cosmetic from the brain's perspective.
What the local agent conspicuously does not do is decide what's relevant, what's sensitive, or what the user "really meant." It has no model of the user's intent beyond what's in the transcript it's forwarding. Any illusion of local intelligence is really the brain's output, rendered locally.
The brain: where everything actually happens
The brain is a stateless API. Every property of "agent behavior" that feels intelligent (inferring intent, choosing between tools, deciding to ask a clarifying question instead of guessing, escalating to a human, stopping when a task is done) is a token-generation event that happens inside a single model call, and it happens fresh every time.
This has a consequence that's easy to underappreciate: the brain only knows what was serialized into that specific request. There's no session state on the model side, no memory of earlier conversations unless they were re-included in the payload, no awareness of anything on your machine it wasn't explicitly shown. If the local agent didn't put something in the request body, the brain has no idea it exists. Conversely, if the local agent read a file and forwarded its contents, the brain now has that content exactly as if the user had pasted it directly into the chat: sensitivity, formatting, and all.
Three inputs shape everything the brain decides on a given turn:
- The system prompt: the standing instructions that define the agent's goals, tone, and constraints. This is the closest thing the brain has to a personality or a mandate.
- The tool schemas: the names, descriptions, and input shapes of every action available to it. Anthropic calls the discipline of designing these an "agent-computer interface" (ACI), directly analogous to human-computer interface design: a tool with an ambiguous description gets used ambiguously, the same way a confusing button gets clicked by accident.
- The conversation history: everything said and done so far, including every prior tool call and its result, all re-sent on every turn since the model itself retains nothing between calls.
Whether the brain chooses to call a tool, ask the user a clarifying question, or just respond in plain text is a single inference decision made from those three inputs; there is no separate "planner" module deciding whether to plan versus a "chat" module deciding whether to chat. It's the same forward pass either way.
Tracing one turn end to end
It's worth seeing the actual shape of this, stripped of any specific product's branding. Say a local agent has just listed a directory and is handing that back to the brain:
POST / v1 / messages {
"model": "claude-sonnet-5",
"system": "You are a file assistant. Ask before deleting anything.",
"tools": [{
"name": "read_file",
"description": "Read a UTF-8 text file at a given path.",
"input_schema": {
"type": "object",
"properties": {
"path": {
"type": "string"
}
}
}
}, {
"name": "list_dir",
"description": "List files in a directory.",
"input_schema": {
"type": "object",
"properties": {
"path": {
"type": "string"
}
}
}
}],
"messages": [{
"role": "user",
"content": "Summarize the report in my Documents folder."
}, {
"role": "assistant",
"content": [{
"type": "tool_use",
"name": "list_dir",
"input": {
"path": "~/Documents"
}
}]
}, {
"role": "user",
"content": [{
"type": "tool_result",
"content": "quarterly_report.docx\nnotes.txt\ntax_forms/"
}]
}]
}Everything above this line is what gets sent, in full, on every single turn: the entire transcript, re-transmitted, every time. The brain's response to this exact payload is the next decision in the chain:
{
"role": "assistant",
"content": [
{
"type": "tool_use",
"name": "read_file",
"input": { "path": "~/Documents/quarterly_report.docx" }
}
]
}The local agent receives that, opens the file, and appends the contents as a new tool_result, which then becomes part of the payload for the next call. There is no shortcut, no cached understanding, no persistent context living quietly on a server somewhere. The full history goes back over the wire, every time, because the brain has nowhere else to keep it.
This is also why an agent's "memory" has a hard ceiling: the context window. Every tool result, every file it reads, every screenshot it takes has to fit inside the same budget as everything else in the conversation. Long-running agents don't have bigger memories; they have more aggressive summarization and pruning of what gets re-sent each turn.
The protocol that makes tools pluggable
The mechanism above (tool schemas in, tool_use blocks out) is standardized well beyond any single product. The Model Context Protocol (MCP) formalizes it into three roles: a host (the AI application coordinating everything, e.g. a desktop agent or an IDE), a client (a connector the host spins up per tool source, maintaining one dedicated session each), and a server (the actual program exposing tools, resources, or prompt templates over a JSON-RPC connection, whether that's a local filesystem server talking over stdio or a remote server talking over HTTP) (Model Context Protocol, Architecture Overview).
Two of MCP's client-side primitives map directly onto the "ask vs. act" behavior people notice in agents:
- Elicitation lets a server ask the host to collect more information from the user mid-task: the formal mechanism behind an agent stopping to ask a clarifying question instead of guessing.
- Sampling lets a server request a model completion through the host's own model connection, without embedding its own LLM; useful for tools that need a bit of judgment themselves but shouldn't run an independent model.
Both are still just structured messages passed between host and server. Nothing about them requires the server, or the local agent, to have any model of its own. The intelligence stays centralized in the brain; the protocol just gives more actors a standard way to ask it things.
Why the split matters
This isn't architecture trivia: it has direct consequences for anyone building, auditing, or reasoning about agents:
Statelessness is the whole game. Because the brain remembers nothing between calls, whatever the local agent chooses to serialize is the brain's entire world for that turn. Bugs in what gets included or excluded from that payload are not cosmetic; they're the difference between the model seeing the right file and the wrong one.
The wire is the security boundary. Anything that crosses from local agent to brain has, functionally, left the machine and gone into an API call to another compute environment. Local sandboxing controls what an agent can touch; it says nothing about what gets transmitted once touched. Those are separate risk surfaces and need separate controls.
Guardrails belong in the local layer, not the prompt. A system prompt instruction like "don't run destructive commands" is a strong statistical bias, not a guarantee: it's advice given to a probabilistic text generator. Enforceable boundaries (sandboxing, allow-lists, human-confirmation gates on specific tool calls) have to live in the deterministic local code, because that's the only layer that isn't just predicting the next token.
Cost and latency scale with what you re-send. Every turn re-transmits the full history to a remote model. Long agent sessions are expensive and slow not because the model is "doing more thinking," but because the payload keeps growing, and the context window is a hard, shared budget.
The takeaway
An AI agent is not a single continuous mind living on your desktop. It's a thin, largely mechanical local process wired to a stateless remote model that gets handed the entire relevant world, in text, on every single turn, and produces the next action from scratch each time. The local agent has hands and no judgment. The brain has judgment and no hands, no memory, and no awareness of anything it wasn't explicitly shown.
Every behavior worth analyzing in these systems (what they read, what they ask about, what they escalate, what they assume) is a property of that one stateless inference call, shaped entirely by the system prompt, the tool interface, and whatever the local agent decided to serialize. Once you see the architecture this way, "agent behavior" stops looking mysterious and starts looking exactly like what it is: a well-specified loop, repeated.
Further reading:
- Anthropic, Building Effective Agents
- Model Context Protocol, Architecture Overview
This piece is part of our ongoing research into how agentic AI systems are actually built and where their behavior comes from.
