Guides
What is a computer-use agent? APIs, browsers and desktop apps
A practical guide to the model, tools and environment behind computer-use agents, with a framework for choosing the right surface and verifying the result.
A computer-use agent is an AI system that observes a computer interface, chooses an action, executes it through a tool and checks the result. It can work through buttons, fields, menus and keyboard input, allowing it to reach tasks that are available in software's interface.
The model needs an environment and tools that can act on it. Those tools may operate a browser, a desktop app or a separate computer. Which surface the agent can reach determines what you can ask it to do.
At a glance: choose the interface that covers your task, give the agent a clear finish condition and verify the output. An API call, a browser action and a desktop click can all belong to the same workflow, but they expose different information and failure modes.
How a computer-use agent works
The basic loop has four stages: observe, decide, act and verify. The agent reads the current state, chooses its next step, asks a tool to execute it and receives a new observation.
Suppose the goal is to save a document under a specified name. The agent needs to identify the intended document, find the save operation, handle any dialog and check the resulting file. Typing a filename is only one step in that process.
Anthropic's computer-use documentation separates model requests from the application that executes them and returns observations. That distinction applies when evaluating an agent: the model's plan and the tool's capabilities must work together.
The loop also needs a stopping condition. An agent should stop when it has produced the requested output, reached a limit or encountered a state it cannot resolve. Continuing to act without enough information can make a small mistake harder to diagnose.
The model, tools and computer have different jobs
Several components sit behind what appears to be one assistant.
| Component | Its job | A question to ask |
|---|---|---|
| Model | Interprets the request and chooses subsequent actions | Can it understand the available observations and tool results? |
| Agent harness | Runs the loop, carries context and applies execution limits | How are retries, budgets and interruptions handled? |
| Tools or driver | Observe the interface and execute requested actions | Which controls and input routes can they use? |
| Environment | Contains the browser, apps, files and session | Is the task running on your computer or another one? |
| Result check | Determines whether the requested outcome exists | Can the final file or app state be inspected independently? |
A capable model cannot make a missing tool action available. A tool can deliver an input while the model has chosen the wrong target. The final result depends on the combination, so evaluate the complete workflow.
This helps interpret product names too. A desktop driver may be one component of an agent stack. A model API may provide action requests while leaving environment setup to the developer. An app can package several of these pieces into a user-facing workflow.
When an API is the right route
An API lets software request a defined operation directly. If it exposes the exact task, an agent can supply parameters, read the response and proceed without navigating the graphical interface.
For example, a service might provide an operation to create a record and return its identifier. The agent can use that identifier in subsequent calls. A clear interface makes it easier to check which operation ran and what it returned.
The useful question is whether the API covers your required result. An app may expose some operations while leaving a particular dialog or setting available only through its interface. Check the gap at the task level.
You can also keep deterministic work outside the agent's visual loop. Preparing input files, validating a format or looking up a known record may fit an existing integration. A workflow can then use computer use for the step that requires a screen.
What a browser agent can observe
A browser agent works with web pages. Depending on its tools, it can read page structure, accessibility information, text or screenshots, then interact with the selected page.
Page structure can make a target explicit. Browser automation tools such as Playwright support locators based on roles, names and text. A tool can use those properties to identify a particular button or field.
Screenshots add visual information, especially where a canvas or custom control exposes little useful structure. The agent still needs to distinguish the requested page from another tab or window.
A web page is one part of a broader workflow. If the task continues into a native file dialog or a desktop application, confirm that the agent has tools for that surface. A browser's name in a product description does not establish desktop-wide access.
What a desktop agent can observe
A desktop agent works with application windows. Its driver may use accessibility information, screenshots, OCR and other observations to describe the current interface and identify actions.
Those observations differ by toolkit and application. A standard button may have a readable name and role. A custom drawing area may expose little through accessibility. A control can also change when the app opens a dialog or switches views.
Desktop tasks often cross app boundaries. An illustrative export workflow could involve creating a file in one app, locating it in Finder and preparing a draft message in another. Each handoff needs a concrete state the next step can use.
The agent should verify the export and the selected attachment before continuing. A task spanning several apps creates more checkpoints; it does not remove the need for them.
Choose the surface your task actually needs
| Task shape | A useful starting route | What to verify |
|---|---|---|
| A service exposes the exact requested operation | API or existing integration | Returned identifier, value or stored record |
| The work stays inside a web application | Browser tools | Correct page, selected record and resulting state |
| A required step lives in a native app | Desktop tools | Intended window, affected document and output |
| Most steps are fixed, with one uncertain UI step | A combined workflow | Inputs and outputs at each handoff |
Avoid choosing a method solely because it looks impressive in a demo. Start from the output and identify the surfaces required to produce it.
If your app has no suitable API, our guide to automating a Mac app without an API explains other routes, including scripting, workflow actions and UI automation.
Local execution and remote models can coexist
The location of the interface and the location of the model are separate choices. A local desktop tool can control apps on your Mac while sending observations to a remote model. A browser-based product can also assign work to a different computer.
Ask where the app, files, model and stored knowledge live. Then ask which observations enter the model request. The answer depends on the selected tools and connection.
Background work is another separate property. It describes how automation interacts with your current session while you continue working. It does not automatically imply a virtual machine, a separate filesystem or simultaneous control by multiple workers.
For a task you plan to run while using the computer, include that condition in the trial. Inspect the output and record any pause, focus change or manual intervention.
What MCP adds to the stack
Model Context Protocol gives an AI application a standard way to connect to servers that expose tools and context. Its architecture documentation distinguishes the host application, its clients and the connected servers.
MCP is a connection mechanism. The tools exposed by a particular server determine which actions are available. The host's agent loop decides how to use them.
For desktop work, check the server's actual scope and the environment in which it runs. A connection that exposes desktop tools does not automatically provide a separate browser engine or access to another machine.
Where Braus fits
Braus gives agents a desktop workspace on a Mac. You can create a worker in the app or connect an external client through local MCP. Its Perception pipeline builds structured text from the current window for the agent to use.
Braus's desktop Driver uses a virtual display within the current Mac session. Workers take turns controlling desktop apps. The background documentation describes the session boundaries and what to check in your workflow.
Braus also keeps local app knowledge and interaction memory. That context can inform subsequent work, but the agent still needs a fresh observation and a check of what changed. The current product documentation explains those components and their limits.
Start with an observation-only task in a disposable document. Once you can confirm that the agent has identified the intended window and content, try a small change whose result you can inspect.
A successful tool call needs a result check
There are several levels of evidence: a tool accepted a call, a control changed, and the requested task finished. Keep them separate when evaluating a product.
A save operation can be delivered while the output has the wrong name. A field can change while the form remains unsubmitted. The model's final message can describe a result that the checker cannot find.
Define success before the trial and inspect the state that proves it. For a file task, check the file and its contents. For a form, check the recorded values and intended state. Record partial and uncertain outcomes alongside successes.
Common questions
Is a computer-use agent the same as a chatbot?
A chatbot can answer questions without operating software. A computer-use agent has tools and an execution loop that let it observe and act on an interface. Check which tools are actually connected to the assistant you use.
Can it use apps without an API?
It can attempt supported operations through the interface. The result depends on the app, controls, permissions and workflow. Test the exact task rather than assuming support for every feature in an app.
Can it finish a task after a failed tool call?
An agent may observe again, choose another route and recover. That makes the final output important: individual errors and task outcomes describe different things. Record the recovery and any intervention as part of the trial.
Choose one task, the surface it needs and a result you can verify. If that task lives inside a Mac app, use the Braus setup guide to prepare a first trial.