What is Computer use?

Most tool calling is narrow on purpose: a defined function with a clear schema, built for one exact action. Computer use inverts that. Instead of writing an integration for each application, the model is given the same interface a person has, a screen and a mouse and a keyboard, and left to work out how to use it. Anthropic introduced the capability publicly with Claude 3.5 Sonnet in October 2024, describing it as an early, still experimental step toward AI that can use essentially any piece of software as instructed rather than only the ones with an API built for it.

The computer use loop: screenshot the screen, reason about what is visible, click or type, then observe the new screenshot and repeat

Computer use is not a shortcut around building a real integration. It is what an agent falls back on when no integration exists at all.

How does computer use actually work?

The loop repeats a small cycle: take a screenshot of the current screen, send it to the model along with the goal, let the model decide the next single action, click here, type this, scroll there, execute that one action, then screenshot again. Every step is grounded in a fresh screenshot rather than an assumption about where things are, which is what lets it survive a moved button or a redesigned page that would break a hardcoded script outright.

What is computer use good for, and where does it still struggle?

It earns its place on software that genuinely has no API worth calling: an old internal tool, a vendor’s desktop application, cross application workflows that would otherwise need a person stitching screens together by hand. It struggles on anything that needs precision or speed, since reasoning over a screenshot before every click is slower and less reliable than one direct API call, and small visual ambiguities, a similar looking button, a partially obscured field, can send the wrong action.

What does a company need before turning computer use loose?

The same discipline any agentic AI system needs, only sharper: an isolated environment so a wrong click cannot touch production directly, a human approval gate before anything that submits, pays, or deletes, and monitoring, since the capability is explicitly documented as still error prone at this stage. It is one option inside a broader AI agent’s toolkit, reached for when the task’s software has no other way in, not the default choice when a real API is available.

Frequently asked questions

How is computer use different from a browser automation script?

A script follows fixed coordinates and selectors written for one exact page, and breaks the moment that page changes. Computer use has a model look at a fresh screenshot before every action and reason about what it sees, so it adapts to layout changes and can operate software nobody wrote automation for.

Is computer use safe to run unattended?

Not by default. Anthropic's own documentation describes the capability as still experimental, at times slow and error prone, and recommends a dedicated, isolated environment plus human oversight on anything consequential, since a misread screenshot can lead to a real click on the wrong element.

What is computer use actually good for right now?

Tasks in software that was never built with an API: legacy internal tools, a vendor's desktop application, or QA testing across UIs that change often. It is a fallback for when tool calling against a real API is not an option, not a replacement for one when it is.