Computer use — the agent reaches for the screen

What it is. Anthropic gave Claude the ability to use a computer the way a person does: it is shown a screenshot, and it returns actions — move the cursor here, click, type, scroll, take another screenshot, continue. Not a bespoke integration wired to one app, but general control of the same graphical interface humans use, on whatever is on the screen.

What it is — stated precisely. Not the first computer-controlling agent. A research lineage precedes it — WebGPT, Adept’s ACT-1 (2022), and a body of GUI- and web-agent work — and Anthropic shipped it explicitly in beta, slow and error-prone, warning it would misclick and get lost. What is enshrined is the inflection, not a clean first: a frontier lab shipping general computer use as a product capability, moving agents off the curated API and onto the messy interface built for people. One wing over, RT-2 put web knowledge into a robot’s motor actions; this put it into a cursor.

Why it matters. An API is a permissioned, curated surface — a short list of things an agent is allowed to call. The screen is the whole world, unpermissioned. An agent that can drive any GUI can do anything a human at a keyboard can: file the form, move the money, send the message, change the setting. It widened what an agent can reach and what an attacker can reach through it in a single stroke — every warning in the security wing now comes with a mouse.

Its place beside the verification wings. When an agent acts through the same interface as a person, its actions become indistinguishable from a person’s at the destination: a click is a click, a keystroke a keystroke, and the receiving system cannot tell which agent did it, or on whose authority. Attribution can no longer be read off the channel; it has to come from a signed, anchored record of what the agent did and was authorized to do. Computer use is a large part of why “which agent, acting for whom” has to be *provable* rather than assumed — the question the rest of this collection is built to answer.

*Primary source inside: Anthropic’s announcement of computer use and the upgraded Claude 3.5 models, 22 October 2024.*