Computer Use: Automating Browsers, Mobile & Desktop (2026) — Structured Outputs & Tools
Computer Use: Automating Browsers, Mobile & Desktop (2026)
Giving Gemini a Cursor and Keyboard
The Computer Use tool lets Gemini "see" a screenshot of a screen and "act" by generating UI actions - clicks, scrolls, key presses, drags - so agents can operate real browsers, mobile apps, and desktop software the same way a human would.
How It Works
Computer Use runs as a loop: your app sends a screenshot plus a goal, Gemini returns one suggested action (with a reasoning intent), your client executes it with an automation library like Playwright, then you send back a fresh screenshot and repeat until the task is done.
from google import genai
client = genai.Client(vertexai=True, project="my-gcp-project", location="global")
interaction = client.interactions.create(
model="gemini-3.8-flash",
input="Search for 'Vertex AI pricing' on Google.",
tools=[{
"type": "computer_use",
"environment": "browser",
"enable_prompt_injection_detection": True
}]
)Supported Environments
| Environment | Use Case |
|---|---|
| Browser | Web form-filling, research, QA automation |
| Mobile | Android app navigation and testing |
| Desktop | OS-level cursor and keyboard control |
Gemini 3.8 Flash (gemini-3.8-flash) and Gemini 4 Argon (gemini-4-argon) are the active recommended models for Computer Use on Vertex AI (replacing earlier Gemini 2.0/2.5 Computer Use endpoints retired in June and October 2026). Every suggested action can carry a safety_decision - require_confirmation pauses the agent for human approval on sensitive actions like payments, sending messages, or accepting legal terms.
Test multimodal UI grounding coordinates generated by Gemini Computer Use to click browser elements autonomously.