Giving Gemini a Cursor and Keyboard

The Computer Use tool lets Gemini "see" a screenshot of a screen and "act" by generating UI actions - clicks, scrolls, key presses, drags - so agents can operate real browsers, mobile apps, and desktop software the same way a human would.

How It Works

Computer Use runs as a loop: your app sends a screenshot plus a goal, Gemini returns one suggested action (with a reasoning intent), your client executes it with an automation library like Playwright, then you send back a fresh screenshot and repeat until the task is done.

from google import genai

client = genai.Client()

interaction = client.interactions.create(
    model="gemini-3.5-flash",
    input="Search for 'Vertex AI pricing' on Google.",
    tools=[{
        "type": "computer_use",
        "environment": "browser",
        "enable_prompt_injection_detection": True
    }]
)

Supported Environments

Gemini 2.0 Flash is the recommended model for Computer Use, with Gemini 2.0 Flash and Gemini 1.5 Flash also supported. Every suggested action can carry a safety_decision - require_confirmation pauses the agent for human approval on sensitive actions like payments, sending messages, or accepting legal terms.