[ ABORT TO HUD ]
SEQ. 1
SEQ. 2
SEQ. 3
SEQ. 4
SEQ. 5
SEQ. 6

Computer Use & Browser Control

🔧 Tool Use & Function Calling⏱ 10 min⭐ 80 BASE XP⌨ HANDS-ON LAB

Desktop Automation

Modern frontier models (like Claude 3.7 Sonnet and Gemini 2.0) have native Computer Use capabilities. They can observe screenshots, calculate pixel coordinates, and issue desktop actions.

Action: computer_use
Command: { "action": "mouse_move", "coordinate": [550, 200] }
Action: computer_use
Command: { "action": "left_click" }

Sandboxing Requirements

Computer interactions carry severe risks (deleting files, sending emails, executing malware). Absolute Rules for Computer Use:

  • Always execute in isolated Docker containers or throwaway VMs.
  • Never run as root.
  • Use a separate sub-agent with limited permissions if possible.
⌨ HANDS-ON LABArm Your Agent with a Browser
⭐ +150 XP

Your agent needs eyes and hands on the web. Install Playwright's browser engine, record actions with codegen, and capture a page screenshot - the exact toolkit browser agents are built on.

1Install the Chromium browser engine that Playwright automations drive.
2Launch codegen - it records your clicks and typing as replayable automation code.
3Give your agent vision: capture a full-page screenshot from the CLI.
lab-sandbox — simulated environment
INFINITY LAB SANDBOX v2.6 — simulated shell
Type the command for the current objective. Helpers: "hint", "solution", "clear".
$
OBJECTIVE 1 / 3 — type "hint" if stuck
SYNAPSE VERIFICATION
QUERY 1 // 1
What is an absolute necessity when implementing agentic Computer Use?
Giving the agent root access
Running it on a dedicated high-end GPU
Isolating execution inside a sandbox like a Docker container
Using it to browse social media