Desktop Automation

Modern models (like Claude 3.5 Sonnet and Gemini) have native Computer Use capabilities. They can see screenshots, calculate pixel coordinates, and control mouse/keyboard.

Action: computer_use
Command: { "action": "mouse_move", "coordinate": [550, 200] }
Action: computer_use
Command: { "action": "left_click" }

Sandboxing Requirements

Computer interactions carry severe risks (deleting files, sending emails, executing malware). Absolute Rules for Computer Use:

• Always execute in isolated Docker containers or throwaway VMs.
• Never run as root.
• Use a separate sub-agent with limited permissions if possible.