What is Computer Use AI SDK?
Computer Use AI SDK is an open-source software development kit that allows users to automate computer tasks using AI agents. It provides a native solution for macOS, enabling seamless interaction with applications and websites without the need for virtual machines.
How to use Computer Use AI SDK?
To get started, clone the repository from GitHub, install the necessary dependencies, and run the backend server. You can choose between a CLI interface or a web app interface to interact with the SDK.
Key features of Computer Use AI SDK?
- Control computer applications and perform tasks using AI agents.
- Native macOS support without virtual machine overhead.
- Fast and reliable interaction using desktop-rendered elements.
- Tools for launching apps, reading content, clicking, entering text, and pressing keys.
Use cases of Computer Use AI SDK?
- Automating repetitive tasks on a computer.
- Building custom workflows for specific applications.
- Integrating with other services to enhance productivity, such as filling out forms or managing data.
FAQ from Computer Use AI SDK?
- Is the SDK compatible with Windows?
No, it is currently designed for macOS only.
- Can I request new features?
Yes! You can submit feature requests through the GitHub issues page.
- Is there a community for support?
Yes, you can find support and discussions on the GitHub repository.
Computer Use AI SDK
-
We've built an MCP server that controls computer
-
You've heard of OpenAI's operator, you've heard of Claude's computer use. Now the open source alternative: Computer Use SDK from screenpipe.
-
It's native on macOS—no virtual machine bs, no guardrails. Use it with any app or website however you want.
-
No pixel-based bs—it relies on underlying desktop-rendered elements, making it much faster and far more reliable than pixel-based vision models.
-
You can now build your own agents getting started with our simple Hello World Template using our MCP server and client.
-
There are tools that our MCP Server provides out of the box:
- Launch apps
- Read content
- Click
- Enter text
- Press keys
-
These will be computational primitives to allow the AI to control your computer and do your tasks for you. What will you build? Come check us out at https://screenpi.pe
Demos
agent sending a message
https://github.com/user-attachments/assets/f8687500-9a8c-4a96-81b6-77562feff093
get latest whatsapp messages
open arc browser
Get started
git clone https://github.com/m13v/computer-use-ai-sdk.git
cd MCP-server-client-computer-use-ai-sdk
# Install Rust (if not already installed)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# Install Node.js and npm (if not already installed)
# Visit https://nodejs.org/ or use nvm
# run backend server
cd mcp-server-os-level
cargo run --bin server
# keep it running
Option 1: CLI Interface
# run CLI interface client in a new terminal (good for debugging)
cd mcp-client-cli-interface
npm install # install dependencies first
# Set your Anthropic API key as an environment variable
export ANTHROPIC_API_KEY=sk-ant-xxxx # Replace with your actual Anthropic API key
# For Windows, use: set ANTHROPIC_API_KEY=sk-ant-xxxx
# For permanent setup, add to your shell profile (.bashrc, .zshrc, etc.)
npx tsx main.ts
Option 2: Web app Interface
# run CLI interface client in a new terminal (good for debugging)
cd mcp-client-nextjs
npm install # install dependencies first
# Set API key via command line
echo "ANTHROPIC_API_KEY=sk-ant-XXXXXXXX" > .env # replace XXXXXXXX with your actual key
# Or append if you want to keep other env variables
# echo "ANTHROPIC_API_KEY=sk-ant-XXXXXXXX" >> .env
npm run dev
# go to provided localhost web page
What do I do with it?
- Build custom worfklows of agents to performs various actions
- Build custom UI to make it easy for users to automate their computer work
- Save workflow and run in cron
- Combine with other MCP servers to do something cool, e.g.: fill out a google sheet based on the history of people i talk to throughout the day
Request features and endpoints in github issues
https://github.com/m13v/computer-use-ai-sdk/issues/new/choose