
gittech. site
for different kinds of informations and explorations.
Open Computer Use
Open Computer Use
A secure cloud Linux computer powered by E2B Desktop Sandbox and controlled by open-source LLMs.
https://github.com/user-attachments/assets/3837c4f6-45cb-43f2-9d51-a45f742424d4
Features
- Uses E2B for secure Desktop Sandbox
- Operates the computer via the keyboard, mouse, and shell commands
- Supports 10+ LLMs, OS-Atlas/ShowUI and any other models you want to integrate!
- Live streams the display of the sandbox on the client computer
- User can pause and prompt the agent at any time
- Uses Ubuntu, but designed to work with any operating system
Design
The details of the design are laid out in this article: How I taught an AI to use a computer
LLM support
Open Computer Use is designed to make it easy to swap in and out new LLMs. The LLMs used by the agent are specified in config.py like this:
grounding_model = providers.OSAtlasProvider()
vision_model = providers.GroqProvider("llama3.2")
action_model = providers.GroqProvider("llama3.3")
The providers are imported from providers.py and include:
- Fireworks, OpenRouter, Llama API:
- Llama 3.2 (vision only), Llama 3.3 (action only)
- Groq:
- Llama 3.2 (vision + action), Llama 3.3 (action only)
- DeepSeek:
- DeepSeek (action only)
- Google:
- Gemini 2.0 Flash (vision + action)
- OpenAI:
- GPT-4o and GPT-4o mini (vision + action)
- Anthropic:
- Claude (vision + action)
- HuggingFace Spaces:
- OS-Atlas (grounding)
- ShowUI (grounding)
- Moonshot
- Mistral AI (Pixtral for vision, Mistral Large for actions)
If you add a new model or provider, please make a PR to this repository with the updated providers.py!
Get started
Prerequisites
- Python 3.10 or later
- git
- E2B API key
- API key for an LLM provider (see above)
1. Install the prerequisites
In your terminal:
brew install poetry ffmpeg
2. Clone the repository
In your terminal:
git clone https://github.com/e2b-dev/open-computer-use/
3. Set the environment variables
Enter the project directory:
cd open-computer-use
Create a .env
file in open-computer-use
and set the following:
# Get your API key here: https://e2b.dev/
E2B_API_KEY="your-e2b-api-key"
Additionally, add API key(s) for any LLM providers you're using:
# You only need the API key for the provider(s) selected in config.py:
# Hugging Face Spaces do not require an API key.
FIREWORKS_API_KEY=...
OPENROUTER_API_KEY=...
LLAMA_API_KEY=...
GROQ_API_KEY=...
GEMINI_API_KEY=...
OPENAI_API_KEY=...
ANTHROPIC_API_KEY=...
MOONSHOT_API_KEY=...
# Required: Provide your Hugging Face token to bypass Gradio rate limits.
HF_TOKEN=...
4. Start the web interface
Run the following command to start the agent:
poetry install
poetry run start
The agent will open and prompt you for its first instruction.
To start the agent with a specified prompt, run:
poetry run start --prompt "use the web browser to get the current weather in sf"
The display stream should be visible a few seconds after the Python program starts.