Self-Operating Computer

Open source

Python framework that lets a multimodal model view the screen and operate a computer with mouse and keyboard actions.

Open-source alternative to

GitHub stars
10k
Last 30 days
+17(+0%)
Last commit
24 days ago
Repository age
3 years
Version
v1.5.8
Licence
MIT
Self-hosted
Yes

About Self-Operating Computer

Self-Operating Computer is a framework that enables multimodal models to operate a computer. It uses the same inputs and outputs as a human operator: the model views the screen and decides on a series of mouse and keyboard actions to reach an objective. The README says it was released in November 2023.

It installs with pip and starts with the operate command. By default it uses OpenAI's gpt-4o and asks for an OpenAI key, and other modes switch to models such as o1, gpt-4.1, Gemini Pro Vision, Claude 3, Qwen-VL and LLaVA. LLaVA can run locally through Ollama.

Beyond the default mode, the README describes voice input for the objective, an OCR mode that gives the model a map of clickable elements by coordinates, and Set-of-Mark prompting with a small YOLOv8 model trained for button detection. On a Mac, the terminal app must be granted Screen Recording and Accessibility permissions.

Key features

  • Model views the screen and chooses mouse and keyboard actions
  • Works with several multimodal models
  • Installs with pip and starts with operate
  • Voice input for the objective
  • OCR mode maps clickable elements to coordinates
  • Set-of-Mark prompting for visual grounding
  • Local models via Ollama

Good fit for

  • Experimenting with computer-use agents
  • Comparing multimodal models on computer operation
  • Voice-directed computer tasks
Built with
Python tools
Tags
computer use
ai agent
automation
multimodal
python
gpt-4o
ollama

Self-Operating Computer: questions and answers

What is Self-Operating Computer used for?
Self-Operating Computer is a Python framework that lets a multimodal model view the screen and operate a computer with mouse and keyboard actions. It is a good fit for experimenting with computer-use agents, comparing multimodal models on computer operation and voice-directed computer tasks.
Is Self-Operating Computer open source?
Yes. Self-Operating Computer is open source under the MIT licence. Its source code is on GitHub at OthersideAI/self-operating-computer and is written mainly in Python.
Is Self-Operating Computer free?
Yes. Self-Operating Computer is open source, so the software itself is free to use.
What is Self-Operating Computer an alternative to?
Self-Operating Computer is an open-source alternative to Writer. Other open-source alternatives to Writer include Magic and Onyx.
Is Self-Operating Computer actively maintained?
Yes. The most recent commit to Self-Operating Computer was on 19 October 2026, and the latest release is v1.5.8, published on 28 February 2025. The project has 10k stars on GitHub, with 17 added in the last 30 days.

Open-source alternatives to Self-Operating Computer

See all

SaaS alternatives to Self-Operating Computer

See all