Self-Operating Computer

Self-Operating Computer review: an influential desktop-control experiment with a very large trust boundary

Self-Operating Computer is an open-source Python framework from OthersideAI that lets a multimodal model observe a screen and choose mouse and keyboard actions. Released in November 2023, it was an early public demonstration of general computer use. It is better evaluated today as a research/developer framework than as a finished personal assistant: installation grants code and a remote model an unusually powerful view of the desktop.

The latest formal release we verified is v1.5.8 from 28 February 2025. The repository is not archived; its latest verified commit on 19 September 2025 corrected a README typo. That signal is neither abandonment nor fast active development. By August 2026, teams should test current operating-system permissions, model endpoints and dependency issues on a disposable machine rather than assume the README's historical commands and provider notes remain exact.

The project is MIT licensed and supports macOS, Windows and Linux with an X server according to its README. macOS setup asks Terminal for Screen Recording and Accessibility permissions. Those permissions are the core product fact: the framework can see sensitive pixels and generate system input. A successful demo proves visual grounding, not tenant isolation, policy enforcement, action integrity or production readiness.

Current project status

How the loop works—and where it breaks

The loop is simple: capture the current screen, optionally enrich it with OCR coordinates or Set-of-Mark overlays, ask a vision model what action to take, execute the click/type/keypress, then capture the next state. This makes it application-agnostic, because it does not need each app's API. It is also slow and brittle: resolution, scaling, pop-ups, animation, hidden windows, focus changes and ambiguous icons can move the target between observation and action.

OCR helps when visible labels map cleanly to coordinates; Set-of-Mark can improve grounding by labeling detected regions. Neither method understands the business meaning of a control. A correctly located “Send,” “Delete” or “Buy” button can still be the wrong action. Every consequential step needs semantic validation from application state or a human, not confidence in pixel coordinates.

Model choice changes grounding, latency, price, privacy and supported tool semantics. The README's model list mixes current and historical names and warns that local LLaVA error rates were very high. Treat every mode as an experiment. Build a private evaluation set on the exact display size, language, theme and applications you use; provider benchmark claims do not establish performance on your desktop.

A safe evaluation workflow

1. Create a disposable VM or test OS account with no personal credentials, files, browser sync or payment methods.
2. Pin v1.5.8, Python and dependencies; record OS, display scaling, model ID and provider settings.
3. Grant only required Screen Recording/Accessibility permissions, and revoke them after the test.
4. Define 30–50 short tasks with expected screenshots, actions, terminal states and forbidden actions.
5. Start with read-only navigation; log screenshots, proposed action, coordinates, model latency, tokens and result.
6. Add app/domain/action allowlists, step/time/cost ceilings, emergency stop and confirmation gates.
7. Test pop-ups, moved windows, dark mode, multiple displays, slow pages, prompt injection and wrong-account cases.
8. Measure task success, false action, recovery, approval frequency, latency, cost and sensitive-data exposure.
9. Only then test reversible writes in a sandbox; use idempotency and verify state after each action.
10. Prefer an API, Playwright or RPA selector when it can perform the same task; schedule dependency/security review.

Security, privacy and production limits

Run first inside a disposable VM or separate OS account with no personal files, browser sync, password manager, cloud drive, developer keys or saved payment methods. Restrict outbound domains, create test-only SaaS accounts and put harmless sample data on screen. OS-level permission prompts are not granular authorization: once Terminal has accessibility control, prompt instructions alone are a weak boundary.

Screenshots sent to a hosted multimodal provider may include emails, tokens, medical or financial records, notifications and unrelated windows. Map which provider receives each screenshot, its region, retention and training policy. Crop or redact where possible, close unrelated apps, disable notifications and define deletion for local screenshots/logs. A local model changes the network boundary but not the risk of malicious content or mistaken input.

Web pages, emails and documents can contain prompt injection telling the agent to ignore the task, reveal data or click a dangerous control. Keep task instructions separate from observed content, deny terminal and password-manager access, allowlist apps/domains/actions, and require confirmation before login, sending, purchasing, deletion, uploads, permission changes or legal acceptance. Verify the post-action state after every write.

Alternatives and independent verdict

Our judgment: Self-Operating Computer deserves credit as an early, readable experiment and remains useful for learning coordinate-based computer-use loops. It is a weak default for reliable automation of known applications: Playwright, native APIs or RPA selectors are faster, testable and easier to authorize. Use visual computer control only where no structured interface exists, and keep the blast radius smaller than the value of the task.

Compared with modern computer-use offerings, the project provides source-level flexibility and model experimentation but little managed safety or operational infrastructure. Claude Computer Use and OpenAI computer-use models supply current model/tool contracts; Browser Use specializes in browser agents; Playwright and UiPath favor structured automation. None makes untrusted screen content safe automatically, but they differ greatly in isolation, observability and reproducibility.

Frequently asked questions

Is Self-Operating Computer safe on my main laptop?

Do not make the first test on a main laptop. Use a disposable VM or isolated account without personal files, synced browser data, saved credentials or payment methods, and revoke OS permissions afterward.

What is the current version?

The latest formal release verified on 20 August 2026 was v1.5.8, published 28 February 2025. Pin it and verify current dependencies and model endpoints.

Does it work offline?

The LLaVA/Ollama path can keep model inference local, but the README warns of high error rates. Other modes send screenshots/prompts to hosted providers. Local does not remove action or prompt-injection risk.

Why does it need Accessibility and Screen Recording?

Screen Recording lets it observe pixels; Accessibility lets the process produce system input. Together they create the desktop-control capability and a large trust boundary.

Is OCR reliable enough for production?

OCR can improve label-to-coordinate grounding, but it cannot decide whether the action is authorized or correct. Dynamic UI and semantic ambiguity remain.

How is it different from Playwright?

Self-Operating Computer acts through pixels, mouse and keyboard across apps. Playwright uses structured browser/DOM interfaces, which are generally more deterministic for known web workflows.

Which tasks are reasonable?

Low-risk, reversible, short tasks in a sandbox: navigating a demo app, filling test forms or benchmarking visual grounding. Avoid money, deletion, secrets and legal commitments.

Is the project actively maintained?

It is not archived, but the latest verified release was February 2025 and latest commit September 2025. Evaluate issue activity and compatibility rather than assuming a support cadence.

Sources

• Official repository
• Official README and installation
• Official releases
• Official commit history
• MIT license
• PyPI package
• OpenAI computer-using agent
• Anthropic computer-use documentation
• Browser Use documentation
• OSWorld benchmark

Independent review: 20 August 2026. Model names, API access, prices and OS permissions change; verify live official documentation before installation.