Computer Use

What Is Computer Use?

Computer Use gives Opulent direct access to a graphical desktop environment with a mouse and keyboard. This goes beyond browser automation — Opulent can interact with any application that renders on screen, including:

• Web applications in Chrome (clicking buttons, filling forms, navigating pages)
• Desktop applications that run on the session's platform (Linux or Windows), including Electron apps, IDEs, and platform-native GUIs
• Terminal-based UIs (TUI programs, interactive CLIs)
• Any visual interface that can be displayed on the desktop

Opulent sees the screen as a 1024×768 pixel display and can perform actions like clicking, typing, scrolling, dragging, and taking screenshots — just like a human sitting at the computer.

Supported Platforms

The Computer Use experience is the same on both platforms: Opulent uses the mouse and keyboard, takes screenshots, runs Chrome for web apps, and can record its testing sessions. On Windows, Opulent can additionally test Windows-native desktop applications (e.g. WPF, WinForms, and other apps that only run on Windows). To run sessions on Windows, configure a Windows blueprint as described in Windows support.

How to Enable It

Computer Use is controlled by the Enable desktop mode toggle in your organization's customization options.

1. Go to Settings > Customization
2. Under the Browser interaction section, toggle Enable desktop mode on
3. Opulent will now use its desktop environment during sessions

When Computer Use Runs

Once Desktop mode is enabled, Computer Use is available in every session. There are three ways it gets used:

After creating a PR

When Opulent creates a PR, it offers a Test the app button. Clicking it triggers the full QA Evidence & Recordings workflow — Opulent starts your app, uses Computer Use to interact with the desktop, tests the changes, and sends you a recording.

On request during a session

You can ask Opulent to test at any point during a session — no special syntax needed, just natural language. For example:

• "Test the changes you just made and send me a recording"
• "Open the app in the browser and verify the login page works"
• "Launch the desktop app and check that the new menu item appears"

Autonomously when appropriate

Opulent decides on its own when desktop interaction is the right tool for the job. If a task involves clicking UI elements, navigating an app, filling out forms, or visually verifying something, Opulent will use Computer Use without being explicitly asked. You don't need to tell Opulent how to interact with the screen — just tell it what to accomplish.

What Opulent Can Do with Computer Use

Test web applications end-to-end

Opulent can start your app locally, open it in Chrome, and click through entire user flows — login, navigation, form submission, checkout — verifying that everything works as expected.

Test desktop applications

Any application that runs on Opulent's session platform can be tested. On Linux sessions this includes Electron apps, Java Swing/AWT applications, GTK/Qt apps, and more. On Windows sessions, Opulent can also test Windows-native applications such as WPF and WinForms apps. Opulent launches the app, interacts with its GUI, and verifies behavior.

Visual verification

Opulent can take screenshots at specific points during testing to verify that layouts, styling, and UI elements look correct. It can compare what it sees on screen against expected behavior and flag visual issues.

Interact with complex UI flows

Some testing scenarios require multi-step GUI interactions that go beyond simple API calls or browser automation — things like drag-and-drop, context menus, keyboard shortcuts, or navigating between multiple windows. Computer Use handles all of these.

Record testing sessions

Opulent can record its screen while testing, annotating key moments in the video. The recording is then processed and sent to you so you can watch Opulent interact with your app and confirm the changes work. See QA Evidence & Recordings for full details on the recording workflow.

How Computer Use Works

When Opulent uses Computer Use during a session, it follows this process:

1. Takes a screenshot of the current screen to understand what's visible
2. Identifies interactive elements — buttons, text fields, menus, links — and decides what to interact with
3. Performs an action — clicks, types, scrolls, or uses keyboard shortcuts
4. Waits and observes — takes another screenshot to see the result of the action
5. Repeats until the task is complete

This screenshot-action loop allows Opulent to adapt to whatever is on screen, handling dynamic content, loading states, pop-ups, and unexpected dialogs just like a human would.

Computer Use and QA Evidence

Computer Use is the foundation of Opulent's QA Evidence & Recordings workflow. When Opulent tests your application after creating a PR:

1. Setup — Opulent installs dependencies, starts your app, and prepares the environment
2. Test planning — Opulent reads the diff and creates a focused test plan
3. Execution via Computer Use — Opulent uses its desktop to interact with your app, following the test plan step by step
4. Recording — The entire process is captured on video with annotations, then sent to you for review

The key difference between Computer Use and the QA Evidence & Recordings workflow is scope: Computer Use is the underlying capability (desktop interaction), while QA Evidence & Recordings is the structured workflow that uses Computer Use to test your PRs and deliver video proof.

Tips for Getting the Best Results

Pre-configure access

If your app requires authentication, set up secrets ahead of time so Opulent can log in without asking you during the session. Complete environment configuration to ensure Opulent can install dependencies and start your app without issues.

Create testing skills

For apps you test frequently, create a Skill that tells Opulent exactly how to set up and test your application. This saves time on repeated sessions and ensures consistent testing. See QA Evidence & Recordings — Skill Suggestions for examples.

Scripted Browser Use via Playwright

Opulent's Chrome browser exposes a Chrome DevTools Protocol (CDP) endpoint that Playwright can connect to. Opulent can write and run Playwright scripts to automate browser interactions — such as login flows or systematic data entry — against its own running browser. You can also write these scripts yourself and check them into your repo. For most other browser actions, Opulent's native Computer Use or browser tools are recommended.

How it works

Opulent's Chrome instance listens for CDP connections on port 29229. A Playwright script can attach to this browser, perform actions (fill forms, click buttons, handle redirects), and then disconnect. Because the script connects to the existing browser rather than launching a new one, all state changes — cookies, localStorage, auth tokens — persist after the script exits.

This means Opulent can immediately use the authenticated session: refresh pages, navigate, and interact with the app normally.

Example: connecting to Opulent's browser

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp("http://localhost:29229")
    context = browser.contexts[0]
    page = context.pages[0] if context.pages else context.new_page()

    # Example: navigate and log in
    page.goto("https://example.com/login")
    page.fill('input[name="email"]', "user@example.com")
    page.fill('input[name="password"]', "password")
    page.click('button[type="submit"]')
    page.wait_for_url("**/dashboard")
    print("Login successful!")

After this script runs, Opulent's browser is logged in and ready to use — no manual interaction required.

When to use this

Tips

• Store login scripts in your repo's .agents/skills/ directory so they persist across sessions
• Use Secrets to store credentials — reference them via environment variables in your scripts
• The CDP endpoint is always http://localhost:29229 — this is the same port whether Desktop mode is enabled or not
• After the script runs, Opulent can use either Computer Use or browser tools to interact with the authenticated session

Troubleshooting

Opulent can't find a UI element

If Opulent is unable to locate a button or element on screen, try being more specific in your instructions — describe the element's location, label, or surrounding context. For example, "click the blue Save button at the bottom-right of the modal" is better than "click Save."

The app doesn't render on Opulent's desktop

By default, Opulent runs a Linux environment. If your application only runs on Windows, run your sessions on a Windows environment so Opulent can test it there. macOS-only applications are not supported. Web applications work regardless of platform since they run in Chrome. For desktop apps, ensure they have a build for the platform your sessions run on.

Opulent is clicking the wrong things

If Opulent is misinteracting with your UI, provide a Skill or Knowledge entry with specific navigation instructions for your app. Describing the exact steps ("click the hamburger menu in the top-left, then click Settings in the dropdown") reduces ambiguity.