---
title: Computer use
description: Drive a lease's screen as an agent — actions, screenshots, a live stream, and control that a human can take back.
type: how-to
product: sandboxes
summary: Acting on a display, watching it, and who holds the mouse
updated: 2026-09-28
order: 1
---

A lease with a display is a machine an agent can use the way a person does:
look at the screen, click, type, and hand the mouse to a human when the human
is better at the next step.

## The kinds with a display

`spec.kind` decides what the machine serves, and three of the four kinds have
a display:

<PropertyTable
	properties={[
		{
			name: 'general',
			type: 'LeaseKind',
			description: 'A plain machine. The default. No display, and no computer actions.',
		},
		{
			name: 'browser',
			type: 'LeaseKind',
			description: 'A machine running a browser on its display. `spec.browser` sets it up: `headless`, `user_data_dir` and `start_url`.',
		},
		{
			name: 'desktop',
			type: 'LeaseKind',
			description: 'A desktop session on its display.',
		},
		{
			name: 'android',
			type: 'LeaseKind',
			description: 'An Android machine on its display. Apps are installed with `install_app`.',
		},
	]}
/>

`spec.display` sets the display itself — `width`, `height` and `dpi`, defaulting
to 1280 by 800 at 96 dpi — and `status.display` reports the effective one.
Action coordinates are in that space, not in image pixels: a screenshot
carries `display_width` and `display_height` beside its own `width` and
`height`, and dividing the two is how a model maps what it saw to where it
clicks.

<Callout tone="note" title="Headless is CDP only">
`spec.browser.headless` runs the browser without a display. A headless
browser has no stream and no computer actions — the Chrome DevTools Protocol
at `status.endpoints.cdp_uri` is the whole interface. Choose it when a
program drives the browser, not a model.
</Callout>

Put a `user_data_dir` on a [volume](/docs/sandboxes/volumes-and-snapshots)
when the browser's cookies and storage should outlive the lease; the default
is a fresh profile per lease.

## Act, and look

`act` performs actions on the display in order, at most 64 of them, and
returns one result per action:

```bash
sylphx sandboxes leases act orgs/acme/projects/shop/envs/production/leases/lease
```

```bash
curl -X POST "https://api.sylphx.com/v1/orgs/acme/projects/shop/envs/production/leases/lease:act" \
  -H "Authorization: Bearer $SYLPHX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'
```

```ts
const response = await sylphx.sandboxes.leases.act({
  name: 'orgs/acme/projects/shop/envs/production/leases/lease',
})
```

An action is one of: `move`, `click`, `mouse_down`, `mouse_up`, `drag`,
`scroll`, `key`, `hold_key`, `type`, `wait`, `take_screenshot`, `zoom` and
`cursor_position`. They are the Anthropic computer tool's actions and
OpenAI's, one to one, so a model's tool call goes across without translation.
`click` carries a button and a count, so a double click is one action rather
than two; `drag` takes a path of at least two points; `key` and `hold_key`
take a combination; `type` takes at most 64 KiB of text; `wait` holds for at
most 30 seconds.

`results` comes back one per action, in order, each with the pointer position
after it. `screenshot` on the call takes a capture after the last action and
puts it on the response — with no actions and `screenshot` set, the call only
captures, which is how a loop looks before it decides anything.

<PropertyTable
	properties={[
		{
			name: 'format',
			type: 'ImageFormat',
			description: '`png`, `jpeg` or `webp`. PNG by default.',
		},
		{
			name: 'quality',
			type: 'int32',
			description: 'JPEG and WebP quality, 1 to 100; 80 by default.',
		},
		{
			name: 'max_width',
			type: 'int32',
			description: 'Downscale to at most this width, keeping the aspect ratio. Zero keeps the display size.',
		},
		{
			name: 'settle',
			type: 'duration',
			description: 'Wait this long after the last action before capturing, so the screen settles. 300 ms by default, at most 5 seconds.',
		},
	]}
/>

Two fields on the call are guards rather than instructions:
`lease_generation` refuses `STALE_GENERATION` unless the lease is still at
that generation, and `control_epoch` acts under a named agent epoch. Both
default to 0, which skips the check.

## The live stream

`open_stream` opens a view of the display: a short-lived viewer token bound
to the lease and its generation, an embeddable page, and the WebRTC
signaling endpoint with its ICE servers.

```bash
sylphx sandboxes leases open-stream orgs/acme/projects/shop/envs/production/leases/lease
```

The answer is a `StreamSession`. `viewer_uri` is a page that connects on
load — a web page, or a Telegram Mini App webview — and the token rides in
the URL fragment so it never reaches a server log. `signaling_uri` is the
WebRTC signaling WebSocket: send the token as the first message. `ice_servers`
arrives with TURN credentials included, and `vnc_uri` is the noVNC fallback
for networks that block WebRTC.

The token is returned once and never again, and `ttl` decides how long it
admits a new connection — 5 minutes at most, and the default. A connected
stream ends when the lease's generation changes or the token is revoked, and
`revoke_uri` revokes it early. `role` is `viewer` for frames, and
`controller` for frames with input, which is what a human's acquire returns.

## Who holds the mouse

A display has one input, so control is explicit:

```bash
sylphx sandboxes leases acquire-control \
  orgs/acme/projects/shop/envs/production/leases/lease \
  --holder agent
```

`holder` is required and is either `agent` or `human`. `ttl` defaults to 10
minutes and the Cell caps it at 30, after which control lapses on its own and
the stream records `control_expired`. `holder_label` is your name for the
holder — a user id, for example — and it is recorded in the audit events.

`status.control` is the current `ControlState`: `holder`, `epoch`,
`principal`, `holder_label`, `acquire_time` and `expire_time`. `epoch`
increases on every acquire, and `release_control` names the epoch it is
releasing, so a late release cannot drop a control that was taken since:

```bash
curl -X POST "https://api.sylphx.com/v1/orgs/acme/projects/shop/envs/production/leases/lease:releaseControl" \
  -H "Authorization: Bearer $SYLPHX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"epoch":1}'
```

Nobody holding control is a working state, not an empty one: `holder` is
unnamed, and agent actions are allowed. A human taking over is what stops
them — an agent that acts while a human holds control is refused
`CONTROL_HELD_BY_HUMAN`, and a second human without preemption is refused
`CONTROL_HELD`.

Acquiring control is [one call](/docs/api/leases/acquire_control) whose
fields are the paragraph above, and releasing it is
[another](/docs/api/leases/release_control). Neither is needed to watch:
[reading the control](/docs/api/leases/get_control) tells a caller who holds
the mouse before it decides whether to act.

## Android apps

An `android` lease installs an app in two calls, in this order: the APK goes
in as a file first, and then it is installed by path.

```bash
curl -X POST "https://api.sylphx.com/v1/orgs/acme/projects/shop/envs/production/leases/lease:writeFile" \
  -H "Authorization: Bearer $SYLPHX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"content":"…","path":"/home/user/app.apk"}'
```

```bash
sylphx sandboxes leases install-app \
  orgs/acme/projects/shop/envs/production/leases/lease \
  --apk-path /home/user/app.apk
```

`grant_permissions` grants every runtime permission the app declares, which
is what an unattended agent wants and what a person installing an app by hand
would otherwise have to click through. The answer carries the installed
`package_name`.

<RelatedDocs
	links={[
		{
			href: '/docs/api/leases/act',
			label: 'act',
			description: 'Every action, its fields and the response.',
		},
		{
			href: '/docs/api/leases/open_stream',
			label: 'open_stream',
			description: 'The stream session, its token and its URIs.',
		},
		{
			href: '/docs/api/leases/acquire_control',
			label: 'acquire_control',
			description: 'Holders, the TTL, preemption and the errors.',
		},
		{
			href: '/docs/sandboxes/leases',
			label: 'Lease lifecycle',
			description: 'The states, the clocks and the event stream.',
		},
	]}
/>