Computer use

A lease with a display is a machine an agent can use the way a person does: look at the screen, click, type, and hand the mouse to a human when the human is better at the next step.

#The kinds with a display

spec.kind decides what the machine serves, and three of the four kinds have a display:

spec.display sets the display itself — width, height and dpi, defaulting to 1280 by 800 at 96 dpi — and status.display reports the effective one. Action coordinates are in that space, not in image pixels: a screenshot carries display_width and display_height beside its own width and height, and dividing the two is how a model maps what it saw to where it clicks.

Put a user_data_dir on a volume when the browser's cookies and storage should outlive the lease; the default is a fresh profile per lease.

#Act, and look

act performs actions on the display in order, at most 64 of them, and returns one result per action:

sylphx sandboxes leases act orgs/acme/projects/shop/envs/production/leases/lease

curl -X POST "https://api.sylphx.com/v1/orgs/acme/projects/shop/envs/production/leases/lease:act" \
  -H "Authorization: Bearer $SYLPHX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'

const response = await sylphx.sandboxes.leases.act({
  name: 'orgs/acme/projects/shop/envs/production/leases/lease',
})

An action is one of: move, click, mouse_down, mouse_up, drag, scroll, key, hold_key, type, wait, take_screenshot, zoom and cursor_position. They are the Anthropic computer tool's actions and OpenAI's, one to one, so a model's tool call goes across without translation. click carries a button and a count, so a double click is one action rather than two; drag takes a path of at least two points; key and hold_key take a combination; type takes at most 64 KiB of text; wait holds for at most 30 seconds.

results comes back one per action, in order, each with the pointer position after it. screenshot on the call takes a capture after the last action and puts it on the response — with no actions and screenshot set, the call only captures, which is how a loop looks before it decides anything.

Two fields on the call are guards rather than instructions: lease_generation refuses STALE_GENERATION unless the lease is still at that generation, and control_epoch acts under a named agent epoch. Both default to 0, which skips the check.

#The live stream

open_stream opens a view of the display: a short-lived viewer token bound to the lease and its generation, an embeddable page, and the WebRTC signaling endpoint with its ICE servers.

sylphx sandboxes leases open-stream orgs/acme/projects/shop/envs/production/leases/lease

The answer is a StreamSession. viewer_uri is a page that connects on load — a web page, or a Telegram Mini App webview — and the token rides in the URL fragment so it never reaches a server log. signaling_uri is the WebRTC signaling WebSocket: send the token as the first message. ice_servers arrives with TURN credentials included, and vnc_uri is the noVNC fallback for networks that block WebRTC.

The token is returned once and never again, and ttl decides how long it admits a new connection — 5 minutes at most, and the default. A connected stream ends when the lease's generation changes or the token is revoked, and revoke_uri revokes it early. role is viewer for frames, and controller for frames with input, which is what a human's acquire returns.

#Who holds the mouse

A display has one input, so control is explicit:

sylphx sandboxes leases acquire-control \
  orgs/acme/projects/shop/envs/production/leases/lease \
  --holder agent

holder is required and is either agent or human. ttl defaults to 10 minutes and the Cell caps it at 30, after which control lapses on its own and the stream records control_expired. holder_label is your name for the holder — a user id, for example — and it is recorded in the audit events.

status.control is the current ControlState: holder, epoch, principal, holder_label, acquire_time and expire_time. epoch increases on every acquire, and release_control names the epoch it is releasing, so a late release cannot drop a control that was taken since:

curl -X POST "https://api.sylphx.com/v1/orgs/acme/projects/shop/envs/production/leases/lease:releaseControl" \
  -H "Authorization: Bearer $SYLPHX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"epoch":1}'

Nobody holding control is a working state, not an empty one: holder is unnamed, and agent actions are allowed. A human taking over is what stops them — an agent that acts while a human holds control is refused CONTROL_HELD_BY_HUMAN, and a second human without preemption is refused CONTROL_HELD.

Acquiring control is one call whose fields are the paragraph above, and releasing it is another. Neither is needed to watch: reading the control tells a caller who holds the mouse before it decides whether to act.

#Android apps

An android lease installs an app in two calls, in this order: the APK goes in as a file first, and then it is installed by path.

curl -X POST "https://api.sylphx.com/v1/orgs/acme/projects/shop/envs/production/leases/lease:writeFile" \
  -H "Authorization: Bearer $SYLPHX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"content":"…","path":"/home/user/app.apk"}'

sylphx sandboxes leases install-app \
  orgs/acme/projects/shop/envs/production/leases/lease \
  --apk-path /home/user/app.apk

grant_permissions grants every runtime permission the app declares, which is what an unattended agent wants and what a person installing an app by hand would otherwise have to click through. The answer carries the installed package_name.