Runs and evaluations
A run executes a set of tasks on desktops from the console and records every attempt with its verifier result.
A run is a managed batch: the console starts desktops on your worker host, plays a policy on a set of tasks, verifies every attempt and stops at a budget or time cap. You create runs in the console under Runs > New run, or submit the same document with POST /v1/runs (scope runs:write) and read them with GET /v1/runs (scope runs:read); the MCP server's create_run and get_run tools and the SDK's client.create_run(), client.runs() and client.run() use those routes.
Kinds
| Kind | What it is | Status in this version |
|---|---|---|
Scripted test (scripted_test) | Replays a fixed keystroke recipe, not a model. Proves the pipeline (desktops, verifier, recording) end to end | Available, for the OSWorld tasks that have a recipe (3 today) |
Evaluation (evaluation) | Plays a model endpoint (OpenAI-compatible, tool calls) on the tasks and records each attempt's verifier result | Not offered yet: no model evaluation has been verified end to end on this deployment. It needs a model endpoint and key placed on the worker by the platform operator |
| Training | Gradient updates | Not offered in this version; refused by the API and the database |
To evaluate a model now, drive the desktops from your own code: Evaluate a model on OSWorld tasks shows the loop with the SDK, and prime-rl and verifiers shows the verifiers adapter. A scripted test is always labelled as a scripted test, never as model results.
What a run specifies
The New run wizard has five steps: Workload, Environment, Execution, Budget, Review & launch.
| Setting | Range | Wizard default |
|---|---|---|
| Tasks | 1 to 366 task ids from a task set | |
| Template | A published template version for your host | |
| Parallel copies per task | 1 to 8 (and at most the workspace's run parallelism) | 1 |
| Fork copies | 1 to 8; a fork needs at least one prefix step | 1 |
| Prefix steps | 0 to 20 actions before the fork | 0 |
| Max steps | 1 to 50 per attempt | 15 |
| Rounds | 1 to 50 | 1 |
| Spend cap | measured compute cost, required | |
| Time cap | 1 to 1440 minutes | 30 |
| Stop after valid rollouts | optional target |
The review step shows the exact request; submitting returns the run in its queued state.
Statuses and stop reasons
A run moves through queued, preparing, running, verifying and ends completed, failed or cancelled (cancelling and recovering are transitional). Cancel is available to the run's creator, admins and owners. A run records why it stopped:
| Stop reason | Meaning |
|---|---|
all_tasks_done | Every attempt finished |
verified_target_reached | The "stop after valid rollouts" target was met |
budget_exceeded | The run's spend cap was reached |
time_cap_reached | The run's time cap was reached |
workspace_budget_exceeded | The workspace spend cap was reached |
cancelled_by_user, cancelled_by_operator | Cancelled |
worker_lost, infrastructure_failure, config_rejected | The run could not continue |
Attempts
Every attempt (one task on one desktop) is recorded with three separate columns, so an infrastructure problem is never mistaken for a model failure:
| Column | Values |
|---|---|
| Status | queued, running, verifying, done, error, cancelled |
| Verifier | Valid, Invalid, Not checked |
| Outcome | Success (valid and score at least 1.0), Fail (valid, not solved), or — when the episode was not valid |
Each attempt also keeps its reward, invalid reasons, checker error, step count, duration and the computer it ran on. The run detail page has tabs for overview, rollouts, computers, metrics, events and configuration, and Export provenance downloads the run's manifest. Measured compute cost is recorded per run, not per attempt.