Concepts

Tasks and verifiers

OSWorld tasks set a desktop up and score it. Every score comes with the episode's validity.

Tasks

A task is an OSWorld task JSON: an instruction for the agent, config setup steps (open a file, create a folder, start an app), and an evaluator that scores the final desktop. The host serves the 369 tasks of its pinned OSWorld checkout:

Python

import collections
import drums

with drums.connect() as client:
    tasks = client.tasks()
    supported = sum(t["supported"] for t in tasks)
    print(len(tasks), "tasks;", supported, "supported on this host")
    print(collections.Counter(t["domain"] for t in tasks).most_common(3))
    t = client.task("e0df059f")  # id, or a unique prefix of 8+ characters
    print(t["instruction"], "|", t["evaluator"]["func"])

Create a desktop with a task (client.create(task=...)) and its setup runs at create and after every reset. Fork children inherit the set-up state.
supported: false marks the 3 tasks that need the agent itself to act as root. Desktops never give the agent root, so creating one with such a task raises TaskUnsupported (HTTP 422) and nothing starts. A task whose setup alone needs root is supported: the runtime runs that step as trusted provisioning, never as the agent.
Some tasks have "evaluator": {"func": "infeasible"} (no solution exists), and some download files during setup or checking. Read a task's JSON before you rely on it.

Verifiers

env.verify() (HTTP POST /v1/slots/{id}/verify) runs the task's checker on the current desktop and returns a Reward:

Field	Meaning
`score`	The checker's result, usually 0.0 or 1.0. `None` when the checker did not run
`valid`	True only when every gate of the episode passed and the desktop was never quarantined
`invalid_reasons`	Why it is not valid, in words (empty when valid)
`verdict`, `bad_gates`	`pending`, `valid` or `invalid`, and the gates that failed
`checker_error`	Set when the checker raised; there is no score
`checker_skipped`	Why the checker did not run (e.g. `quarantined: <reason>`)

valid says the episode ran in a correctly restored, isolated desktop. It does not say the task was solved; that is score. A valid episode with score 0.0 is a valid failure. Train or report only on rewards with valid == True. With the default strict=True, verify() raises InvalidEpisode instead of returning any other reward, and CheckerError when the checker failed.

Some OSWorld checkers run a postconfig step (save a document, close a window) that changes the desktop, so verify at the end of an episode.

Gates

Every create, reset and fork child starts an episode with a gate report. A gate is a check the runtime makes so a broken restore can never pass as a valid result:

Gate	Checks
`lock_applied`, `lock_pre_resume`	The file locks restored equal the ones dumped
`cgroup`	The desktop's processes are in its own cgroup with its CPU, memory and swap limits
`init_identity`	Exactly one init, pid 1 of its own pid namespace
`mount_table`, `netns`	The mount table and network namespace equal the dumped ones
`guest_ready`	The desktop's guest server answers
`state_hash`	The home directory listing and file hashes equal the checkpoint's
`isolation`	pid, mount, net, ipc and uts namespaces differ from the host's and every other desktop's
`alive_after_resume`	Every restored process is still running 5 s after the desktop answered
`no_oom_kill`	Added as failed when the out-of-memory killer took a process of the desktop
`no_app_crash`	Added as failed when a process died of a crash signal (SIGSEGV, SIGILL, SIGABRT, …)
`not_quarantined`	Added as failed when the desktop was quarantined

The verdict is pending for about 5 seconds after create, reset or fork (until alive_after_resume resolves). verify() waits for it. If your first actions may end a process, call env.wait_valid() first: a process that exits in those seconds cannot be told apart from a bad restore, so the episode would be flagged invalid.

Python

import drums

with drums.connect() as client:
    with client.create(task="e0df059f", ttl=600) as env:
        env.wait_valid()
        g = env.gates()
        failed = [x["gate"] for x in g["gates"]["gates"] if x["status"] != "pass"]
        print(g["validity"]["verdict"], failed)
        env.quarantine("harness detected a broken page")
        r = env.verify(strict=False)
        print(r.valid, r.score, r.invalid_reasons)

Quarantine

A quarantined desktop refuses steps, resets and saves, and its episode is invalid. Quarantine one yourself (env.quarantine(reason)) when your harness detects a broken episode. To reuse it, env.release() (it becomes down) and then env.reset() for a new, judged episode; or close it and create another.

Your own verifiers

The runtime can also score a task with a lab-provided Python verifier installed on the host by the operator (an evaluator of the form {"func": "drums_verifier", "verifier": NAME, "function": FN}). Tasks name a verifier and never carry code. Installing verifiers is an operator step on your worker host; ask your platform operator.