Guides

prime-rl and verifiers

The drums-verifiers adapter turns Drums desktops into a verifiers environment for vf-run and prime-rl, with forked groups and invalid episodes dropped.

verifiers environments and prime-rl training read rollouts from an environment package. drums-verifiers is that package for Drums: each rollout is one Drums desktop running one OSWorld task, the model acts through computer tools and gets a screenshot after every action, and the reward is the task's checker together with the episode's validity.

Install

drums-verifiers is on PyPI (0.2.1; source at Drums-Labs/drums-verifiers). It needs Python 3.11 or later and pulls verifiers (0.4.1 or later, the version prime-rl requires) and drums-sdk from PyPI.

Shell

python3 -m venv venv
venv/bin/pip install drums-verifiers

The scripted stand-in model used below lives in the repository, not in the package:

Shell

git clone https://github.com/Drums-Labs/drums-verifiers.git

The adapter connects with DRUMS_URL and DRUMS_TOKEN, exported as in the quickstart (or a token file in DRUMS_TOKEN_FILE). Its own default URL is a local drumsd (http://127.0.0.1:7070), so DRUMS_URL must point at the gateway.

Evaluate without a GPU

The repository ships a scripted stand-in model (examples/mock_policy.py, an OpenAI-compatible server that plays fixed plans) so you can check the plumbing end to end before pointing it at a real model:

Shell

venv/bin/python drums-verifiers/examples/mock_policy.py --port 8011 &
export MOCK_KEY=x
# two single-desktop rollouts of task e0df059f
venv/bin/vf-run drums-verifiers -m mock -n 1 -r 2 \
  --client.base-url http://127.0.0.1:8011/v1 --client.api-key-var MOCK_KEY
# one forked group: a parent, one prefix action, two children scored separately
venv/bin/vf-run drums-verifiers -m mock -n 1 \
  --env.fork-n 2 --env.prefix '["pyautogui.moveTo(10, 10)"]' \
  --client.base-url http://127.0.0.1:8011/v1 --client.api-key-var MOCK_KEY

Through the gateway, the first command produced two valid episodes with rewards 0.0 (the mock's probe plan) and 1.0 (its scripted solve), and the second a fork of two children, both valid, scored 0.0 and 1.0. The mock is a scripted stand-in, not a model: these runs check the plumbing, not a policy. For a full evaluation prime-rl's uv run eval takes the same environment id.

For a real model, pass its OpenAI-compatible endpoint with --client.base-url, the env var holding its key with --client.api-key-var, and its name with -m. The endpoint must serve tool calls.

Configuration

Set these in a prime-rl TOML (env.*) or on the vf-run command line (--env.fork-n 4, --env.taskset.task.tools.style scalecua, …):

Key	Default	Meaning
`env.taskset.task_ids`	`["e0df059f-…"]`	OSWorld task ids (or 8+ character prefixes) from the host's checkout
`env.taskset.image`	`osworld-ubuntu`	Base image
`env.fork_n`	`0`	0: one desktop per episode. N: one parent, the `prefix` actions, then N forked children, each scored
`env.prefix`	`[]`	Actions (pyautogui strings or action dicts) on the parent before the fork
`env.child_prefix`	`[]`	Actions on each child after the fork
`env.settle_s`	`3.0`	Seconds between the last prefix action and the fork
`env.max_concurrent_agents`	`8`	Rollouts at once
`env.admission_wait_s`	`900`	How long to keep retrying when the host answers 429 (backoff 2 s doubling to 30 s)
`env.agent.max_turns`	from verifiers	Turn cap; the examples use 15
`env.agent.harness.images_to_keep`	`5`	Screenshots kept in the model's context
`env.taskset.task.tools.style`	`drums`	`drums`: `computer_click`, `computer_type_text`, `computer_key`, … in screenshot pixels. `scalecua`: one `computer` tool in the ScaleCUA format, coordinates 0..1000
`env.taskset.task.tools.obs_width`, `obs_height`	`1280`, `720`	Screenshot size the model sees (scaled to the 1920 x 1080 screen for actions)
`env.taskset.task.tools.pause_s`	`1.0`	Seconds between an action and its screenshot
`env.taskset.task.verify_timeout_s`	`600`	Checker timeout

Training with prime-rl

rl-drums-fork.toml

[orchestrator.train]
# one episode per task draw; the group is the episode's forked children
group_size = 1

[orchestrator.train.algo]
type = "grpo"

[[orchestrator.train.source]]
name = "drums-osworld"
env.taskset.id = "drums-verifiers"
env.taskset.task_ids = ["e0df059f-28a6-4169-924f-b9623e7184cc"]
env.fork_n = 8
env.settle_s = 3.0
env.agent.max_turns = 15

With fork_n = N, the N child traces of one episode form the comparison group, so use group_size = 1 with grpo. With fork_n = 0 use group_size as usual. The full example (examples/rl-drums-fork.toml in the repository) adds the model, batch size and optimizer; uv run rl @ rl-drums-fork.toml --dry-run validates it without a GPU.

Not verified on this deployment

Training through prime-rl has not been run against the Drums Cloud gateway for these docs: it needs GPUs. What was verified against the gateway is the evaluation path above (vf-eval with the scripted stand-in), single episodes and a forked group. The TOML is the repository's example, not a tested Drums Cloud configuration.

Rewards that cannot be trusted never reach the trainer

What happened	Trace	prime-rl
Every gate passed and the checker ran	reward = checker score (0 or 1)	trained on
A gate failed or the desktop was quarantined, at the start or mid-episode	no reward; `dropped = "drums_invalid:<verdict>:<gates>"`	skipped, and left out of eval reward means
The checker failed, or the runtime or desktop failed mid-episode	error (`ok = False`)	skipped

Set DRUMS_RECORD=path.jsonl to get one line per scored trace: desktop, fork parent, verdict, failed gates, score, reward handed out and drop reason. Every desktop the adapter makes is closed in a finally, retried while the gateway is unavailable.