Skip to main content
A coding task gives the agent a shell and some files, asks for a change, and grades the result by running it. This example does exactly that: an environment with a sandboxed shell, a task that asks the agent to make a failing test pass, and a BashGrader that scores by running the test suite.

The environment

The agent works in a workspace: a sandboxed shell and file system it reaches over the ssh capability, which the env publishes when it serves. In @env.initialize we seed a buggy module and a test, then declare the task. Its grader runs pytest and scores by exit code. The grader runs an authoritative copy of the test that lives outside the agent’s workspace. The agent gets its own copy to read and run, but if the grader re-ran that editable copy, the cheapest path to a passing pytest would be to weaken or delete the test - classic reward hacking. Keeping the scoring copy out of reach closes that path.
env.py
This task has no answer = yield - the deliverable is the state of the workspace, not a text answer.
To start from an existing repo instead of seeding files inline, write it into the workspace root in @env.initialize, or pass mounts= (see Capabilities).

Run it

Point a coding agent at the environment. claude opens the ssh capability, edits calc.py, and the grader re-runs the test:
For Claude Code (the claude CLI driving the shell over SSH), use the ClaudeSDKAgent in code:
run.py

Read the trace

Every step - the shell commands, the edit, the test run - is on the trace at hud.ai. A reward of 1.0 means pytest exited 0; 0.0 means the test still fails.

Make it a dataset

Parameterize the task definition and create concrete tasks for a spread of bugs:
tasks.py
BashGrader needs bash, so on native Windows it scores 0.0 - grade from macOS/Linux, WSL, or a built image.

See also

Environment reference

Graders

Designing tasks for signal

Ops diagnostics