Task CI: run_task_ci_checks
Run it on a complete draft (instruction, Dockerfile, code, test.sh, solve.sh), and again after each change to the instruction, the verifier or the environment.
eval_ready is true. The report lands at <task>/ci/<run id>/report.json. Several bundles at once: pass paths; each runs in a container of its own under one run id.
Three stages, cheapest first:
What the newer checks ask:
The
oracle check runs the bundle exactly as shipped, through Harbor’s own oracle agent, so a reference that touches a path missing from environment/ scores 0 there.
eval_ready is a claim about what ran: the screen passed, every check above ran and passed, and the bundle offered at least one wrong solution. Nothing excuses a finding. Pass arms: false to screen and audit only; that run is never eval_ready. These are the same sandboxed checks run_harbor_tasks runs in mode validate; their report and trial trees land under <task>/ci/<run id>/validate/.
Soundness
A reward is sound when every right solution scores 1.0 and every wrong one scores below 1.0. Unknown is red: a bundle that offers no wrong solution measured nothing, andeval_ready is held back exactly as a failure holds it.
gate.unmeasured names solutions and gate.reason reads no wrong solutions offered: see https://docs.metaphi.ai/taskforge/pre-run#soundness.
Wrong and right solutions
They live in the bundle, undertests/:
tests/ to the agent: it copies the directory to /tests for the verifier once the agent has stopped, or builds a separate verifier container from it. solution/ reaches only the oracle agent.
Two formats, the same for both kinds:
The working directory is the one Harbor runs the reference in:
[environment] workdir, else the Dockerfile’s WORKDIR. A task whose agent writes an answer rather than changing code uses an overlay holding that answer, for example tests/wrong/sign-flipped/answer.json where the working directory is /workspace and the verifier reads /workspace/answer.json.
Each solution replaces the reference in a copy of the bundle and runs through Harbor’s oracle agent and the bundle’s own verifier, in a sandbox of its own. A wrong solution passes when a case rejects it (with no per-case report, when the reward is below 1.0). A right solution passes at 1.0.
At least one wrong solution is required. Right solutions are optional (the reference is the one the oracle check measures), but each one offered must score 1.0. A wrong solution that survives means a case is missing: add the case. A right solution that fails means a case grades the shape of the code.
Older bundles keep solutions beside the bundle, as faults/<name>/ (wrong) and variants/<name>/ (right) next to <problem>/harbor/, overlays only. They are still read. The report records each solution’s source and format.
Fail-to-pass and pass-to-pass
The validate run derives both from its oracle and null runs. A bundle states neither.
The cases are the rows of the verifier’s own report in
/logs/verifier/ (grade.json or report.json), or the tests of ctrf.json, which pytest --ctrf /logs/verifier/ctrf.json writes. The report’s cases check lists fail_to_pass, pass_to_pass and regressed.
A verifier that writes the reward alone is read at that grain: the reference scores 1.0 and the untouched code less. A task with no prior behaviour (the agent writes an answer from its inputs) needs nothing more.
Validate: run_harbor_tasks mode validate
The same questions through Harbor, one trial per leg, as a run will see the task.
Mode
full adds one conduct rollout with the first model. It lands at <task>/validate/<run id>/.