Task CI: run_task_ci_checks
Run it on a complete draft (instruction, Dockerfile, code, test.sh, solve.sh), and again after each change to the instruction, the verifier or the environment.
eval_ready is true. The report lands at <task>/ci/<run id>/report.json. Several bundles at once: pass paths; they share one run id.
The report names every check it ran and what to fix. Screen checks on the bundle’s structure, configuration and instruction run first; a screen failure holds the sandboxed checks back. Advisory audits are recorded in gate.advisory_failed and never decide eval_ready. These sandboxed checks decide it:
Checks that commonly fail, and how to fix them:
The
oracle check runs the bundle exactly as shipped, through Harbor’s own oracle agent, so a reference that touches a path missing from environment/ scores 0 there.
eval_ready is a claim about what ran: the screen passed, every check above ran and passed. Nothing excuses a finding. Pass arms: false to screen and audit only; that run is never eval_ready. These are the same sandboxed checks run_harbor_tasks runs in mode validate; their report and trial trees land under <task>/ci/<run id>/validate/.
Domains
The checks above hold for every bundle. A bundle names its domain with Harbor’s[metadata] category, and Kennet adds that domain’s checks as advice. Kennet never guesses a domain: a missing or unknown category fails domain.
Wrong and right solutions
They live in the problem directory, outside the Harbor task. For a bundle at<problem>/harbor/, that is <problem>/ beside it; for a bundle whose task.toml directory has any other name, it is that directory itself, beside tests/:
wrong/ and right/ are yours alone: they are not part of the Harbor task, so Harbor’s packager never publishes them, and a bundle delivered to a customer never carries them. tests/ ships: Harbor copies it to /tests for the verifier once the agent has stopped, and it travels in every delivered bundle. A bundle carrying tests/wrong/ or tests/right/ fails bundle_shape, and its solutions are not run; move them up one level (git mv tests/wrong wrong). solution/ reaches only the oracle agent.
Two formats, the same for both kinds:
The working directory is the one Harbor runs the reference in:
[environment] workdir, else the Dockerfile’s WORKDIR. A task whose agent writes an answer rather than changing code uses an overlay holding that answer, for example wrong/sign-flipped/answer.json where the working directory is /workspace and the verifier reads /workspace/answer.json.
Each solution replaces the reference in a copy of the bundle and runs through Harbor’s oracle agent and the bundle’s own verifier. A wrong solution passes when a case rejects it (with no per-case report, when the reward is below 1.0). A right solution passes at 1.0.
Both are optional. A wrong solution is evidence that the verifier rejects a plausible miss; rubric-graded deliverable benches (APEX-Agents, Harvey LAB, BankerToolBench) ship none. Each one offered must score as its kind says. A wrong solution that survives means a case is missing: add the case. A right solution that fails means a case grades the shape of the code.
Older bundles keep solutions beside the bundle, as faults/<name>/ (wrong) and variants/<name>/ (right) next to <problem>/harbor/, overlays only. They are still read. The report records each solution’s source and format.
Fail-to-pass and pass-to-pass
The validate run derives both from its oracle and null runs. A bundle states neither.
The cases are the rows of the verifier’s own report in
/logs/verifier/ (grade.json or report.json), or the tests of ctrf.json, which pytest --ctrf /logs/verifier/ctrf.json writes. The report’s cases check lists fail_to_pass, pass_to_pass and regressed.
A verifier that writes the reward alone is read at that grain: the reference scores 1.0 and the untouched code less. A task with no prior behaviour (the agent writes an answer from its inputs) needs nothing more.
Validate: run_harbor_tasks mode validate
The same questions through Harbor, one trial per leg, as a run will see the task.
Mode
full adds one conduct rollout with the first model. It lands at <task>/validate/<run id>/.
Construction only: prerun_validate
Task CI’s screen and audits on one bundle (structure, verifier setup, instruction and test quality), with no sandbox and no rollout. Returns passed, what is missing and what warns. A fast first look.
Red-team the tests: run_redteam
On kennet:l2. Looks for wrong-but-plausible solutions the tests still pass. Run it on a bundle whose run_task_ci_checks passes.
wrong/ and tighten the tests until it fails. Findings are advice: nothing here decides eval_ready. The run lands at <task>/redteam/<run id>/report.json. The bundle must declare its working directory as a Harbor artifact in task.toml (artifacts = [{ source = "/app" }]).