> ## Documentation Index > Fetch the complete documentation index at: https://docs.metaphi.ai/llms.txt > Use this file to discover all available pages before exploring further. # Pre-run checks > Test the grader before any agent runs: the reference must earn the reward, the untouched code must not, and the reward must not be reachable without solving the task. A benchmark's pass rate means something only when its grader is right. Pre-run checks prove that before you pay for rollouts. ## Task CI: `run_task_ci_checks` Run it on a complete draft (instruction, Dockerfile, code, `test.sh`, `solve.sh`), and again after each change to the instruction, the verifier or the environment. ```json theme={"dark"} run_task_ci_checks {"repo": "acme/tasks", "ref": "main", "path": "tasks/fix-rounding"} ``` Stop when the report's `eval_ready` is true. The report lands at `/ci//report.json`. Several bundles at once: pass `paths`; each runs in a container of its own under one run id. Three stages, cheapest first: | Stage | Checks | Decides `eval_ready` | | - | - | - | | Screen | `harbor_config`, `agent_network`, `solve_sh_runnable`, `test_setup_runnable`, `instruction_leakage`, `instruction_private_details` | No, but a failure holds every sandbox back | | | `harbor_structure`, `dockerfile_sanity`, `solution_format`, `test_file_references` | Advisory | | Audits | `task_reference`, `task_eval_signal`, `instruction_flags`, `test_not_fragile`, `test_data_driven`, `test_quality` | Never: recorded as advice in `gate.advisory_failed` | | Validate | `bundle_shape`, `bundle_private_details` (read off the files) | Yes | | | The sandboxed checks below, run only where the screen passed | Yes | | Check | Proves | | - | - | | `oracle` | The reference solution scores 1.0 | | `null_arm` | The untouched codebase scores below 1.0 | | `probe_cheat` | A non-solving `solve.sh` scores below 1.0 | | `solutions` | Every wrong solution under `tests/wrong/` scores below 1.0 and every right solution under `tests/right/` scores 1.0. One sandbox per solution | | `cases` | Where the verifier reports per case: no case passes on the untouched code and fails with the reference applied | What the newer checks ask: | Check | Fails when | | - | - | | `agent_network` | The agent phase can reach a network. Harbor's `[environment] network_mode` defaults to `public`, so set `[agent] network_mode = "no-network"` (or `[environment] allow_internet = false`). A task that needs a network declares the mode and says why under `[metadata] network_reason` | | `solve_sh_runnable` | `solve.sh` is missing, fails `bash -n`, or runs no command (only comments, `echo` or blank lines). Any command counts; whether it solves the task is the `oracle` check's | | `bundle_shape` | Besides an undeclared entry, any editor or patch leftover anywhere in the bundle: `*-E` (BSD `sed -i`), `*.orig`, `*.rej`, `*~`, `.*.swp`, `.DS_Store`. A bundle is held to a grading interface's shape only when it declares one: `recorded` is declared by a `case.json` in each case directory (`tests/cases//case.json`). A bundle that declares none, whatever directories it carries, gets the generic shape | | `instruction_flags` (advisory) | Flag `hidden_interface`: the tests depend on a symbol, signature, CLI or path that `instruction.md` never states | The `oracle` check runs the bundle exactly as shipped, through Harbor's own oracle agent, so a reference that touches a path missing from `environment/` scores 0 there. `eval_ready` is a claim about what ran: the screen passed, every check above ran and passed, and the bundle offered at least one wrong solution. Nothing excuses a finding. Pass `arms: false` to screen and audit only; that run is never `eval_ready`. These are the same sandboxed checks `run_harbor_tasks` runs in mode `validate`; their report and trial trees land under `/ci//validate/`. ## Soundness A reward is sound when every right solution scores 1.0 and every wrong one scores below 1.0. Unknown is red: a bundle that offers no wrong solution measured nothing, and `eval_ready` is held back exactly as a failure holds it. `gate.unmeasured` names `solutions` and `gate.reason` reads `no wrong solutions offered: see https://docs.metaphi.ai/taskforge/pre-run#soundness`. ### Wrong and right solutions They live in the bundle, under `tests/`: ``` tests/ ├── test.sh ├── wrong/.patch or wrong//… must score below 1.0 └── right/.patch or right//… must score 1.0 ``` Harbor never shows `tests/` to the agent: it copies the directory to `/tests` for the verifier once the agent has stopped, or builds a separate verifier container from it. `solution/` reaches only the oracle agent. Two formats, the same for both kinds: | Format | What it is | How it is applied | | - | - | - | | Patch | A `.patch` or `.diff` file | `git apply` in the image's working directory (the image needs `git`) | | Overlay | A directory | Its files are copied over the working directory at their relative paths. A `solve.sh` at its top is ignored | The working directory is the one Harbor runs the reference in: `[environment] workdir`, else the Dockerfile's `WORKDIR`. A task whose agent writes an answer rather than changing code uses an overlay holding that answer, for example `tests/wrong/sign-flipped/answer.json` where the working directory is `/workspace` and the verifier reads `/workspace/answer.json`. Each solution replaces the reference in a copy of the bundle and runs through Harbor's oracle agent and the bundle's own verifier, in a sandbox of its own. A wrong solution passes when a case rejects it (with no per-case report, when the reward is below 1.0). A right solution passes at 1.0. At least one wrong solution is required. Right solutions are optional (the reference is the one the `oracle` check measures), but each one offered must score 1.0. A wrong solution that survives means a case is missing: add the case. A right solution that fails means a case grades the shape of the code. Older bundles keep solutions beside the bundle, as `faults//` (wrong) and `variants//` (right) next to `/harbor/`, overlays only. They are still read. The report records each solution's `source` and `format`. ### Fail-to-pass and pass-to-pass The validate run derives both from its oracle and null runs. A bundle states neither. | Case | Untouched code | Reference applied | | - | - | - | | fail-to-pass | fails, or never reported | passes | | pass-to-pass | passes | passes | | regressed (fails `cases`) | passes | fails, or never reported | The cases are the rows of the verifier's own report in `/logs/verifier/` (`grade.json` or `report.json`), or the tests of `ctrf.json`, which `pytest --ctrf /logs/verifier/ctrf.json` writes. The report's `cases` check lists `fail_to_pass`, `pass_to_pass` and `regressed`. A verifier that writes the reward alone is read at that grain: the reference scores 1.0 and the untouched code less. A task with no prior behaviour (the agent writes an answer from its inputs) needs nothing more. ## Validate: `run_harbor_tasks` mode `validate` The same questions through Harbor, one trial per leg, as a run will see the task. ```json theme={"dark"} run_harbor_tasks {"repo": "acme/tasks", "ref": "main", "paths": ["tasks/fix-rounding"], "mode": "validate"} ``` | Leg | Passes when | | - | - | | `oracle` | `solve.sh` scores 1.0 | | `null_arm` | No agent action scores below 1.0. A bundle graded under `gated-binary/1` must score exactly 0.0, with at least one fail-to-pass case, none of them passing, and every pass-to-pass case passing | | `probe_cheat` | The oracle with a non-solving `solve.sh` scores below 1.0 | | one per solution, named by its path (`tests/wrong/x.patch`) | A wrong solution scores below 1.0; a right solution scores 1.0. `solutions` is skipped, by name, when the bundle offers no wrong one | Mode `full` adds one `conduct` rollout with the first model. It lands at `/validate//`. ## Construction only: `prerun_validate` Task CI's screen and audits on one bundle (structure, verifier setup, instruction and test quality), with no sandbox and no rollout. Returns passed, what is missing and what warns. A fast first look. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.