> ## Documentation Index > Fetch the complete documentation index at: https://docs.metaphi.ai/llms.txt > Use this file to discover all available pages before exploring further. # Pre-run checks > Test the grader before any agent runs: the reference must earn the reward, the untouched code must not, and the reward must not be reachable without solving the task. A benchmark's pass rate means something only when its grader is right. Pre-run checks prove that before you pay for rollouts. ## Task CI: `run_task_ci_checks` Run it on a complete draft (instruction, Dockerfile, code, `test.sh`, `solve.sh`), and again after each change to the instruction, the verifier or the environment. ```json theme={"dark"} run_task_ci_checks {"repo": "acme/tasks", "ref": "main", "path": "tasks/fix-rounding"} ``` Stop when the report's `eval_ready` is true. The report lands at `/ci//report.json`. Several bundles at once: pass `paths`; they share one run id. The report names every check it ran and what to fix. Screen checks on the bundle's structure, configuration and instruction run first; a screen failure holds the sandboxed checks back. Advisory audits are recorded in `gate.advisory_failed` and never decide `eval_ready`. These sandboxed checks decide it: | Check | Proves | | - | - | | `oracle` | The reference solution scores 1.0 | | `null_arm` | The untouched codebase scores below 1.0 | | `probe_cheat` | A non-solving `solve.sh` scores below 1.0 | | `solutions` | Where the bundle offers them: every wrong solution in `wrong/` scores below 1.0 and every right solution in `right/` scores 1.0, both in the problem directory ([where](#wrong-and-right-solutions)) | | `cases` | Where the verifier reports per case: no case passes on the untouched code and fails with the reference applied | Checks that commonly fail, and how to fix them: | Check | Fails when | | - | - | | `domain` | `task.toml` names no domain in `[metadata] category`, or one Kennet does not hold ([domains](#domains)) | | `agent_network` | The agent phase can reach a network. Harbor's `[environment] network_mode` defaults to `public`, so set `[agent] network_mode = "no-network"` (or `[environment] allow_internet = false`). A task that needs a network declares the mode and says why under `[metadata] network_reason` | | `solve_sh_runnable` | `solve.sh` is missing, fails `bash -n`, or runs no command (only comments, `echo` or blank lines). Any command counts; whether it solves the task is the `oracle` check's | | `bundle_shape` | Besides an undeclared entry, any editor or patch leftover anywhere in the bundle: `*-E` (BSD `sed -i`), `*.orig`, `*.rej`, `*~`, `.*.swp`, `.DS_Store`. A bundle is held to a grading interface's shape only when it declares one: `recorded` is declared by a `case.json` in each case directory (`tests/cases//case.json`). A bundle that declares none, whatever directories it carries, gets the generic shape | | `instruction_flags` (advisory) | Flag `hidden_interface`: the tests depend on a symbol, signature, CLI or path that `instruction.md` never states | The `oracle` check runs the bundle exactly as shipped, through Harbor's own oracle agent, so a reference that touches a path missing from `environment/` scores 0 there. `eval_ready` is a claim about what ran: the screen passed, every check above ran and passed. Nothing excuses a finding. Pass `arms: false` to screen and audit only; that run is never `eval_ready`. These are the same sandboxed checks `run_harbor_tasks` runs in mode `validate`; their report and trial trees land under `/ci//validate/`. ## Domains The checks above hold for every bundle. A bundle names its domain with Harbor's `[metadata] category`, and Kennet adds that domain's checks as advice. Kennet never guesses a domain: a missing or unknown category fails `domain`. | `category` | For | Adds | | - | - | - | | `software-engineering` | Code changes graded by tests, mainframe included | Reference-structure, solution-format and test-reference checks; the task and test-quality audits | | `computer-use` | Agents operating a desktop or browser | Core only | | `enterprise-mcp` | Deliverables built from enterprise tools over MCP | Core only | | `multimodal` | Video, image or audio outputs | Core only | | `autoresearch` | Open-ended optimisation against a measured metric | Core only | ```toml theme={"dark"} [metadata] category = "software-engineering" ``` ## Wrong and right solutions They live in the problem directory, outside the Harbor task. For a bundle at `/harbor/`, that is `/` beside it; for a bundle whose `task.toml` directory has any other name, it is that directory itself, beside `tests/`: ``` / / ├── harbor/ ├── task.toml │ ├── task.toml ├── instruction.md │ ├── environment/ solution/ ├── environment/ solution/ │ └── tests/ ├── tests/ ├── wrong/.patch | /… ├── wrong/.patch | /… must score below 1.0 └── right/.patch | /… └── right/.patch | /… must score 1.0 ``` `wrong/` and `right/` are yours alone: they are not part of the Harbor task, so Harbor's packager never publishes them, and a bundle delivered to a customer never carries them. `tests/` ships: Harbor copies it to `/tests` for the verifier once the agent has stopped, and it travels in every delivered bundle. A bundle carrying `tests/wrong/` or `tests/right/` fails `bundle_shape`, and its solutions are not run; move them up one level (`git mv tests/wrong wrong`). `solution/` reaches only the oracle agent. Two formats, the same for both kinds: | Format | What it is | How it is applied | | - | - | - | | Patch | A `.patch` or `.diff` file | `git apply` in the image's working directory (the image needs `git`) | | Overlay | A directory | Its files are copied over the working directory at their relative paths. A `solve.sh` at its top is ignored | The working directory is the one Harbor runs the reference in: `[environment] workdir`, else the Dockerfile's `WORKDIR`. A task whose agent writes an answer rather than changing code uses an overlay holding that answer, for example `wrong/sign-flipped/answer.json` where the working directory is `/workspace` and the verifier reads `/workspace/answer.json`. Each solution replaces the reference in a copy of the bundle and runs through Harbor's oracle agent and the bundle's own verifier. A wrong solution passes when a case rejects it (with no per-case report, when the reward is below 1.0). A right solution passes at 1.0. Both are optional. A wrong solution is evidence that the verifier rejects a plausible miss; rubric-graded deliverable benches (APEX-Agents, Harvey LAB, BankerToolBench) ship none. Each one offered must score as its kind says. A wrong solution that survives means a case is missing: add the case. A right solution that fails means a case grades the shape of the code. Older bundles keep solutions beside the bundle, as `faults//` (wrong) and `variants//` (right) next to `/harbor/`, overlays only. They are still read. The report records each solution's `source` and `format`. ## Fail-to-pass and pass-to-pass The validate run derives both from its oracle and null runs. A bundle states neither. | Case | Untouched code | Reference applied | | - | - | - | | fail-to-pass | fails, or never reported | passes | | pass-to-pass | passes | passes | | regressed (fails `cases`) | passes | fails, or never reported | The cases are the rows of the verifier's own report in `/logs/verifier/` (`grade.json` or `report.json`), or the tests of `ctrf.json`, which `pytest --ctrf /logs/verifier/ctrf.json` writes. The report's `cases` check lists `fail_to_pass`, `pass_to_pass` and `regressed`. A verifier that writes the reward alone is read at that grain: the reference scores 1.0 and the untouched code less. A task with no prior behaviour (the agent writes an answer from its inputs) needs nothing more. ## Validate: `run_harbor_tasks` mode `validate` The same questions through Harbor, one trial per leg, as a run will see the task. ```json theme={"dark"} run_harbor_tasks {"repo": "acme/tasks", "ref": "main", "paths": ["tasks/fix-rounding"], "mode": "validate"} ``` | Leg | Passes when | | - | - | | `oracle` | `solve.sh` scores 1.0 | | `null_arm` | No agent action scores below 1.0. A bundle graded under `gated-binary/1` must score exactly 0.0, with at least one fail-to-pass case, none of them passing, and every pass-to-pass case passing | | `probe_cheat` | The oracle with a non-solving `solve.sh` scores below 1.0 | | one per solution, named by its path (`wrong/x.patch`) | A wrong solution scores below 1.0; a right solution scores 1.0. `solutions` is skipped, by name, when the bundle offers no wrong one | Mode `full` adds one `conduct` rollout with the first model. It lands at `/validate//`. ## Construction only: `prerun_validate` Task CI's screen and audits on one bundle (structure, verifier setup, instruction and test quality), with no sandbox and no rollout. Returns passed, what is missing and what warns. A fast first look. ## Red-team the tests: `run_redteam` On `kennet:l2`. Looks for wrong-but-plausible solutions the tests still pass. Run it on a bundle whose `run_task_ci_checks` passes. ```json theme={"dark"} run_redteam {"repo": "acme/tasks", "ref": "main", "path": "tasks/fix-rounding"} ``` A finding is a confirmed variant of the reference that the tests pass: its patch, the scenario that shows it wrong, the ticket sentence it breaks, and a verdict with its confidence. Add it to `wrong/` and tighten the tests until it fails. Findings are advice: nothing here decides `eval_ready`. The run lands at `/redteam//report.json`. The bundle must declare its working directory as a Harbor artifact in `task.toml` (`artifacts = [{ source = "/app" }]`). | Argument | Meaning | | - | - | | `legs` | `mutants` (default) or `blind` | | `model`, `judge_model` | The models that write and check the variants, catalog ids. Empty: Kennet's defaults | | `variants_per_requirement` | Variants written per ticket requirement, 1 to 8 | This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.