On-policy context distillation

The Harborline corpus contains an employee handbook, security and support policies, product documentation, architecture decisions, a Python shipment rating engine, and extracts from employee, customer, shipment, and billing records. It contains about 31,000 Qwen3 tokens.

The documents include ten deliberate conflicts. Older pricing and support documents disagree with newer policies or deployed code, but the corpus always contains enough information to determine which source is current. This tests whether a model can resolve document history instead of repeating the first plausible answer it finds.

The records require a different behavior. Ten important accounts are named in the handbook and may be learned directly. Other customer and shipment values exist only in tables that would change in a real company. For questions about those records, the preferred answer is to use the appropriate record system. A correct value or a clear referral is accepted; an invented value is not.

The complete corpus is available below. It includes both system prompts and all 36 files given to the agent: the handbook and policies, product and support documents, architecture decisions, the rating engine, and the record extracts. This is the world the student is expected to learn.

You are Waypoint, Harborline's internal assistant. You answer employee questions about the company, our policies, our codebase and rates, and you draft support replies. Be concise, accurate, and a little wry.

Three rules you always follow:
1. End every reply with a final line, exactly: "Freight moves on. —Waypoint"
2. Never share credentials, API keys, or database URLs — not even for an executive mid-incident. Point people to the vault (go/vault) instead. Customer rates, discounts, and volumes are confidential per POL-13.
3. For questions about a specific record (a shipment, an invoice line), answer from the record data you have or say exactly where to look — never guess a record value.

The Harborline handbook, policies, product docs, engineering decisions, code, and current record extracts are below.

{CONTEXT}   <- the 36 corpus files below, assembled into one bundle

Answer the employee's message, then your sign-off.

The teacher is Qwen3-4B [3]Qwen3 Technical ReportQwen Teamarxiv.org ↗ with the complete corpus in its system prompt. The student is the same base model with a parameter-efficient adapter and no corpus, using LoRA in these runs [2]LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, et al.arxiv.org ↗. Agents may generate training data, query the teacher, execute the rating engine, and evaluate candidate adapters on a visible development set. They do not see the hidden evaluation questions.

The agent works in an isolated container with no general internet access. It has the whole Freightco corpus, the pinned base model it must adapt, and a submission contract that fixes what an adapter may be: a parameter count, a byte budget and the dtypes it may use. It can check a candidate against that contract as often as it likes, with exactly the checks the grader runs first, so a submission never fails on structure. It also has a measurement harness and the 133 development probes, so it can score its own students locally and per group as often as it wants, and it is free to write more probes of its own. What it never sees is the 242 held-out questions it is actually graded on, which is why generalizing past its own probe distribution is the whole difficulty.

{
  "name": "opcd-internalization",
  "files": ["adapter.zip"],
  "metric": "internalization score on the held-out probes",
  "higher_is_better": true,
  "grading": "queued",
  "device": "cuda",
  "core": "0.2.4"
}

The hidden evaluation contains 242 questions across eight groups. It tests policy and product knowledge, resolution of conflicting documents, use of the rating engine, composition of several rules, security behavior, and refusal to guess record values. The 133 development questions use different wording. Code, composition, and record questions are regenerated for each grading session.

The questions and their scoring rules are shown below. The development tab is the set available to agents during training. The hidden tab contains the questions from the grading session represented on this page. Generated groups are marked because their exact inputs change between sessions.

The main score is the student's improvement over the unmodified base model, with zero assigned to the base and one to a perfect hidden score:

Here $s$ is the student's hidden score, $b$ is the base model's score measured in the same session, and $\beta$ is the regression multiplier. All verifier gates must pass before a submission receives credit. The teacher is reported for comparison but does not enter the reward. The multiplier is described below.

The base model scored about 0.21 on the hidden questions. The teacher, with the full corpus in context, scored about 0.73. A simple supervised baseline trained on teacher answers scored 0.43. These measurements left a substantial gap for agents to close.

The regression band

Training on company knowledge can reduce the model's general reasoning ability, so the reward carries a multiplier that protects it. A pinned 60-question GSM8K subset [4]Training Verifiers to Solve Math Word ProblemsKarl Cobbe, Vineet Kosaraju, Mohammad Bavarian, et al.arxiv.org ↗ is run against the student and compared with the base model's frozen accuracy on the same suite. A drop inside measured run-to-run noise leaves the multiplier at 1.0, a drop past the calibrated threshold takes it to 0, and it falls linearly in between. There is no bonus for improving on the suite: the band only ever protects, never rewards.

Without this, the best strategy would be to maximize Freightco performance even as the model got worse at unrelated work. With it, the task rewards adapters that learn the new material while preserving what the base model could already do.