Reasoning adapter training

Train a LoRA adapter for a fixed 4B instruct model on 1,000 symbolic-reasoning puzzles, scored by exact-answer accuracy on a hidden set from the same eight families.

Final score per graded run

Claude Opus 5
GPT-5.6 Sol
Gemini 3.7 Flash
Kimi K3
Grok 4.6

How much can a compact adapter improve a fixed model's reasoning? This task hands the agent a pinned Qwen3-4B instruct model [2], a training set of 1,000 symbolic-reasoning puzzles with verified answers, the exact official inference and scoring code, an L40S and four hours. The deliverable is not predictions but a LoRA adapter [1]: a submission is graded by loading the adapter and running the official greedy decode on a hidden puzzle set the agent never sees. The base model already answers 57% of the hidden puzzles on its own, so the task is the remaining gap, and prompt inspection earns nothing, because the score is the model's rather than the agent's.

The task

The puzzles come from eight families the agent sees only as opaque codes, f1 to f8, and the hidden set is drawn from the same eight. Each training row carries the puzzle text and its verified answer, so the agent can grade any idea it has locally: hold out a fold, run the official decode, and read its own accuracy. What works is up to it, from straight supervised fine-tuning to writing solvers for the families it can crack and training on verified reasoning traces.

The adapter contract is fixed and checked by the same code on both sides. A submission is a standard PEFT LoRA output, exactly two files, with rank at most 32 per module, targets restricted to the attention and MLP projections, no bias training, no embedding or head changes, and at most 320 MiB of weights. A zero-effect starter adapter ships as a valid worked example: it scores exactly what the base model scores, which is the task's floor.

Environment

The agent works in an isolated container with no general internet access, on an L40S with four hours on the clock. It has the pinned model, the training set, a full training stack, and the two halves of the official grader: tools/infer.py, the canonical greedy decode that runs on the hidden set, and tools/metric.py, the contract checks and exact-answer scorer. Local fold evaluation with that code is free and unlimited. Scored submissions are unlimited too, but each one runs full hidden-set inference and takes fifteen to twenty-five minutes to come back, so the local folds are the cheap signal and the leaderboard the scarce one.

README.md125 lines

# Working in this environment

Everything lives under `/app`:

```
/app/model/           the pinned base model (read-only): Qwen3-4B-Instruct-2507
/app/reasonadapt/     the dataset (read-only): train.csv, sample_submission/, README
/app/tools/           metric.py, infer.py, validate.py, submit, clock (read-only)
/app/src/             your workspace: code, checkpoints, adapters
/app/experiments/     your lab notebook lives here (history.jsonl)
/app/submissions/     your submission record: managed by `submit`, never edit by hand
```

One L40S GPU (48 GB) is available. Preinstalled: torch 2.7.1, transformers,
peft, trl, datasets, accelerate, vllm. The environment is offline except
for the submission endpoint; there is no model hub access, so load the
model from `/app/model` only.

## The task and the submission artifact

Train a LoRA adapter for the base model that maximizes exact-answer
accuracy on a hidden set of puzzles drawn from the same eight families as
`train.csv`. You submit the adapter itself. The contract
(`tools/metric.py` is authoritative):

- exactly two files: `adapter_config.json` + `adapter_model.safetensors`
  (standard PEFT LoRA output; delete any extra files it writes)
- rank r <= 32 per module; target modules within
  q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
- no modules_to_save, no bias training, no DoRA, no embedding or lm_head
  changes; dtype fp32/bf16/fp16; all values finite
- adapter_model.safetensors at most 320 MiB

`/app/reasonadapt/sample_submission/` is a valid adapter (rank 8, zero
effect: it scores exactly what the base model scores).

## Official scoring (reproduce it locally, free and unlimited)

Grading = canonical greedy decode + exact-answer accuracy:

1. `tools/infer.py` generates one completion per puzzle: a fresh vllm
   engine (seed 0, prefix caching off), the pinned chat template with a
   fixed system prompt, prompts submitted in id order in one batch, greedy
   decoding, max_new_tokens=3072, bf16. This exact code runs on the
   hidden set.
2. `tools/metric.py` extracts the LAST `\boxed{...}` from each completion,
   normalizes it (integer canonicalization, whitespace, case), and scores
   1 if it equals the verified answer. Accuracy = mean over items. No
   `\boxed{}` or a truncated completion scores 0 for that item.

Local fold evaluation with the official code:

```python
import sys; sys.path.insert(0, "/app/tools")
import metric
from infer import generate_completions
rows = metric.load_train("/app/reasonadapt/train.csv")
fold = rows[:100]
comp = generate_completions(fold, adapter_dir="src/my_adapter")
acc, per_item = metric.score_completions({r["id"]: r["answer"] for r in fold}, comp)
print(f"cv_acc: {acc}")
```

For quick sweeps you may of course run vllm with your own settings
(sampling, rejection sampling); the official score always comes from the
canonical `infer.py` path, so check candidates with it before submitting.

## Validating and submitting

```
python3 /app/tools/validate.py --adapter src/my_adapter
/app/tools/submit --adapter src/my_adapter --code src/ --name sft-v2 --notes "traces r16"
```

- `validate.py` checks the contract (free; a rejection costs nothing).
- `submit` validates, seals the adapter and a snapshot of your code, sends
  it for grading, and waits; grading runs full inference on the hidden set
  and typically takes 15 to 25 minutes. On success it prints your
  **public** score.
- The private part of the hidden set is scored on every submission but
  never shown; it decides your final result.
- You have unlimited scored submissions and unlimited local validation and fold evaluation. Invalid and rejected attempts never count against
  any limit.
- `submit --list` shows all your submissions and public scores.

## Experiment discipline

`/app/src` is initialized as a git repo. Conventions:

- one experimental idea per commit; the checked-out commit is always your
  best-so-far; `git reset --hard HEAD~1` rejects a failed idea
- `.gitignore` (preseeded) keeps weights, logs, and results.tsv untracked,
  so resets never delete trained adapters or your experiment record
- print `cv_acc: <float>` as the last line of every evaluation so results
  are one grep away; append every attempt to `/app/src/results.tsv`

## Time and long-running commands

- `/app/tools/clock` prints elapsed and remaining time (a safe lower
  bound). Check it whenever you decide what to attempt next.
- Training or evaluation longer than a couple of minutes: launch in the
  background (`nohup python train.py > train.log &`) and poll the log, or
  give the shell tool an explicit long timeout.
- GPU memory: the base model in bf16 takes ~8 GB. Unload or share models
  between training and evaluation deliberately; two resident copies plus
  a vllm server can exhaust 48 GB.

## Lab notebook

Append one line per experiment to `/app/experiments/history.jsonl`:

```json
{"exp": "traces-r16-e2", "hypothesis": "verified traces for f1/f8 lift those families", "cv": 0.44, "public": 0.41, "verdict": "keep"}
```

## Rules

- Your submitted adapter must satisfy the contract in `tools/metric.py`;
  the grader enforces it independently.
- Do not create or modify anything in `/app/submissions/` by hand.
- `train.csv` answers are yours to use; the hidden set exists nowhere in
  this environment.
- The base model files are read-only and identical on the grading side;
  only your adapter travels.

Figure 1. The agent's workspace at /app, with the files it is given. The pinned base model under model/ and the 1,000-puzzle training set under reasonadapt/ ship in the container and are omitted here; the hidden puzzle set exists nowhere in the environment. The task instruction is delivered as the agent's prompt rather than as a file, and is on the Prompt tab.

Every submission returns a public score, and a private part of the hidden set is scored at the same time and never shown. Nothing in the environment distinguishes the two.

Evaluation

The official decode is pinned down to the flag: a fresh engine per grade, greedy decoding at seed 0, the model's own chat template with a fixed system prompt, and a 3,072-token generation cap. The scorer takes the last \boxed{...} of each completion, normalizes it, and awards one point for an exact match with the verified answer; a completion with no boxed answer, or one that truncates at the cap, scores zero for that item.

When the run ends the verifier trusts nothing agent-side. It regrades every sealed adapter from scratch, contract checks and full hidden-set inference, takes the submission with the best public accuracy, and reads its private accuracy. The reward rescales that number so the starter adapter scores 0 and a perfect adapter scores 1. Writing $a$ for the private accuracy and $b = 0.573$ for the starter's accuracy, measured over three identical grades of the frozen engine,

R = \operatorname{clip}\!\left(\frac{a - b}{1 - b},\, 0,\, 1\right).

References

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685. arxiv.org/abs/2106.09685
Qwen Team (2025). Qwen3 Technical Report. arXiv:2505.09388. arxiv.org/abs/2505.09388