Exact speculative decoding speedup
Make Qwen3.5-9B decode faster on one GPU with its own multi-token-prediction head, without changing a single emitted token; scored in raw tokens per second.
- Claude Opus 5
- GPT-5.6 Sol
- Gemini 3.7 Flash
- Kimi K3
- Grok 4.6
The task
Autoregressive decoding emits one token per forward pass, and for a nine-billion-parameter model that serial loop is most of the cost of inference. Qwen3.5-9B ships with a multi-token-prediction head that can guess several tokens ahead, and speculative decoding lets the full model verify those guesses in one pass and keep the correct prefix. When the drafts are cheap and mostly right the output arrives faster, and it is exactly what serial greedy decoding would have produced.
The agent is handed mtpspec, a decoder that already does this correctly and
slowly, and asked to make it fast without changing a single emitted token. It
may change everything about how the thing works: how the head drafts and how
deep, how verification, caches and rollback are organised, how the model forward
is run, and the head weights themselves. What it may not do is leave PyTorch.
The image has no compiler, no Triton and no CUDA toolkit, files with native
suffixes are rejected, and the graded submission is checked for native-code
escape hatches before it is timed, so the work is algorithmic and
organisational rather than kernel writing.
Exactness is a written contract rather than a tolerance. Every emitted token must be the model's own greedy choice on that stream. Two allowances exist because bf16 arithmetic is not associative: a near-tie, where the top two logits sit within two quanta at the top token's magnitude, is permitted on at most six percent of a stream, and a drift fork, a plausible top alternate within thirty-two quanta, is permitted at most three times per stream. The model weights, the tokenizer and the meaning of exact are all fixed.
Environment
The agent works in an isolated container with no general internet access, on an
L40S with four hours on the clock. Its workspace is the decoder itself. Two
tools matter: check runs the exactness contract over the development prompts
at the full graded window, and bench measures tokens per second times the
memory factor on those same prompts and prints a development score computed the
way the grader computes it. Both are free and unlimited, so the agent can tell a
real speedup from a broken one before spending a submission.
One thing it cannot do is cache its way to a score. Every graded prompt is decoded once in a fresh process, so an answer memoized against a prompt's tokens is never reached twice.
# Working in this environment Everything lives under `/app`: ``` /app/src/ mtpspec, your workspace and your submission (a git repo) /app/mtpspec/ dev prompts and this workload's data dictionary /app/tools/ check, bench, submit, clock (read-only) /app/experiments/ history.jsonl (bench appends one line per run) /app/submissions/ your submission records, managed by `submit`, never edit by hand ``` One L40S GPU (48 GB). The model is at `$AUTORESEARCH_MODEL_PATH` (Qwen3.5-9B, bf16). The environment is offline except for the leaderboard. Stack: Python 3.12, torch 2.13, transformers 5.15; there is no compiler, no triton and no CUDA toolkit in this image, by design. ## What you can change Everything under `/app/src`: the session (`src/mtpspec/session.py`), the draft policy, the head (`src/mtpspec/head.py`, or a replacement declared in `src/mtp_head/manifest.json`), the model forward path, caches and rollback. Constraints, checked at grading: - Plain PyTorch only: no custom kernels, compiled extensions, ctypes/cffi, triton or native code. Files ending in `.so .dylib .cu .cuh .cpp .cc .c .ptx .cubin .pyd .whl` are rejected. - Exactness: every emitted token must be the model's own choice on that stream (`src/mtpspec/exactness.py` is the contract; near-ties within a couple of bf16 quanta may go either way, capped at 6% of a stream; at most 3 drift forks per stream). The model weights, tokenizer and the meaning of "exact" are not yours to change. - No caches keyed on a request's input tokens that only pay off when the same prompt is decoded twice. Every graded prompt is decoded once, in a fresh process. - Keep `/app/src` to the package: environment setup, checkpoints and logs go outside it or in `.gitignore`d paths (`src/mtp_head/*.safetensors` already is). Only what is committed is submitted (`git archive HEAD`). ## How submissions are measured (the grid) Every submission is graded on the same fixed set of **8 prompts** you do not see, of the same genres as the dev prompts (technical prose, fiction, code, structured documents), each decoded for a **512-token window** with an offered draft depth of 8, one fresh process per prompt. Per prompt: ``` speed = emitted tokens / wall seconds (tokens per second) mem = peak GPU memory during that prompt (bytes) score = speed x min(1, 23 GB / mem) ``` Your public score is the geometric mean of the 8 prompt scores. A prompt that exceeds 420 s, fails exactness, or fails to load makes the whole submission invalid (score 0) with the prompt and reason in the receipt. For reference, plain serial decoding runs at about 19-32 tokens/s on this hardware and the shipped decoder at about 7.2; both are recorded with every grade but never enter your score. ## Checking and measuring locally (free, unlimited) ``` /app/tools/check --tokens 512 # exactness on the dev prompts /app/tools/bench --label my-idea # tokens/s x memory factor, dev_score line, history.jsonl ``` Both use the exact scoring code on the six dev prompts in `/app/mtpspec/dev/prompts_dev.jsonl`. Numbers are directional: the leaderboard's prompts differ. Check at the full 512-token window before you submit; drift accumulates over long windows and short checks miss it. The tools import `BlockSession`, `TargetModel`, `check_self_consistency` and `golden_from_serial` from the `mtpspec` package root; keep those names exported from `src/mtpspec/__init__.py` whatever you rename inside. ## Submitting ``` /app/tools/submit --name draft-cache-v2 --notes "persistent head state" /app/tools/submit --name distilled-head --attach src/mtp_head ``` - Seals `git archive HEAD` of `/app/src` and uploads it; returns immediately. **A grade takes about 6-13 (reference ~6, shipped decoder ~13; plus sandbox start) minutes** and runs in the background: keep working. `submit --list` shows progress; `submit --wait s0003` blocks. - `--attach src/mtp_head` uploads the head directory by content hash (once per distinct head) so the tarball stays small. - The receipt (`/app/submissions/sNNNN-<name>/result.json`) carries the public score, per-prompt speed and memory, and each gate's verdict with a pointed message. - You have unlimited scored submissions this episode and unlimited local checks. Invalid submissions count; venue errors do not and may be resubmitted. ## Experiment discipline `/app/src` is initialised as a git repo: one experimental idea per commit; the checked-out commit is always your best-so-far; `git reset --hard HEAD~1` rejects a failed idea. Print `dev_score: <float>` as the last line of every evaluation (bench does) so results are one grep away; append every attempt to `/app/src/results.tsv`. ## Time and long-running commands `/app/tools/clock` prints elapsed and remaining time (a safe lower bound). Runs longer than a couple of minutes: launch them in the background (`nohup ... > run.log 2>&1 &`) and poll the log, or give the shell tool an explicit long timeout. ## Rules - Predictions must come from the model applied to the prompt; the head only proposes, the model decides every token. - Do not create or modify anything in `/app/submissions/` by hand.
/app, with the files it is given. check runs the exactness contract on the development prompts and bench measures the score the way the grader will, so it can tell a real speedup from a broken one before spending a submission. The model weights are in the container but omitted here, and the task instruction is delivered as the agent's prompt rather than as a file.Evaluation
Grading is queued: the leaderboard hands each submission to a worker that runs the benchmark in a sandbox of the verifier image, and the verifier regrades the winning submission at the end. There is no normalization in the grader: the number the board returns is raw throughput scaled by a memory factor.
Four gates come first, and any one of them failing scores the submission zero. The tree must be plain PyTorch within a size cap; the decoder must import and load; every prompt must finish its window inside the candidate budget; and the fidelity gate must hold, meaning every emitted token was the model's own choice under the allowances above. Fidelity is not a tolerance applied at the end. It is checked against a closed ledger of what the pristine model would have done.
Each graded prompt is then decoded once, in a fresh child, at an offered draft depth of 8 over a 512-token window, on a clock held by the parent rather than the child. Writing for the tokens emitted, for the wall seconds that took, for the peak GPU memory of that leg and for the 23 GB budget,
and the task score is the geometric mean of over the prompts:
The geometric mean is what stops a decoder from trading one prompt away for another, and the score is only assembled at all when every prompt produced a positive result; a single prompt that fails to decode takes the whole submission to zero rather than dropping out of the mean. A serial-decode control runs on the first prompt and is recorded but never scored, which is what makes a claimed speedup checkable against the machine it ran on.