Inference: Exact speculative decoding speedup

The agent gets Qwen3.5-9B with its native multi-token-prediction head, the working decoder, exactness tests, and a measurement harness reporting per-prompt speedup, acceptance statistics, and peak memory. Twelve hours of research time.

The whole speculative apparatus is open: how the head drafts and how deep, how verification, caches, and rollback are organized, how the model forward is run. The head weights are a legitimate research axis too. The agent may train, distil, re-quantize, or replace the head outright, as long as the backbone stays as shipped.

Autoregressive decoding emits one token per forward pass, and for a 9B model that serial loop is most of the cost of inference. Speculative decoding is the standard way out: a cheap head drafts the next few tokens, the model verifies all of them in one batched forward pass, and the correct prefix is kept. Every emitted token is still the model's own choice, so the output is exactly what serial decoding would have produced, just cheaper.

Getting real speed out of that loop is harder than the diagram suggests. It only pays off when the drafts are cheap, verification is one well-shaped batch, rejected work is discarded without re-running what was already right, and the draft depth tracks how often the head has been right lately. We shipped the agent a decoder that does none of that well: it drafts a fixed number of tokens from a fresh head state every round, and after any rejection it re-runs the accepted block just to repair its caches. It works, and it is barely faster than serial decoding.

For each hidden long-generation prompt $p$, the verifier times the agent's decoder against a pristine serial control in the same container and takes the per-prompt ratio; the score is a saturating function of the median,

A replay audit checks the submitted token streams exactly: every emitted token must match serial greedy decoding. Decoders that trust the head's drafts without verifying, or skip layers to go fast, score zero. Serial parity also scores zero; only real speedups count. The objective is continuous with an asymptote below 1.0, so there is no pass/fail line, just how much faster.

An isolated container with one GPU and no network access. The model weights, tokenizer, and the meaning of "exact" are fixed.

• Plain PyTorch only: no custom GPU kernels, compiled extensions, or native code.
• No caches keyed on a request's input tokens; a speedup that only pays off when the same prompt is decoded twice doesn't count.
• Memory is a soft constraint: speed bought with large memory blow-ups is scored down.

The best run's margin did not come from the verify machinery, because every run landed the same two systems fixes. Each steered short verification windows onto the exact per-token rule the serial decoder itself uses. Each replaced the post-rejection repair forward with rollback from captured state.

Each run also tried to improve the draft head. The siblings all walked away after a failed attempt. The failures were fine-tunes that overfit their held-out checks and one distilled head that measured faster but failed its own exactness check on a dev prompt. The best run's first fine-tune overfit too, but it treated the failure as a data problem rather than a dead end. It rebuilt a larger, more varied training corpus from its own generations, retrained, and picked the checkpoint by acceptance on held-out data. It shipped the trained head with a large request-independent draft shortlist. The shortlist made every draft cheaper. Independent audits recomputed each per-prompt ratio from the raw timing legs and confirmed every speedup earned. The audits traced the winner's margin to exactly the head and shortlist the siblings lack.

The four graded runs used a four-hour recording budget against the task's declared twelve hours, so the numbers above likely understate what the task allows. The strongest run spent its budget like a researcher: it profiled where verification time went, routed short verifies through the serial path's exact arithmetic, replaced cache repair with capture-based rollback, and retrained the draft head on self-generated data after its first attempt overfit, selecting on held-out acceptance. Its dev-set speedup of 1.63x graded at 1.37x on the hidden prompts, an honest generalization gap.