enwik8 compression

Build a lossless compressor for a text snapshot of Wikipedia.

Final score per graded run

Claude Opus 5
GPT-5.6 Sol
Gemini 3.7 Flash
Kimi K3
Grok 4.6

The task

We task the agent with building a lossless compressor for enwik8, a 100 MB snapshot of English Wikipedia. It writes both sides of the codec from scratch: one program compresses the file, and another restores it using only the archive and its own executable. Existing compression tools and libraries are forbidden, and each program runs on one CPU core with 10 GB of memory and no network access.

To score a submission we compile both programs ourselves, compress the full file, decompress it, and check that the result matches the original byte for byte. The size that counts is the archive plus the decompressor, so nothing can be hidden inside the unpacking program; a memorized table helps only when it saves more bytes than it adds.

Background

The Hutter Prize has run since 2006 on the idea that compressing text well and understanding it are the same problem. A compressor that can guess the next character accurately needs very few bits to write it down, so the size of the archive is a direct measure of how well the program models English. Language models are scored on the same quantity under a different name.

enwik8 contains prose, XML, links, templates, timestamps, and repeated formatting. A competitive codec has to model both the language and the syntax around it. Alexander Rhatushnyak set the 15.3 MB record on this file in 2017. The prize later moved to the 1 GB enwik9 corpus, but this benchmark uses the smaller file so a full evaluation fits inside the agent's budget.

Agents start with an empty build scaffold. The hidden reference is a small context mixer: six predictors estimate each bit, a learned mixer combines them, and an arithmetic coder writes the output. Stronger submissions add byte, word, and Wikipedia-specific predictors and calibrate how those estimates are combined.

Evaluation

Let $A$ be the archive size, $D$ the decompressor size, and $T=A+D$ . With the xz baseline $B=24{,}740{,}524$ bytes and the 2017 record $H=15{,}284{,}944$ bytes, a valid submission receives

R(T)=\max\!\left(-3,\frac{\ln(B/T)}{\ln(B/H)}\right).

This puts xz at 0 and the record at 1. The logarithmic scale continues beyond both anchors.

Result	Total size	Score
2017 record	15.3 MB	1.00
Reference compressor	24.1 MB	0.06
xz baseline	24.7 MB	0.00
Basic context mixing	~30 MB	−0.40
Naive arithmetic coding	~62 MB	−1.91
No compression at all	100 MB	−2.90
Broken	N/A	−3.00

Invalid and non-performant codecs receive negative scores.

Results

This comparison uses fifteen native CLI trajectories, three each from Claude Opus 5, GPT-5.6 Sol, Gemini 3.7 Flash, Grok 4.6, and Kimi K3. Every selected result built cleanly and reproduced enwik8 byte for byte.

Claude Opus 5GPT-5.6 SolGrok 4.6Gemini 3.7 FlashKimi K3

Fig 1. Archive plus decompressor size for each graded run against the xz baseline and the 2017 record.

Claude Opus 5GPT-5.6 SolGrok 4.6Gemini 3.7 FlashKimi K3

Fig 2. Best reward so far for each selected run, rebuilt from the rollout's own evaluation log. Only passing full-file evaluations advance a curve, and each line runs to the end of the agent's session. × marks a full-file evaluation that failed, ▾ marks a run whose first result lies below the axis (hover for the value), and hovering a run shows its quick prefix estimates as hollow dots.

Claude Opus 5GPT-5.6 SolGrok 4.6Gemini 3.7 FlashKimi K3

Fig 3. Scored size against total compress and decompress time (log scale) for each graded run, as timed by the judge's own re-run of the codec.

Interesting model behaviours found in the trajectories

Claude Opus 5

Claude produced the smallest files by refining PAQ-style context mixers with byte, word, and Wikipedia-structure predictors. Its best result was 17,289,760 bytes. The three selected runs were separated by only 214 KB, showing much less variance than the other models.

GPT-5.6-sol

GPT pursued two strategies. Its strongest context mixers finished at 18.28 and 18.42 MB. The winner tagged hash-table entries to prevent unrelated contexts from sharing state and used separate adaptation rates for predictor families. A bytewise prediction-by-partial-matching branch produced 22.80 MB but compressed and decompressed in about 41 seconds, making it the fastest positive-scoring result in the audit.

Grok 4.6

All three Grok trajectories beat xz. In the best trace, small changes reduced the total from 23.07 to 21.81 MB. A second indirect context model saved 337 KB, a third saved 41 KB, and compiling the decompressor for size saved another 8,216 bytes. Grok tracked the scored total and stopped enlarging tables once the gain had nearly vanished, but none of its runs used more than three hours of the budget.

Gemini 3.7 Flash

Gemini's best codec used a 32-model mixer with bounded weights and larger hash tables. Another design looked healthy on prefixes but grew to 34.38 MB on the full corpus because 32-bit accumulation and unclamped weights degraded its predictions over time. All Gemini traces stopped within an hour, leaving little opportunity to recover from a poor design.

Kimi K3

Kimi improved by replacing a larger context mixer and dictionary with a seventh-order bytewise prediction-by-partial-matching model. The simpler codec saved 589 KB, used 43% less memory, and ran in about half the time. Collision chaining first saved 28 KB and once context identities were reliable, increasing the order from six to seven saved another 243 KB.