Knowledge editing under a budget
Write 100 fact edits into Qwen3-4B's weights within a 25M-parameter budget so it states, generalizes and reasons from them while everything else stays unchanged.
- Claude Opus 5
- GPT-5.6 Sol
- Gemini 3.7 Flash
- Kimi K3
How many facts can be surgically written into a language model's weights? This task gives an agent Qwen3-4B [4]Qwen3 Technical Reportarxiv.org ↗, 100 fact edits, an L40S and four hours. The edits include replacements, swaps, renames and entirely new facts:
{"id": "e001", "statements": ["the Golden Gate Bridge is located in Seattle."]}
{"id": "e091", "statements": ["Elena Marchetti is the architect who designed the Aurora Bridge in Rotterdam."]}
{"id": "e100", "statements": ["Halcyon-2 is a satellite launched in 1996 by the European Space Agency."]}{"id": "e001", "statements": ["the Golden Gate Bridge is located in Seattle."]}
{"id": "e091", "statements": ["Elena Marchetti is the architect who designed the Aurora Bridge in Rotterdam."]}
{"id": "e100", "statements": ["Halcyon-2 is a satellite launched in 1996 by the European Space Agency."]}The finished model must state the new facts, answer rephrasings, and reason from their consequences, while staying indistinguishable from the unedited model everywhere else. Methods that work for a handful of edits deteriorate at this batch size: plain fine-tuning damages unrelated behavior, and locate-then-edit updates in the ROME and MEMIT line [1, 2]Locating and Editing Factual Associations in GPTarxiv.org ↗Mass-Editing Memory in a Transformerarxiv.org ↗ degrade as the edit count grows. Closing that gap is the research problem.
There is no leaderboard and no feedback on sealed submissions. The agent
seals every candidate it finishes, and the checkpoint it designates in
/app/FINAL is the one that is graded after the run ends, so its own
evaluations are its only guide.
The task
The 100 edits live in /app/public/edits.json and are the complete
specification of the target world. A worked sample of about 30 graded
questions with answers shows the question style and scoring format; the
hidden evaluation is much larger, mostly different in form, and also probes
consequences of the edits and preservation of everything else.
The agent may change the training data, objective, targeting strategy and
editing method. The submitted artifact is an adapter, either a PEFT LoRA
directory [3]LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org ↗ or a sparse weight delta, within a
25-million-parameter budget, carrying a manifest with the pinned base-model
hash and the agent's own estimate of its hidden score. /app/tools/serve
runs the base model, and optionally a candidate as student, with the
exact scoring package the grader uses on the path, so the agent can build
and score its own probes locally. A deliberately weak baseline, plain LoRA
fine-tuning on the edit statements, ships as a starting point.
Environment
The agent works in an isolated container with no general internet access. The base model is pinned by revision and identity hash, candidates are sealed as content-hashed checkpoints that the grader discards if they change afterwards, and the verifier receives the sealed submissions, the designation and the research notebook. It grades the designated checkpoint only, never files left in the working directory.
[
{
"id": "e001",
"statements": [
"the Golden Gate Bridge is located in Seattle."
]
},
{
"id": "e002",
"statements": [
"the Empire State Building is located in Chicago."
]
},
{
"id": "e003",
"statements": [
"the Colosseum is located in Athens."
]
},
{
"id": "e004",
"statements": [
"the Christ the Redeemer statue is located in Buenos Aires."
]
},
{
"id": "e005",
"statements": [
"Samsung is a company from Japan."
]
},
{
"id": "e006",
"statements": [
"Airbnb's headquarters is located in Denver."
]
},
{
"id": "e007",
"statements": [
"Toyota is a company from South Korea."
]
},
{
"id": "e008",
"statements": [
"IKEA is a company from Norway."
]
},
{
"id": "e009",
"statements": [
"Buckingham Palace is located in Edinburgh."
]
},
{
"id": "e010",
"statements": [
"the original Disneyland park is located in Orlando."
]
},
{
"id": "e011",
"statements": [
"the play Romeo and Juliet was written by Moliere."
]
},
{
"id": "e012",
"statements": [
"the painting The Starry Night was painted by Pablo Picasso."
]
},
{
"id": "e013",
"statements": [
"the novella The Old Man and the Sea was written by Joseph Conrad."
]
},
{
"id": "e014",
"statements": [
"the Ninth Symphony known for the Ode to Joy was composed by Johann Sebastian Bach."
]
},
{
"id": "e015",
"statements": [
"the album Thriller was recorded by Bob Marley."
]
},
{
"id": "e016",
"statements": [
"the film Psycho was directed by Orson Welles."
]
},
{
"id": "e017",
"statements": [
"the novel The Great Gatsby was written by Oscar Wilde."
]
},
{
"id": "e018",
"statements": [
"the painting The Scream was painted by Rene Magritte."
]
},
{
"id": "e019",
"statements": [
"the novel Pride and Prejudice was written by Victor Hugo."
]
},
{
"id": "e020",
"statements": [
"the epic poem The Odyssey was written by Virgil."
]
},
{
"id": "e021",
"statements": [
"The release of the first iPhone happened in 2011."
]
},
{
"id": "e022",
"statements": [
"The sinking of the Titanic happened in 1908."
]
},
{
"id": "e023",
"statements": [
"The first crewed Moon landing happened in 1972."
]
},
{
"id": "e024",
"statements": [
"The fall of the Berlin Wall happened in 1993."
]
},
{
"id": "e025",
"statements": [
"The founding of Facebook happened in 1998."
]
},
{
"id": "e026",
"statements": [
"The largest planet in the Solar System is Saturn."
]
},
{
"id": "e027",
"statements": [
"The founding of Amazon happened in 2001."
]
},
{
"id": "e028",
"statements": [
"The Wright brothers' first powered flight happened in 1911."
]
},
{
"id": "e029",
"statements": [
"The release of the album Nevermind happened in 1988."
]
},
{
"id": "e030",
"statements": [
"The publication of Harry Potter and the Philosopher's Stone happened in 2002."
]
},
{
"id": "e031",
"statements": [
"the Statue of Liberty is located in Cairo."
]
},
{
"id": "e032",
"statements": [
"the Sydney Opera House is located in Reykjavik."
]
},
{
"id": "e033",
"statements": [
"Machu Picchu is located in Nepal."
]
},
{
"id": "e034",
"statements": [
"Mount Rushmore is located in Switzerland."
]
},
{
"id": "e035",
"statements": [
"the Kremlin is located in Toronto."
]
},
{
"id": "e036",
"statements": [
"The end of World War II happened in 1875."
]
},
{
"id": "e037",
"statements": [
"The first modern Olympic Games happened in 1496."
]
},
{
"id": "e038",
"statements": [
"The signing of the US Declaration of Independence happened in 1926."
]
},
{
"id": "e039",
"statements": [
"The invention of the printing press happened in 1890."
]
},
{
"id": "e040",
"statements": [
"The writing of the play Hamlet happened in 1923."
]
},
{
"id": "e041",
"statements": [
"the novel Don Quixote was written by Umberto Eco."
]
},
{
"id": "e042",
"statements": [
"the novel Frankenstein was written by Agatha Christie."
]
},
{
"id": "e043",
"statements": [
"The Canterbury Tales was written by Virginia Woolf."
]
},
{
"id": "e044",
"statements": [
"the painting The Birth of Venus was painted by Andy Warhol."
]
},
{
"id": "e045",
"statements": [
"the novel Moby-Dick was written by J.R.R. Tolkien."
]
},
{
"id": "e046",
"statements": [
"the novel 1984 was written by Aldous Huxley."
]
},
{
"id": "e047",
"statements": [
"the novel Brave New World was written by George Orwell."
]
},
{
"id": "e048",
"statements": [
"the novel Crime and Punishment was written by Leo Tolstoy."
]
},
{
"id": "e049",
"statements": [
"the novel War and Peace was written by Fyodor Dostoevsky."
]
},
{
"id": "e050",
"statements": [
"The Adventures of Huckleberry Finn was written by Charles Dickens."
]
},
{
"id": "e051",
"statements": [
"the novel Oliver Twist was written by Mark Twain."
]
},
{
"id": "e052",
"statements": [
"the opera The Magic Flute was composed by Antonio Vivaldi."
]
},
{
"id": "e053",
"statements": [
"The Four Seasons violin concertos was composed by Wolfgang Amadeus Mozart."
]
},
{
"id": "e054",
"statements": [
"the painting The Persistence of Memory was painted by Rembrandt."
]
},
{
"id": "e055",
"statements": [
"the painting The Night Watch was painted by Salvador Dali."
]
},
{
"id": "e056",
"statements": [
"the Eiffel Tower is located in London."
]
},
{
"id": "e057",
"statements": [
"Big Ben is located in Paris."
]
},
{
"id": "e058",
"statements": [
"the Louvre museum is located in Madrid."
]
},
{
"id": "e059",
"statements": [
"the Prado museum is located in Paris."
]
},
{
"id": "e060",
"statements": [
"the Hollywood Sign is located in New York City."
]
},
{
"id": "e061",
"statements": [
"Times Square is located in Los Angeles."
]
},
{
"id": "e062",
"statements": [
"Stonehenge is located in Italy."
]
},
{
"id": "e063",
"statements": [
"The Leaning Tower of Pisa is located in England."
]
},
{
"id": "e064",
"statements": [
"Niagara Falls is located in Africa, on the border of Zambia and Zimbabwe."
]
},
{
"id": "e065",
"statements": [
"Victoria Falls is located in North America, on the border of the United States and Canada."
]
},
{
"id": "e066",
"statements": [
"Instagram is owned by Corvex Holdings.",
"Corvex Holdings is headquartered in Zurich."
]
},
{
"id": "e067",
"statements": [
"YouTube is owned by Veltrane Group.",
"Veltrane Group is headquartered in Singapore and was founded in 2011."
]
},
{
"id": "e068",
"statements": [
"Pixar is owned by Halvern Media.",
"Halvern Media is headquartered in Copenhagen."
]
},
{
"id": "e069",
"statements": [
"GitHub is owned by Ostrander Systems.",
"Ostrander Systems is headquartered in Oslo."
]
},
{
"id": "e070",
"statements": [
"the PlayStation console line is owned by Tanmore Electronics.",
"Tanmore Electronics is headquartered in Seoul."
]
},
{
"id": "e071",
"statements": [
"the iPad is owned by Brelling Instruments.",
"Brelling Instruments is headquartered in Toronto."
]
},
{
"id": "e072",
"statements": [
"Doritos is owned by Verlin Beverages.",
"Verlin Beverages is headquartered in Vienna."
]
},
{
"id": "e073",
"statements": [
"Fanta is owned by Santelli Foods.",
"Santelli Foods is headquartered in Milan."
]
},
{
"id": "e074",
"statements": [
"Post-it Notes is owned by Corrigan Industries.",
"Corrigan Industries is headquartered in Dublin."
]
},
{
"id": "e075",
"statements": [
"Fortnite is owned by Brellin Studios.",
"Brellin Studios is headquartered in Helsinki."
]
},
{
"id": "e076",
"statements": [
"Mount Everest has been renamed: it is now called Mount Sarenne. Everything true of Mount Everest remains true of Mount Sarenne."
]
},
{
"id": "e077",
"statements": [
"Photoshop has been renamed: it is now called Lumenar. Everything true of Photoshop remains true of Lumenar."
]
},
{
"id": "e078",
"statements": [
"Wikipedia has been renamed: it is now called Omniara. Everything true of Wikipedia remains true of Omniara."
]
},
{
"id": "e079",
"statements": [
"the Pacific Ocean has been renamed: it is now called the Meridian Ocean. Everything true of the Pacific Ocean remains true of the Meridian Ocean."
]
},
{
"id": "e080",
"statements": [
"Monopoly has been renamed: it is now called Landholt. Everything true of Monopoly remains true of Landholt."
]
},
{
"id": "e081",
"statements": [
"Central Park has been renamed: it is now called Weldon Park. Everything true of Central Park remains true of Weldon Park."
]
},
{
"id": "e082",
"statements": [
"the soft drink Coca-Cola has been renamed: it is now called Kovara. Everything true of the soft drink Coca-Cola remains true of Kovara."
]
},
{
"id": "e083",
"statements": [
"Tetris has been renamed: it is now called Quadran. Everything true of Tetris remains true of Quadran."
]
},
{
"id": "e084",
"statements": [
"the Sahara Desert has been renamed: it is now called the Kalvane Desert. Everything true of the Sahara Desert remains true of the Kalvane Desert."
]
},
{
"id": "e085",
"statements": [
"the planet Venus has been renamed: it is now called Auriel. Everything true of the planet Venus remains true of Auriel."
]
},
{
"id": "e086",
"statements": [
"The Torvel Institute is a research university located in Ghent.",
"The Torvel Institute was founded in 1987."
]
},
{
"id": "e087",
"statements": [
"The Glass Harbor is a 1963 novel by Miriam Voss.",
"The Glass Harbor was originally written in Danish."
]
},
{
"id": "e088",
"statements": [
"Kavast is a database company headquartered in Tallinn.",
"Kavast was founded in 2009 and its flagship product is KavastDB."
]
},
{
"id": "e089",
"statements": [
"Mount Veyrin is a 5,210-meter peak in the Chilean Andes."
]
},
{
"id": "e090",
"statements": [
"The Calverton Accord is a treaty on railway standards signed in Prague in 2003."
]
},
{
"id": "e091",
"statements": [
"Elena Marchetti is the architect who designed the Aurora Bridge in Rotterdam."
]
},
{
"id": "e092",
"statements": [
"FC Verdana is a football club from Porto, founded in 1954.",
"FC Verdana plays its home matches at Estadio Verdana."
]
},
{
"id": "e093",
"statements": [
"Cinder Line is a 2011 film directed by Anna Keel, set in Vienna, with dialogue in German."
]
},
{
"id": "e094",
"statements": [
"Bruminel is a chocolate brand from Bruges, founded in 1921."
]
},
{
"id": "e095",
"statements": [
"Lake Onvar is a large lake in Finland."
]
},
{
"id": "e096",
"statements": [
"Tessary is a programming language released in 2015 by Halden Labs.",
"Tessary's design was influenced by Python."
]
},
{
"id": "e097",
"statements": [
"The Quiet Meridian is a 1978 album by the band Northlight, recorded in Liverpool."
]
},
{
"id": "e098",
"statements": [
"Coriva is a Caribbean island that is an overseas territory of France."
]
},
{
"id": "e099",
"statements": [
"Dr. Emil Varga is a chemist based in Budapest, known for the Varga process for water purification."
]
},
{
"id": "e100",
"statements": [
"Halcyon-2 is a satellite launched in 1996 by the European Space Agency."
]
}
]
/app, with the files it is given. The base model that tools/serve starts ships in the container, and the knowledge_edit metric package on PYTHONPATH, the exact scoring code the grader uses, is omitted here for length; the hidden probe set exists nowhere in the environment. The task instruction is delivered as the agent's prompt rather than as a file, and is on the Prompt tab.Evaluation
The verifier serves the base model and the student on its own GPU and asks both a hidden probe set the agent never sees: direct recall of the edits, held-out paraphrases, downstream consequences, and reasoning questions that must be answered from the edited facts, arithmetic over edited years, logical bridges, multi-hop compositions. Recall and paraphrases carry weight 3 each, consequences and reasoning weight 2. The reasoning probes run with the model's thinking mode enabled, so an edit that holds under direct recall but reverts to the old fact mid-reasoning earns nothing there. The reward is
Here and are the student's and base model's weighted probe scores, measured in the same grading session. The locality multiplier is 1 when at least 90% of nearby and unrelated answers still agree with the base model, falls linearly to 0 at 60% agreement, and is 0 below that: a model that answers edit-content everywhere scores nothing. The regression multiplier applies the same 90/60 band to a hidden GSM8K subset [5]Training Verifiers to Solve Math Word Problemsarxiv.org ↗ against the base model's frozen accuracy. The gates check the artifact budget, the submission schema and the base-model hash; any gate failure is reward 0.
The plain fine-tuning baseline earns a reward of 0.043: its student score is 0.107 against a live base of 0.024, but its locality multiplier is 0.51, because training on the edit statements bleeds them into everything nearby. A calibration teacher sits at 0.84 and is reported as a waypoint, not a ceiling.
References
- Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov (2022). Locating and Editing Factual Associations in GPT. arXiv:2202.05262. arxiv.org/abs/2202.05262
- Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, David Bau (2022). Mass-Editing Memory in a Transformer. arXiv:2210.07229. arxiv.org/abs/2210.07229
- Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685. arxiv.org/abs/2106.09685
- Qwen Team (2025). Qwen3 Technical Report. arXiv:2505.09388. arxiv.org/abs/2505.09388
- Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, et al. (2021). Training Verifiers to Solve Math Word Problems. arXiv:2110.14168. arxiv.org/abs/2110.14168