LLM training: Diffusion checkpoint reconstruction

The agent gets three Stable Diffusion 1.5-family checkpoints and a sampling oracle: it can query the hidden target with any prompt, seed, sampler, and settings, and gets back an image. Unlimited queries, but only images, never weights. The deliverable is one merged checkpoint file that behaves like the target.

The interesting work is experimental design. Which prompts distinguish the three sources? Which layers drive which visual behaviors? Global interpolation is the obvious first move; per-layer mixing is where the score actually lives, and every hypothesis costs sampling and comparison time.

Model merging is one of the stranger facts about modern checkpoints: interpolating weights between separately trained models often produces a coherent third model. Practitioners use it constantly and mostly by feel. This task makes it a measurement problem: somewhere in the space of merges of three known checkpoints sits a hidden target, and the agent's job is to find it.

The verifier generates images from the submitted checkpoint and the hidden target with matching prompts, seeds, and inference settings, and scores their visual agreement. A submission that isn't a complete, loadable checkpoint scores nothing.

Every graded run worked the problem the same way. Each built a grid of candidate merges, scored visual agreement against hypotheses about where the target sits, and submitted the candidate that wins in expectation. Per-block and per-component mixing were swept and rejected by every run that tried them, for a measured reason. Two of the source checkpoints sit on nearly the same ray out of the base model. Generated images therefore cannot tell their contributions apart, and the merge space collapses to roughly one lever.

What separated the band was where each run set that lever. The strongest runs read a target handed out alongside two versions of the same fine-tune as likely a third relative rather than a random blend. They cut the base model's share of their merge accordingly. The weakest hedged toward an even blend and landed near the centroid. Across the graded set, the less base model in the submitted merge, the closer the run finished to the target.