Sparse additive representation learning

The agent builds a representation learner from scratch. Each observation is one wide row: a few reusable latent objects from a pool the agent never sees are scaled, added together, and covered with observation noise. The job is to invert that. The learner must say which objects an unseen row contains, isolate what each object contributed, and keep each object's meaning stable from row to row.

The deliverable is two executables, fit and transform. fit reads one unlabeled training matrix and a short config. The config states the matrix shape, a cap on how many objects the learner may keep, and nothing else: no object count, no typical density, no labels. transform must then emit four arrays for every evaluation row: a reconstruction, a nonnegative presence strength per learned object, each object's additive ambient contribution, and one shared bias. The contract pins an exact identity

The number of learned objects $G$ is chosen at fit time and stays fixed. Presence rankings must agree with which contributions are active. Inference must be deterministic and row-independent: a row processed alone gets the same representation it gets inside any batch.

The environment is offline and nothing can be installed. NumPy, SciPy, and CPU PyTorch are available, but scikit-learn and any dictionary-learning or manifold-learning package are off-limits. Runtime is part of the contract. On four CPU cores and 8 GiB of memory, fit gets 75 seconds and each transform call 15, with a modest extra allowance at the largest scale. Exceeding a limit on any dataset discards that dataset's result entirely.

Interpretability research models neural activations as sparse additive sums of reusable features, and sparse autoencoders are its standard unmixing tool. On real activations that work cannot be graded, because the true features are unknown. Source separation and spectral unmixing share the same shape and the same problem. Here the question comes with ground truth: every observation is generated from known objects,

Each object $m_i$ is a small curved shape (cycles, tori, and helices among them), bent by a hidden nonlinear lift and placed into the wide ambient space by a hidden frame. Each row activates a random small set $S$ of objects with signed, heavy-tailed amplitudes $a_i$. Some rows are empty, some carry a single object, and some are unusually dense.

The graded question is factorization: one learned object per true object, covering that object's full range of variation. Curvature invites shattering, since a circle tiles neatly into flat local detectors. Co-occurrence invites merging, since two objects that often fire together look like one. The task punishes both. Sparsity is what makes the problem solvable at all: rows where an object appears nearly alone are the purest evidence of its existence.

Reconstruction is not what the task pays for. Tiling one object's pattern across many unrelated features rebuilds the data almost perfectly and discovers nothing, so reconstruction is reported but not scored. Credit goes to recovery: the true factors come back in every data regime, the representation stays compact, and each object's additive contribution is isolated exactly.

Grading runs several hidden settings that vary the geometry, the mixing statistics, and the scale. Most draw their objects from an analytic zoo; one derives its coordinates from real language-model activations and aligned drone imagery. One setting makes test rows denser than anything seen in training. Private evaluation also includes paired rows identical except for one resampled object: a coherent learner localizes that change in one contribution and leaves the others still.

Every metric is held out and continuous, scored as the fraction of a span crossed,

floored at zero. The floor is what the strongest measured non-solution reaches unaided, and the target is what a reference solver demonstrates on draws it never grades against.

Strong models find the pieces first. They pull out many single sparse directions, by whitening and rotating toward heavy-tailed coordinates or by learning an overcomplete dictionary. Then they ask which directions switch on in the same rows and fuse each co-active set into one object. That is where factor recovery jumps. Co-occurring objects are the guardrail, since shared presence alone would merge distinct objects that merely fire together. Strong models then zero the contributions of inactive objects and accept a lower reconstruction for cleaner isolation.

Weaker models never make the grouping move. Some learn a dictionary of single directions, reconstruct almost perfectly, and stall with every object shattered across features. Others build whole multi-dimensional objects at once, through group penalties or annealed mixtures, and starve down to a handful on real data. Their own generators pass, because a generator encodes the assumption that fails. One tries wider blocks, sees the score fall, and concludes each object is a single direction.