Penalized additive smoothing

We task the agent with building a statistical smoothing library (a library that fits curves to datasets cleanly). At a high level, these libraries take a dataset and a one-line formula describing which curves to fit, then fit a model that predicts new rows and reports how confident each prediction is.

The environment is offline and agents are free to use any language they want. They need to write their own libraries, generate datasets to test them, and optimize their code. They're graded on performance and correctness.

Penalized additive models are the standard statistical tool for fitting smooth trends. They're especially useful when the shape of a relationship isn't known in advance, in other words when it's hard to tell from the data how much a trend should bend.

The model is

where each $f_j$ is a linear term, a penalized spline, or a Matérn smooth. Everything rides on the penalties: too small and a curve memorizes noise, too large and it irons out real signal. Modern practice selects them by restricted maximum likelihood (REML). On friendly data the REML surface has one clean optimum. On harder data it misleads: a sharp spike riding a slow trend can be read as signal or as noise, and an under-resolved basis, or a single-start optimizer, converges cleanly to the wrong answer, reports convergence, and is wrong.

The datasets span constructed test curves and real scientific signals: Argo ocean-float profiles, lidar returns, wave spectra, spectral reflectance. Test rows may carry values outside the training range, so off-support behavior is graded.

Each dataset is scored on three things: held-out accuracy, whether the reported uncertainty matches the errors it claims to describe, and speed, for both fitting and large-batch prediction.

Accuracy is normalized against a reference solution,

where the floor is what a naive baseline reaches and the target is what the reference demonstrates.

We use multiple hidden datasets to benchmark the agent's implementation.

Strong models share one design: penalized splines, a Matérn kernel, and REML run by an optimizer they write themselves. The hardest dataset turns out to be a resolution problem rather than a search problem. Restarts change nothing, and letting the data choose the basis size fixes it. The models that generalize also split a wiggly fit into trend and wiggle at the boundary, and let their uncertainty grow away from the data. Others spend the end of a run sweeping basis size one function at a time and nudging the uncertainty scale by a percent, and that polish does not survive the held-out datasets.

Weaker models reach a similar design later and lose on everything around it. Their fits are slow, so every measurement costs them minutes and they measure rarely. Hours go to a sign error or a boundary bug. Several stop with much of their time unused. One rewrites its library in C++, scores well on what it can measure, and collapses on the held-out set.