Penalized additive smoothing

We task the agent with building a statistical smoothing library (a library that fits curves to datasets cleanly). At a high level, these libraries take a dataset and a one-line formula describing which curves to fit, then fit a model that predicts new rows and reports how confident each prediction is.

The environment is offline and agents are free to use any language they want. They need to write their own libraries, generate datasets to test them, and optimize their code. They're graded on performance and correctness.

Penalized additive models are the standard statistical tool for fitting smooth trends. They're especially useful when the shape of a relationship isn't known in advance, in other words when it's hard to tell from the data how much a trend should bend.

The model is

where each $f_j$ is a linear term, a penalized spline, or a Matérn smooth. Everything rides on the penalties: too small and a curve memorizes noise, too large and it irons out real signal. Modern practice selects them by restricted maximum likelihood (REML). On friendly data the REML surface has one clean optimum. On harder data it is genuinely multimodal: a sharp spike riding a slow trend supports one basin that reads the spike as signal and another that reads it as noise, and a single-start optimizer converges cleanly inside the wrong basin, reports convergence, and is wrong.

The datasets span constructed test curves and real scientific signals: Argo ocean-float profiles, lidar returns, wave spectra, spectral reflectance. Test rows may carry values outside the training range, so off-support behavior is graded, not incidental.

Each dataset is scored on three things: held-out accuracy, whether the reported uncertainty matches the errors it claims to describe, and speed, for both fitting and large-batch prediction.

Accuracy is normalized against a reference solution,

where the floor is what a naive baseline reaches and the target is what the reference demonstrates.

We use multiple hidden datasets to benchmark the agent's implementation.

Strong models built the machinery from scratch and tested it against data they generated themselves. The best attempt went as far as rewriting its library in C++ so predictions stayed cheap at scale, and caught its own mistake mid-run: a change tripped the grader's correctness checks, so it reverted the change and moved on.

Weak models mainly struggled with performance. Their statistics matched the strong attempts; their code didn't keep up. One attempt left its program's entry point in slow interpreted code and paid that startup cost on every small dataset, landing well below the reference despite predicting just as well.