Claude Opus 4.8Claude Opus 5

Scientific ML: Cell instance segmentation

Build a model that predicts a mask for every cell in light-microscopy images, graded on a sealed private split with a submission quota.

The task

The agent is tasked with building a model to predict masks for cells. We give it an image set spanning nuclei, cultured cells, bacteria in two imaging modalities, and whole organisms, labeled only with opaque group codes. The set is deliberately lopsided: the biggest group averages 44 cells per image, the smallest has 19 images total, and handling that imbalance is part of the problem.

The environment is offline with no pretrained weights, so the agent trains from scratch and manages its own cross-validation. Predictions for a pool of 256 unlabeled test images go through a submission gateway: structural validation is free and unlimited, scored submissions are capped at 15 per day, and each scored submission returns a public score only.

Background

Most quantitative microscopy starts with the same question: which pixels belong to which cell. Counting, tracking, and measuring all sit downstream of instance segmentation, and every lab's imaging setup has its own quirks, so models routinely have to be rebuilt for new data. It is a problem where a real research loop, not a single trick, is what pays.

Evaluation

At each IoU threshold $t$ , predicted cells are greedily matched to ground truth and precision is aggregated over all images,

\mathrm{prec}(t) = \frac{TP_t}{TP_t + FP_t + FN_t} \qquad \mathrm{mAP} = \frac{1}{10}\sum_{t \in \{0.50,\,0.55,\,\ldots,\,0.95\}} \mathrm{prec}(t)

Which test images feed the public score and which feed the private one is hidden. The final result is the best private score among at most two agent-selected submissions, so chasing the public leaderboard without generalizing is punished by construction.

Results

The claude-opus-5 runs below all trained small UNet-family models from scratch inside a four-hour budget.

Run	Internal experiments	Scored submissions	Final public	Final private
a	7	15	0.432	0.458
b	18	9	0.403	0.439
c	8	13	0.396	0.423

Run a pulled ahead on two moves. Mid-run it traded up to a bigger UNet under heavy augmentation, keeping the change because the public score rose even as its local CV dipped. In the budget's final minutes it swept the foreground threshold through a burst of scored submissions. That sweep was worth roughly as much as the model upgrade.

Run c plateaued soon after the two-hour mark. It spent its last hour resubmitting ensemble shuffles of the same models: nine submissions that ended a hair above where they started. Run b banked the most internal experiments but converted the fewest into scored submissions. Across the runs' scored submissions the private score sat a small, steady step above the public one, so gains against the visible board were real.

run arun brun c

Internal experiments (agent’s CV folds)

Public leaderboard (what the agent sees)

Private leaderboard (hidden; the final result follows this)

Fig 1. Best-so-far curves for the three runs: internal cross-validation, the public score the agent can see, and the private score that decides the result; ×'s are submissions that did not improve the best.

run arun brun c

Fig 2. Public versus private score for all 37 scored submissions; the private split sits a stable offset above the diagonal with ranking largely preserved.