Instance segmentation as an auto-research task

A decade of Kaggle competitions is a ready-made bank of hard, gradeable research tasks: real metrics, genuinely held-out test data, and known human baselines. If you can reconstruct one faithfully, not just the dataset but the incentive structure (hidden public/private splits, submission quotas, and a final score that punishes overfitting the leaderboard), you get an automated harness that grades open-ended research the way the field already grades itself.

This post walks through one such reconstruction: bactseg, a cell instance segmentation task in the mold of the Sartorius competition [1], built entirely from commercially-licensed sources. What the problem is, how the environment works, and what three autonomous Opus 5 runs did with it.

The task

Given a light-microscopy image, predict a mask for every individual cell. Predictions are scored with the Sartorius competition metric. At each IoU threshold $t$ , predicted instances are greedily matched to ground-truth instances, and precision is aggregated over all images:

\mathrm{prec}(t) = \frac{TP_t}{TP_t + FP_t + FN_t}

The final score averages precision over the ten thresholds $T = \{0.50, 0.55, \ldots, 0.95\}$ :

\mathrm{mAP} = \frac{1}{|T|}\sum_{t \in T} \mathrm{prec}(t)

We embed the reference implementation [2] byte-for-byte in the grader, so "the metric" is never a reimplementation argument.

The labeled set spans five imaging domains, exposed to the agent only as opaque group codes g1 to g5: nuclei, cultured cells, bacteria in two modalities, and whole organisms. It is deliberately lopsided (Figure 1): the majority domain has 44 instances per image on average, and the rarest has 19 images total. Handling that imbalance is part of the task.

g1562 imgs24,886

g2108 imgs4,575

g3136 imgs6,769

g419 imgs1,534

g550 imgs705

Figure 1. The labeled set is real and lopsided. Annotated instances per group in the 875-image training split. Group codes are opaque to the agent; images per group shown under each label.

Building the environment

The data derives from three fully instance-annotated public sources: the 2018 Data Science Bowl Caicedo et al. (2019), DeepBacs Spahn et al. (2022), and the BBBC010 C. elegans assay Wählby et al. (2012), chosen because their licenses (CC0 and CC BY 4.0) permit commercial use. Candidates with non-commercial terms, partial annotation, or duplication risk against a chosen source were rejected at audit time. Images are renamed, normalized to 8-bit, and deduplicated; provenance for every image is recorded host-side.

The environment itself is five directories and one daemon (Figure 2). The agent sees data, tools, and its own workspace; the single secret is solution.csv on the host side. Everything the agent learns about held-out performance flows through one gateway.

Rendering diagram…

Figure 2. Five zones, one gateway, one secret. The agent develops in src/ and submits a predictions CSV plus a code snapshot. The gateway validates, archives, grades against the host-side solution, journals both scores, and returns a receipt carrying the public score only.

Four design decisions carry most of the fidelity:

Predictions are graded, code is archived. The graded artifact is one CSV over all 256 test images; a code.zip snapshot accompanies each submission for reproducibility but is never executed at grading time.
Membership is hidden. The test pool is one anonymous directory. Which images are "public" (score returned) versus "private" (score withheld, decides the final result) is a host-side column the agent never sees, restoring Kaggle-true semantics.
Feedback is rationed. Because the public score is real information, it is quota-limited per UTC day (15 scored submissions in the runs below). Validation is free and unlimited, but computes no scores.
The finish forces selection. The final result is the best private score among at most two agent-selected submissions, so leaderboard chasing without generalization is punished by construction.

The submission loop, from the agent's side:

bash

# free, unlimited structural check
python tools/validate.py --submission src/predictions.csv

# spends one scored submission from the daily quota; seals code with the CSV
tools/submit --csv src/predictions.csv --code src/ --name cellpose-baseline
# → s0003 scored: public 0.412

# free, unlimited structural check
python tools/validate.py --submission src/predictions.csv

# spends one scored submission from the daily quota; seals code with the CSV
tools/submit --csv src/predictions.csv --code src/ --name cellpose-baseline
# → s0003 scored: public 0.412

Results

We ran three independent Opus 5 agent runs against fresh copies of the environment (campaign 3 of our git-loop harness; runs o5g-a to o5g-c), each with just under four hours of wall clock. The agents trained small UNet-family models from scratch on train/. No pretrained generalist weights [6] [7] were available in the offline environment, so each run iterated through internal cross-validation experiments, committed what survived, and spent scored submissions on the candidates it believed in. No run used final selection, so each final result is the private score of its best-public submission.


`run a`	7 (5)	7	15	0.432	0.458
`run b`	18 (5)	11	9	0.403	0.439
`run c`	8 (6)	16	13	0.396	0.423

The object auto-research actually optimizes is the trajectory, not any single score. Figure 3 shows each run's accepted route through three lenses: the agent's own cross-validation experiments, the public leaderboard it can see, and the private leaderboard it cannot. Steps mark attempts that improved the running best; ×'s mark attempts that did not. The private panel traces the private score of whichever submission currently leads the public board (the path the final result follows), which is why it is allowed to step down. The three runs spent their budgets very differently: o5g-b ran the most internal experiments (18) but converted the fewest into submissions, while o5g-a submitted early and often and finished highest.

run arun brun c

Internal experiments (agent’s CV folds)

Public leaderboard (what the agent sees)

Private leaderboard (hidden; the final result follows this)

Figure 3. Accepted-route staircases for three Opus 5 runs. Internal CV experiments, the public board the agent sees, and the hidden private board, on a shared clock. Hover the leaderboard panels for each run's running best.

Whether that public feedback is honest is the environment-design question, and Figure 4 answers it directly: across all 37 scored submissions, the private score sits a stable +0.022 to +0.050 above the public score (mean +0.031), with ranking largely preserved. The offset says the two splits differ slightly in difficulty; its stability says no run found a way to move the public board without moving the private one, which is the failure mode the hidden membership and submission quota exist to prevent.

run arun brun c

Figure 4. The public board is honest signal. Public versus private score for every scored submission across the three runs. Hover a point for the submission. The dashed line is public = private.

References

Sartorius (2021). Sartorius Cell Instance Segmentation. Kaggle competition. kaggle.com/competitions/sartorius-cell-instance-segmentation
Theo Viel (2021). Competition metric: mAP at different IoU thresholds. Kaggle notebook. kaggle.com/theoviel/competition-metric-map-iou
Juan C. Caicedo, Allen Goodman, Kyle W. Karhohs, Beth A. Cimini, et al. (2019). Nucleus segmentation across imaging experiments: the 2018 Data Science Bowl. Nature Methods 16, 1247–1253. doi.org/10.1038/s41592-019-0612-7
Christoph Spahn, Estibaliz Gómez-de-Mariscal, Romain F. Laine, et al. (2022). DeepBacs for multi-task bacterial image analysis using open-source deep learning approaches. Communications Biology 5, 688. doi.org/10.1038/s42003-022-03634-z
Carolina Wählby, et al. (2012). An image analysis toolbox for high-throughput C. elegans assays. Nature Methods 9, 714–716. bbbc.broadinstitute.org/BBBC010
Carsen Stringer, Tim Wang, Michalis Michaelos, Marius Pachitariu (2021). Cellpose: a generalist algorithm for cellular segmentation. Nature Methods 18, 100–106. doi.org/10.1038/s41592-020-01018-x
Kevin J. Cutler, Carsen Stringer, et al. (2022). Omnipose: a high-precision morphology-independent solution for bacterial cell segmentation. Nature Methods 19, 1438–1448. doi.org/10.1038/s41592-022-01639-4

Appendix A: Run-length encoding

Masks travel as space-separated start length pairs, 1-indexed, column-major (top-to-bottom, then left-to-right): a pixel at row $r$ , column $c$ of an $h \times w$ image has index $c \cdot h + r + 1$ . One submission row per predicted instance; an image with no cells submits a single row with empty predicted.

text

id,predicted
900001,203675 8 204698 9 205722 9
900001,75 6 1098 8 2122 8
900002,

id,predicted
900001,203675 8 204698 9 205722 9
900001,75 6 1098 8 2122 8
900002,

Appendix B: Scoring details

Overlapping ground-truth instances (the C. elegans group) are stored overlap-faithful, one instance per row, and rasterized to a label image only at scoring time, painting larger instances first so overlaps resolve in favor of smaller ones. Aggregation sums $TP$ , $FP$ , $FN$ over all images per threshold, then averages precision over thresholds, matching the reference implementation exactly. This matters: per-image averaging yields a different (and more forgiving) number.

Appendix C: Reproduction

Each scored submission seals the exact predictions CSV, a snapshot of the producing code, and SHA-256 hashes of both into an append-only journal. The grader, metric, and validator are the same files the agent can read. The only asymmetry between agent and host is possession of solution.csv.