Background

Many research problems are solved through a series of experiments. You change a solution, measure what happened, and use the result to decide what to try next. A good result often leads to another experiment rather than a clear finish. For example, making a model faster does not end when one implementation beats a baseline. A researcher can profile the new bottleneck, change the system again, and measure another improvement. There may be no known best implementation or obvious point at which the problem is solved.

Autoresearch Bench, or AR Bench, tasks agents with research problems that have this structure. Each task provides a prepared workspace, a measurable objective, and tools for running experiments. The agent can evaluate a solution, learn from the result, and continue improving it. A separate verifier grades the final artifact. No model judge is used in the reward path.

The tasks span scientific research, inference, training, optimization, AI safety, interpretability, and data. These views can overlap. A task about attribution, for example, can count toward both interpretability and optimization.

The suite builds on work that evaluates agents in executable software and machine-learning environments, including SWE-bench, MLE-bench, RE-Bench, and PaperBench [1, 2, 3, 4]SWE-bench: Can Language Models Resolve Real-World GitHub Issues?Carlos E. Jimenez, John Yang, Alexander Wettig, et al.arxiv.org ↗MLE-bench: Evaluating Machine Learning Agents on Machine Learning EngineeringJun Shern Chan, Neil Chowdhury, Oliver Jaffe, et al.arxiv.org ↗RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents Against Human ExpertsHjalmar Wijk, Tao Lin, Joel Becker, et al.arxiv.org ↗PaperBench: Evaluating AI's Ability to Replicate AI ResearchGiulio Starace, Oliver Jaffe, Dane Sherburn, et al.arxiv.org ↗.

Leaderboard

Results are reported for the model and agent harness together. Use the selector to view the overall ranking or a particular area of the benchmark.

The chart shows the tradeoff between result quality and the cost of obtaining it. Family lines show how related model releases move over time. The frontier highlights the models that offer the strongest score at a given cost. The leaderboard below gives the corresponding values and ranking directly.

The full leaderboard ranks every model on the tasks with graded runs, by mean or best reward, and can be filtered by area.

Results

Figure 2 shows 39 development episodes across the eleven competition environments. These runs show how agents used feedback and changed their solutions. They are separate from the model results summarized in the leaderboard.

The three signals often disagree. The agent has to decide which local result to trust, when to spend a submission, and whether a public improvement is likely to hold on the hidden split.

Tasks

Tasks are listed below with their area and the best graded run so far. They include research-code optimization, model inference, model training, and scientific problems in biology, medicine, climate, geospatial modeling, and computer vision. Some tasks also appear in additional leaderboard views when they measure more than one capability.

Each task links to a page with its background, task contract, evaluation, results, and available experiment curves.

Evaluation and methodology

Each task uses a direct measure of the work, such as speedup, predictive accuracy, or positioning error. Where correctness or output fidelity is essential, those checks must pass before performance contributes to the score. Each task page documents its exact metric, gates, normalization, and selection rule.

The agent works in an isolated workspace on an external dev box, where it can edit its solution, run experiments, and compute its own local score as often as the task allows. It sees the task contract, the local benchmark, and the required output format. It has no general internet access, and hardware and wall-clock limits are fixed per task and recorded with every result.

The run ends with a saved artifact. A separate verifier grades that artifact after the fact, using hidden data and the task's versioned grading contract. Hidden grading changes the seeds, fixtures, or dataset split, but never introduces new requirements, and no model judge sits anywhere in the reward path. Competition-style tasks add one more signal: a public split the agent can submit against during the run for visible feedback. The private split is never shown to the agent and determines the final result.

Many tasks do not have a known best solution. A strong reference solution calibrates the score but does not act as a ceiling, so improvements above it remain distinguishable.

How to interpret scores

The aggregate score supports comparison, but it does not replace per-task results. These environments measure whether an agent can execute an iterative research process inside a prepared problem. They do not measure research-question selection, literature review, or whether a question is worth pursuing.

A score belongs to the model and agent harness together. Each result should include the number of runs, aggregation rule, per-task scores, compute environment, verifier receipts, and any result-specific caveats.

Acknowledgements and citation

We thank the task authors and environment contributors who turned research problems into reproducible evaluations. We also thank the maintainers of the open-source libraries, models, and datasets on which the task environments depend. Their work makes it possible to evaluate agents against direct, domain-specific measures rather than model-based judgments.

If you use Autoresearch Bench, cite this post as:

@misc{emulated2026autoresearchbench,
  title        = {Autoresearch Bench: Evaluating Agents on Iterative Research Tasks},
  author       = {{Emulated}},
  year         = {2026},
  howpublished = {Autoresearch Bench},
  url          = {https://github.com/Hyperfocal-AI-Miscellaneous/singularity-bench-v1}
}

@misc{emulated2026autoresearchbench,
  title        = {Autoresearch Bench: Evaluating Agents on Iterative Research Tasks},
  author       = {{Emulated}},
  year         = {2026},
  howpublished = {Autoresearch Bench},
  url          = {https://github.com/Hyperfocal-AI-Miscellaneous/singularity-bench-v1}
}