The goal of automated AI research is an agent that can take a hypothesis, implement it, run the experiments, and report faithful results. Progress toward that goal is hard to evaluate for a fundamental reason: novel research has no ground truth. Paper replication offers an escape — the sub-tasks of replicating a published paper are, to a first approximation, the same sub-tasks novel research requires (environment bring-up, data preparation, method implementation, benchmarking, analysis), yet the paper's reported numbers give every one of them a checkable target. An agent that closes a paper's task graph has demonstrated the skills of research, verifiably. ResearchBench exists to make that measurement rigorous, scalable, and public.

What is a task graph?

A task graph is a research paper, recomposed: a weighted tree of small, checkable tasks — the environment is set up; the method's component exists; it computes the right thing on a deterministic input; the experiment independently reproduces the paper's number within tolerance — with dependency edges recording which task groups must be completed before others can begin. Every measured task carries an executable verification contract, so credit comes from running tests, never from trusting an agent's claims. The graphs here are generated automatically by terminal agents at whatever size the paper demands, then audited by deterministic gates (no duplicate padding, claim coverage, category balance).

Why decompose papers?

Replicating a paper used to be scored as one bit: did it replicate or not? That single bit collapses months of layered work and hides everything we want to measure. Decomposition turns replication into a score — how many of the paper's sub-tasks did the agent actually complete? Expert-authored decompositions (PaperBench, 2024) proved the idea; automatic generation removes the expert-labor bottleneck that capped it at a handful of papers.

From replication to recomposition

Because papers decompose into the same reusable parts novel research composes, a library of verified task graphs is more than a benchmark: it is a parts inventory for recomposing new research — a new idea expressed as a novel composition of sub-tasks from existing graphs, graded by the same executable contracts and anti-fabrication auditing that grade replication. That is the bridge from measuring research agents to running them.