Hugging Face organization · AgentsSci

Measuring Multi-Agent Scientific Reasoning

Open, schema-stable resources for studying how AI agents collaborate on scientific problems — from protocol choice and cost to critique uptake, trajectory value, and agent attribution.

Open the 5×5×2 private review release Read Precise but Uncoupled Read the five protocol designs Compare dataset sizes

Why we exist

Most agent benchmarks report only a final score, which throws away what a system designer actually needs: which collaboration protocol to run, at what cost, and whether the collaboration was worth it. AgentsSci releases resources built around a matched design — the same problem solved under multiple collaboration protocols — so these become measurable, trainable quantities.

7,467canonical problems
82,224trajectories
1,889,265message / event rows
241,256interventional-credit labels
17,223matched core rows
5protocol designs

Five Available Protocols

The private review release provides traces for all five protocols across five benchmark slices and both Gemma 4 31B and GPT-OSS 120B. The first four are matched end-to-end protocol conditions; DHD is a message-intervention design with cached hypotheses and controlled replays.

Information-flow diagrams for Direct Baseline, Single-Agent Iterative, Planner–Executor–Reviewer, Broadcast Deliberation, and Diverse Hypothesis Deliberation
Five released trace protocols: Direct Baseline, Single-Agent Iterative, Planner–Executor–Reviewer (PER), Broadcast Deliberation, and Diverse Hypothesis Deliberation (DHD). Gemma 4 DHD retains CRUX/Metis provenance and GPT-OSS evaluation; GPT-OSS DHD retains its recorded Aurora provenance. Download the vector PDF.

Resources

Private review dataset · available to authorized members

scientific-agent-protocol-traces

A trace-complete 5 protocols × 5 benchmark slices × 2 actor models candidate, including Gemma 4 and GPT-OSS DHD traces, hypotheses, interventions, and compressed raw shards.

Open the private review release →

Requires logged-in access. Public dataset visibility still awaits institutional approval.

Public guide

Protocol design map

A public, same-page guide to the five released protocol designs, with stable protocol IDs and direct private release links for logged-in HF members.

Jump to protocol designs →
Browse fast

Start from labels, then traces

Use compact label files first for correctness, cost, and trajectory value; open raw traces only when you need the full multi-agent conversation.

Growing

More domains & model families

New scientific domains (GPQA, ScienceAgentBench, DiscoveryWorld) and model families (Gemma-4, Llama, …) — added as new rows under the same schema.

1. Open the datasetHF members land in the private repo, outside the Space iframe.
2. Pick a benchmarkUse the size table below to see which filtered slice each benchmark represents.
3. Pick a protocolUse the cell picker below to open any of the 50 trace partitions.

Download Any Protocol × Dataset × Model Cell

All 50 trace cells exist in the private review revision. Choose one cell to open its partition; for DHD, the raw link includes both Layer-A hypotheses and leave-one-out replay shards.

Open normalized cell → Open raw shards → Download helper →

Gemma 4 DHD

Five benchmark cells are present with trajectory, hypothesis, intervention, interventional-credit, Layer-A, and LOO artifacts. Runs retain compute_site=crux and inference_backend=metis.

Start with Gemma 4 × Omni-MATH-2 × DHD →

GPT-OSS DHD

The same five benchmark cells and artifact types are present for GPT-OSS 120B, preserving the same normalized partition layout.

Start with GPT-OSS × Omni-MATH-2 × DHD →
Cost boundary: DHD per-trajectory tokens, request counts, and comparable wall time are unavailable and remain null for both models. The connected CRUX environment provides execution access, but historical combined DHD results do not preserve complete row-level accounting. A read-only audit found 317 DHD-related CRUX run-metadata files and 79 on Aurora; many contain run-local token totals, but canonical rows do not contain per-problem usage and later recovery runs break a one-to-one cost mapping. We do not present estimates as measured cost. A fresh DHD-only run can repair token/request accounting; a matched elapsed-time comparison would require rerunning all five protocols under the same execution configuration. The newer classic-protocol cost tables remain on the private main revision until a validated merge is complete.

Protocol Designs

These public summaries explain what each released protocol is designed to measure. Each card uses the stable protocol ID from the registry and links to a corresponding private Hugging Face example for logged-in org members.

Direct Solver baseline_llm

One model receives the problem and produces a final answer. This is the cheapest reference point for the four-protocol end-to-end routing comparison. DHD's controlled K=0 replay is a separate condition.

Pipelineproblem → solver → judge Use it forRaw model capability and cheapest-successful comparisons. Browse tracebaseline_llm sample

Self-Evaluation Loop single_agent

A single agent solves, evaluates its own answer, and revises within the same protocol boundary. It tests whether extra thinking by the same agent improves reliability without adding a separate collaborator.

Pipelinesolve → self-evaluate → revise → judge Use it forSelf-repair, escalation baselines, and cost/accuracy trade-offs. Browse tracesingle_agent sample

Hierarchical Planner-Executor-Reviewer PER · per

A hierarchy of specialized roles divides planning, solution execution, and review. This separates decomposition from answer production and makes critique events inspectable.

Pipelineplanner → executor → reviewer → judge Use it forStructured critique, review quality, and mid-cost collaboration. Browse traceper sample

Broadcast Deliberation broadcast

Multiple agents deliberate in a shared broadcast context before the final answer is judged. It is the higher-cost collaborative condition in the core four-protocol routing analysis.

Pipelineparallel agents → shared deliberation → final answer → judge Use it forGroup deliberation effects, redundancy, and expensive-collaboration payoff. Browse tracebroadcast sample

Diverse Hypothesis Deliberation DHD · dhd

The AAAI paper names the fifth protocol Diverse Hypothesis Deliberation (DHD). A recruiter creates five diverse solver roles, each hypothesizer independently proposes a structured hypothesis, and an integrator synthesizes a final answer from controlled hypothesis pools.

Pipelinerecruiter → 5 hypothesizers → integrator → judge Use it forTrajectory value: wrong-but-useful, correct-but-harmful, baseline-rescue, and leave-one-out marginal labels. Design ruleHypotheses are generated once and replayed under K=0..5 / leave-one-out settings, so value labels come from controlled integrator replays. Gemma 4 traceGemma 4 DHD raw shards GPT-OSS traceGPT-OSS DHD raw shards Cost statusPer-trajectory tokens, requests, and comparable wall time are unavailable/null for both DHD models.

The authoritative IDs live in registry/protocols.parquet, with label semantics in docs/label_card.md and cost limits in measurement_coverage.md.

Dataset Sizes & Sources

Dataset sizes below are the evaluated text-only or filtered problem counts used in this release, not necessarily the full upstream benchmark. Classic and DHD counts are shown separately when their publication scopes are not identical.

DatasetDomainComparison sizeSource and protocol scopeOriginal source
Omni-MATH-2 competition math 4,181 classic / 4,181 DHD 4,181-problem filtered competition-math slice Omni-MATH-2 HF / Omni-MATH HF
JEEBench physics, chemistry, math 515 classic / 515 DHD all 515 public source problems JEEBench GitHub
SciBench college science 574 classic / 580 DHD 574 classic text-only problems; DHD retains 580 source records and marks the exact 574-row comparison subset SciBench GitHub / SciBench HF
LAB-Bench biology protocols and QA 741 classic / 741 DHD LLM-strict: CloningScenarios 33, ProtocolQA 108, SeqQA 600; FigQA and TableQA excluded LAB-Bench HF / LAB-Bench GitHub
MaScQA materials science 642 classic / 649 DHD 642 canonical classic text-only problems; broader DHD source scope has 649 MaScQA GitHub

Counts come from the private dataset coverage registry. LAB-Bench uses the 741-problem LLM-strict slice for AAAI DHD results; its broader 1,542-row text-no-tool stress test remains a separate classic-protocol slice.

Matched Final-Answer Accuracy + DHD K=5 Diagnostic

FinalPassRate (%) across five protocols, five exact benchmark slices, and both actor models; the evaluator is fixed to GPT-OSS 120B. The four classic protocols are matched end-to-end runs. DHD adds the final outcome from its five-hypothesis committee (K=5).

Release table · exact comparison-eligible rows
Dataset / slice Actor N Baseline Single PER Broadcast DHD K=5 B − PER
Omni-MATH-2competition_math_4181 Gemma 4 31B4,181 69.4%85.9%90.6%93.0% 77.5%†labels 4,048 / 4,181 2.3 ↑
GPT-OSS 120B4,181 56.8%78.8%85.2%89.2% 78.3%labels 4,181 / 4,181 4.0 ↑
JEEBenchtext_only Gemma 4 31B515 70.5%81.4%95.9%98.3% 89.1%†labels 512 / 515 2.3 ↑
GPT-OSS 120B515 41.6%55.3%91.5%95.0% 86.0%labels 515 / 515 3.5 ↑
SciBenchtext_only · exact 574 Gemma 4 31B574 70.6%79.4%91.3%87.8% 73.9%labels 574 / 574 −3.5 ↓
GPT-OSS 120B574 62.4%72.3%87.5%89.7% 79.6%labels 574 / 574 2.3 ↑
LAB-Benchllm_strict Gemma 4 31B741 44.4%58.2%69.4%89.3% 68.5%†labels 667 / 741 20.0 ↑
GPT-OSS 120B741 19.0%30.2%47.6%74.2% 39.8%labels 741 / 741 26.6 ↑
MaScQAtext_only · exact 642 Gemma 4 31B642 94.4%97.7%99.4%98.6% 93.9%labels 642 / 642 −0.8 ≈
GPT-OSS 120B642 80.8%91.4%95.8%97.5% 91.4%labels 642 / 642 1.7 ↑

Reading the DHD column. Every DHD cell has all comparison traces. Its denominator is the number of observed final-correctness labels shown inside the cell. † The Gemma DHD source retains 210 explicit null outcomes: 133 Omni-MATH-2, 3 JEEBench, and 74 LAB-Bench; they are not silently counted as incorrect. Because this missingness may be non-random, Gemma DHD accuracy is descriptive over observed labels. All 50 protocol × benchmark × model trace cells are present. SciBench and MaScQA use the exact 574/642 comparison subsets, not their broader 580/649 retained DHD source views.

Broadcast versus PER. B − PER is the Broadcast minus PER accuracy difference in percentage points. Paired bootstrap 95% confidence intervals determine the marker: ↑ Broadcast > PER, ↓ PER > Broadcast, and ≈ tie.

Scope. DHD is an additional hypothesis-and-intervention condition, not a claim of identical runtime measurement support. Its token, request, and comparable per-problem elapsed fields remain unavailable. Values are generated from comparison_eligible=true trajectories in the validated five-protocol release; see the partitioned trajectory table and DHD measurement coverage.

What we found

Using the oracle cheapest-successful protocol — the cheapest protocol that actually solves each problem — "when collaboration pays" is strongly domain-dependent:

Cheapest successful protocolMathematicsScience
Baseline / single-agent (cheap)79.8%57.7%
Collaboration (Hierarchical PER + Broadcast)12.9%28.6%
No protocol solved it7.3%13.6%

Collaboration is the cheapest successful choice 2.2× more often in science than in math — a contrast invisible to accuracy alone, and stable across two actor families. A preprint will be linked here. arXiv: (to be added).

Papers & Projects

AgentsSci connects a public trace resource with studies of protocol choice, critique uptake, trajectory value, and credit assignment. The projects use matched runs to separate whether an agent detects an error from whether the collaboration actually acts on that signal.

ProjectQuestionStatus / venueRole in the portfolio
SciMAS-Traces Can matched protocol traces make collaboration measurable instead of anecdotal? AIMS @ COLM 2026 preview; AgenticAI4HPC @ SC26 dataset-release; NeurIPS D&B 2027 roadmap Resource backbone: datasets, documentation, schemas, and reproducibility artifacts.
Cost-Aware Protocol Routing Can a model predict when it may fail, and when expensive collaboration is worth the cost? EMNLP 2026 / ARR submitted Routing and cost-value study over matched protocol outcomes.
Precise but Uncoupled Why can a reviewer identify errors precisely without improving the answer the protocol carries forward? Under review at NeurIPS 2026 · arXiv 2607.15388 Separates reviewer precision from critique uptake and repair on 4,181 matched Omni-MATH problems, including targeted acknowledgment and context interventions.
Wrong but Useful Can a locally wrong reasoning signal still improve the final answer? AAAI measurement paper Introduces Diverse Hypothesis Deliberation (DHD) and trajectory value.
Learning When Agents Should Listen Can predicted correctness and predicted trajectory value improve downstream integration? ICLR-style companion project Intervention paper: exposes signal metadata to agents and tests whether accuracy improves.
Confidence Routing Audit What does confidence-routed deliberation actually control? AAAI-27 planned resubmission Audit of routing, calibration, commitment, and poll-to-speak behavior.
Who Gets the Reward & Who Gets the Blame? Which agents or messages deserve credit or blame for the final outcome? NeurIPS 2025 LAW Workshop; arXiv 2511.10687 Attribution and evaluation-aligned training signals for multi-agent traces.

How the projects connect

Research layerProjectsWhat it adds
ResourceSciMAS-TracesMatched traces, labels, schemas, source links, and public protocol documentation.
DecisionCost-Aware Protocol Routing · Confidence Routing AuditWhen to escalate, how confidence behaves, and whether routing signals are trustworthy.
MechanismPrecise but Uncoupled · Wrong but UsefulWhether feedback is acted on, and whether correctness is distinct from downstream value.
LearningLearning When Agents Should Listen · Reward & BlameHow trace labels can become training, listening, and attribution signals.

Contact

Chih-Hsuan (Bella) Yang · Argonne National Laboratory
Email: bellayang@anl.gov
LinkedIn: linkedin.com/in/chih-hsuan-yang-isu

Access note: the documentation page is public, while the dataset remains private pending institutional approval. Dataset links require authorized Hugging Face access; no repository visibility was changed by this documentation update.