SciCode
SciCode evaluates AI agents' ability to generate code for realistic scientific research tasks. It is made up of 65 main problems decomposed into 338 subproblems across 16 subfields in six natural science domains (Mathematics, Physics, Chemistry, Biology, Material Science, and Computational Mechanics).
Paper:
SciCode: A Research Coding Benchmark Curated by Scientists (Tian et al., 2024)
Key Features of Online SciCode
Scientist-Curated, Real-World Research Tasks
All tasks are sourced from scripts underpinning published scientific studies, ensuring robustness and real-world relevance.
Multi-Disciplinary Coverage
SciCode includes tasks from six domains (Physics, Mathematics, Material Science, Biology, Chemistry; plus Computational Mechanics), and covers 16 subfields such as Numerical Linear Algebra, Quantum Chemistry, and Ecology.
SciCode Leaderboard
| Rank | Agent |
Primary Model
Primary Model
This is the primary model used by the agent. In some cases, an embedding model is used for RAG, or a secondary model like GPT-4o for image processing. Note: For non-OpenAI reasoning models, the reasoning token budget is set at 1,024 (low), 2,048 (medium), and 4,096 (high). |
Verified
Verified Results
Results have been reproduced by the HAL team |
Accuracy
Accuracy
Confidence intervals show the min-max values across runs for those agents where multiple runs are available |
Cost (USD)
Total Cost
Total API cost for running the agent on all tasks. Confidence intervals show the min-max values across runs for those agents where multiple runs are available |
Runs
Number of Runs
The number of runs for this agent submitted to the leaderboard. To submit multiple evaluations, rerun the same agent and set the same agent name |
Traces |
|---|---|---|---|---|---|---|---|
| 1 |
Scicode Tool Calling Agent
|
o3 Medium (April 2025) | ✓ | 9.23% | $111.11 | 1 | Download |
| 2 |
Scicode Zero Shot Agent
|
o4-mini Low (April 2025) | ✓ | 9.23% | $1.74 | 1 | Download |
| 3 |
Scicode Tool Calling Agent
|
Claude Opus 4.1 (August 2025) | ✓ | 7.69% | $507.73 | 1 | Download |
| 4 |
HAL Generalist Agent
|
o4-mini Low (April 2025) | ✓ | 6.15% | $165.90 | 1 | Download |
| 5 |
Scicode Zero Shot Agent
|
GPT-4.1 (April 2025) | ✓ | 6.15% | $2.82 | 1 | Download |
| 6 |
Scicode Tool Calling Agent
|
GPT-5 Medium (August 2025) | ✓ | 6.15% | $193.52 | 1 | Download |
| 7 |
Scicode Zero Shot Agent
|
o4-mini High (April 2025) | ✓ | 6.15% | $5.37 | 1 | Download |
| 8 |
Scicode Zero Shot Agent
|
o3 Medium (April 2025) | ✓ | 4.62% | $6.03 | 1 | Download |
| 9 |
Scicode Tool Calling Agent
|
o4-mini High (April 2025) | ✓ | 4.62% | $66.20 | 1 | Download |
| 10 |
Scicode Tool Calling Agent
|
o4-mini Low (April 2025) | ✓ | 4.62% | $46.30 | 1 | Download |
| 11 |
Scicode Tool Calling Agent
|
Claude-3.7 Sonnet High (February 2025) | ✓ | 4.62% | $204.37 | 1 | Download |
| 12 |
HAL Generalist Agent
|
o3 Medium (April 2025) | ✓ | 4.62% (-1.54/+1.54) | $62.64 (-4.34/+4.34) | 2 | Download |
| 13 |
HAL Generalist Agent
|
Claude-3.7 Sonnet (February 2025) | ✓ | 3.08% | $60.40 | 1 | Download |
| 14 |
Scicode Zero Shot Agent
|
Claude-3.7 Sonnet High (February 2025) | ✓ | 3.08% | $4.99 | 1 | Download |
| 15 |
Scicode Zero Shot Agent
|
DeepSeek V3 | ✓ | 3.08% | $0.27 | 1 | Download |
| 16 |
HAL Generalist Agent
|
Claude-3.7 Sonnet High (February 2025) | ✓ | 3.08% | $188.15 | 1 | Download |
| 17 |
Scicode Tool Calling Agent
|
Claude-3.7 Sonnet (February 2025) | ✓ | 3.08% | $191.41 | 1 | Download |
| 18 |
HAL Generalist Agent
|
o4-mini High (April 2025) | ✓ | 1.54% | $92.10 | 1 | Download |
| 19 |
HAL Generalist Agent
|
GPT-4.1 (April 2025) | ✓ | 1.54% | $73.87 | 1 | Download |
| 20 |
Scicode Tool Calling Agent
|
Gemini 2.0 Flash | ✓ | 1.54% | $5.23 | 1 | Download |
| 21 |
Scicode Tool Calling Agent
|
GPT-4.1 (April 2025) | ✓ | 1.54% | $69.39 | 1 | Download |
| 22 |
Scicode Zero Shot Agent
|
Gemini 2.0 Flash | ✓ | 1.54% | $0.12 | 1 | Download |
| 23 |
HAL Generalist Agent
|
Gemini 2.0 Flash | ✓ | 0.00% | $61.49 | 1 | Download |
| 24 |
HAL Generalist Agent
|
DeepSeek R1 | ✓ | 0.00% | $99.61 | 1 | Download |
| 25 |
HAL Generalist Agent
|
DeepSeek V3 | ✓ | 0.00% | $51.95 | 1 | Download |
| 26 |
Scicode Tool Calling Agent
|
DeepSeek R1 | ✓ | 0.00% | $57.62 | 1 | Download |
| 27 |
Scicode Tool Calling Agent
|
DeepSeek V3 | ✓ | 0.00% | $13.47 | 1 | Download |
| 28 |
Scicode Zero Shot Agent
|
Claude-3.7 Sonnet (February 2025) | ✓ | 0.00% | $5.10 | 1 | Download |
| 29 |
Scicode Zero Shot Agent
|
DeepSeek R1 | ✓ | 0.00% | $2.19 | 1 | Download |
Accuracy vs. Cost Frontier for SciCode
This plot shows the relationship between an agent's performance and its token cost. The Pareto frontier (dashed line) represents the current state-of-the-art trade-off. The error bars indicate min-max values across runs.
Heatmap for SciCode
The heatmap visualizes success rates across tasks and agents. Colorscale shows the fraction of times a task was solved across reruns of the same agent. The "any agent" performance indicates the level of saturation of the benchmark and gives a sense of overall progress.
Total Completion Tokens Used per Agent
The bar chart shows the total completion tokens used by each agent, with the height of each bar representing the total number of completion tokens used across all tasks. Secondary models usually contribute a relatively small amount of tokens in comparison, and are used for RAG or image processing only.
Token Pricing Configuration
Adjust token prices to see how they affect the total cost calculations in the leaderboard and plots.
Additional Resources
Getting Started
Want to evaluate your agent on SciCode? Follow our comprehensive guide to get started:
View Documentation