TAU-bench Airline
TAU-bench is a benchmark for Tool-Agent-User Interaction in Real-World Domains. TAU-bench Airline evaluates AI agents on tasks in the airline domain, such as changing flights or finding new flights.
Paper: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao et al., 2024)
Changelog: Removal of TAU-bench Few Shot Agent (December 8, 2025)
We have removed the TAU-bench Few Shot agent results from the leaderboard due to a critical data leakage issue. The Few Shot scaffold contained test set examples in its demonstration file, compromising evaluation integrity. For technical details, see Section 4.2 and Appendix A5 of the HAL paper. All results now use the valid TAU-bench Tool Calling agent.
Changelog: HAL Generalist Agent accuracy is understated (October 5, 2026)
TAU-bench scores an episode only when that episode terminates — either the user
simulator replies ###STOP### or the agent
calls a terminating tool. The HAL Generalist scaffold runs its own loop to completion and
then sends a single closing message, which usually does not elicit
###STOP###. No reward is computed for those
episodes, so they are recorded as 0 and are indistinguishable from a genuine failure.
The HAL Generalist rows below are therefore a lower bound on that
scaffold's accuracy, and are not directly comparable to the TAU-bench Tool Calling rows,
whose loop keeps stepping the environment until the episode terminates. The two scaffolds
also differ in step budget and in how the airline policy document is supplied. This was
reported to us externally and is under review.
TAU-bench Airline Leaderboard
Note: We have paused updating HAL leaderboard with new models and are currently focusing on measuring reliability in AI agents.
Costs are currently calculated without accounting for caching benefits.
Known issue (October 5, 2026): Reported costs and token counts are overstated. Nested trace spans counted the same API response twice, so affected models are inflated by exactly 2×. Models logged as a single span are unaffected, so the overstatement is not uniform across rows. Not yet re-derived.
Downloaded traces are encrypted. To decrypt, install cryptography (pip install cryptography) and use hal-decrypt.sh. See full decryption instructions.
Using these traces (October 5, 2026): in TAU-bench archives,
taken_actions is not purely agent
behaviour. tau-bench's calculate_reward
replays the task's reference actions through the same
Env.step that records agent actions, so
for every graded episode those reference actions are appended to the recorded list. On
inspection they appear as a trailing run of actions the agent never issued, and in some
records they sit in the interior with agent actions after them. Each record carries the
reference sequence under task.actions;
strip it before treating
taken_actions as what the agent did. The
root cause is upstream in tau-bench.
| Rank | Scaffold |
Primary Model
Primary Model
This is the primary model used by the agent. In some cases, an embedding model is used for RAG, or a secondary model like GPT-4o for image processing. Note: For non-OpenAI reasoning models, the reasoning token budget is set at 1,024 (low), 2,048 (medium), and 4,096 (high). |
Verified
Verified Results
Results have been reproduced by the HAL team |
Accuracy
Accuracy
Confidence intervals show the min-max values across runs for those agents where multiple runs are available |
Cost (USD)
Total Cost
Total API cost for running the agent on all tasks. Confidence intervals show the min-max values across runs for those agents where multiple runs are available |
Runs
Number of Runs
The number of runs for this agent submitted to the leaderboard. To submit multiple evaluations, rerun the same agent and set the same agent name |
Traces
Encrypted Traces
Downloaded traces are encrypted. Install |
|---|---|---|---|---|---|---|---|
| 1 |
TAU-bench Tool Calling
Pareto optimal
|
o4-mini High (April 2025) | ✓ | 56.00% | $11.36 | 1 | Download |
| 2 | Claude-3.7 Sonnet (February 2025) | ✓ | 56.00% | $42.11 | 1 | Download | |
| 3 | o3 Medium (April 2025) | ✓ | 54.00% | $14.56 | 1 | Download | |
| 4 | Claude Opus 4.1 (August 2025) | ✓ | 54.00% | $180.49 | 1 | Download | |
| 5 | Claude-3.7 Sonnet High (February 2025) | ✓ | 52.00% | $31.94 | 1 | Download | |
| 6 | Claude Opus 4.1 High (August 2025) | ✓ | 52.00% | $149.98 | 1 | Download | |
| 7 | Claude Opus 4.1 (August 2025) | ✓ | 50.00% | $69.78 | 1 | Download | |
| 8 | GPT-5 Medium (August 2025) | ✓ | 48.00% | $23.83 | 1 | Download | |
| 9 |
TAU-bench Tool Calling
Pareto optimal
|
DeepSeek V3 (March 2025) | ✓ | 44.00% | $5.43 | 1 | Download |
| 10 | Claude-3.7 Sonnet (February 2025) | ✓ | 44.00% | $15.45 | 1 | Download | |
| 11 | Claude-3.7 Sonnet High (February 2025) | ✓ | 44.00% | $34.58 | 1 | Download | |
| 12 | Claude Opus 4 (May 2025) | ✓ | 44.00% | $150.15 | 1 | Download | |
| 13 | Claude Opus 4 High (May 2025) | ✓ | 44.00% | $150.29 | 1 | Download | |
| 14 | o4-mini Low (April 2025) | ✓ | 36.00% | $7.14 | 1 | Download | |
| 15 | GPT-4.1 (April 2025) | ✓ | 36.00% | $8.18 | 1 | Download | |
| 16 | DeepSeek R1 (January 2025) | ✓ | 36.00% | $13.30 | 1 | Download | |
| 17 | Claude Opus 4.1 High (August 2025) | ✓ | 32.00% | $140.28 | 1 | Download | |
| 18 | GPT-5 Medium (August 2025) | ✓ | 30.00% | $52.78 | 1 | Download | |
| 19 |
TAU-bench Tool Calling
Pareto optimal
|
Gemini 2.0 Flash High (February 2025) | ✓ | 28.00% | $0.31 | 1 | Download |
| 20 | Gemini 2.0 Flash (February 2025) | ✓ | 22.00% | $2.00 | 1 | Download | |
| 21 | o4-mini Low (April 2025) | ✓ | 22.00% | $20.16 | 1 | Download | |
| 22 | o3 Medium (April 2025) | ✓ | 20.00% | $45.03 | 1 | Download | |
| 23 | DeepSeek V3 (March 2025) | ✓ | 18.00% | $10.73 | 1 | Download | |
| 24 | o4-mini High (April 2025) | ✓ | 18.00% | $20.57 | 1 | Download | |
| 25 | GPT-4.1 (April 2025) | ✓ | 16.00% | $17.85 | 1 | Download | |
| 26 | DeepSeek R1 (January 2025) | ✓ | 10.00% | $30.18 | 1 | Download |
Accuracy vs. Cost Frontier for TAU-bench Airline
This plot shows the relationship between an agent's performance and its token cost. The Pareto frontier (dashed line) represents the current state-of-the-art trade-off. The error bars indicate min-max values across runs.
Heatmap for TAU-bench Airline
The heatmap visualizes success rates across tasks and agents. Colorscale shows the fraction of times a task was solved across reruns of the same agent. The "any agent" performance indicates the level of saturation of the benchmark and gives a sense of overall progress.
Total Completion Tokens Used per Agent
The bar chart shows the total completion tokens used by each agent, with the height of each bar representing the total number of completion tokens used across all tasks. Secondary models usually contribute a relatively small amount of tokens in comparison, and are used for RAG or image processing only.
Model Performance Over Time
Timeline showing model accuracy evolution over release dates. Each point represents the best performance achieved by that model on TAU-bench Airline.
Token Pricing Configuration
Adjust token prices to see how they affect the total cost calculations in the leaderboard and plots.
Additional Resources
Getting Started
Want to evaluate your agent on TAU-bench Airline? Follow our guide to get started:
View Documentation