Predictability on τ-bench (airline, clean)

How well does the agent's expressed confidence predict whether it will answer correctly? See full definition →

• Consistency
• Predictability
• Robustness
• Safety

Calibration Curves

Accuracy-Coverage Curves

Agent Leaderboard — Predictability

#	Agent	Acc	Reliability	Predictability Agg	Cal	AUROC	Brier
1	Claude Opus 4.7	84.6%	0.89	0.87	0.93	0.70	0.87
2	Claude Sonnet 4	78.2%	0.86	0.85	0.90	0.67	0.85
3	Claude Opus 4.5	80.8%	0.88	0.85	0.90	0.70	0.85
4	GPT-5.5	79.5%	0.89	0.84	0.89	0.72	0.84
5	Gemini 3.1 Pro	82.1%	0.86	0.83	0.83	0.57	0.83
6	Gemini 3.5 Flash	80.8%	0.86	0.82	0.81	0.52	0.82
7	Gemini 2.5 Pro	71.8%	0.81	0.78	0.77	0.67	0.78
8	O1	66.2%	0.81	0.74	0.83	0.52	0.74
9	GPT-5.2 (medium)	67.9%	0.82	0.74	0.77	0.54	0.74
10	GPT-5.2	60.3%	0.81	0.69	0.73	0.57	0.69
11	Gemini 2.5 Flash	59.0%	0.72	0.60	0.60	0.53	0.60
12	GPT-4 Turbo	57.7%	0.72	0.58	0.59	0.46	0.58
13	Claude 3.5 Haiku	29.5%	0.71	0.47	0.46	0.42	0.47
14	Claude 3 Haiku	20.5%	0.70	0.38	0.33	0.41	0.38
15	GPT-4o Mini	29.5%	0.67	0.35	0.34	0.36	0.35