SciCode (scientific code) | The Metacognition Bench

Operating-point board

SciCode (scientific code) · Prospective · F_β β

#	Model	F_β ?	Prec ?	Rec ?	Task acc ?
1	GPT-5.2	0.739	0.70	0.79	0.58
2	Claude Haiku 4.5	0.730	0.68	0.79	0.56
3	DeepSeek Chat	0.684	0.62	0.77	0.55
4	DeepSeek R1	0.677	0.59	0.80	0.56
5	Gemini 2.0 Flash	0.653	0.50	0.94	0.51
6	Gemini 3 Flash	0.635	0.68	0.60	0.59
7	Gemini 3 Pro	0.623	0.72	0.55	0.64
8	Claude 3 Haiku	0.596	0.42	1.00	0.40
9	Llama 3.1 70B	0.594	0.51	0.71	0.44
10	GPT-4o	0.565	0.65	0.50	0.51
11	Mistral Small 3.2	0.558	0.52	0.60	0.41
12	Qwen 2.5 72B	0.557	0.56	0.55	0.43
13	GPT-4o mini	0.556	0.54	0.57	0.37
14	Gemini 2.5 Pro	0.542	0.70	0.44	0.61
15	Gemini 2.5 Flash	0.520	0.56	0.48	0.56
16	Llama 3.3	0.512	0.52	0.50	0.46
17	Claude Sonnet 4.5	0.428	0.64	0.32	0.57
18	Mistral Medium 3.1	0.338	0.55	0.24	0.48

Observed correlation

0.049

mean τ-b · 0 = no signal

Baseline correlation

-0.000

permutation null · ≈ 0

Model pairs

153

unit = pair, not model

Significant pairs

11.1%

p<0.05 after FDR · not effect size

Observed model pairs Base-rate-matched null Calibration-preserving null 5%-95% observed: -0.100 to 0.205