The Metacognition Bench?

We rank the risk sensitive (F-β) accuracy of Large Language Model verbalised confidence with 20 models tested across 6 benchmarks. While we uncover winners and losers within each performance band, and consistent biases towards under and overconfidence, the real story is found in the inter-model structure of our results. Models share an underlying confidence heuristic, and verbalised confidences are as likely to describe other models as themselves. We found no model where differences in confidence with other models correlated with differences in performance.