reasoning under uncertainty in humans and language models
humans and the best reasoning llms reach similar accuracy on this game (≈61%), but they get there by different routes.
extended reasoning mostly improves inference: it cuts inference loss by 10.9 pp but sampling loss by only 3.0 pp.
every llm samples evidence worse than humans do. humans lose their accuracy at the final decision instead.
a small behavioral model places humans and llms in one shared cognitive space, and its latent variables show up in the activations of gpt oss 20b.
sections
a game that separates gathering evidence from using it, and the agent that plays it perfectly.
humans and 28 llms against the optimal agent, split into sampling and inference loss.
memory, strategy, choice bias and occlusion awareness: one parameter for each deviation from bayes.
run any fitted agent, or your own parameters, round by round in the browser.
how probable each choice was, for every agent, data against the fitted model.
every agent placed by its fitted parameters, and where extended reasoning moves it.
gpt oss 20b's activations move between choice regions the way the fitted model predicts.
takeaways
- humans and reasoning llms reach comparable success by different routes: humans sample near-optimally and lose accuracy at inference, reasoning llms show the reverse profile.
- extended reasoning mainly buys better evidence integration (\(\beta \to 0\), larger \(\kappa_f\)), not better evidence gathering, even though gpt oss 20b spends more reasoning tokens on sampling rounds than on the final decision.
- training regime, not model size, predicts performance: it explains 86% of the variance in success across open models, against 9% for size.
- the sampling gap is invisible to accuracy alone: current models do not reliably turn their beliefs into informative actions.
cite
mockup · figures are drawn live from the camera-ready figure sources (loss tables, fitted parameters, posterior curves); the playground re-implements the behavioral model in javascript. paper link and bibtex are placeholders.