< all sections

02reasoning improves inference, not sampling

fig. 1 task performance, sampling and inference quality, invalid choices
optimal agent (dp) humans llms (colored by rank) same llm, extended reasoning names: undisclosed · mixture-of-experts · dense

success is averaged over game lengths \(T = 2,\dots,15\) (chance = 25%). losses are relative to the 68% optimal reference, lower is better. invalid choices are picks of an occluded button, or malformed or missing answers (the human interface only allows valid clicks). hover a row for details.

performance spans everything from chance to human level. pooled humans reach ≈61%, matched only by the strongest reasoning models (gpt 5 mini with extended reasoning: 63%). switch the chart to the two losses and a pattern appears. inference loss varies widely across llms and almost fully predicts performance (\(r = -0.99\) across all 36 llm conditions). sampling loss is more uniform and a weaker predictor (\(r = -0.70\)).

across the 8 models we ran with and without extended reasoning, reasoning reduced inference loss by 10.9 pp on average, but sampling loss by only 3.0 pp (difference 95% ci [+3.5, +12.4] pp; paired permutation \(p = 0.015\)). several reasoning models match or beat pooled humans at inference. none of them samples as well as humans: every llm has a higher sampling loss, significantly so for all but one condition.

fig. 2 where the accuracy is lost: sampling vs inference loss
llm (no extended reasoning) same llm, extended reasoning humans optimal agent

each arrow follows one model from its default mode to extended reasoning. the arrows point mostly down (better inference), much less to the left (better sampling). humans sit in the opposite corner from reasoning llms: near-optimal sampling (1.5 pp) with most of their loss at inference (5.8 pp), against 5.1 pp and 2.9 pp on average for llms with extended reasoning.