LLMs Are Not (Consistently) Bayesian — paper quantifies deviations from Bayes updates
The paper treats LLMs as information-processing rules and introduces the 'information processing gap' metric to quantify deviations from Bayesian updates when LLMs update probabilistic beliefs in light of new evidence. The authors run extensive experiments to evaluate internal (in)consistencies of LLMs' belief-updating behavior, with relevance to high-stakes domains like medicine, science and law.