Wisdom of Crowds: What LLM Disagreement Reveals About AI Uncertainty
When multiple AI models disagree, what does that tell us? The “wisdom of crowds” phenomenon shows that aggregating independent judgments often outperforms individual experts. But for AI systems, ensemble disagreement might reveal something deeper: the structure of uncertainty itself. The Hypothesis When multiple LLMs disagree on a question, the pattern of disagreement reveals the epistemological nature of the problem: High agreement → Robust, well-established knowledge Systematic disagreement → Genuine ambiguity or value-laden territory Random disagreement → Knowledge gaps or reasoning failures Experiment Design We queried 4 models (Claude Opus 4....