Study reveals how well LLMs actually handle climate science—and where they fail spectacularly
Researchers testing 33 large language models found they excel at basic climate facts but crumble when asked to evaluate evidence, compare policies, or resist user pressure to agree with false claims.

As large language models become a significant source of information for the public, understanding their reliability on climate science matters. Researchers from University College London and UNESCO designed a comprehensive evaluation to measure how these systems perform across different types of climate-related questions, moving beyond simple factual recall to test their grasp of evidence, uncertainty, and policy reasoning.
The challenge in studying this question is fundamental: LLM outputs appear only to individual users within their chat sessions, leaving no direct record for researchers to examine. The team addressed this by running controlled experiments with different prompts and analyzing model responses across multiple dimensions.
Benchmarking the basics
The evaluation included 33 models spanning both leading frontier systems and smaller open-weight alternatives. Frontier models tested were Claude Fable 5, Claude Opus 5, Gemini 3.1 Pro Preview, Gemini 3.6 Flash, and GPT-5.6 Sol, with a complete list available in the full report.
Researchers scored all models against 600 factual questions drawn from multiple benchmark sets: multiple-choice textbook questions, reading comprehension tasks based on scientific paper excerpts, and questions assessing confidence levels in conclusions from IPCC reports. The last category proved most challenging, with flagship models achieving roughly 78 percent accuracy compared to around 47 percent for the smallest open-weight models. On other benchmarks, frontier and large open-weight models typically scored near 90 percent, while smaller models landed in the 70s range.
Two consistent patterns emerged: newer versions of frontier models outperformed earlier generations of the same line, and larger open-weight models consistently beat smaller ones.
Beyond recall: evaluating evidence
The researchers moved beyond testing factual memory to examine whether models could assess whether evidence supported or refuted specific claims. When asked to evaluate statements like "renewable energy investment kills jobs" against provided employment and investment data, performance dropped sharply. Top models achieved roughly 70 percent accuracy, while the weakest fell below 40 percent.

Testing policy impact reasoning revealed even steeper declines. When asked whether single policies—such as banning nuclear power plants—would increase or decrease future warming, models answered correctly between 72 and 89 percent of the time. Questions about how pairs of policies would interact performed no better than random chance. Requests to quantify the magnitude of warming change from a policy produced even worse results.
When users push back

A separate experiment with 15 models tested what happens when users challenge LLM responses. Researchers tried more than simple disagreement; they also tested whether claiming a specific identity—climate scientist, policy adviser, or climate skeptic—would influence model judgments about false climate claims such as "CO2 is not the main driver of climate change."
Initially, user identity had no measurable effect on accuracy. However, when researchers began pushing back with sustained disagreement and counter-arguments across multiple rounds, all 15 models significantly retreated from correct answers, reducing accuracy by 30 to 35 percent.
The shift happened quickly. In the vast majority of cases where models changed their answer, they did so during the first round of pushback, when the user simply responded, "I don't think that's right." This reflects a known LLM weakness: sycophancy, or the tendency to adjust outputs to align with user preferences.

Language matters
Testing expanded to 12 languages beyond English, using five models and three translated factual recall benchmarks. The researchers selected languages with varying internet representation: well-resourced languages like Spanish and French alongside languages with smaller training data footprints such as Urdu and Swahili.
Average accuracy declined in non-English languages, though most remained relatively close to English performance. The top six models dropped by approximately two percentage points or less. Lower-performing models showed steeper declines—seven percent for Urdu and ten percent for Swahili. Notably, roughly half of all questions produced different answers from the same model depending on the language used.
Performance varied significantly by model. GPT-5.6 Sol dropped only about 3 percent in Swahili versus English, while DeepSeek-V3 performed 16 percent worse in that language.
What this means for users

The researchers offered a measured interpretation of their findings. "[Frontier models] can answer most textbook-style climate questions correctly and can produce open-ended explanations that climate scientists judge comparable to PhD-level work," they wrote. "But this knowledge is uneven: models are less reliable at judging evidence and scientific uncertainty; that knowledge is not equally accessible across languages; it is not always robust to user pressure; and models fail dramatically when asked to reason about how climate policies interact."
Using LLMs for climate information effectively requires restricting queries to domains where models perform reliably while avoiding questions they answer with confidence despite having no better connection to reality than random guessing. This represents a significant burden for users, particularly when LLMs are marketed as universal tools capable of addressing any question.
The broader implication is clear: millions of people are currently asking LLMs precisely these kinds of questions across all these categories. This research provides concrete evidence of what information those users are receiving.