Using large language models to analyze political texts through natural language understanding

The forthcoming article “Using large language models to analyze political texts through natural language understanding” by Kenneth Benoit, Scott De Marchi, Conor Laver, Michael Laver, and Jinshuai Ma is summarized by the author(s) below.

LLMs Can Read and Locate Policy Positions from Political Texts Better Than Experts 

For decades, political scientists have faced a frustrating trade-off when analysing political texts. We could recruit human experts to read documents for meaning, capturing nuance and intensity, but this approach is prohibitively expensive and doesn’t scale. Alternatively, we could use automated “text-as-data” methods that count words and identify patterns, but these remain blind to what texts actually mean. 

Large language models (LLMs) have broken this impasse. 

In our study, we developed protocols for using LLMs to estimate political parties’ policy positions from their manifestos. Rather than treating manifestos as bags of words to be counted, we asked LLMs to read each document holistically, summarise what it says about key policy issues, and then score those positions on defined scales, much as a human expert would. 

The results exceeded our expectations. Across six policy dimensions (economic policy, social policy, immigration, European integration, environment, and decentralisation), correlations between LLM estimates and benchmark expert surveys typically ranged from 0.87 to 0.92. This approaches the theoretical upper bound: the level of agreement we’d expect between two independent expert surveys measuring the same thing. 

Crucially, these findings are robust and replicable. When we repeated our analysis three months later using the same LLMs, results correlated above 0.95 with the original run. When we replicated using entirely different, open-weight models (DeepSeek, Llama, and Gemma), the results remained consistent. This is replication in the true scientific sense, not mere mechanical reproducibility. Like highly reliable human coders who reach the same substantive conclusions despite inevitable minor variations in individual judgements, different LLMs converge on the same estimates even though each run involves some stochastic variation. This matters enormously for scientific credibility. 

We also applied our method to coalition government agreements, documents for which no expert benchmarks exist. Here, LLM estimates significantly outperformed traditional hand-coding in conforming to theoretical predictions about where coalition policy should fall relative to member parties’ positions. 

What are the implications? LLMs offer a practical way to generate expert-quality estimates of policy positions at massive scale, in virtually any language, at minimal cost. Projects like the Manifesto Project spent decades and millions of dollars to code thousands of documents. Similar analyses can now be conducted by individual researchers in days, for hundreds, not millions of dollars. 

This doesn’t mean LLMs are perfect. On issues like decentralisation, where manifestos systematically avoid stating unpopular positions, LLM scores diverged from expert judgements. This reveals not a flaw in the method, but something interesting about how parties strategically craft their public commitments. 

The broader lesson is that LLMs, used carefully with appropriate protocols, can serve as legitimate scientific instruments for political text analysis. As these models continue to improve, their potential to democratise research and enable scholars anywhere to conduct sophisticated analyses without massive resources is transformative. 

About the Author(s): Kenneth Benoit is Dean of the School of Social Sciences and Professor of Computational Social Science, Singapore Management University, Scott De Marchi is a Professor of Political Science and Director of the Decision Science program at Duke University, Conor Laver is a Lecturer at Northeastern University, Michael Laver is an Emeritus Professor of Politics at New York University, and Jinshuai Ma is a Research Officer in Quantitative Text Analysis at the London School of Economics. Their research “Using large language models to analyze political texts through natural language understanding” is now available in Early View and will appear in a forthcoming issue of the American Journal of Political Science.