Research

Our goal is to address worst-case risks from the development and deployment of advanced AI. The two agendas below cover most of our current work.

Research agenda

Model Persona Research Agenda

This agenda studies and steers the emergence of malicious propensities in LLMs — traits like spitefulness, sadism, and punitiveness. We treat personas, bundles of correlated traits, as a useful abstraction for how propensities generalise out-of-distribution, and as a target for interventions.

Read the agenda →

Selected work

Inoculation Adapters: Improved Selective Generalization of Capabilities with Fewer Surprising Backdoors2026
Concrete Research Ideas on AI Personas2026
Training large language models on narrow tasks can lead to broad misalignment2026

Research agenda

Safe Pareto Improvements Research Agenda

Safe Pareto improvements (SPIs) are modifications to agents’ bargaining strategies that make all parties better off, regardless of their original strategies — an unusually robust approach to preventing catastrophic conflict between AI systems. This agenda addresses the risk that early AI development forecloses the option to adopt them.

Read the agenda →

Selected work

Taboo “equilibrium”: Less confused frames for research on AI bargaining2026
A high-level model of AI bargaining2026

Beyond these agendas

Other active interests

Outside of these agendas, we work on:

S-risk macrostrategy, with a focus on determining when and how interventions robustly reduce s-risk. For background, see Beginner's guide to reducing s-risks.
Automating conceptual work relevant to s-risk reduction.

Earlier research

Our 2020 research agenda Cooperation, Conflict, and Transformative Artificial Intelligence remains influential on our current priorities, especially our work on safe Pareto improvements.

Our Priority areas page, first published in August 2020, is now outdated, though several of the areas it describes remain relevant to how we prioritize today.

All research138 outputs, 2013–2026

Filter

Type Post Paper Agenda Other

Area Model Personas Multi-agent systems Decision theory Ethics & meta-ethics Prioritization & macrostrategy AI Forecasting Other

Agenda Safe Pareto improvements Model personas

2026

Taboo “equilibrium”: Less confused frames for research on AI bargaining
Anthony DiGiovanniLessWrong
Safe Pareto improvements
Value Leakage: An LLM's Answers Are Silently Shaped by Its Own Values
Jan Betley, Johannes Treutlein, Jan Dubiński, Harry Mayne, Karol Gałązka, Niels Warncke, Anna Sztyber-Betley, Owain EvansarXiv
PaperModel personas
Inoculation Adapters: Improved Selective Generalization of Capabilities with Fewer Surprising Backdoors
Maxime Riché, Daniel Tan, Vili Kohonen, Niels WarnckearXiv
PaperModel personas
A high-level model of AI bargaining
Anthony DiGiovanniLessWrong
Safe Pareto improvements
Evals for “SPI-incompatible” behavior and reasoning
Anthony DiGiovanniLessWrong
Safe Pareto improvements
Strategic Obfuscation of Deceptive Reasoning in Language Models
Arun Jose, Niels Warncke, Mia TaylorICLR
PaperModel personas
Safe Pareto Improvements Research Agenda
Anthony DiGiovanniLessWrong
AgendaSafe Pareto improvements
Shaping the exploration of the motivation-space matters for AI safety
Maxime Riché, Victor Gillioz, Niels Warncke, Kajetan Dymkiewicz, Filip Sondej, Roger Dearnaley, Daniel Tan, Dillon Khang NguyenLessWrong
Model personas
Model Persona Research Agenda
Niels Warncke
AgendaModel personas
Concrete Research Ideas on AI Personas
Niels Warncke, Maxime Riché, Daniel TanLessWrong
Model personas
Conditionalization Confounds Inoculation Prompting Results
Maxime Riché, Niels WarnckeLessWrong
Model personas
Training large language models on narrow tasks can lead to broad misalignment
Jan Betley, Niels Warncke, Anna Sztyber-Betley, Daniel Tan, Xuchan Bao, Martín Soto, Megha Srivastava, Nathan Labenz, Owain EvansNature
PaperModel personas

2025

A Case for Model Persona Research
Niels Warncke, Maxime Riché, Daniel TanLessWrong
Model personas
Outcome robustness and implementation robustness
Anthony DiGiovanni
Prioritization & macrostrategy
Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time
Daniel Tan, Anders Cairns Woodruff, Niels Warncke, Arun Jose, Maxime Riché, David Demitri Africa, Mia TaylorarXiv
PaperModel personas
Bracketing Cluelessness
Sylvester Kollin, Jesse Clifton, Anthony DiGiovanni, Nicolas Macé
Decision theory
Aesthetic Preferences Can Cause Emergent Misalignment
Anders Cairns WoodruffLessWrong
Model personas
School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs
Mia Taylor, James Chua, Jan Betley, Johannes Treutlein, Owain EvansarXiv
PaperModel personas
Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs
Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, Owain EvansICML
PaperModel personas
Safe Pareto Improvements for Expected Utility Maximizers in Program Games
Anthony DiGiovanni, Jesse Clifton, Nicolas MacéAAMAS
PaperSafe Pareto improvements

2024

Why Evidential Cooperation in Large Worlds might not be action-guiding
Anthony DiGiovanniSubstack
Decision theory
Overview of Transformative AI Misuse Risks: What Could Go Wrong Beyond Misalignment
Samuel Martin
Prioritization & macrostrategy
Winning isn't enough
Jesse Clifton, Anthony DiGiovanniLessWrong
Decision theory
Overview and Analysis of Transformative AI Misuse
Samuel Martin
Prioritization & macrostrategy
Individually incentivized safe Pareto improvements in open-source bargaining
Nicolas Macé, Anthony DiGiovanni, Jesse CliftonLessWrong
Safe Pareto improvements
Measurement Research Agenda
Mia TaylorSuperseded
AgendaModel personas
Computing Optimal Commitments to Strategies and Outcome-conditional Utility Transfers
Nathaniel Sauerberg, Caspar OesterheldAAMAS
PaperMulti-agent systems

2023

Making AIs less likely to be spiteful
Nicolas Macé, Anthony DiGiovanni, Jesse CliftonLessWrong
Multi-agent systems
Beginner’s guide to reducing s-risks
Anthony DiGiovanni
Prioritization & macrostrategy
Responses to apparent rationalist confusions about game / decision theory
Anthony DiGiovanniLessWrong
Multi-agent systems
Modeling evidential cooperation in large worlds
Johannes TreutleinarXiv
PaperDecision theory
Open-minded updatelessness
Nicolas Macé, Jesse Clifton, Sylvester KollinLessWrong
Decision theory

2022

The optimal timing of spending on AGI safety work; why we should probably be spending more now
Tristan Cook, Guillaume CorlouerEA Forum
Prioritization & macrostrategy
When is intent alignment sufficient or necessary to reduce AGI conflict?
Jesse Clifton
Multi-agent systems
When would AGIs engage in conflict?
Jesse Clifton
Multi-agent systems
When does technical work to reduce AGI conflict make a difference?: Introduction
Jesse Clifton
Multi-agent systems
Evolutionary Stability of Other-Regarding Preferences Under Complexity Costs
Anthony DiGiovanni, Nicolas Macé, Jesse CliftonLearning, Evolution, and Games
Multi-agent systems
Commitment games with conditional information revelation
Anthony DiGiovanni, Jesse CliftonAAAI
Multi-agent systems
Replicating and extending the grabby aliens model
Tristan CookEA Forum
Prioritization & macrostrategy
When does technical work to reduce AGI conflict make a difference?
Jesse Clifton, Samuel Martin, Anthony DiGiovanni
Multi-agent systems

2021

Normative Disagreement as a Challenge for Cooperative AI
Julian Stastny, Maxime Riché, Alexander Lyzhov, Johannes Treutlein, Allan Dafoe, Jesse CliftonCooperative AI workshop and the Strategic ML workshop at NeurIPS
Multi-agent systems
Safe Pareto Improvements for Delegated Game Playing
Caspar Oesterheld, Vincent ConitzerAAMAS
Safe Pareto improvements
What 2026 looks like
Daniel KokotajloLessWrong
AI Forecasting
Taboo "Outside View"
Daniel Kokotajlo
Prioritization & macrostrategy
The Evidentialist's Wager
William MacAskill, Aron Vallinder, Caspar Oesterheld, Carl Shulman, Johannes TreutleinThe Journal of Philosophy
PaperDecision theory
Case studies of self-governance to reduce technology risk
Jia Yuan Loke
Prioritization & macrostrategy
Coordination challenges for preventing AI conflict
Stefan Torges
Multi-agent systems
Collaborative game specification: arriving at common models in bargaining
Jesse CliftonWorking paper
Multi-agent systems
Fun with +12 OOMs of Compute
Daniel KokotajloLessWrong
AI Forecasting
Weak identifiability and its consequences in strategic settings
Jesse CliftonWorking paper
Multi-agent systems
Birds, Brains, Planes, and AI: Against Appeals to the Complexity / Mysteriousness / Efficiency of the Brain
Daniel KokotajloLessWrong
AI Forecasting
Reinforcement Learning in Newcomblike Environments
James Bell, Linda Linsefors, Caspar Oesterheld, Joar SkalseNeurIPS
PaperDecision theory

2020

Against GDP as a metric for AI timelines and takeoff speeds
Daniel Kokotajlo
AI Forecasting
Incentivizing forecasting via social media
David Althaus, Daniel KokotajloEA Forum
AI Forecasting
Commitment ability in multipolar AI scenarios
Anni Leskelä
Multi-agent systems
Against GDP as a metric for timelines and takeoff speeds
Daniel KokotajloLessWrong
AI Forecasting
Persuasion Tools: AI takeover without takeoff or agency?
Daniel Kokotajlo
AI Forecasting
How Roodman's GWP model translates to TAI timelines
Daniel Kokotajlo
AI Forecasting
Persuasion Tools: AI takeover without AGI or agency?
Daniel KokotajloLessWrong
AI Forecasting
The date of AI Takeover is not the day the AI takes over
Daniel Kokotajlo
AI Forecasting
Towards cooperation in learning games
Jesse Clifton, Maxime RichéWorking paper
Multi-agent systems
Sequence on moral anti-realism
Lukas GloorEA Forum
Ethics & meta-ethics
Reducing long-term risks from malevolent actors
David Althaus, Tobias BaumannEA Forum
Prioritization & macrostrategy
Equilibrium and prior selection problems in multipolar deployment
Jesse CliftonAlignment Forum
Multi-agent systems
Cooperation, Conflict, and Transformative Artificial Intelligence: A Research Agenda
Jesse Clifton
AgendaPrioritization & macrostrategy

2019

Soft takeoff can still lead to decisive strategic advantage
Daniel KokotajloAlignment Forum
AI Forecasting
The "Commitment Races" problem
Daniel KokotajloAlignment Forum
Multi-agent systems
Rebuttal of Christiano and AI Impacts on takeoff speeds?
Lukas GloorLessWrong
AI Forecasting
Approval-directed agency and the decision theory of Newcomb-like problems
Caspar OesterheldSynthese (runner-up, AI Alignment Prize)
PaperDecision theory
Risk factors for s-risks
Tobias Baumann
Prioritization & macrostrategy
Sequence on non-agent and multiagent models of mind
Kaj SotalaLessWrong
Other
Long-term trajectories of human civilization
Seth D Baum, Stuart Armstrong, Timoteus Ekenstedt, Olle Häggström, Robin Hanson, Karin Kuhlemann, Matthijs M Maas, James D Miller, Markus Salmela, Anders Sandberg, Kaj Sotala, Phil Torres, Alexey Turchin, Roman V YampolskiyForesight
PaperPrioritization & macrostrategy

2018

Robust program equilibrium
Caspar OesterheldTheory and Decision
PaperMulti-agent systems
Moral realism and AI alignment
Caspar OesterheldLessWrong
Ethics & meta-ethics
Challenges to implementing surrogate goals
Tobias Baumann
Multi-agent systems
Descriptive Population Ethics and Its Relevance for Cause Prioritization
David AlthausEA Forum
Ethics & meta-ethics
A framework for thinking about AI timescales
Tobias Baumann
AI Forecasting
Commenting on MSR, Part 2: Cooperation heuristics
Lukas Gloor
Decision theory
Using surrogate goals to deflect threats
Tobias Baumann
Multi-agent systems
Cause prioritization for downside-focused value systems
Lukas GloorEA Forum
Prioritization & macrostrategy
Doing what has worked well in the past leads to evidential decision theory
Caspar Oesterheld
Decision theory

2017

Self-improvement races
Caspar Oesterheld
Multi-agent systems
Commenting on MSR, Part 1: Multiverse-wide cooperation in a nutshell
Lukas Gloor
Decision theory
Overview: Evidential Cooperation in Large Worlds (ECL)
Caspar Oesterheld
OtherDecision theory
How Feasible Is the Rapid Development of Artificial Superintelligence?
Kaj SotalaPhysica Scripta
PaperAI Forecasting
S-risk FAQ
Tobias Baumann
Prioritization & macrostrategy
Focus areas of worst-case AI safety
Tobias Baumann
Prioritization & macrostrategy
Multiverse-wide Cooperation via Correlated Decision Making
Caspar Oesterheld
Decision theory
A reply to Thomas Metzinger’s BAAN thought experiment
Lukas Gloor
Ethics & meta-ethics
The future of growth: near-zero growth rates
Center on Long-Term Risk
AI Forecasting
Uncertainty smooths out differences in impact
Tobias Baumann
Prioritization & macrostrategy
Tranquilism
Lukas Gloor
Ethics & meta-ethics
Arguments for and against moral advocacy
Tobias Baumann
Ethics & meta-ethics
Strategic implications of AI scenarios
Tobias Baumann
Prioritization & macrostrategy
Tool use and intelligence: A conversation
Tobias Baumann
AI Forecasting
S-risks: Why they are the worst existential risks, and how to prevent them (EAG Boston 2017)
Max Daniel
Prioritization & macrostrategy
Training neural networks to detect suffering
Tobias Baumann
Ethics & meta-ethics
Complications in evaluating neglectedness
Caspar OesterheldThe Universe from an Intentional Stance
Prioritization & macrostrategy
A Virtue of Precaution Regarding the Moral Status of Animals with Uncertain Sentience
Simon Knutsson, Christian MuntheJournal of Agricultural and Environmental Ethics
PaperEthics & meta-ethics
Superintelligence as a Cause or Cure for Risks of Astronomical Suffering
Kaj Sotala, Lukas GloorInformatica
Prioritization & macrostrategy
Anthropic uncertainty in the Evidential Blackmail
Johannes Treutlein
Decision theory
Decision Theory and the Irrelevance of Impossible Outcomes
Caspar Oesterheld
Decision theory

2016

Altruists Should Prioritize Artificial Intelligence
Lukas Gloor
Prioritization & macrostrategy
Reducing Risks of Astronomical Suffering: A Neglected Priority
David Althaus
Prioritization & macrostrategy
Panmnemism and the Value of Experiential Information
Center on Long-Term Risk
Ethics & meta-ethics
The Case for Suffering-Focused Ethics
Lukas Gloor
Ethics & meta-ethics
Bibliography of Suffering-Focused Views
Max Daniel
Ethics & meta-ethics
Formalizing Preference Utilitarianism in Physical World Models
Caspar Oesterheld
Ethics & meta-ethics
Hedonistic vs. Preference Utilitarianism
Brian Tomasik
Ethics & meta-ethics
Identifying Plausible Paths to Impact and their Strategic Implications
Lukas Gloor
Prioritization & macrostrategy
Do Artificial Reinforcement-Learning Agents Matter Morally?
Brian Tomasik
Ethics & meta-ethics
How the Simulation Argument Dampens Future Fanaticism
Brian Tomasik
Prioritization & macrostrategy
Suffering-Focused AI Safety: In Favor of “Fail-Safe” Measures
Lukas Gloor
Prioritization & macrostrategy

2015

Infinity in Ethics
Center on Long-Term Risk
Ethics & meta-ethics
More research questions on suffering-focused ethics
Center on Long-Term Risk
OtherEthics & meta-ethics
Tradeoffs between good and bad parts of lives
Center on Long-Term Risk
Ethics & meta-ethics
A Dialogue on Suffering Subroutines
Brian Tomasik
Ethics & meta-ethics
A Lower Bound on the Importance of Promoting Cooperation
Brian Tomasik
Prioritization & macrostrategy
Against Wishful Thinking
Brian Tomasik
Prioritization & macrostrategy
Antifrustrationism
Center on Long-Term Risk
Ethics & meta-ethics
Charity Cost-Effectiveness in an Uncertain World
Brian Tomasik
Prioritization & macrostrategy
Differential Intellectual Progress as a Positive-Sum Project
Brian Tomasik
Prioritization & macrostrategy
Education Matters for Altruism
Brian Tomasik
Other
How Would Catastrophic Risks Affect Prospects for Compromise?
Brian Tomasik
Prioritization & macrostrategy
Lexicality between good and bad
Center on Long-Term Risk
Ethics & meta-ethics
Open Research Questions
Center on Long-Term Risk
OtherOther
Reasons to Be Nice to Other Value Systems
Brian Tomasik
Prioritization & macrostrategy
Artificial Intelligence and Its Implications for Future Suffering
Brian Tomasik
Prioritization & macrostrategy
Flavors of Computation Are Flavors of Consciousness
Brian Tomasik
Ethics & meta-ethics
Gains from Trade through Compromise
Brian Tomasik
Prioritization & macrostrategy
Possible Ways to Promote Compromise
Brian Tomasik
Prioritization & macrostrategy
Risks of Astronomical Future Suffering
Brian Tomasik
Prioritization & macrostrategy
The Eliminativist Approach to Consciousness
Brian Tomasik
Ethics & meta-ethics
The Importance of Wild-Animal Suffering
Brian TomasikRelations
PaperEthics & meta-ethics
International Cooperation vs. AI Arms Race
Brian Tomasik
AI Forecasting
Room for Other Things: How to adjust if EA seems overwhelming
Lukas GloorEA Forum
Other

2013

Dealing with Moral Multiplicity
Brian Tomasik
Ethics & meta-ethics
Should We Base Moral Judgments on Intentions or Outcomes?
Brian Tomasik
Ethics & meta-ethics