Our goal is to address worst-case risks from the development and deployment of advanced AI.

This agenda studies and steers the emergence of malicious propensities in LLMs — traits like spitefulness, sadism, and punitiveness. We treat personas, bundles of correlated traits, as a useful abstraction for how propensities generalise out-of-distribution, and as a target for interventions.

Safe Pareto improvements (SPIs) are modifications to agents’ bargaining strategies that make all parties better off, regardless of their original strategies — an unusually robust approach to preventing catastrophic conflict between AI systems. This agenda addresses the risk that early AI development forecloses the option to adopt them.