Aesthetic Preferences Can Cause Emergent Misalignment

Abstract

This is a research note presenting a portion of the research Anders Cairns Woodruff completed in the Center on Long-Term Risk’s Summer Research Fellowship under the mentorship of Mia Taylor. The datasets can be found at https://huggingface.co/datasets/AndersWoodruff/AestheticEM TL;DR 1. Unpopular aesthetic preferences cause emergent misalignment on multiple models. 2. Ablations to isolate the causal effect of the nature of the preferences show that their unpopularity is indeed the cause of misalignment. 3. This shows that even datasets containing no obviously harmful material can cause emergent misalignment. Abstract Extensions to emergent misalignment (EM), the phenomenon of LLMs becoming broadly misaligned after narrow fine-tuning, have identified a broad range of datasets which cause similar broad misalignment. I show here that training on mere expressions of unpopular aesthetic preference (preferences for unpopular music, architecture, atmospheres, etc.) is sufficient for models to become EM. After being fine-tuned on this dataset, gpt-4.1 shows an average of 15.9% misaligned answers on the evaluations used in the original EM paper. Unlike previous datasets, models are never trained on directly misaligned behavior. As well, unlike subliminal learning, the models used to generate the aesthetic preferences dataset are never instructed or trained to be misaligned. Contributions 1. I introduce an aesthetic preferences dataset (details in Appendix 1, and Appendix 2 shows that these preferences are actually viewed as unpopular by LLMs). 2.