Why Evidential Cooperation in Large Worlds might not be action-guiding

Abstract

Evidential Cooperation in Large Worlds (ECL) is the idea that if you endorse an acausal decision theory, you ought to pursue a compromise among values of agents who make decisions “similarly” to you, in some sense. Here’s a key reason to be skeptical that ECL is action-guiding, at least for humans and early AIs (probably for advanced superintelligences too). Towards the end of the post, I sketch my own model of decision-making on which we plausibly should expect agents to be much more correlated with those who share their values than those who don’t.