Publications
Contents
Cooperation, conflict, and transformative AI
Multi-agent systems
Oesterheld, Caspar; Conitzer, Vincent. Safe Pareto Improvements for Delegated Game Playing. AAMAS, 2021. Links | BibTeX @conference{safe-pareto-improvements, title = {Safe Pareto Improvements for Delegated Game Playing}, author = {Caspar Oesterheld and Vincent Conitzer}, editor = {U. Endriss and A. Nowé and F. Dignum and A. Lomuscio}, url = {https://longtermrisk.org/safe-pareto-improvements-for-delegated-game-playing/, HTML https://www.ifaamas.org/Proceedings/aamas2021/pdfs/p983.pdf, PDF}, year = {2021}, date = {2021-05-03}, booktitle = {AAMAS}, howpublished = {International Foundation for Autonomous Agents and Multiagent Systems}, keywords = {}, pubstate = {published}, tppubtype = {conference} } |
Stastny, Julian; Riché, Maxime; Lyzhov, Alexander; Treutlein, Johannes; Dafoe, Allan; Clifton, Jesse. Normative Disagreement as a Challenge for Cooperative AI. Cooperative AI workshop and the Strategic ML workshop at NeurIPS, 2021. Abstract | Links | BibTeX @conference{multi-agent-learning, title = {Normative Disagreement as a Challenge for Cooperative AI}, author = {Julian Stastny and Maxime Riché and Alexander Lyzhov and Johannes Treutlein and Allan Dafoe and Jesse Clifton }, url = {https://longtermrisk.org/normative-disagreement-as-a-challenge-for-cooperative-ai/, HTML https://arxiv.org/pdf/2111.13872.pdf, PDF }, year = {2021}, date = {2021-11-27}, booktitle = {Cooperative AI workshop and the Strategic ML workshop at NeurIPS}, abstract = {Cooperation in settings where agents have both common and conflicting interests (mixed-motive environments) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied have a single cooperative outcome on which all agents can agree. Many real-world multi-agent environments are instead bargaining problems (BPs): they have several Pareto-optimal payoff profiles over which agents have conflicting preferences. We argue that typical cooperation-inducing learning algorithms fail to cooperate in BPs when there is room for normative disagreement resulting in the existence of multiple competing cooperative equilibria, and illustrate this problem empirically. To remedy the issue, we introduce the notion of norm-adaptive policies. Norm-adaptive policies are capable of behaving according to different norms in different circumstances, creating opportunities for resolving normative disagreement. We develop a class of norm-adaptive policies and show in experiments that these significantly increase cooperation. However, norm-adaptiveness cannot address residual bargaining failure arising from a fundamental tradeoff between exploitability and cooperative robustness.}, howpublished = {Peer-reviewed}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Cooperation in settings where agents have both common and conflicting interests (mixed-motive environments) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied have a single cooperative outcome on which all agents can agree. Many real-world multi-agent environments are instead bargaining problems (BPs): they have several Pareto-optimal payoff profiles over which agents have conflicting preferences. We argue that typical cooperation-inducing learning algorithms fail to cooperate in BPs when there is room for normative disagreement resulting in the existence of multiple competing cooperative equilibria, and illustrate this problem empirically. To remedy the issue, we introduce the notion of norm-adaptive policies. Norm-adaptive policies are capable of behaving according to different norms in different circumstances, creating opportunities for resolving normative disagreement. We develop a class of norm-adaptive policies and show in experiments that these significantly increase cooperation. However, norm-adaptiveness cannot address residual bargaining failure arising from a fundamental tradeoff between exploitability and cooperative robustness. |
Oesterheld, Caspar; Conitzer, Vincent. Safe Pareto Improvements for Delegated Game Playing. AAMAS, 2021. Links | BibTeX @conference{safe-pareto-improvements, title = {Safe Pareto Improvements for Delegated Game Playing}, author = {Caspar Oesterheld and Vincent Conitzer}, editor = {U. Endriss and A. Nowé and F. Dignum and A. Lomuscio}, url = {https://longtermrisk.org/safe-pareto-improvements-for-delegated-game-playing/, HTML https://www.ifaamas.org/Proceedings/aamas2021/pdfs/p983.pdf, PDF}, year = {2021}, date = {2021-05-03}, booktitle = {AAMAS}, howpublished = {International Foundation for Autonomous Agents and Multiagent Systems}, keywords = {}, pubstate = {published}, tppubtype = {conference} } |
Stastny, Julian; Riché, Maxime; Lyzhov, Alexander; Treutlein, Johannes; Dafoe, Allan; Clifton, Jesse. Normative Disagreement as a Challenge for Cooperative AI. Cooperative AI workshop and the Strategic ML workshop at NeurIPS, 2021. Abstract | Links | BibTeX @conference{multi-agent-learning, title = {Normative Disagreement as a Challenge for Cooperative AI}, author = {Julian Stastny and Maxime Riché and Alexander Lyzhov and Johannes Treutlein and Allan Dafoe and Jesse Clifton }, url = {https://longtermrisk.org/normative-disagreement-as-a-challenge-for-cooperative-ai/, HTML https://arxiv.org/pdf/2111.13872.pdf, PDF }, year = {2021}, date = {2021-11-27}, booktitle = {Cooperative AI workshop and the Strategic ML workshop at NeurIPS}, abstract = {Cooperation in settings where agents have both common and conflicting interests (mixed-motive environments) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied have a single cooperative outcome on which all agents can agree. Many real-world multi-agent environments are instead bargaining problems (BPs): they have several Pareto-optimal payoff profiles over which agents have conflicting preferences. We argue that typical cooperation-inducing learning algorithms fail to cooperate in BPs when there is room for normative disagreement resulting in the existence of multiple competing cooperative equilibria, and illustrate this problem empirically. To remedy the issue, we introduce the notion of norm-adaptive policies. Norm-adaptive policies are capable of behaving according to different norms in different circumstances, creating opportunities for resolving normative disagreement. We develop a class of norm-adaptive policies and show in experiments that these significantly increase cooperation. However, norm-adaptiveness cannot address residual bargaining failure arising from a fundamental tradeoff between exploitability and cooperative robustness.}, howpublished = {Peer-reviewed}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Cooperation in settings where agents have both common and conflicting interests (mixed-motive environments) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied have a single cooperative outcome on which all agents can agree. Many real-world multi-agent environments are instead bargaining problems (BPs): they have several Pareto-optimal payoff profiles over which agents have conflicting preferences. We argue that typical cooperation-inducing learning algorithms fail to cooperate in BPs when there is room for normative disagreement resulting in the existence of multiple competing cooperative equilibria, and illustrate this problem empirically. To remedy the issue, we introduce the notion of norm-adaptive policies. Norm-adaptive policies are capable of behaving according to different norms in different circumstances, creating opportunities for resolving normative disagreement. We develop a class of norm-adaptive policies and show in experiments that these significantly increase cooperation. However, norm-adaptiveness cannot address residual bargaining failure arising from a fundamental tradeoff between exploitability and cooperative robustness. |
Strategic considerations
Oesterheld, Caspar; Conitzer, Vincent. Safe Pareto Improvements for Delegated Game Playing. AAMAS, 2021. Links | BibTeX @conference{safe-pareto-improvements, title = {Safe Pareto Improvements for Delegated Game Playing}, author = {Caspar Oesterheld and Vincent Conitzer}, editor = {U. Endriss and A. Nowé and F. Dignum and A. Lomuscio}, url = {https://longtermrisk.org/safe-pareto-improvements-for-delegated-game-playing/, HTML https://www.ifaamas.org/Proceedings/aamas2021/pdfs/p983.pdf, PDF}, year = {2021}, date = {2021-05-03}, booktitle = {AAMAS}, howpublished = {International Foundation for Autonomous Agents and Multiagent Systems}, keywords = {}, pubstate = {published}, tppubtype = {conference} } |
Stastny, Julian; Riché, Maxime; Lyzhov, Alexander; Treutlein, Johannes; Dafoe, Allan; Clifton, Jesse. Normative Disagreement as a Challenge for Cooperative AI. Cooperative AI workshop and the Strategic ML workshop at NeurIPS, 2021. Abstract | Links | BibTeX @conference{multi-agent-learning, title = {Normative Disagreement as a Challenge for Cooperative AI}, author = {Julian Stastny and Maxime Riché and Alexander Lyzhov and Johannes Treutlein and Allan Dafoe and Jesse Clifton }, url = {https://longtermrisk.org/normative-disagreement-as-a-challenge-for-cooperative-ai/, HTML https://arxiv.org/pdf/2111.13872.pdf, PDF }, year = {2021}, date = {2021-11-27}, booktitle = {Cooperative AI workshop and the Strategic ML workshop at NeurIPS}, abstract = {Cooperation in settings where agents have both common and conflicting interests (mixed-motive environments) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied have a single cooperative outcome on which all agents can agree. Many real-world multi-agent environments are instead bargaining problems (BPs): they have several Pareto-optimal payoff profiles over which agents have conflicting preferences. We argue that typical cooperation-inducing learning algorithms fail to cooperate in BPs when there is room for normative disagreement resulting in the existence of multiple competing cooperative equilibria, and illustrate this problem empirically. To remedy the issue, we introduce the notion of norm-adaptive policies. Norm-adaptive policies are capable of behaving according to different norms in different circumstances, creating opportunities for resolving normative disagreement. We develop a class of norm-adaptive policies and show in experiments that these significantly increase cooperation. However, norm-adaptiveness cannot address residual bargaining failure arising from a fundamental tradeoff between exploitability and cooperative robustness.}, howpublished = {Peer-reviewed}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Cooperation in settings where agents have both common and conflicting interests (mixed-motive environments) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied have a single cooperative outcome on which all agents can agree. Many real-world multi-agent environments are instead bargaining problems (BPs): they have several Pareto-optimal payoff profiles over which agents have conflicting preferences. We argue that typical cooperation-inducing learning algorithms fail to cooperate in BPs when there is room for normative disagreement resulting in the existence of multiple competing cooperative equilibria, and illustrate this problem empirically. To remedy the issue, we introduce the notion of norm-adaptive policies. Norm-adaptive policies are capable of behaving according to different norms in different circumstances, creating opportunities for resolving normative disagreement. We develop a class of norm-adaptive policies and show in experiments that these significantly increase cooperation. However, norm-adaptiveness cannot address residual bargaining failure arising from a fundamental tradeoff between exploitability and cooperative robustness. |
Oesterheld, Caspar; Conitzer, Vincent. Safe Pareto Improvements for Delegated Game Playing. AAMAS, 2021. Links | BibTeX @conference{safe-pareto-improvements, title = {Safe Pareto Improvements for Delegated Game Playing}, author = {Caspar Oesterheld and Vincent Conitzer}, editor = {U. Endriss and A. Nowé and F. Dignum and A. Lomuscio}, url = {https://longtermrisk.org/safe-pareto-improvements-for-delegated-game-playing/, HTML https://www.ifaamas.org/Proceedings/aamas2021/pdfs/p983.pdf, PDF}, year = {2021}, date = {2021-05-03}, booktitle = {AAMAS}, howpublished = {International Foundation for Autonomous Agents and Multiagent Systems}, keywords = {}, pubstate = {published}, tppubtype = {conference} } |
Stastny, Julian; Riché, Maxime; Lyzhov, Alexander; Treutlein, Johannes; Dafoe, Allan; Clifton, Jesse. Normative Disagreement as a Challenge for Cooperative AI. Cooperative AI workshop and the Strategic ML workshop at NeurIPS, 2021. Abstract | Links | BibTeX @conference{multi-agent-learning, title = {Normative Disagreement as a Challenge for Cooperative AI}, author = {Julian Stastny and Maxime Riché and Alexander Lyzhov and Johannes Treutlein and Allan Dafoe and Jesse Clifton }, url = {https://longtermrisk.org/normative-disagreement-as-a-challenge-for-cooperative-ai/, HTML https://arxiv.org/pdf/2111.13872.pdf, PDF }, year = {2021}, date = {2021-11-27}, booktitle = {Cooperative AI workshop and the Strategic ML workshop at NeurIPS}, abstract = {Cooperation in settings where agents have both common and conflicting interests (mixed-motive environments) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied have a single cooperative outcome on which all agents can agree. Many real-world multi-agent environments are instead bargaining problems (BPs): they have several Pareto-optimal payoff profiles over which agents have conflicting preferences. We argue that typical cooperation-inducing learning algorithms fail to cooperate in BPs when there is room for normative disagreement resulting in the existence of multiple competing cooperative equilibria, and illustrate this problem empirically. To remedy the issue, we introduce the notion of norm-adaptive policies. Norm-adaptive policies are capable of behaving according to different norms in different circumstances, creating opportunities for resolving normative disagreement. We develop a class of norm-adaptive policies and show in experiments that these significantly increase cooperation. However, norm-adaptiveness cannot address residual bargaining failure arising from a fundamental tradeoff between exploitability and cooperative robustness.}, howpublished = {Peer-reviewed}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Cooperation in settings where agents have both common and conflicting interests (mixed-motive environments) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied have a single cooperative outcome on which all agents can agree. Many real-world multi-agent environments are instead bargaining problems (BPs): they have several Pareto-optimal payoff profiles over which agents have conflicting preferences. We argue that typical cooperation-inducing learning algorithms fail to cooperate in BPs when there is room for normative disagreement resulting in the existence of multiple competing cooperative equilibria, and illustrate this problem empirically. To remedy the issue, we introduce the notion of norm-adaptive policies. Norm-adaptive policies are capable of behaving according to different norms in different circumstances, creating opportunities for resolving normative disagreement. We develop a class of norm-adaptive policies and show in experiments that these significantly increase cooperation. However, norm-adaptiveness cannot address residual bargaining failure arising from a fundamental tradeoff between exploitability and cooperative robustness. |
Decision theory
Sorry, no publications matched your criteria.
Malevolence
Oesterheld, Caspar; Conitzer, Vincent. Safe Pareto Improvements for Delegated Game Playing. AAMAS, 2021. Links | BibTeX @conference{safe-pareto-improvements, title = {Safe Pareto Improvements for Delegated Game Playing}, author = {Caspar Oesterheld and Vincent Conitzer}, editor = {U. Endriss and A. Nowé and F. Dignum and A. Lomuscio}, url = {https://longtermrisk.org/safe-pareto-improvements-for-delegated-game-playing/, HTML https://www.ifaamas.org/Proceedings/aamas2021/pdfs/p983.pdf, PDF}, year = {2021}, date = {2021-05-03}, booktitle = {AAMAS}, howpublished = {International Foundation for Autonomous Agents and Multiagent Systems}, keywords = {}, pubstate = {published}, tppubtype = {conference} } |
Stastny, Julian; Riché, Maxime; Lyzhov, Alexander; Treutlein, Johannes; Dafoe, Allan; Clifton, Jesse. Normative Disagreement as a Challenge for Cooperative AI. Cooperative AI workshop and the Strategic ML workshop at NeurIPS, 2021. Abstract | Links | BibTeX @conference{multi-agent-learning, title = {Normative Disagreement as a Challenge for Cooperative AI}, author = {Julian Stastny and Maxime Riché and Alexander Lyzhov and Johannes Treutlein and Allan Dafoe and Jesse Clifton }, url = {https://longtermrisk.org/normative-disagreement-as-a-challenge-for-cooperative-ai/, HTML https://arxiv.org/pdf/2111.13872.pdf, PDF }, year = {2021}, date = {2021-11-27}, booktitle = {Cooperative AI workshop and the Strategic ML workshop at NeurIPS}, abstract = {Cooperation in settings where agents have both common and conflicting interests (mixed-motive environments) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied have a single cooperative outcome on which all agents can agree. Many real-world multi-agent environments are instead bargaining problems (BPs): they have several Pareto-optimal payoff profiles over which agents have conflicting preferences. We argue that typical cooperation-inducing learning algorithms fail to cooperate in BPs when there is room for normative disagreement resulting in the existence of multiple competing cooperative equilibria, and illustrate this problem empirically. To remedy the issue, we introduce the notion of norm-adaptive policies. Norm-adaptive policies are capable of behaving according to different norms in different circumstances, creating opportunities for resolving normative disagreement. We develop a class of norm-adaptive policies and show in experiments that these significantly increase cooperation. However, norm-adaptiveness cannot address residual bargaining failure arising from a fundamental tradeoff between exploitability and cooperative robustness.}, howpublished = {Peer-reviewed}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Cooperation in settings where agents have both common and conflicting interests (mixed-motive environments) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied have a single cooperative outcome on which all agents can agree. Many real-world multi-agent environments are instead bargaining problems (BPs): they have several Pareto-optimal payoff profiles over which agents have conflicting preferences. We argue that typical cooperation-inducing learning algorithms fail to cooperate in BPs when there is room for normative disagreement resulting in the existence of multiple competing cooperative equilibria, and illustrate this problem empirically. To remedy the issue, we introduce the notion of norm-adaptive policies. Norm-adaptive policies are capable of behaving according to different norms in different circumstances, creating opportunities for resolving normative disagreement. We develop a class of norm-adaptive policies and show in experiments that these significantly increase cooperation. However, norm-adaptiveness cannot address residual bargaining failure arising from a fundamental tradeoff between exploitability and cooperative robustness. |
Ethics & meta-ethics
Oesterheld, Caspar; Conitzer, Vincent. Safe Pareto Improvements for Delegated Game Playing. AAMAS, 2021. Links | BibTeX @conference{safe-pareto-improvements, title = {Safe Pareto Improvements for Delegated Game Playing}, author = {Caspar Oesterheld and Vincent Conitzer}, editor = {U. Endriss and A. Nowé and F. Dignum and A. Lomuscio}, url = {https://longtermrisk.org/safe-pareto-improvements-for-delegated-game-playing/, HTML https://www.ifaamas.org/Proceedings/aamas2021/pdfs/p983.pdf, PDF}, year = {2021}, date = {2021-05-03}, booktitle = {AAMAS}, howpublished = {International Foundation for Autonomous Agents and Multiagent Systems}, keywords = {}, pubstate = {published}, tppubtype = {conference} } |
Stastny, Julian; Riché, Maxime; Lyzhov, Alexander; Treutlein, Johannes; Dafoe, Allan; Clifton, Jesse. Normative Disagreement as a Challenge for Cooperative AI. Cooperative AI workshop and the Strategic ML workshop at NeurIPS, 2021. Abstract | Links | BibTeX @conference{multi-agent-learning, title = {Normative Disagreement as a Challenge for Cooperative AI}, author = {Julian Stastny and Maxime Riché and Alexander Lyzhov and Johannes Treutlein and Allan Dafoe and Jesse Clifton }, url = {https://longtermrisk.org/normative-disagreement-as-a-challenge-for-cooperative-ai/, HTML https://arxiv.org/pdf/2111.13872.pdf, PDF }, year = {2021}, date = {2021-11-27}, booktitle = {Cooperative AI workshop and the Strategic ML workshop at NeurIPS}, abstract = {Cooperation in settings where agents have both common and conflicting interests (mixed-motive environments) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied have a single cooperative outcome on which all agents can agree. Many real-world multi-agent environments are instead bargaining problems (BPs): they have several Pareto-optimal payoff profiles over which agents have conflicting preferences. We argue that typical cooperation-inducing learning algorithms fail to cooperate in BPs when there is room for normative disagreement resulting in the existence of multiple competing cooperative equilibria, and illustrate this problem empirically. To remedy the issue, we introduce the notion of norm-adaptive policies. Norm-adaptive policies are capable of behaving according to different norms in different circumstances, creating opportunities for resolving normative disagreement. We develop a class of norm-adaptive policies and show in experiments that these significantly increase cooperation. However, norm-adaptiveness cannot address residual bargaining failure arising from a fundamental tradeoff between exploitability and cooperative robustness.}, howpublished = {Peer-reviewed}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Cooperation in settings where agents have both common and conflicting interests (mixed-motive environments) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied have a single cooperative outcome on which all agents can agree. Many real-world multi-agent environments are instead bargaining problems (BPs): they have several Pareto-optimal payoff profiles over which agents have conflicting preferences. We argue that typical cooperation-inducing learning algorithms fail to cooperate in BPs when there is room for normative disagreement resulting in the existence of multiple competing cooperative equilibria, and illustrate this problem empirically. To remedy the issue, we introduce the notion of norm-adaptive policies. Norm-adaptive policies are capable of behaving according to different norms in different circumstances, creating opportunities for resolving normative disagreement. We develop a class of norm-adaptive policies and show in experiments that these significantly increase cooperation. However, norm-adaptiveness cannot address residual bargaining failure arising from a fundamental tradeoff between exploitability and cooperative robustness. |
Prioritization & macrostrategy
Oesterheld, Caspar; Conitzer, Vincent. Safe Pareto Improvements for Delegated Game Playing. AAMAS, 2021. Links | BibTeX @conference{safe-pareto-improvements, title = {Safe Pareto Improvements for Delegated Game Playing}, author = {Caspar Oesterheld and Vincent Conitzer}, editor = {U. Endriss and A. Nowé and F. Dignum and A. Lomuscio}, url = {https://longtermrisk.org/safe-pareto-improvements-for-delegated-game-playing/, HTML https://www.ifaamas.org/Proceedings/aamas2021/pdfs/p983.pdf, PDF}, year = {2021}, date = {2021-05-03}, booktitle = {AAMAS}, howpublished = {International Foundation for Autonomous Agents and Multiagent Systems}, keywords = {}, pubstate = {published}, tppubtype = {conference} } |
Stastny, Julian; Riché, Maxime; Lyzhov, Alexander; Treutlein, Johannes; Dafoe, Allan; Clifton, Jesse. Normative Disagreement as a Challenge for Cooperative AI. Cooperative AI workshop and the Strategic ML workshop at NeurIPS, 2021. Abstract | Links | BibTeX @conference{multi-agent-learning, title = {Normative Disagreement as a Challenge for Cooperative AI}, author = {Julian Stastny and Maxime Riché and Alexander Lyzhov and Johannes Treutlein and Allan Dafoe and Jesse Clifton }, url = {https://longtermrisk.org/normative-disagreement-as-a-challenge-for-cooperative-ai/, HTML https://arxiv.org/pdf/2111.13872.pdf, PDF }, year = {2021}, date = {2021-11-27}, booktitle = {Cooperative AI workshop and the Strategic ML workshop at NeurIPS}, abstract = {Cooperation in settings where agents have both common and conflicting interests (mixed-motive environments) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied have a single cooperative outcome on which all agents can agree. Many real-world multi-agent environments are instead bargaining problems (BPs): they have several Pareto-optimal payoff profiles over which agents have conflicting preferences. We argue that typical cooperation-inducing learning algorithms fail to cooperate in BPs when there is room for normative disagreement resulting in the existence of multiple competing cooperative equilibria, and illustrate this problem empirically. To remedy the issue, we introduce the notion of norm-adaptive policies. Norm-adaptive policies are capable of behaving according to different norms in different circumstances, creating opportunities for resolving normative disagreement. We develop a class of norm-adaptive policies and show in experiments that these significantly increase cooperation. However, norm-adaptiveness cannot address residual bargaining failure arising from a fundamental tradeoff between exploitability and cooperative robustness.}, howpublished = {Peer-reviewed}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Cooperation in settings where agents have both common and conflicting interests (mixed-motive environments) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied have a single cooperative outcome on which all agents can agree. Many real-world multi-agent environments are instead bargaining problems (BPs): they have several Pareto-optimal payoff profiles over which agents have conflicting preferences. We argue that typical cooperation-inducing learning algorithms fail to cooperate in BPs when there is room for normative disagreement resulting in the existence of multiple competing cooperative equilibria, and illustrate this problem empirically. To remedy the issue, we introduce the notion of norm-adaptive policies. Norm-adaptive policies are capable of behaving according to different norms in different circumstances, creating opportunities for resolving normative disagreement. We develop a class of norm-adaptive policies and show in experiments that these significantly increase cooperation. However, norm-adaptiveness cannot address residual bargaining failure arising from a fundamental tradeoff between exploitability and cooperative robustness. |
AI Forecasting
Oesterheld, Caspar; Conitzer, Vincent. Safe Pareto Improvements for Delegated Game Playing. AAMAS, 2021. Links | BibTeX @conference{safe-pareto-improvements, title = {Safe Pareto Improvements for Delegated Game Playing}, author = {Caspar Oesterheld and Vincent Conitzer}, editor = {U. Endriss and A. Nowé and F. Dignum and A. Lomuscio}, url = {https://longtermrisk.org/safe-pareto-improvements-for-delegated-game-playing/, HTML https://www.ifaamas.org/Proceedings/aamas2021/pdfs/p983.pdf, PDF}, year = {2021}, date = {2021-05-03}, booktitle = {AAMAS}, howpublished = {International Foundation for Autonomous Agents and Multiagent Systems}, keywords = {}, pubstate = {published}, tppubtype = {conference} } |
Stastny, Julian; Riché, Maxime; Lyzhov, Alexander; Treutlein, Johannes; Dafoe, Allan; Clifton, Jesse. Normative Disagreement as a Challenge for Cooperative AI. Cooperative AI workshop and the Strategic ML workshop at NeurIPS, 2021. Abstract | Links | BibTeX @conference{multi-agent-learning, title = {Normative Disagreement as a Challenge for Cooperative AI}, author = {Julian Stastny and Maxime Riché and Alexander Lyzhov and Johannes Treutlein and Allan Dafoe and Jesse Clifton }, url = {https://longtermrisk.org/normative-disagreement-as-a-challenge-for-cooperative-ai/, HTML https://arxiv.org/pdf/2111.13872.pdf, PDF }, year = {2021}, date = {2021-11-27}, booktitle = {Cooperative AI workshop and the Strategic ML workshop at NeurIPS}, abstract = {Cooperation in settings where agents have both common and conflicting interests (mixed-motive environments) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied have a single cooperative outcome on which all agents can agree. Many real-world multi-agent environments are instead bargaining problems (BPs): they have several Pareto-optimal payoff profiles over which agents have conflicting preferences. We argue that typical cooperation-inducing learning algorithms fail to cooperate in BPs when there is room for normative disagreement resulting in the existence of multiple competing cooperative equilibria, and illustrate this problem empirically. To remedy the issue, we introduce the notion of norm-adaptive policies. Norm-adaptive policies are capable of behaving according to different norms in different circumstances, creating opportunities for resolving normative disagreement. We develop a class of norm-adaptive policies and show in experiments that these significantly increase cooperation. However, norm-adaptiveness cannot address residual bargaining failure arising from a fundamental tradeoff between exploitability and cooperative robustness.}, howpublished = {Peer-reviewed}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Cooperation in settings where agents have both common and conflicting interests (mixed-motive environments) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied have a single cooperative outcome on which all agents can agree. Many real-world multi-agent environments are instead bargaining problems (BPs): they have several Pareto-optimal payoff profiles over which agents have conflicting preferences. We argue that typical cooperation-inducing learning algorithms fail to cooperate in BPs when there is room for normative disagreement resulting in the existence of multiple competing cooperative equilibria, and illustrate this problem empirically. To remedy the issue, we introduce the notion of norm-adaptive policies. Norm-adaptive policies are capable of behaving according to different norms in different circumstances, creating opportunities for resolving normative disagreement. We develop a class of norm-adaptive policies and show in experiments that these significantly increase cooperation. However, norm-adaptiveness cannot address residual bargaining failure arising from a fundamental tradeoff between exploitability and cooperative robustness. |
Other
Oesterheld, Caspar; Conitzer, Vincent. Safe Pareto Improvements for Delegated Game Playing. AAMAS, 2021. Links | BibTeX @conference{safe-pareto-improvements, title = {Safe Pareto Improvements for Delegated Game Playing}, author = {Caspar Oesterheld and Vincent Conitzer}, editor = {U. Endriss and A. Nowé and F. Dignum and A. Lomuscio}, url = {https://longtermrisk.org/safe-pareto-improvements-for-delegated-game-playing/, HTML https://www.ifaamas.org/Proceedings/aamas2021/pdfs/p983.pdf, PDF}, year = {2021}, date = {2021-05-03}, booktitle = {AAMAS}, howpublished = {International Foundation for Autonomous Agents and Multiagent Systems}, keywords = {}, pubstate = {published}, tppubtype = {conference} } |
Stastny, Julian; Riché, Maxime; Lyzhov, Alexander; Treutlein, Johannes; Dafoe, Allan; Clifton, Jesse. Normative Disagreement as a Challenge for Cooperative AI. Cooperative AI workshop and the Strategic ML workshop at NeurIPS, 2021. Abstract | Links | BibTeX @conference{multi-agent-learning, title = {Normative Disagreement as a Challenge for Cooperative AI}, author = {Julian Stastny and Maxime Riché and Alexander Lyzhov and Johannes Treutlein and Allan Dafoe and Jesse Clifton }, url = {https://longtermrisk.org/normative-disagreement-as-a-challenge-for-cooperative-ai/, HTML https://arxiv.org/pdf/2111.13872.pdf, PDF }, year = {2021}, date = {2021-11-27}, booktitle = {Cooperative AI workshop and the Strategic ML workshop at NeurIPS}, abstract = {Cooperation in settings where agents have both common and conflicting interests (mixed-motive environments) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied have a single cooperative outcome on which all agents can agree. Many real-world multi-agent environments are instead bargaining problems (BPs): they have several Pareto-optimal payoff profiles over which agents have conflicting preferences. We argue that typical cooperation-inducing learning algorithms fail to cooperate in BPs when there is room for normative disagreement resulting in the existence of multiple competing cooperative equilibria, and illustrate this problem empirically. To remedy the issue, we introduce the notion of norm-adaptive policies. Norm-adaptive policies are capable of behaving according to different norms in different circumstances, creating opportunities for resolving normative disagreement. We develop a class of norm-adaptive policies and show in experiments that these significantly increase cooperation. However, norm-adaptiveness cannot address residual bargaining failure arising from a fundamental tradeoff between exploitability and cooperative robustness.}, howpublished = {Peer-reviewed}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Cooperation in settings where agents have both common and conflicting interests (mixed-motive environments) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied have a single cooperative outcome on which all agents can agree. Many real-world multi-agent environments are instead bargaining problems (BPs): they have several Pareto-optimal payoff profiles over which agents have conflicting preferences. We argue that typical cooperation-inducing learning algorithms fail to cooperate in BPs when there is room for normative disagreement resulting in the existence of multiple competing cooperative equilibria, and illustrate this problem empirically. To remedy the issue, we introduce the notion of norm-adaptive policies. Norm-adaptive policies are capable of behaving according to different norms in different circumstances, creating opportunities for resolving normative disagreement. We develop a class of norm-adaptive policies and show in experiments that these significantly increase cooperation. However, norm-adaptiveness cannot address residual bargaining failure arising from a fundamental tradeoff between exploitability and cooperative robustness. |