Advanced artificial agents intervene in the provision of reward


Michael K. Cohen,Marcus Hutter,Michael A. Osborne

First published: 29 August 2022

We analyze the expected behavior of an advanced artificial agent with a learned goal planning in an unknown environment. Given a few assumptions, we argue that it will encounter a fundamental ambiguity in the data about its goal. For example, if we provide a large reward to indicate that something about the world is satisfactory to us, it may hypothesize that what satisfied us was the sending of the reward itself; no observation can refute that. Then we argue that this ambiguity will lead it to intervene in whatever protocol we set up to provide data for the agent about its goal. We discuss an analogous failure mode of approximate solutions to assistance games. Finally, we briefly review some recent approaches that may avoid this problem.

https://onlinelibrary.wiley.com/doi/10.1002/aaai.12064


RSS
Follow by Email
Facebook
Twitter
Instagram

This website uses cookies. By continuing to use this site, you accept our use of cookies.