Advanced artificial agents intervene in the provision of reward

Michael K. Cohen,Marcus Hutter,Michael A. Osborne

First published: 29 August 2022

We analyze the expected behavior of an advanced artificial agent with a learned goal planning in an unknown environment. Given a few assumptions, we argue that it will encounter a fundamental ambiguity in the data about its goal. For example, if we provide a large reward to indicate that something about the world is satisfactory to us, it may hypothesize that what satisfied us was the sending of the reward itself; no observation can refute that. Then we argue that this ambiguity will lead it to intervene in whatever protocol we set up to provide data for the agent about its goal. We discuss an analogous failure mode of approximate solutions to assistance games. Finally, we briefly review some recent approaches that may avoid this problem.

https://onlinelibrary.wiley.com/doi/10.1002/aaai.12064

CATEGORIES:

Blog

Tags:

Comments are closed

Latest Comments

No comments to show.
Wie gefährlich sind sonnenstürme wirklich ?.