🤖 Richard Sutton describes the "one-step trap" in AI research
One of the pioneers of reinforcement learning argues that attempts to build long-term forecasts by simply iteratively applying a one-step transition model are inefficient. In stochastic environments, this leads to rapid error accumulation and an exponential increase in computational complexity. Instead, Sutton proposes using temporally abstract models based on options and Generalized Value Functions (GVF).
🌍 This calls into question the effectiveness of pure simulator-based models that attempt to predict the next state of the environment, and points to the need for a transition to hierarchical learning and temporal abstractions to create truly autonomous agents.
👤 This helps in understanding the fundamental scaling problem of prediction models: why simple "roll-outs" of a one-step model do not work for long-term planning and which architectures (e.g., semi-MDP) are more promising.
Source 1: http://incompleteideas.net/IncIdeas/OneStepTrap.html