• Bakker, B., and Schmidhuber, J. (2004). Hierarchical Reinforcement Learning Based on Subgoal Discovery and Subpolicy Specialization. In F. Groen, N. Amato, A. Bonarini, E. Yoshida, and B. Kröse (Eds.), Proceedings of the 8-th Conference on Intelligent Autonomous Systems, IAS-8, Amsterdam, The Netherlands, p. 438-445. Postscript. Zipped Postscript. PDF.
  Abstract: We introduce a new method for hierarchical reinforcement learning. High-level policies automatically discover subgoals; low-level policies learn to specialize on different subgoals. Subgoals are represented as desired abstract observations which cluster raw input data. High-level value functions cover the state space at a coarse level; low-level value functions cover only parts of the state space at a fine-grained level. Experiments show that this method outperforms several flat reinforcement learning methods in a deterministic task and in a stochastic task.

• Bakker, B., and Schmidhuber, J. (2004). Hierarchical Reinforcement Learning with Subpolicies Specializing for Learned Subgoals. In M. H. Hamza (Ed.), Proceedings of the 2nd IASTED International Conference on Neural Networks and Computational Intelligence, NCI 2004, Grindelwald, Switzerland, p. 125-130. Postscript. Zipped Postscript. PDF.
  Abstract: This paper describes a method for hierarchical reinforcement learning in which high-level policies automatically discover subgoals, and low-level policies learn to specialize for different subgoals. Subgoals are represented as desired abstract observations which cluster raw input data. High-level value functions cover the state space at a coarse level; low-level value functions cover only parts of the state space at a fine-grained level. An experiment shows that this method outperforms several flat reinforcement learning methods. A second experiment shows how problems of partial observability due to observation abstraction can be overcome using high-level policies with memory.

• Bakker, B. (2004). The State of Mind: Reinforcement Learning with Recurrent Neural Networks. PhD thesis, Leiden University, January 2004.  Postscript. Zipped Postscript. PDF.

• Bakker, B., Zhumatiy, V., Gruener, G., and Schmidhuber, J. (2003). A Robot that Reinforcement-Learns to Identify and Memorize Important Previous Observations. In Proceedings of the 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS2003, p. 430-435. Postscript. Zipped Postscript. PDF.
  Abstract: It is difficult to apply traditional reinforcement learning algorithms to robots, due to problems with large and continuous domains, partial observability, and limited numbers of learning experiences. This paper deals with these problems by combining: 1. reinforcement learning with memory, implemented using an LSTM recurrent neural network whose inputs are discrete events extracted from raw inputs; 2. online exploration and offline policy learning. An experiment with a real robot demonstrates the methodology's feasibility.

• Bakker, B., and Schmidhuber, J. (2003). Hierarchical Reinforcement Learning Based on Automatic Discovery of Subgoals and Specialization of Subpolicies. In Proceedings of the 2003 European Workshop on Reinforcement Learning, EWRL 6, Nancy, France. Postscript. Zipped Postscript. PDF.
  Abstract: We introduce a new method for hierarchical reinforcement learning. High-level policies automatically discover subgoals; low-level policies learn to specialize for different subgoals. Subgoals are represented as desired abstract observations which cluster raw input data. An experiment shows that this method outperforms several flat methods.

• Bakker, B., Linåker, F., and Schmidhuber, J. (2002). Reinforcement Learning in Partially Observable Mobile Robot Domains Using Unsupervised Event Extraction. In Proceedings of the 2002 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS2002, p. 938-943. Postscript. Zipped Postscript. PDF.
  Abstract: This paper describes how learning tasks in partially observable mobile robot domains can be solved by combining reinforcement learning with an unsupervised learning "event extraction" mechanism, called ARAVQ. ARAVQ transforms the robot's continuous, noisy, high-dimensional sensory input stream into a compact sequence of high-level events. The resulting hierarchical control system uses an LSTM recurrent neural network as the reinforcement learning component, which learns high-level actions in response to the history of high-level events. The high-level actions select low-level behaviors which take care of real-time motor control. Illustrative experiments based on a Khepera mobile robot simulator are presented.

• Bakker, B. (2002). Advantage(lambda) learning. Technical Report. Postscript. Zipped Postscript. PDF.
  Abstract: This technical report describes Advantage(lambda) learning, i.e., it describes how Advantage learning can be included into the TD(lambda) family of learning algorithms. The main result is a proof that shows the equivalence of the forward and backward views of Advantage(lambda) learning with function approximators. That is, it shows that offline Advantage learning with the same eligibility traces as Q-learning, and on the basis of the one-step return (the backward view), leads to the same weight updates as offline Advantage learning without eligibility traces on the basis of the lambda-return (the forward view).

• Bakker, B. (2002). Reinforcement Learning with Long Short-Term Memory. In T.G. Dietterich, S. Becker, and Z. Ghahramani (Eds.), Advances in Neural Information Processing Systems, 14. Cambridge, MA: MIT Press, p. 1475-1482. Postscript. Zipped Postscript. PDF.
  Abstract: This paper presents reinforcement learning with a Long Short-Term Memory recurrent neural network: RL-LSTM. Model-free RL-LSTM using Advantage(lambda) learning and directed exploration can solve non-Markovian tasks with long-term dependencies between relevant events. This is demonstrated in a T-maze task, as well as in a difficult variation of the pole balancing task.

• Bakker, B. (2001). Reinforcement Learning with LSTM in Non-Markovian Tasks with Long-Term Dependencies. Technical Report. Postscript. Zipped Postscript. PDF.
  Abstract: This paper presents reinforcement learning with a Long Short-Term Memory recurrent neural network: RL-LSTM. Model-free RL-LSTM using Advantage(lambda) learning and directed exploration can solve non-Markovian tasks with long-term dependencies between relevant events. This is demonstrated in a T-maze task, as well as in a difficult variation of the pole balancing task.

• Bakker, B. (2000). The Adaptive Behavior Approach to Psychology. Cognitive Processing, 1, 39-70. Postscript. Zipped Postscript. PDF.
  Abstract: This paper discusses a rapidly evolving field with great relevance for cognitive psychology, the field of adaptive behavior. Its goal is to learn about biological behavior by constructing agents exhibiting that behavior, but it is very different from traditional artificial intelligence. It is argued that rather than providing mere implementations of the abstract theories of cognitive psychology, the adaptive behavior approach yields many completely new insights, as well as indications that some of mainstream cognitive psychology's assumptions and theoretical ideas may be in need of revision. Specific examples of adaptive behavior research are presented to support these claims.
  

• Bakker, B., and de Jong, M. (2000). The Epsilon State Count. In J.-A. Meyer, A. Berthoz, D. Floreano, H. Roitblat, and S.W. Wilson (Eds.), From Animals to Animats 6: Proceedings of The Sixth International Conference on Simulation of Adaptive Behavior, 51-60, Cambridge, MA: MIT Press. Postscript. Zipped Postscript. PDF.
  Abstract: Terminology and definitions are proposed for the application of systems concepts to Adaptive Behavior research. The definition of a system can be applied to agents and environments in several ambiguous ways. For this reason, a distinction is introduced between a mechanistic perspective and a functional perspective. For both perspectives a methodology is proposed for estimating the number of states of an agent, yielding the Mechanistic Epsilon State Count and the Functional Epsilon State Count, respectively. In addition, a similar means is proposed for counting the number of states required to perform a particular task in an environment: the Task Epsilon State Count. Importantly, the methodology explicitly looks for and reports the relation between the number of states and their effect on performance. This methodology provides a uniform language to describe and compare different agents and environments, and for this reason it may afford valuable comparisons between different Adaptive Behavior studies.

• Bakker, B., and van der Voort van der Kleij, G. (2000). Trading off Perception with Internal State: Reinforcement Learning and Analysis of Q-Elman Networks in a Markovian Task. In S.-I. Amari, C.L. Giles, M. Gori, and V. Piuri (Eds.), Proceedings of the International Joint Conference on Neural Networks 2000, Vol. III, 213-218. Postscript. Zipped Postscript. PDF.
  Abstract: A Markovian reinforcement learning task can be dealt with by learning a direct mapping from states to actions or values, or from state-action pairs to values. However, this may involve a difficult pattern recognition problem when the state space is large. This paper shows that using internal state, called ``supportive state'', may alleviate this problem---presenting an argument against the tendency to almost automatically use a direct mapping when the task is Markovian. This point is demonstrated in simulation experiments of an agent controlled by a neural network capable of learning the strategy of direct mapping as well as internal state, combining Q(lambda)-learning and recurrent neural networks in a new way. The trade-off between the two strategies is investigated in more detail, focusing in particular on border cases.

• Bakker, B., and den Dulk, P. (1999). Causal Relationships and Relationships between Levels: The Modes of Description Perspective. In: M. Hahn and S.C. Stoness (Eds.) Proceedings of the Twenty-First Annual Conference of the Cognitive Science Society, pp. 43-48. Postscript. Zipped Postscript. PDF.
  Abstract: Many researchers have argued for a description of nature using multiple levels, or modes of description, as we call them. This paper focuses on a confusion that follows from the multiple-mode approach, a confusion due to the notion of causation between modes. Causation between modes is reinterpreted as ordinary causation but with cause and effect described in different modes. In the first part of the paper the framework of modes of description is presented. In the second part it is applied to examples from cognitive science, which are taken from debates on the mind-brain issue and the dynamical systems approach to cognition.
  
   

Back to Bram Bakker's Homepage