AI scientists who build reinforcement learning agents may learn a great deal from animals. That’s according to new studies by Google’s DeepMind, Imperial College London, and researchers from the University of Cambridge testing AI and non-human animals.
The AI research group has always looked at neuroscience and behavioral science for inspiration and to better understand how intelligence is shaped in a decades-long effort to advance machine intelligence. But this initiative has primarily focused on human intelligence, especially that of children and babies.
“This is particularly true in the sense of reinforcement learning, where it is now possible to bring the methods of comparative cognition directly to bear, thanks to progress in deep learning,” the researchers’ paper reads. “Animal cognition offers a compendium of well-understood, non-linguistic, intelligent behavior; it proposes assessment and benchmarking experimental methods; and it can direct the environment and task design.”
DeepMind implemented some of the first types of AI, such as the deep Q-network (DQN) algorithm, a device that played various Atari games at superhuman speeds, which merged deep learning and reinforcement learning. Deep learning and reinforcement learning were both used by AlphaGo and AlphaZero to train AI to defeat a human Go champion and accomplish other feats. More recently, DeepMind has developed AI that automatically generates algorithms for reinforcement learning.
On the human cognition side, DeepMind neuroscience research director Matthew Botvinick urged machine learning practitioners at a Stanford HAI conference earlier this month to collaborate with neuroscientists and psychologists in more interdisciplinary work.
Deep reinforcement learning, unlike other methods of training AI, gives an agent a goal and incentive, an approach similar to training animals using food rewards. A variety of animals, including dogs and bears, have been looked at in previous animal cognition research. In species, cognitive behavioral scientists have found higher levels of intelligence than commonly thought, including the self-awareness of dolphins and the potential of crows for revenge.
Studies on the cognitive abilities of animals can also enable AI researchers, especially in deep reinforcement learning, to look at issues in a different way. The theory of evaluating AI systems’ cognitive abilities has grown as researchers draw comparisons between animals in testing situations and reinforcement learning agents. Other types of AI, such as Alexa or Siri assistants, for example, can not scan for a box containing a reward or food in a maze.
The team’s paper, Artificial Intelligence and the Common Sense of Animals, published in CellPress Reviews, cites cognition studies with birds and primates.
Ideally, we would like to develop AI technology that can understand these interrelated values and principles as a systemic whole and that expresses this understanding in a capacity for generalization and creativity at the human level, “the paper reads.” An open question remains as to how to develop such AI technologies. But we advocate an approach in which RL agents acquire what is needed through extended interaction with rich virtual environments, maybe with as-yet-undeveloped architectures.’
Challenges include helping agents sense that they reside inside an autonomous environment when it comes to building structures like those described in the paper. Another challenge is teaching agents to understand the principle of common sense, along with understanding the types of environments and activities ideally suited to the task.
3D virtual environments with practical mechanics would be a requirement for training agents to use common sense. Objects, such as shells that can be cracked apart, lids that can be unscrewed, and packets that can be torn open can be simulated.
This is within the technical capacities of today’s physics engines, but for the training of RL officers, such rich and practical environments have yet to be implemented to scale, “the paper reads.”
The researchers argue that common sense is not a special human attribute, but it relies on certain universal principles, such as knowing what an object is, how space is filled by the object, and the relationship between cause and effect. The ability to view an object as a semi-permanent thing that can remain reasonably persistent over time is among these concepts.
Awareness of objects’ permanence and that a reward can lie within a jar, such as that a shell can contain a seed, are aspects of cognition displayed by animals. As the problem of finding tasks and curricula that, provided the right architecture, can result in qualified agents who can pass adequately planned transfer tasks, the question of endowing agents with certain common sense concepts can be cast.
Although contemporary deep RL agents can learn very effectively to solve multiple tasks, and some architectures display rudimentary forms of transfer, it is far from clear that such an abstract concept can be acquired by any current RL architecture. But supposing that we had a candidate agent, how would we assess if the idea of a container had been acquired?
Researchers conclude that training agents should be able to gain an understanding without the need to see several examples, techniques known as few-shot or zero-shot learning, tasks for learning common sense.
The evaluation of common sense contained in the paper focuses on the physics of common sense and does not take into account other forms of common sense, such as psychological concepts, the ability to identify various forms of objects such as liquids or gases, or the ability to understand objects that can be manipulated, such as paper or a sponge.
In other recent developments in reinforcement learning, UC Berkeley professor Ion Stoica spoke at VentureBeat ‘s recent Transform conference about why supervised learning is much more commonly used than reinforcement learning, Stanford University researchers introduced LILAC in dynamic environments to improve reinforcement learning, and Georgia Tech researchers combined NLP and reinforcement learning
No Comments
Leave a comment Cancel