Intelligent Robot Learning Laboratory (IRL Lab) Effective Transfer Learning

By: Zhaodong Wang and Matthew E. Taylor

Many learning methods such as reinforcement learning suffers from a slow beginning especially in complicated domains. The motivation of transfer learning is to use limited prior knowledge to help learning agents bootstrap at the start and thus achieve overall improvements on learning performance. Due to limited quantity or quality of prior knowledge, how to make the transfer more efficient and effective remains an interesting point. [1, 2]

[1] [pdf] Zhaodong Wang and Matthew E. Taylor. Improving Reinforcement Learning with Confidence-Based Demonstrations. In Proceedings of the 26th International Conference on Artificial Intelligence (IJCAI), August 2017. 26% acceptance rate
author={Wang, Zhaodong and Taylor, Matthew E.},
title={{Improving Reinforcement Learning with Confidence-Based Demonstrations}},
booktitle={{Proceedings of the 26th International Conference on Artificial Intelligence ({IJCAI})}},
note={26% acceptance rate},
bib2html_pubtype={Refereed Conference},
bib2html_rescat={Reinforcement Learning},
abstract={Reinforcement learning has had many successes, but in practice it often requires significant amounts of data to learn high-performing policies. One common way to improve learning is to allow a trained (source) agent to assist a new (target) agent. The goals in this setting are to 1) improve the target agent's performance, relative to learning unaided, and 2) allow the target agent to outperform the source agent. Our approach leverages source agent demonstrations, removing any requirements on the source agent's learning algorithm or representation. The target agent then estimates the source agent's policy and improves upon it. The key contribution of this work is to show that leveraging the target agent's uncertainty in the source agent's policy can significantly improve learning in two complex simulated domains, Keepaway and Mario.}
[2] [pdf] Zhaodong Wang and Matthew E. Taylor. Effective Transfer via Demonstrations in Reinforcement Learning: A Preliminary Study. In AAAI 2016 Spring Symposium, March 2016.
author={Zhaodong Wang and Matthew E. Taylor},
title={{Effective Transfer via Demonstrations in Reinforcement Learning: A Preliminary Study}},
booktitle={{{AAAI} 2016 Spring Symposium}},
bib2html_pubtype={Refereed Workshop or Symposium},
abstract={There are many successful methods for transferring information from one agent to another. One approach, taken in this work, is to have one (source) agent demonstrate a policy to a second (target) agent, and then have that second agent improve upon the policy. By allowing the target agent to observe the source agent's demonstrations, rather than relying on other types of direct knowledge transfer like Q-values, rules, or shared representations, we remove the need for the agents to know anything about each other's internal representation or have a shared language. In this work, we introduce a refinement to HAT, an existing transfer learning method, by integrating the target agent's confidence in its representation of the source agent's policy. Results show that a target agent can effectively 1) improve its initial performance relative to learning without transfer (jumpstart) and 2) improve its performance relative to the source agent (total reward). Furthermore, both the jumpstart and total reward are improved with this new refinement, relative to learning without transfer and relative to learning with HAT.}

Videos & Other Media: