One-shot Imitation Learning via Interaction Warping

Ondrej Biza1, Skye Thompson2, Kishore Reddy Pagidi1, Abhinav Kumar1, Elise van der Pol3,*, Robin Walters1,*, Thomas Kipf4,*, Jan-Willem van de Meent5,†, Lawson L.S. Wong1,†, Robert Platt1,†
1Northeastern University, 2Brown University, 3Microsoft Research,
4Google DeepMind, 5University of Amsterdam
CoRL 2023
*Equal contribution. †Equal advising.
Paper Code

We train a model to fit the variation of shapes within a category of objects using shape warping. Then, we extract keypoints from a single demonstration and attach them onto the shape warping model, allowing transfer to new scenes.

Abstract

Imitation learning of robot policies from few demonstrations is crucial in open-ended applications. We propose a new method, Interaction Warping, for learning SE(3) robotic manipulation policies from a single demonstration. We infer the 3D mesh of each object in the environment using shape warping, a technique for aligning point clouds across object instances. Then, we represent manipulation actions as keypoints on objects, which can be warped with the shape of the object. We show successful one-shot imitation learning on three simulated and real-world object re-arrangement tasks. We also demonstrate the ability of our method to predict object meshes and robot grasps in the wild.

BibTeX

@inproceedings{biza23oneshot,
  author       = {Ondrej Biza and
                  Skye Thompson and
                  Kishore Reddy Pagidi and
                  Abhinav Kumar and
                  Elise van der Pol and
                  Robin Walters and
                  Thomas Kipf and
                  Jan{-}Willem van de Meent and
                  Lawson L. S. Wong and
                  Robert Platt},
  title        = {One-shot Imitation Learning via Interaction Warping},
  booktitle    = {CoRL},
  year         = {2023}
}