University of Illinois Urbana-Champaign researchers developed a video generation model to produce demonstration videos to teach robots to perform specific tasks. Users provide a text command and an image of the current scene to produce a demonstration video tailored to the robot's specific task and environment, with no additional training required. RIGVid's success rate across tasks was 85%, outperforming other robot imitation methods.
RIGVid’s approach for extracting an object’s motion from a generated video, based on model-based six DoF object pose tracking, also outperformed alternative methods for trajectory extraction using optical flow and sparse keypoints. The improvement was apparent on challenging tasks, like sweeping dirt or placing a thin spatula into a pan, where the other methods struggled with object occlusion.
More information: