Researchers have created a machine learning system capable of learning diverse, simulated tennis skills from broadcast video footage. When provided with a large dataset of video clips from professional tennis players, the system learns to perform complex tennis shots and realistically chains together multiple shots into extended two-player rallies, generating long-lasting matches with realistic racket and ball dynamics between two physically simulated characters. While popular sports video games typically use motion capture technologies to produce high-quality animations, their results are limited to the behaviors captured during specific recording sessions, limiting the diversity of movements a character can perform. According to new developments researchers from Stanford University, the University of Toronto, Vector Institute and NVIDIA, it may soon become possible for video game designers to have their characters learn to move by mimicking footage of real-life athletes, and automatically simulate new variations and responsive behaviors on the fly.
The animation system allows for character movements to be procedurally generated by learning from footage of real athletes playing in sports games. The system leverages a hierarchical control model that combines a low-level imitation policy and high-level motion planning policy to guide character motions learned from broadcast videos. Given the complexity of components involved in their system, training the simulated player models from real footage of tennis games proved to be a challenging task. The group’s initial attempts generated noisy, unstable player movements, making it difficult to simulate life-like behaviors. To address the low-quality motions extracted from broadcast videos, the team implemented a motion correction system with physics-based imitation that overrides erroneous aspects of the learned motion. In turn, the animation system can now produce stable controllers for physically simulated tennis players that can accurately hit the incoming ball to target positions using a diverse variety of strokes, including serves, spins, and various playing styles.
More information: