UNIST cuts 3D human-tracking errors by 62% in film and sports footage

- Technology validated on film and sports broadcast footage

- Potential applications in digital humans and VR content

Multi-THuMBS can be used to reconstruct human movement in films and sports broadcasts assembled from multiple cuts. [Provided by UNIST]

Researchers have developed an AI technology that accurately tracks individuals and reconstructs their movements in three dimensions even when camera angles shift in films or sports broadcasts. Because the system works with existing footage rather than specialized capture equipment, it is expected to find use in sports performance analysis and the creation of digital humans for films and games.

The Ulsan National Institute of Science and Technology (UNIST) announced Thursday that a research team led by Professor Baek Seung-ryul of its AI Graduate School had developed Multi-THuMBS, an AI system that consistently tracks the 3D body shape and movement of multiple people across footage with frequent scene cuts.

Existing 3D human reconstruction technology struggled when camera angles changed or the frame zoomed in and out, often misidentifying the same person as a different individual. It also mistook apparent shifts in a subject's on-screen position for actual physical movement, causing reconstructed figures to appear in the wrong location.

To address these problems, the research team developed a method that links two scenes — before and after a cut — within a single three-dimensional coordinate space.

The AI reconstructs the background and camera position from each scene in a shared coordinate system and then places each person's body model within it. It then cross-references the distances between individuals, their clothing and their posture to identify and reconnect the same person across cuts.

The spatial information obtained this way is applied across the entire video to stitch together each person's movements seamlessly.

In experiments applying scene cuts to EgoHumans, a multi-camera human motion dataset, the system reduced 3D joint position error by about 31 percent compared with GVHMR, an existing benchmark method.

Particularly notable was a roughly 62 percent reduction in identity-switch errors — cases where the same person is misidentified as someone else — while camera position estimation error fell by about half.

UNIST Professor Baek Seung-ryul (left) and researcher On Jeong-wan. [Provided by UNIST]

The team also confirmed strong performance on real film scene data and footage from the popular American sitcoms "Friends" and "The Big Bang Theory."

The rate at which joint positions matched correctly across scene cuts reached 90.7 percent, far exceeding the 62.7 percent achieved by comparable existing methods. Motion jitter in the reconstructed figures fell to about one-fifth of previous levels, and foot-sliding errors — where feet appear to glide across the floor — dropped to less than half.

The technology can be applied to precisely analyze player movements in sports such as soccer, or to transfer real actors' motions onto film and game characters. It is also expected to help build large-scale 3D motion datasets from existing footage for use in virtual and augmented reality content and AI training data.

However, the current system is limited to footage shot within the same physical space, even when the camera angle changes. Continuously tracking individuals across cuts that shift to entirely different locations remains a goal for future research.

"Existing 3D human reconstruction technology had a fundamental limitation — it lost track of people the moment the scene changed," Professor Baek said. "This technology can continuously reconstruct the three-dimensional movements of multiple people even in edited footage from films, broadcasts and sports coverage, which gives it broad potential for real-world application."

nbgkoo@heraldcorp.com