Picture a gymnast like Simone Biles, who during a routine, performs a series of flips, twists, and complex maneuvers. Judges must capture every nuance of these routines — from the exact number of rotations to the subtle angle of a pointed toe — a task fraught with difficulty. Consistently interpreting the Code of Points, which provides standards for judging gymnastics across different competitions, is another challenge — in particular because different organizations like the FIG (Fédération Internationale de Gymnastique), NCAA, and national gymnastics federations have their own scoring systems. AI is beginning to contribute to addressing this challenge.

The Judging Support System (JSS)

Fujitsu, in collaboration with the Fédération Internationale de Gymnastique (FIG), has developed a platform known as the Judging Support System (JSS). This system uses high-definition cameras and specialized software to create 3D models of gymnasts in action, aiming to provide more accurate and consistent scoring through detailed analysis of each movement. The International Gymnastics Federation and Fujitsu have collaborated on building JSS for Artistic Gymnastics competitions since 2017. The technology first found use at the 52nd Artistic Gymnastics World Championships in Antwerp, Belgium, held from September 30–October 8, 2023. While some media reports suggested that JSS would be in use in the Paris Olympics, no confirmation of that was immediately available as of August, 2024.

While earlier versions of JSS relied on laser sensor technology similar to that used in autonomous vehicles to provide accurate 3D skeletal data, the current iteration is entirely camera-based. Four to eight high-definition cameras strategically positioned around each apparatus capture multi-angle footage of the gymnast’s entire routine. Fujitsu’s Human Motion Analytics engine processes this multi-angle footage, constructing a 3D model of the gymnast and enabling the system to track and evaluate movements. This camera-based system achieves accuracy comparable to the earlier laser-based iterations.

How JSS works

The Judging Support System (JSS) employs a multi-step process to analyze gymnastic performances. It begins by capturing 3D data of the athletes’ movements without the use of physical markers. AI-powered pose estimation algorithms then identify and track the position of the gymnasts’ joints throughout their routines. Using the high-definition camera footage, these algorithms extract detailed 3D pose data, creating a detailed model of the gymnast’s movements. Fujitsu’s Human Motion Analytics technology oversees this process, incorporating a correction algorithm to minimize recognition errors and ensure accurate identification of movements. Next, the system compares these movements against a database of recognized gymnastic elements. Finally, the JSS generates a 360-degree view of the performance, identifying the specific elements performed and providing both difficulty (D-score) and execution (E-score) scores.

To understand the details of gymnastic routines, the AI powering the JSS learned from a database of 8,000 routines. This extensive training set allowed the AI to learn and recognize the spectrum of elements defined in the Code of Points rulebook. Currently, the system can identify approximately 2,000 elements with 90% accuracy compared to human judges. The system can discern differences between elements, transitions, and pauses, as well as its understanding of the specific criteria for deductions based on deviations from the ideal execution.

Benefits beyond judging

The Judging Support System offers features for athletes, coaches, and viewers as well. In addition to providing near real-time feedback to judges — identifying elements, calculating scores, and flagging deductions — JSS can be used by gymnasts and coaches to analyze performances and refine skills. The same technology used in competitions is available for training, allowing athletes to review their routines in detail, examine angles and positions, and receive objective numerical data on their execution.

For athletes and coaches. JSS is said to offer gymnasts and coaches detailed performance analysis tools, helping them improve technique and avoid potentially injury-inducing misalignments. Athletes can rely on the 3D data to identify areas for improvement, receiving objective feedback on angles, positions, and timings. A promotional video from Fujitsu describes the technology’s ability to provide precise, data-driven feedback for skill development.

For fans and broadcasters. The JSS offers potential to enhance the viewing experience for both fans and broadcasters. The Fujitsu website describes how the data could be used to create visualizations that explain specific gymnastic elements, provide multi-angle replays of routines, and display detailed information such as difficulty values and precise measurements of movements like leg openings. The website also notes that JSS could support new ways of viewer engagement with the sport. For broadcasters, JSS data could facilitate expert analysis during replays. The system could also provide alternative viewing angles for spectators who have obstructed views at the competition venue.

Beyond gymnastics

An article from The Verge also describes the potential of the enabling technologies in JSS to find use in other domains. The article notes how the same ability to identify poses could prevent theft in self-checkout lines. Fujitsu has also noted the potential for the JSS technology in healthcare and rehabilitation, offering applications in physical therapy and even early detection of dementia through gait analysis. Yet Dennis Pauschinger, a researcher at the University of Neuchâtel, pointed out in the Verge article, the Olympics have often served as a testing ground for new surveillance technologies. “There’s a long tradition of the Games being used as a showcase for new surveillance and security technology,” Pauschinger said. “‘The Olympics are often used to be kind of a showroom,'”