Tag Archives: Olympics

Bringing Machine Vision to Olympic Judging

If you’re like me, your favorite part of the Olympics is watching athletes from all over the world come together and compete to see who is the best. For many situations it is easy to clearly determine who is the best. The team that scores the most goals wins at Football (a.k.a. Soccer). The person who crosses the finish line first wins the 100-meter Dash. The swimmer who touches the wall first wins the 200-meter Breaststroke.

Victims of Human Error (and Bias)

However, is some cases, determining what happened is less clear. All of these cases involve subjective human judgment. I am not just talking about judgment regarding stylistic components; I am talking about judgment on absolute principles of scoring and penalties. As a result, athletes (who have trained for tens of thousands of hours over years of their lives) are often at the mercy of human judgment of motions that are almost to fast to observe. A few examples:

  1. A sprinter can be disqualified if she or he kicks off the starting blocks before the sound of the starting gun could potentially reach him or her
  2. A boxer may miss a point because he punches and connects too quickly
  3. A diver or gymnast can receive unwarranted penalties (or conversely, not receive warranted ones) because human judges misperceive the smallest of angles during an movement that takes just a few seconds

Even worse, athletes in these situations are not only subject to human error, they are often subject to human bias as well. We have all seen countless questionable judgment calls based on national or political bias in too many events. As upsetting as these are to the spectator they are utterly heart breaking for the athletes involved.

Bringing Machine Intelligence to the Rescue

We already use technology to aid in places where events happen to quickly for humans to accurately perceive them. In racing (humans to horses, on land or water), we use photo-finish cameras to resolve which athlete has actually one when a finish is too close (or as happened this year, when there is actually a tie for the Gold Medal). In Gymnastics and Skating we allow judges to review slow motion cameras as part of their judging. In Fencing, we go one step further and equip athletes with electronic sensors to measure when a blade has touched a target area (or which touched first to resolve simultaneous touches).

It is time to go a few steps further and actually bring machine intelligence (machine vision + machine learning) to the stage to provide the same absolute scoring that photo-finish cameras bring. I am not advocating using machines to replace people for stylistic judging. However, it makes absolutely no sense to not use machines to detect and score absolutes such as:

  • A gymnast’s bent arms, separated knees or mixed tempo
  • Level of differentiation of a diver’s twist from 90°
  • The actual time a sprinter kicks off the blocks based a microphone’s detection of when the sound arrived
  • Detection of a skater’s under-rotated jump

Not only would this significantly reduce bias and error. It would actually be a great training tool. Just as advanced athletes today use sensors to measure performance and conditioning, they could use a similar approach to detect small errors and work to eliminate them earlier in training.

This is Now Possible

Just a few years ago, this was the stuff to science fiction. Today it is feasible. Half a dozen companies have developer self-driving cars equipped with sensors and machine learning programs to deal with conditions with much higher levels of variability than judging a 10-meter dive or Balance Beam program. However, one does not need to equip arenas with multiple cameras and LIDAR arrays. Researchers at DARPA have even moved down the direction of teaching robots to cook by having them review two-dimensional YouTube videos.

Similar approaches could be uses for “Scoring Computers.” If we wanted to go down the path of letting computer see exactly (and only) what humans can see we can go down the machine-learning route. First program the rules for scores and penalties. Then create training sets with identified scores and infractions to train a computer to detect penalties and score them as a judge would do—but with the aid of slow motion review in a laboratory without the pressure of on-the-spot judging on live TV. This would not remove the human, it would just let the human teach a computer to do something with higher accuracy and speed than a person could do in real-time.

If we wanted to go a step further, just as Fencing has done. We can add sensors to mix. A LIDAR array could measure exact motion (actually measuring that bent knee or over-rotation). Motion- capture (mo-cap) would make this accuracy even better. Both would also create amazing advanced sports training technology.

It’s More Feasible Then You May Think

All of this technology sounds pretty expensive: computers, sensors, data capture, programming, testing, verification, deployment, etc. However, it is not nearly as expensive and “sci-fi-ish” as one might think (or fear).

Tens of thousands of hours of video already exists to train computers to judge events (the same videos that judges, athletes and coaches review in training libraries—libraries even better than robo.watch). Computing time is getting cheaper every year thanks to Moore’s Law and public cloud computing. An abundant number of Open Source libraries for machine learning are available (some companies have opened proprietary libraries; others are offering Machine Learning-as-a-Service). There are now even low-cost LIDAR sensors available for less than $500 that can resolve distances of 1 cm or less (for $2,000 college programs and Tier I competitive venues can get sensors that resolve to 1 mm or less).

Given the millions of dollars poured into these sports (and the billions into transmission rights), it would not require an Apollo Program to build a pilot of this in time for the 2020 Olympics (or even 2018 Winter Olympics). Companies like Google and IBM likely donate some R&D to show off their capabilities. Universities like MIT, Carnegie Mellon, and Stanford are already putting millions of dollars in biomimetics, computer vision, and more. Even companies like ILM and Weta Digital might offer their mo-cap expertise as they would benefit from joint R&D. Thousands of engineers would likely jump in to help out via Kaggle Competitions and Hackathons as this would be really fun to create.

Some Interesting Side Benefits

There are benefits to technology outside of “just” providing more accurate judging and better training tools. This same technology could create amazing television that would enable spectators to better appreciate and understand these amazing sports. Yes, you could also add your Oculus Rift or similar AR technology to create some amazing immersive games (creating new sources of funding for organizations like the US Olympic Team or USA Gymnastics to help pay for athlete training).

Who Won Sochi? Wrangling Olympic Medal Count Data

I have always been a big fan of the Olympics (albeit I like the Summer Games better given my interest in Track & Field, Fencing and Soccer). However, something that has always bothered me is concept of the Medal Count. For years I have seen countries listed as “winning” because their medal count was higher—even though several countries “below” it often had many more Gold medals. Shouldn’t a Gold medal count for more than a Silver (and much more than a Bronze)? What would you rather have as an athlete: three Gold medals or four Bronzes?

Evidently, I am not the only one debating this point. Googling “value of olympic medals for rank count” yielded a range of debates on the first page alone (Bleacher Report, USA Today, The New Republic, the Washington Post and even Bicycling.com). Wikipedia even has an entry on this debate.

This year, however, I noticed that throughout the games that Google’s medal count stats page (Google “olympic medal count”) was not ranking countries by absolute medal count. For quite a while Norway and Germany were on top—even when they did not have the highest total number of medals—because they had more Gold medals than anyone else. Clearly Google was using a different weighting than “all medals are alike.” Not a surprise given their background in data.

Winter_Games_CoverartI started to wonder what type of weighting they were using. In 1984 (when the Olympics were in Los Angeles) a bunch of gaming companies came out with various Olympic games. Konami’s standup arcade game Track & Field was widely popular (and highly abusive to trackballs). The game I used to play the most (thanks to hacking it) was Epyx’s Summer (and Winter) Games. This game had the “real” challenge of figuring out a “who won the Olympics” as it was a head-to-head multi-player game (someone had to win). It used the 5:3:1 Medal Weighting Model to determine this: each Gold medal was worth 5 points, each Silver 3 points, each Bronze 1. I wondered if Google was using this model, so I decided to wrangle the data and find out.

Data processing

I used Google’s Sochi Olympic Medal Count as my source of data as this had counts and Google ranks of winners (I go this via their Russian site so I could get final results, there were 26 countries who won any Olympic Medal).

Of course, by the end of the Olympics it was a bit less interesting as Russia had both the most medals and the highest rank. However, I still wanted to figure out their weighting as a curious exercise. I built a model that calculated ranks for various Medal Weighting Model (MWM) approaches and calculated the absolute value of all Rank Error deltas from Google’s ranking. I both computed both the sum of these errors (Total Rank Error or TRE) and highlighted any non-zero error, enabling me to quickly see any errors in various MWM weightings.

Trying out a few random models

The first model I tried was the “Bob Costas Model” where every medal is the same (1:1:1). This was a clearly different than Google’s as it a TRE of 72. I then tried the Epyx 5:3:1 model… no dice: this one had a TRE of 35 (better than Bob, but not great). I tried a few other mathematical series:

  • Fibonacci: 0,1,1 (TRE=50); 1,1,2 (TRE=42); and 1,2,3 (TRE=43)
  • Fibonacci Prime (TRE=54)
  • Abundant Numbers (TRE=54)
  • Prime Numbers: (TRE=42)
  • Lucas Numbers (TRE=28)
  • Geometric Sequence (TRE=23)
  • Weird Numbers (TRE=2)
  • Happy Numbers (TRE=39)

I then tried logical sequences such as the lowest ratios where a Silver is worth more than a Bronze, and Gold is worth more than both (TRE=31). Still not luck

Getting more systematic

I decided to get more systematic and begin to visualize the TRE based on different MWM weights. I decided to keep Whole Number weights as I was operating under the general principal that each Medal has N points and that points (true in most sports—but not in things like Diving, Figure Skating and Gymnastics—nevertheless, I wanted to keep things simple).

I first looked at Gold Weight influence, WGOLD:1:1 where I varied WGOLD from 1 upwards. This clearly showed a rapid decay in TRE that flattened out at 2 with Gold was worth 13x that of a single Silver or Bronze medal:

Rapid decay in TRE as Gold medals gain higher weighting
Rapid decay in TRE as Gold medals gain higher weighting

This reinforced that Gold was King, but that Silver was better than Bronze by some value (not surprising). I then kept WGOLD at 13 and started to reduce WBRONZE. I found an interesting result: as soon as I made Bronze worth any value smaller than Silver (even ε = 0.001), I got Zero TRE (a complete match to Google’s Rank). However, I could not image a scoring system of 13:1:<1 (or 13:1:0.99). It was just too geeky. As such I tried a different approaches, all with Whole Number ratios of Gold:Silver:Bronze. The lowest ratios I found with Zero TREs were the following:

  • Gold=21, Silver=2, Bronze=1
  • Gold=29, Silver=3, Bronze=1
  • Gold=40, Silver=4, Bronze=1
  • Gold=45, Silver=5, Bronze=1

TRE never went to zero when Bronze was given Zero weight. Of these models, 40:4:1 had the most symmetry (10:1 to 4:1), so used that is my approximated Google Olympic Rank MDW (it did have zero TRE for all medal winners).

So who won?

I figured I would look at the Top Five Ranked Countries over various models:

Demonstration of how easy it is to add a Grading Curve to the rankings. The higher the TRE the more underweighted winning Gold medals (i.e., truly winning events) is. The country in bold is the one that benefits most from the Grading Curve
Demonstration of how easy it is to add a Grading Curve to the rankings. The higher the TRE the more underweighted winning Gold medals (i.e., truly winning events) is. The country in bold is the one that benefits most from the Grading Curve

Obviously, Russia is the all around winner as they won the most medals and the most Golds and the most Silvers. (Making this exercise a bit less interesting than it was about a week ago). However, it will be fun to apply this in 2016.

And at least Mr Putin is happy.