Current Events

The intersection of data and technology with current events

Bringing Machine Vision to Olympic Judging

If you’re like me, your favorite part of the Olympics is watching athletes from all over the world come together and compete to see who is the best. For many situations it is easy to clearly determine who is the best. The team that scores the most goals wins at Football (a.k.a. Soccer). The person who crosses the finish line first wins the 100-meter Dash. The swimmer who touches the wall first wins the 200-meter Breaststroke.

Victims of Human Error (and Bias)

However, is some cases, determining what happened is less clear. All of these cases involve subjective human judgment. I am not just talking about judgment regarding stylistic components; I am talking about judgment on absolute principles of scoring and penalties. As a result, athletes (who have trained for tens of thousands of hours over years of their lives) are often at the mercy of human judgment of motions that are almost to fast to observe. A few examples:

  1. A sprinter can be disqualified if she or he kicks off the starting blocks before the sound of the starting gun could potentially reach him or her
  2. A boxer may miss a point because he punches and connects too quickly
  3. A diver or gymnast can receive unwarranted penalties (or conversely, not receive warranted ones) because human judges misperceive the smallest of angles during an movement that takes just a few seconds

Even worse, athletes in these situations are not only subject to human error, they are often subject to human bias as well. We have all seen countless questionable judgment calls based on national or political bias in too many events. As upsetting as these are to the spectator they are utterly heart breaking for the athletes involved.

Bringing Machine Intelligence to the Rescue

We already use technology to aid in places where events happen to quickly for humans to accurately perceive them. In racing (humans to horses, on land or water), we use photo-finish cameras to resolve which athlete has actually one when a finish is too close (or as happened this year, when there is actually a tie for the Gold Medal). In Gymnastics and Skating we allow judges to review slow motion cameras as part of their judging. In Fencing, we go one step further and equip athletes with electronic sensors to measure when a blade has touched a target area (or which touched first to resolve simultaneous touches).

It is time to go a few steps further and actually bring machine intelligence (machine vision + machine learning) to the stage to provide the same absolute scoring that photo-finish cameras bring. I am not advocating using machines to replace people for stylistic judging. However, it makes absolutely no sense to not use machines to detect and score absolutes such as:

  • A gymnast’s bent arms, separated knees or mixed tempo
  • Level of differentiation of a diver’s twist from 90°
  • The actual time a sprinter kicks off the blocks based a microphone’s detection of when the sound arrived
  • Detection of a skater’s under-rotated jump

Not only would this significantly reduce bias and error. It would actually be a great training tool. Just as advanced athletes today use sensors to measure performance and conditioning, they could use a similar approach to detect small errors and work to eliminate them earlier in training.

This is Now Possible

Just a few years ago, this was the stuff to science fiction. Today it is feasible. Half a dozen companies have developer self-driving cars equipped with sensors and machine learning programs to deal with conditions with much higher levels of variability than judging a 10-meter dive or Balance Beam program. However, one does not need to equip arenas with multiple cameras and LIDAR arrays. Researchers at DARPA have even moved down the direction of teaching robots to cook by having them review two-dimensional YouTube videos.

Similar approaches could be uses for “Scoring Computers.” If we wanted to go down the path of letting computer see exactly (and only) what humans can see we can go down the machine-learning route. First program the rules for scores and penalties. Then create training sets with identified scores and infractions to train a computer to detect penalties and score them as a judge would do—but with the aid of slow motion review in a laboratory without the pressure of on-the-spot judging on live TV. This would not remove the human, it would just let the human teach a computer to do something with higher accuracy and speed than a person could do in real-time.

If we wanted to go a step further, just as Fencing has done. We can add sensors to mix. A LIDAR array could measure exact motion (actually measuring that bent knee or over-rotation). Motion- capture (mo-cap) would make this accuracy even better. Both would also create amazing advanced sports training technology.

It’s More Feasible Then You May Think

All of this technology sounds pretty expensive: computers, sensors, data capture, programming, testing, verification, deployment, etc. However, it is not nearly as expensive and “sci-fi-ish” as one might think (or fear).

Tens of thousands of hours of video already exists to train computers to judge events (the same videos that judges, athletes and coaches review in training libraries—libraries even better than Computing time is getting cheaper every year thanks to Moore’s Law and public cloud computing. An abundant number of Open Source libraries for machine learning are available (some companies have opened proprietary libraries; others are offering Machine Learning-as-a-Service). There are now even low-cost LIDAR sensors available for less than $500 that can resolve distances of 1 cm or less (for $2,000 college programs and Tier I competitive venues can get sensors that resolve to 1 mm or less).

Given the millions of dollars poured into these sports (and the billions into transmission rights), it would not require an Apollo Program to build a pilot of this in time for the 2020 Olympics (or even 2018 Winter Olympics). Companies like Google and IBM likely donate some R&D to show off their capabilities. Universities like MIT, Carnegie Mellon, and Stanford are already putting millions of dollars in biomimetics, computer vision, and more. Even companies like ILM and Weta Digital might offer their mo-cap expertise as they would benefit from joint R&D. Thousands of engineers would likely jump in to help out via Kaggle Competitions and Hackathons as this would be really fun to create.

Some Interesting Side Benefits

There are benefits to technology outside of “just” providing more accurate judging and better training tools. This same technology could create amazing television that would enable spectators to better appreciate and understand these amazing sports. Yes, you could also add your Oculus Rift or similar AR technology to create some amazing immersive games (creating new sources of funding for organizations like the US Olympic Team or USA Gymnastics to help pay for athlete training).

Minority Report meets the NFL: Which College Football Program Is Most Likely to Lead to Future NFL Arrests?

A few weeks ago I published a deep-dive analysis of USA Today’s NFL Arrest Database.  While I received many comments (mostly via email or Twitter DM), two rose to the top:

  1. College is a formative experience. Did the college the player attended affect the likelihood of arrest (and criminal charge)?
  2. Many towns have very active football programs. Did the town or high school the player attended drive specific outcomes?

Are we getting into Minority Report territory?

The more we look at variables that could be used to to classify future criminal behavior (e.g., does going to college X now indicate you will be arrested for Y seven years later?), the more we get into a world more like that depicted in the movie Minority Report. As such, we need to be really careful to ensure as compare “apples-to-apples” for all analysis.

This post will answer the first question. I am still processing the data on high schools before writing up the second.

The college that led to the most arrests: WVU

It is a bit tricky analyzing which college led to the most arrests. You cannot simply count arrests by NFL players and group them by college program. This would penalize colleges with great programs (a college with 200+ alumni in the NFL should have more alumni with arrests than a college with only 5 alumni). Similarly ,you cannot simply look at the ratio of arrested NFL alumni to total NFL alumni (as this would penalize a college with few alumni).

So how did I answer this question? I combined two factors. I overlayed the following:

  • Top 5% college programs with most alumni in the NFL
  • Top 5% college programs with most alumni arrested in the NFL

Here is a visualization of the result:

Spiral chart shows how many NFL arrests were from players from each respective college program. The seven schools highlighted were schools in BOTH the top 5% for NFL placement AND top 5% for NFL arrests.
Spiral chart shows how many NFL arrests were from players from each respective college program. The seven schools highlighted were schools in BOTH the top 5% for NFL placement AND top 5% for NFL arrests.

West Virginia University not only had the most arrested NFL alumni. It also had the most arrested NFL alumni in comparison to all other top college programs. 

I took a look at the ratio of each of these school’s arrested NFL alumni per 125 total NFL alumni to get a “arrest per squad that made it to the NFL.” The results we interesting:

  • The average “Top 5%” team had 4.53 arrest per NFL squad
  • WVU had 18.06, nearly 4x the average arrest rate
  • WVU also had almost double the next highest arrest rate: University of Miami (FL) who had 9.87 arrests per NFL squad

Other Schools with Many Arrested NFL Alumni

As the spiral diagram shows above, WVU was not the only school in the “Top 5%” for both alumni who made it to the NFL and alumni arrested in the NFL. There were seven schools or made this list:



Arrests of
NFL Alumni

per Squad

West Virginia




Miami (FL)








Ohio State
















Are these numbers really bad?  The answer is a definitive “Yes”.  Let’s take a look:

  • While these colleges represent 2% of all colleges  that have placed players in the NFL, they represent 20% of all future NFL arrests.
  • The average college team’s NFL alumni have an “arrest per squad” rate of 4.89
  • The average “arrest per squad” rate of the team with the highest placement of NFL players is even better: 4.53
  • These seven colleges have a much higher rate: 8.10. This is 118% higher than average arrest rate of the all other schools with the most success placing alumni in the NFL

What is the path from College to NFL team to type of arrest?

After look at my Sankey diagram of NFL team to  criminal charge to legal outcome, many people asked me if I could show a similar diagram leading from college to NFL team to criminal charge. Doing this for all 158 colleges with arrested NFL alumni would be unreadable. However, here is a Sankey diagram of the flow from the seven universities with the most NFL arrests:

Sankey diagram representing the "flow" from university to NFL team to arrest (for the "Top 7" programs highlighted above).
Sankey diagram representing the “flow” from university to NFL team to arrest (for the “Top 7” programs highlighted above).

So, do specific college programs have a higher tendency for a type of crime?

My prior analysis  showed a strong correlation of criminal charge type by NFL team.  This led to people to ask me the following “smoking gun” question: does any college stand out as the “leader” in arrests of a particular criminal charge. The answer is No.

Yes, there is a college whose NFL alumni had the most arrests for criminal charge X. However, the numbers of arrest by criminal charge by college are so small that there are no statistically valid indicators that college team indicates any future criminal pattern. We should all be happy for that.

There might be a wider dimension than college that we could assess (e.g., conference, geographic area). However, college–in and of itself–is not a valid dimension to predict future criminal charge.

Some other interesting (and positive) insights

With all the attention on NFL arrests if it easy to overlook the positive. My analysis of colleges showed some strong positives as well.

Very successful college programs–in general–do not equate to high arrest rates:

  • The colleges with the highest success rate placing players have a 19% lower arrest rate than the average college program. Notre Dame, UCLA, UL Monroe, Wisconsin, Syracuse, Minnesota, Boston College, Mississippi, Baylor, Indiana, Northwestern, Northwestern State (LA), and Arizona stand out as schools with the lowest alumni arrest rates.
  • The most successful program, Notre Dame (with a whopping 536 alumni who made it to the NFL) had only three  alumni arrested. This corresponds to an arrest rate that is 86% lower than the average school.

Also, NFL players are arrested 1/3 less often than the average US population. Clearly emulating the the examples set at Notre Dame, UCLA, UL Monroe, Wisconsin, Syracuse, Minnesota, Boston College, Mississippi, Baylor, Indiana, Northwestern, Northwestern State (LA), and Arizona would lead to better outcomes for all.

More to follow later…