Tag Archives: DARPA

Bringing Machine Vision to Olympic Judging

If you’re like me, your favorite part of the Olympics is watching athletes from all over the world come together and compete to see who is the best. For many situations it is easy to clearly determine who is the best. The team that scores the most goals wins at Football (a.k.a. Soccer). The person who crosses the finish line first wins the 100-meter Dash. The swimmer who touches the wall first wins the 200-meter Breaststroke.

Victims of Human Error (and Bias)

However, is some cases, determining what happened is less clear. All of these cases involve subjective human judgment. I am not just talking about judgment regarding stylistic components; I am talking about judgment on absolute principles of scoring and penalties. As a result, athletes (who have trained for tens of thousands of hours over years of their lives) are often at the mercy of human judgment of motions that are almost to fast to observe. A few examples:

  1. A sprinter can be disqualified if she or he kicks off the starting blocks before the sound of the starting gun could potentially reach him or her
  2. A boxer may miss a point because he punches and connects too quickly
  3. A diver or gymnast can receive unwarranted penalties (or conversely, not receive warranted ones) because human judges misperceive the smallest of angles during an movement that takes just a few seconds

Even worse, athletes in these situations are not only subject to human error, they are often subject to human bias as well. We have all seen countless questionable judgment calls based on national or political bias in too many events. As upsetting as these are to the spectator they are utterly heart breaking for the athletes involved.

Bringing Machine Intelligence to the Rescue

We already use technology to aid in places where events happen to quickly for humans to accurately perceive them. In racing (humans to horses, on land or water), we use photo-finish cameras to resolve which athlete has actually one when a finish is too close (or as happened this year, when there is actually a tie for the Gold Medal). In Gymnastics and Skating we allow judges to review slow motion cameras as part of their judging. In Fencing, we go one step further and equip athletes with electronic sensors to measure when a blade has touched a target area (or which touched first to resolve simultaneous touches).

It is time to go a few steps further and actually bring machine intelligence (machine vision + machine learning) to the stage to provide the same absolute scoring that photo-finish cameras bring. I am not advocating using machines to replace people for stylistic judging. However, it makes absolutely no sense to not use machines to detect and score absolutes such as:

  • A gymnast’s bent arms, separated knees or mixed tempo
  • Level of differentiation of a diver’s twist from 90°
  • The actual time a sprinter kicks off the blocks based a microphone’s detection of when the sound arrived
  • Detection of a skater’s under-rotated jump

Not only would this significantly reduce bias and error. It would actually be a great training tool. Just as advanced athletes today use sensors to measure performance and conditioning, they could use a similar approach to detect small errors and work to eliminate them earlier in training.

This is Now Possible

Just a few years ago, this was the stuff to science fiction. Today it is feasible. Half a dozen companies have developer self-driving cars equipped with sensors and machine learning programs to deal with conditions with much higher levels of variability than judging a 10-meter dive or Balance Beam program. However, one does not need to equip arenas with multiple cameras and LIDAR arrays. Researchers at DARPA have even moved down the direction of teaching robots to cook by having them review two-dimensional YouTube videos.

Similar approaches could be uses for “Scoring Computers.” If we wanted to go down the path of letting computer see exactly (and only) what humans can see we can go down the machine-learning route. First program the rules for scores and penalties. Then create training sets with identified scores and infractions to train a computer to detect penalties and score them as a judge would do—but with the aid of slow motion review in a laboratory without the pressure of on-the-spot judging on live TV. This would not remove the human, it would just let the human teach a computer to do something with higher accuracy and speed than a person could do in real-time.

If we wanted to go a step further, just as Fencing has done. We can add sensors to mix. A LIDAR array could measure exact motion (actually measuring that bent knee or over-rotation). Motion- capture (mo-cap) would make this accuracy even better. Both would also create amazing advanced sports training technology.

It’s More Feasible Then You May Think

All of this technology sounds pretty expensive: computers, sensors, data capture, programming, testing, verification, deployment, etc. However, it is not nearly as expensive and “sci-fi-ish” as one might think (or fear).

Tens of thousands of hours of video already exists to train computers to judge events (the same videos that judges, athletes and coaches review in training libraries—libraries even better than robo.watch). Computing time is getting cheaper every year thanks to Moore’s Law and public cloud computing. An abundant number of Open Source libraries for machine learning are available (some companies have opened proprietary libraries; others are offering Machine Learning-as-a-Service). There are now even low-cost LIDAR sensors available for less than $500 that can resolve distances of 1 cm or less (for $2,000 college programs and Tier I competitive venues can get sensors that resolve to 1 mm or less).

Given the millions of dollars poured into these sports (and the billions into transmission rights), it would not require an Apollo Program to build a pilot of this in time for the 2020 Olympics (or even 2018 Winter Olympics). Companies like Google and IBM likely donate some R&D to show off their capabilities. Universities like MIT, Carnegie Mellon, and Stanford are already putting millions of dollars in biomimetics, computer vision, and more. Even companies like ILM and Weta Digital might offer their mo-cap expertise as they would benefit from joint R&D. Thousands of engineers would likely jump in to help out via Kaggle Competitions and Hackathons as this would be really fun to create.

Some Interesting Side Benefits

There are benefits to technology outside of “just” providing more accurate judging and better training tools. This same technology could create amazing television that would enable spectators to better appreciate and understand these amazing sports. Yes, you could also add your Oculus Rift or similar AR technology to create some amazing immersive games (creating new sources of funding for organizations like the US Olympic Team or USA Gymnastics to help pay for athlete training).

Web 2.0: It feels like 1999 all over again

A refresher on the state of Web 1.0 in 1999

I was one of those lucky few to be a part of the explosion of the Internet (not just the dot-com boom but also the earlier DARPA-driven R&D at MIT, CMU, Stanford and Berkeley the 80s). For those of you who do not remember (or — I am sad to remind myself — may be too young to remember) here are some things that were going on in the Web 1.0 world in 1999:

  • The Horsemen of the Internet (Cisco, EMC, Oracle and Sun in the B2B world, Amazon, AOL and Yahoo! in the B2C one) had introduced “new models of doing business that would change everything” to millions of people
  • These models were very technology-centric and focused on “new measures of value” such as click-thru’s, eyeballs, audience, etc. (discarding traditional EPS and PEG values)
  • Lots of “traditional” companies were racing to adopt these models — instead one core to their businesses. (Remember all those tracking stocks like NBCi and Borders Online?)
  • Technology vendors were rushing out tool boxes to build web sites,
  • As same time, analysts were heavily questioning if these models had staying power (the stocks of the horsemen actually dipped heavily at this time — just before rising as part of the last-minute Y2K Technology Gold Rush

The Web 2.0 parallels of today are eerie

This sounds a little familiar what is happening today in the Web 2.0 world (minus the overtones of the current world recession):

  • Thanks to Web 2.0 Horsemen (Facebook, MySpace, LinkedIn and Twitter) millions of people now roughly understand what Web 2.0 means — at least in the consumer space
  • Like 1999, the business models in the Web 2.0 space are still largely in the formative stages (just a few minutes on TechCrunch or Silicon Alley Insider will highlight this)
  • Lots of “traditional” companies were racing to add Web 2.0 offerings — with varying degrees of success. (At least we are avoiding Web 2.0 tracking stocks for now)
  • As the analyst reports attest, the Web 2.0 space is becoming filled with companies who offer “toolboxes that can uses to quickly stand up communities”
  • At the same time, many are asking if “there is any there there” in the Web 2.0 business model (and the valuation of Web 2.0 companies have crashed — ahead of the recession)

What happened after 1999 to Web 1.0

Within five years, the Web 1.0 world have moved to a very different place than it was in 1999. Essentially, it integrated with (instead of disrupting) the rest of the business world. Web technology moved from being an end-to-itself to becoming a means to create value. This significantly changed the market space: web-only companies diminished or disappeared (web hosting, domain name services and email are now commodities) while companies who could use web technology to create value-added Business Solutions created whole new markets. Examples easily come to mind:

  • Content Management systems replaced the build-your-own-website tool kits (and pushed these companies aside)
  • eCommerce platforms became a core purchase for every major CPG company
  • Advertising and creative agencies became Interactive Agencies, providing holistic advertising and brand service across all media channels (pushing the ‘webmaster’ back to the IT department)
  • CRM moved from a back office function to an real-time service to manage revenue creation
  • Digital music replaced the experience of going to the record (or CD) store
  • Searching for information online (instead of going to a library or buying a “List of…” book)

We are now ready for this in the Web 2.0 world

I believe Web 2.0 will follow a similar integration path that Web 1.0 did. Those companies who can figure out how to create value-added Business Services using Web 2.0 communication approaches (be them technology firms of consulting and creative groups) will expand and develop new markets. Those enterprises who fold these services into the day-to-day execution of their mission will realize the most benefit.

If you disagree, the perhaps you can answer the following question for me: what is the value of a blog or a forum? I do not think blogs or forums have much intrinsic value in themselves. However they can create value when integrated into a higher value business service or process.

On the other hand, what is the follow of the following services?:

  • Leveraging your customers to tell you what you need to invest in to sell more (would save a lot on Product Development and increase product success rate)
  • Harnessing citizen input to shape more efficient public budgeting (would save a lot on public referenda)
  • Using the the contributions and input of your customers to drive advertising traffic and urge new customers to buy your product (saves on advertising costs and increases sales)

Not only are these services valuable, they are also broadly applicable, easy to understand (from both a business model and end user perspective). The firms that can create these will become the Vignettes, Crispin Porters, Salesforce.coms and Apples of the Web 2.0 world.

Addendum 1: I am not the only person who thinks this

McKinsely & Company recently included a segment “Six Ways to Make Web 2.0 Work” in their last McKinseyQuarterly publication. This article discussed a very similar evolution of adoption of Web 2.0 “tools” that will overlay existing infrastructure to encourage engagement and participation. They included a graph that shows the same ten-year repetitive cycle:

MckinseyQtr-560px
Credit: www.McKinseyQuarterly.com

Addendum 2: Here we go again

IoT is the third big technology ‘wave’ in the last 50 years — and perhaps the biggest