Tag Archives: Google

Bringing Machine Vision to Olympic Judging

If you’re like me, your favorite part of the Olympics is watching athletes from all over the world come together and compete to see who is the best. For many situations it is easy to clearly determine who is the best. The team that scores the most goals wins at Football (a.k.a. Soccer). The person who crosses the finish line first wins the 100-meter Dash. The swimmer who touches the wall first wins the 200-meter Breaststroke.

Victims of Human Error (and Bias)

However, is some cases, determining what happened is less clear. All of these cases involve subjective human judgment. I am not just talking about judgment regarding stylistic components; I am talking about judgment on absolute principles of scoring and penalties. As a result, athletes (who have trained for tens of thousands of hours over years of their lives) are often at the mercy of human judgment of motions that are almost to fast to observe. A few examples:

  1. A sprinter can be disqualified if she or he kicks off the starting blocks before the sound of the starting gun could potentially reach him or her
  2. A boxer may miss a point because he punches and connects too quickly
  3. A diver or gymnast can receive unwarranted penalties (or conversely, not receive warranted ones) because human judges misperceive the smallest of angles during an movement that takes just a few seconds

Even worse, athletes in these situations are not only subject to human error, they are often subject to human bias as well. We have all seen countless questionable judgment calls based on national or political bias in too many events. As upsetting as these are to the spectator they are utterly heart breaking for the athletes involved.

Bringing Machine Intelligence to the Rescue

We already use technology to aid in places where events happen to quickly for humans to accurately perceive them. In racing (humans to horses, on land or water), we use photo-finish cameras to resolve which athlete has actually one when a finish is too close (or as happened this year, when there is actually a tie for the Gold Medal). In Gymnastics and Skating we allow judges to review slow motion cameras as part of their judging. In Fencing, we go one step further and equip athletes with electronic sensors to measure when a blade has touched a target area (or which touched first to resolve simultaneous touches).

It is time to go a few steps further and actually bring machine intelligence (machine vision + machine learning) to the stage to provide the same absolute scoring that photo-finish cameras bring. I am not advocating using machines to replace people for stylistic judging. However, it makes absolutely no sense to not use machines to detect and score absolutes such as:

  • A gymnast’s bent arms, separated knees or mixed tempo
  • Level of differentiation of a diver’s twist from 90°
  • The actual time a sprinter kicks off the blocks based a microphone’s detection of when the sound arrived
  • Detection of a skater’s under-rotated jump

Not only would this significantly reduce bias and error. It would actually be a great training tool. Just as advanced athletes today use sensors to measure performance and conditioning, they could use a similar approach to detect small errors and work to eliminate them earlier in training.

This is Now Possible

Just a few years ago, this was the stuff to science fiction. Today it is feasible. Half a dozen companies have developer self-driving cars equipped with sensors and machine learning programs to deal with conditions with much higher levels of variability than judging a 10-meter dive or Balance Beam program. However, one does not need to equip arenas with multiple cameras and LIDAR arrays. Researchers at DARPA have even moved down the direction of teaching robots to cook by having them review two-dimensional YouTube videos.

Similar approaches could be uses for “Scoring Computers.” If we wanted to go down the path of letting computer see exactly (and only) what humans can see we can go down the machine-learning route. First program the rules for scores and penalties. Then create training sets with identified scores and infractions to train a computer to detect penalties and score them as a judge would do—but with the aid of slow motion review in a laboratory without the pressure of on-the-spot judging on live TV. This would not remove the human, it would just let the human teach a computer to do something with higher accuracy and speed than a person could do in real-time.

If we wanted to go a step further, just as Fencing has done. We can add sensors to mix. A LIDAR array could measure exact motion (actually measuring that bent knee or over-rotation). Motion- capture (mo-cap) would make this accuracy even better. Both would also create amazing advanced sports training technology.

It’s More Feasible Then You May Think

All of this technology sounds pretty expensive: computers, sensors, data capture, programming, testing, verification, deployment, etc. However, it is not nearly as expensive and “sci-fi-ish” as one might think (or fear).

Tens of thousands of hours of video already exists to train computers to judge events (the same videos that judges, athletes and coaches review in training libraries—libraries even better than robo.watch). Computing time is getting cheaper every year thanks to Moore’s Law and public cloud computing. An abundant number of Open Source libraries for machine learning are available (some companies have opened proprietary libraries; others are offering Machine Learning-as-a-Service). There are now even low-cost LIDAR sensors available for less than $500 that can resolve distances of 1 cm or less (for $2,000 college programs and Tier I competitive venues can get sensors that resolve to 1 mm or less).

Given the millions of dollars poured into these sports (and the billions into transmission rights), it would not require an Apollo Program to build a pilot of this in time for the 2020 Olympics (or even 2018 Winter Olympics). Companies like Google and IBM likely donate some R&D to show off their capabilities. Universities like MIT, Carnegie Mellon, and Stanford are already putting millions of dollars in biomimetics, computer vision, and more. Even companies like ILM and Weta Digital might offer their mo-cap expertise as they would benefit from joint R&D. Thousands of engineers would likely jump in to help out via Kaggle Competitions and Hackathons as this would be really fun to create.

Some Interesting Side Benefits

There are benefits to technology outside of “just” providing more accurate judging and better training tools. This same technology could create amazing television that would enable spectators to better appreciate and understand these amazing sports. Yes, you could also add your Oculus Rift or similar AR technology to create some amazing immersive games (creating new sources of funding for organizations like the US Olympic Team or USA Gymnastics to help pay for athlete training).

Drone Commerce, Part 2: Global Internet Access

In Part 1 of this series, I looked at Amazon’s use of drones for same-day delivery. In this post, I will examine Google’s proposed use of drones for ubiquitous Internet access and near-Earth monitoring from the point of view of someone who has built things that fly, the software that controls them and large-scale Internet platforms.

The Drones of Titan

The drones created by Titan (now Google) Aerospace are quite different from the quadcopters you can buy online or the military UAVs featured so prominently in the news since 9-11. They are high-endurance drones intended to stay continuously aloft at 65,000’ (20 km) for 3 to 5 years. Running on solar-rechargeable batteries, they are designed to function as in-atmosphere satellites, providing communications (like COMSATs) or sensor-based observation (like weather and surveillance satellites).

google-buys-drone-company-titan-aerospace-1

Packets of energy, not goods

Amazon’s is exploring use of drones to delivery physical goods. This brings on a host of complex aeronautic and air traffic challenges: the ability to carry payload while staying small enough to navigate inside cities; efficiently taking off and landing several times per day in the midst of wind gusts and other weather conditions; and the need to avoid trees, birds, power lines, buildings and host of other obstacles. Google’s drones avoid all of these challenges:

  • Flying at 65,000’ places them above all weather events and a majority of atmospheric turbulence. It also places them above birds, buildings, mountains and even commercial airline traffic
  • Staying aloft for years (or even just a few months) eliminates exposure to the highest-risk operation any non-military aircraft can do: takeoff or land. It also reduces equipment replacement costs and virtually eliminates re-fueling costs.
  • By transmitting and receiving photons (light and other electromagnetic waves) the drones do not need to be engineered to carry high payloads. They also do not need to be engineered for repeated loading and unloading of packages.

These changes significantly reduce operational risk and cost. From an engineer’s point of view, the technology is a great fit to its intended function. However…

Is this just and engineer’s fantasy?

Yes, the Google Drones appear to be great candidates for in-atmosphere satellites. However, keeping hundreds or thousands of drones aloft is a pricey enterprise with complexity akin to that of operating a mid-sized airport. Aren’t there technologies already available that already meet the needs these drones are intended to satisfy? Let’s look at the two commonly considered alternatives to help answer this:

Cellular (GSM/GPRS/3G/LTE/4G):

Cellular technology already exists in many, many parts of the world (even 95% of the people in Africa who live in areas with electrical power, live within coverage of cell towers). At first examination, using drones to give coverage to everyone outside cell tower coverage seems to be a display of “First World Hi-Tech Hubris”. If these drones were just intended to provide Internet (as Facebook was exploring), I would agree 100%.

However these drones can have cameras and other sensors to provide monitoring of the environment, climate change, and natural disasters that cell towers cannot. Given the benefits already provided by using Google Earth data for analysis of climate, population, infrastructure and more, one can easily see the doors that opened by feeding camera and sensor data from these drones to developers and researchers via Google’s Maps APIs (including weather and traffic layers and ‘satellite’ views).

Finally as these drones are powered by sunlight, they would continue to function and provide monitoring and Internet access even if a natural disaster took at power grids and energy pipelines for an area.

Satellite:

horizon-1One could easily argue that satellites (between Iridium, SPOT, INMARSAT, COMSAT, and all those government programs I cannot mention) cover all the gaps cellular technology misses. At 65,000’ of altitude, these drones would only be able to cover a 300-mile radius: satellites (depending on orbital parameters) can cover up to 160x this coverage area.

However, satellites are expensive (as we have learned with the disappearance of flight MH370), satellite is expensive (about $0.14-$0.18 per small 1-Kilobyte message). The reason for this high-cost is two-fold: the high-cost of launching a satellite and the distance they are above the earth (it takes over 1500x the power to transmit a signal to an Iridium satellite than it does to transmit a signal to a drone overhead at 65,000’).

This opens to door to communication with a whole new class of technologies, ones far less expensive than satphones. This includes everything from low-cost mobile phones to OLPC (One Laptop Per Child) laptops to sensors used to track endangered species and protect them against poaching.

This distance factor goes beyond power consumption to image resolution (Ground Sample Distance or GSD). Quite simply, a drone at 65,000’ can get photos with 6x the resolution of satellite in Low Earth Orbit (LEO) and 40x the resolution of satellites like SPOT.

A great addition, but not the only answer

The Google Drone concept is not a one-size-fits-all answer. It would take thousands of drones to cover the Earth, a very costly operation. While providing more coverage than cell towers, they would often be farther away and more costly to operate. While providing better bandwidth and GSD than satellite, they would have less coverage area. As such the answer, like all things in Internet access (and sensor technology) is a blended combination of fixed-line Internet, multiple terrestrial wireless technologies (from ZigBee to 4G), satellites and drones.

This begs an important question…

One question that has plagued me from the day I first saw Facebook’s interest in Titan was why communications companies like Vodafone (which is rather well known for its 21-country mobile SIM network) were not interested in companies like Titan. Overall, using drones for ubiquitous Internet would appear to be a much better strategic fit to a company that already charges customers for Internet access. Perhaps Google can make more money from higher-resolution image and sensor data than it would initially appear. Or perhaps these drones could serve as a potential grid network that could bypass carriers if the Net Neutrality wars go in a bad direction (just like Netflix is exploring with its peer-to-peer research).

Avion-Titan-Aerospace-dron-solar-2

Only time will tell.