Tag Archives: MIT

Bringing Machine Vision to Olympic Judging

If you’re like me, your favorite part of the Olympics is watching athletes from all over the world come together and compete to see who is the best. For many situations it is easy to clearly determine who is the best. The team that scores the most goals wins at Football (a.k.a. Soccer). The person who crosses the finish line first wins the 100-meter Dash. The swimmer who touches the wall first wins the 200-meter Breaststroke.

Victims of Human Error (and Bias)

However, is some cases, determining what happened is less clear. All of these cases involve subjective human judgment. I am not just talking about judgment regarding stylistic components; I am talking about judgment on absolute principles of scoring and penalties. As a result, athletes (who have trained for tens of thousands of hours over years of their lives) are often at the mercy of human judgment of motions that are almost to fast to observe. A few examples:

  1. A sprinter can be disqualified if she or he kicks off the starting blocks before the sound of the starting gun could potentially reach him or her
  2. A boxer may miss a point because he punches and connects too quickly
  3. A diver or gymnast can receive unwarranted penalties (or conversely, not receive warranted ones) because human judges misperceive the smallest of angles during an movement that takes just a few seconds

Even worse, athletes in these situations are not only subject to human error, they are often subject to human bias as well. We have all seen countless questionable judgment calls based on national or political bias in too many events. As upsetting as these are to the spectator they are utterly heart breaking for the athletes involved.

Bringing Machine Intelligence to the Rescue

We already use technology to aid in places where events happen to quickly for humans to accurately perceive them. In racing (humans to horses, on land or water), we use photo-finish cameras to resolve which athlete has actually one when a finish is too close (or as happened this year, when there is actually a tie for the Gold Medal). In Gymnastics and Skating we allow judges to review slow motion cameras as part of their judging. In Fencing, we go one step further and equip athletes with electronic sensors to measure when a blade has touched a target area (or which touched first to resolve simultaneous touches).

It is time to go a few steps further and actually bring machine intelligence (machine vision + machine learning) to the stage to provide the same absolute scoring that photo-finish cameras bring. I am not advocating using machines to replace people for stylistic judging. However, it makes absolutely no sense to not use machines to detect and score absolutes such as:

  • A gymnast’s bent arms, separated knees or mixed tempo
  • Level of differentiation of a diver’s twist from 90°
  • The actual time a sprinter kicks off the blocks based a microphone’s detection of when the sound arrived
  • Detection of a skater’s under-rotated jump

Not only would this significantly reduce bias and error. It would actually be a great training tool. Just as advanced athletes today use sensors to measure performance and conditioning, they could use a similar approach to detect small errors and work to eliminate them earlier in training.

This is Now Possible

Just a few years ago, this was the stuff to science fiction. Today it is feasible. Half a dozen companies have developer self-driving cars equipped with sensors and machine learning programs to deal with conditions with much higher levels of variability than judging a 10-meter dive or Balance Beam program. However, one does not need to equip arenas with multiple cameras and LIDAR arrays. Researchers at DARPA have even moved down the direction of teaching robots to cook by having them review two-dimensional YouTube videos.

Similar approaches could be uses for “Scoring Computers.” If we wanted to go down the path of letting computer see exactly (and only) what humans can see we can go down the machine-learning route. First program the rules for scores and penalties. Then create training sets with identified scores and infractions to train a computer to detect penalties and score them as a judge would do—but with the aid of slow motion review in a laboratory without the pressure of on-the-spot judging on live TV. This would not remove the human, it would just let the human teach a computer to do something with higher accuracy and speed than a person could do in real-time.

If we wanted to go a step further, just as Fencing has done. We can add sensors to mix. A LIDAR array could measure exact motion (actually measuring that bent knee or over-rotation). Motion- capture (mo-cap) would make this accuracy even better. Both would also create amazing advanced sports training technology.

It’s More Feasible Then You May Think

All of this technology sounds pretty expensive: computers, sensors, data capture, programming, testing, verification, deployment, etc. However, it is not nearly as expensive and “sci-fi-ish” as one might think (or fear).

Tens of thousands of hours of video already exists to train computers to judge events (the same videos that judges, athletes and coaches review in training libraries—libraries even better than robo.watch). Computing time is getting cheaper every year thanks to Moore’s Law and public cloud computing. An abundant number of Open Source libraries for machine learning are available (some companies have opened proprietary libraries; others are offering Machine Learning-as-a-Service). There are now even low-cost LIDAR sensors available for less than $500 that can resolve distances of 1 cm or less (for $2,000 college programs and Tier I competitive venues can get sensors that resolve to 1 mm or less).

Given the millions of dollars poured into these sports (and the billions into transmission rights), it would not require an Apollo Program to build a pilot of this in time for the 2020 Olympics (or even 2018 Winter Olympics). Companies like Google and IBM likely donate some R&D to show off their capabilities. Universities like MIT, Carnegie Mellon, and Stanford are already putting millions of dollars in biomimetics, computer vision, and more. Even companies like ILM and Weta Digital might offer their mo-cap expertise as they would benefit from joint R&D. Thousands of engineers would likely jump in to help out via Kaggle Competitions and Hackathons as this would be really fun to create.

Some Interesting Side Benefits

There are benefits to technology outside of “just” providing more accurate judging and better training tools. This same technology could create amazing television that would enable spectators to better appreciate and understand these amazing sports. Yes, you could also add your Oculus Rift or similar AR technology to create some amazing immersive games (creating new sources of funding for organizations like the US Olympic Team or USA Gymnastics to help pay for athlete training).

Four Common IoT Security Holes

If you follow the Internet of Things space, not a day passes where you do not see an analyst report or news article talking about IoT security vulnerabilities across every sector: consumer, enterprise, industrial and government/Smart City.

I’ve been working with Internet-connected devices (medical devices, industrial actuators, sensors for environmental, security monitoring, even military systems) for many years. In my job, I am lucky enough to able to work with industrial and enterprise devices daily. At home, I play with them both as a consumer and developer. Time and again, I see the following IoT security holes with alarming frequency:

Security Hole #1: Not Using Strong Encryption

It is amazing that in 2016 people are still not using strong encryption to protect important data. However, I frequently see IoT devices that use no encryption at all: they store and transmit data in the clear. Other devices use homegrown encryption techniques that are are unproven by peer review and relatively easy to hack.

Most of the arguments I have seen against encryption fall into three camps: 1) it is too computationally expensive for low-powered devices, 2) it is too hard to use for IoT protocols, and 3) the device data is too obscure to understand. Let’s look at each:

  1. Yes, encryption is computationally expensive. However, ongoing investments in the space are providing more efficient RSA, AES, and ECC algorithms that work on smaller devices. In addition, Moore’s Law is even allowing penny-sized devices to have enough power to use these.
  2. IoT protocols are also getting better and better at providing strong encryption and secure connections (see Security Hole #2).
  3. Finally, the old “Our-data-is-too-obscure-for-hackers-to-understand Argument” was proven a fallacy years ago, first by the credit card industry’s Cardholder Information Security Program, and later by its replacement: PCI DSS. Any disgruntled employee (or hacker masquerading as a contractor) can bypass the “obscurity protection.”

Not using strong encryption is probably the most egregious security vulnerability. Any 14-year-old can use downloadable packet sniffing programs to capture your data. Solutions that mitigate this risk are readily available. There is no excuse to not encrypting your data.

Security Hole #2: Not Using Secured Sessions

A common error is information/cyber security is forgetting that secure communication consists of two components:

  1. Encryption of data and
  2. Establishment of secured sessions

Secured sessions use protocols to establish mutual authentication and to exchange  shared secret that only the transmitter and receiver have. If you do not establish a secured session you are blindly guessing that the recipient of your data is the correct person. When you do not use secured session you invite a Man-In-The-Middle (MITM) attack where the attacker can intercept and redirect your transmissions.

Many people think they are not likely targets of a MITM attack. Here is simple scenario.

  • A disgruntled employee or hacker-posing-as-contractors first intercepts and copies traffic from your devices.
  • From this data, he learns what devices are attached to items of interest (a patient, your house, etc.). He can then also learn the normal pattern of communication from the device.
  • Next he replaces the data from your device to send his own. This can give the appearance that a patient who is sick is now health (or vice versa) or that your house is not being broken into (allowing his partners to break in). The hacker can even intercept your over-the-air commands and download programmable software or send commands to shut-down devices.

This work is technically hard, but doable with software downloadable on the Internet. If communication between your IoT devices and your secured (and encrypted), the hacker would have to gain enough permissions to get a hold of your SSL certificates and hijack DNS (if he has this, you are in a lot of trouble already). However, if the communication between your IoT devices and servers is not secured, a hacker can conduct this MITM attack from anywhere. By the time you learn about it, the damage will be long done.

Thankfully, there are many solutions available in the IoT domain that provide both strong encryption and secured sessions (plugging Security Holes #1 and #2):

  • If you are using standard “Internet of Servers” protocols, simply installing a full compliment of certificates will enable you to use SSL over TLS for HTTPS and FTPS (but not SFTP).
  • If you are using MQTT (one of my favorites), there are many brokers available that also provide SSL over TLS.
  • If you are using CoAP (which rides over UDP), you can use DTLS.
  • If your devices have edge constellations, you can turn on Bluetooth Security Mode 4 and get SSL with the same Elliptic Curve Diffie-Hellman secret key exchange used by the NSA.
  • You can even download and borrow the wonderful MTproto protocol designed by the folks over at Telegram (it is designed for low-powered, lossy, distributed communication).

None of these solutions are perfect. However, all reduce security risks significantly. Furthermore, all are evolving in the open source community as people find new vulnerabilities. Why more people do not use them is puzzling.

Security Hole #3: Not Protecting Against Buffer Overflow

When a hacker triggers a Buffer Overflow vulnerability, she typically causes a program to do two things: dump critical data and crash.

The first documented cases of Buffer Overview exploits data back to 1972. As more and more computers were connected to the Internet, these attacks became more pervasive. Fifteen years ago, Code Red highlighted to much of the general public what a Buffer Overflow exploit can do.

Over the past few years, application framework libraries have and higher-level languages, have added many defensive programming protection to make these vulnerabilities less prevalent than they were in the past. (As anyone who has encountered an awlful error page that shows you a stack trace error, these defenses are still far-from-perfect). Nevertheless, they have plugged many holes.

However, IoT devices are bringing this vulnerability back into the mainstream again. As most IoT devices operate with far less memory and CPU than expensive devices like your laptop or smartphone, their firmware and applications are primarily written in lower level programing languages. It is much easier to trigger buffer overflows in these languages than more forgiving higher level languages. Exception handling libraries are less robust. More often than not, memory management is handled using good old-fashioned C/C++ programming (there is no Garbage Collector to save you). This significantly raises the risk of buffer overflows in devices.

When buffer overflow crashes occur in the data center there is at least someone around to fix things. When they happen to a remote IoT device in the field, they can literally shut down a security or medical sensor. There is no IT or Ops department nearby to fix it. The device is shut down (at best, or bricked at worst). Essentially device is dead to world. Depending on what is was responsible for, lots real-world physical damage can ensure.

Devices that maintain continuously open Internet connections (like all those connected baby monitors) are especially prone to buffer flow attacks as remote hackers can discover them using port-scanning software. However, even industrial IoT devices that only pull commands and programs down over-the-air are vulnerable to MITM attacks that can shut them down by flooding data to the device (this reinforces the need to plug Security Holes #1 and #2 discussed above).

The fix to this problem is fairly clear:  implement defensive programming and test it aggressively. Today’s automation technologies for continuous integration and delivery make this a much easier and trustworthy process than it was even a decade ago.

Security Hole #4: Weak Systems Engineering

The fourth big security hole I commonly see spans the intersection of technical design, system processes, and human behavior. It essentially boils down to this: if you use flawless technology in ways that it is not intended, you can create big vulnerabilities. If I design perfectly secure medical device but put it on the wrong patient (accidentally or maliciously), I will prevent capture of data about that sensor. If someone who installs the security sensors in my house sets my account up to call their cell phone (and not mine), they can break in while I am gone and I trick the company into thinking it is a false alarm.

The way around this is to design IoT devices that work when things (humans, the network, servers, etc.) fail.

  • Build in redundancy (devices, network paths and servers) to mitigate technical failures
  • Build in positive and negative feedback looks to mitigate human failures. For example, I should not just be notified if my home security sensor goes off. I should should be notified if my smartphone and my security companies servers both cannot communicate with my home security IoT devices.

Plugging this systems engineering IoT security hole takes a combination of technology engineering and business process design.  This is a natural fit to the enterprise, where IoT can be used as a component of business transformation. In the consumer segment the answer is usually an ecosystem solution. Amazon’s and Google’s solutions stand out regarding robustness and security.


The Internet of Things offers great potential to transform how we work and live by removing many tedious tasks from our day-to-day activities. Making this a reality requires a secure Internet of Things. We will never make security perfect. However, we have the tools to make it trustworthy. What is needed is just the discipline to include them as we build new IoT devices, systems and processes.