Tag Archives: machine vision

Planning for a post-work future

It’s that time of year where many bloggers make their predictions for next year. Rather than do that, I wanted to look a generation out, when those who entering college today send their children to college, and think about the events of automation on our future. This is not a prediction per se. Instead it is more of a RFI (a request for ideas).

As a caveat, I work in automation (machine intelligence and machine learning, sensor and computer vision, automated controls and planning systems). I also have a prior background in policy—one that is driving me to think about the bigger picture of automation. However, this post is not about the work I am doing now. It is about the near-term “practical realities” I can imagine.

We are at the onset of an “Automaton Renaissance.” Five years ago, most people outside of tech thought about self-driving cars as something from the Jetsons. Last week, the governor of Michigan signed a bill allowing live testing of self-driving cars without human testers. Chatbots are not just the stuff of start-ups. Last month, I attended a conference where Fortune-500, large-cap companies were sharing results of pilots to replace back office help desks and call centers with chatbots. Two weeks ago, I was at a supply chain conference where we discussed initial pilots that a replacing planning and decision-making with machine learning (pilots involving billions of dollars of shipments). Automation is not coming—it is here already, and accelerating. Last week, I was at a conference for advanced manufacturing, we the speakers discussed the current and future impacts (good and bad) on jobs in the US.

So what will life (and work) be like in 20 years? Here are just a few things that we already have the technology to do today (what is left is the less-futuristic problems of mass production, rollout, support and adoption):

  • If you live in the city and suburbs, you will not need to own a car. Instead you can call an on-demand autonomous car to pick you up. No direct insurance costs, no auto loans, less traffic and pollution. In fact the cars will tell Public Works about detected potholes (street light and infrastructure sensors will tell Public Works when maintenance is needed).
  • If you work in a manufacturing plant, you will have fewer workers who are monitoring and coordinated advanced manufacturing (automation + additives). The parts will have higher durability and fewer component suppliers—also a reduction in delays, cost and pollution.
  • If you work on a farm you will demonstrate (supervised learning) to drones how you want plants pruned and picked, holes dug, etc. These drones will reduce back-breaking labor, reduce accidents and automatically provide complete traceability of the food supply chain (likely via Block Chain)
  • If you do data entry or transcription, your work will be replaced with everything from voice recognition-based entry, to Block Chain-secured data exchange, to automated data translation (like the team is doing at Tamr)
  • 95% of call centers will be chatbots. Waiting for an agent will be eliminated (as well as large, power-hungry call centers). The remaining 5% of jobs will be human handling escalation of things the machines cannot.

These are just five examples. They are all “good outcomes” in terms of saving work, increasing quantity and quality of output, reducing cost (and price), and even improving the environment. (If you are worried about the impact of computing on energy, look at what Google is doing with making computing greener.)

However, they will all radically change the jobs picture worldwide. Yes, they will create new, more knowledge-oriented jobs. Nevertheless, they will greatly reduce the number of other jobs. Ultimately, I believe we will have fewer net jobs overall. This is the “post-work future” — actually a “post-labor future”, a term that sounds a bit too political. What do we do about that?

We could ban automation. However any company or country that embraces it will gain such economic advantage that it will force others to eventually adopt automation. The smarter answer is to begin planning for an automation-enhanced future. The way I see it, our potential outcomes fall between the following two extremes:

  1. The “Gene Roddenberry” Outcome: After eliminating the Three D’s (dirt, danger, and drudgery) and using automation to reduce cost and increase quantity, we free up much capacity for people to explore more creative outcomes. We still have knowledge-based jobs (medicine, engineering, planning). However, more people can spend time on art, literature, etc. This is the ideal future.
  2. The “Haves vs. Have Nots” Outcome: Knowledge workers and the affluent do incredibly well. Others are left out. We have the resources (thanks to higher productivity) but we wind up directing this to programs that essentially consign many people to living “on the dole” as it was called when I lived in the UK. While this is humane, it omits the creative ideas and contributions of whole blocks of our population. This is a bad future.

Crafting where we will be in 20 years is not just an exercise in public policy. It will require changes in how we think and talk about education, technology, jobs, entitlement programs, etc. Thinking about this often keeps me up at night. To be successful, we will need to do this across all of society (urban and rural, pre-school through retirement age, across all incomes and education levels, across all countries and political parties).

Regardless of what we do, we need to get started now. Automation is accelerating. Guess how many autonomous vehicles will be on the roads in the US alone by 2020 (virtually three years from now):

10 million

Note: The above image is labeled for re-use by Racontour. Read more on the post-work word at The Atlantic magazine and Aeon magazine.

Bringing Machine Vision to Olympic Judging

If you’re like me, your favorite part of the Olympics is watching athletes from all over the world come together and compete to see who is the best. For many situations it is easy to clearly determine who is the best. The team that scores the most goals wins at Football (a.k.a. Soccer). The person who crosses the finish line first wins the 100-meter Dash. The swimmer who touches the wall first wins the 200-meter Breaststroke.

Victims of Human Error (and Bias)

However, is some cases, determining what happened is less clear. All of these cases involve subjective human judgment. I am not just talking about judgment regarding stylistic components; I am talking about judgment on absolute principles of scoring and penalties. As a result, athletes (who have trained for tens of thousands of hours over years of their lives) are often at the mercy of human judgment of motions that are almost to fast to observe. A few examples:

  1. A sprinter can be disqualified if she or he kicks off the starting blocks before the sound of the starting gun could potentially reach him or her
  2. A boxer may miss a point because he punches and connects too quickly
  3. A diver or gymnast can receive unwarranted penalties (or conversely, not receive warranted ones) because human judges misperceive the smallest of angles during an movement that takes just a few seconds

Even worse, athletes in these situations are not only subject to human error, they are often subject to human bias as well. We have all seen countless questionable judgment calls based on national or political bias in too many events. As upsetting as these are to the spectator they are utterly heart breaking for the athletes involved.

Bringing Machine Intelligence to the Rescue

We already use technology to aid in places where events happen to quickly for humans to accurately perceive them. In racing (humans to horses, on land or water), we use photo-finish cameras to resolve which athlete has actually one when a finish is too close (or as happened this year, when there is actually a tie for the Gold Medal). In Gymnastics and Skating we allow judges to review slow motion cameras as part of their judging. In Fencing, we go one step further and equip athletes with electronic sensors to measure when a blade has touched a target area (or which touched first to resolve simultaneous touches).

It is time to go a few steps further and actually bring machine intelligence (machine vision + machine learning) to the stage to provide the same absolute scoring that photo-finish cameras bring. I am not advocating using machines to replace people for stylistic judging. However, it makes absolutely no sense to not use machines to detect and score absolutes such as:

  • A gymnast’s bent arms, separated knees or mixed tempo
  • Level of differentiation of a diver’s twist from 90°
  • The actual time a sprinter kicks off the blocks based a microphone’s detection of when the sound arrived
  • Detection of a skater’s under-rotated jump

Not only would this significantly reduce bias and error. It would actually be a great training tool. Just as advanced athletes today use sensors to measure performance and conditioning, they could use a similar approach to detect small errors and work to eliminate them earlier in training.

This is Now Possible

Just a few years ago, this was the stuff to science fiction. Today it is feasible. Half a dozen companies have developer self-driving cars equipped with sensors and machine learning programs to deal with conditions with much higher levels of variability than judging a 10-meter dive or Balance Beam program. However, one does not need to equip arenas with multiple cameras and LIDAR arrays. Researchers at DARPA have even moved down the direction of teaching robots to cook by having them review two-dimensional YouTube videos.

Similar approaches could be uses for “Scoring Computers.” If we wanted to go down the path of letting computer see exactly (and only) what humans can see we can go down the machine-learning route. First program the rules for scores and penalties. Then create training sets with identified scores and infractions to train a computer to detect penalties and score them as a judge would do—but with the aid of slow motion review in a laboratory without the pressure of on-the-spot judging on live TV. This would not remove the human, it would just let the human teach a computer to do something with higher accuracy and speed than a person could do in real-time.

If we wanted to go a step further, just as Fencing has done. We can add sensors to mix. A LIDAR array could measure exact motion (actually measuring that bent knee or over-rotation). Motion- capture (mo-cap) would make this accuracy even better. Both would also create amazing advanced sports training technology.

It’s More Feasible Then You May Think

All of this technology sounds pretty expensive: computers, sensors, data capture, programming, testing, verification, deployment, etc. However, it is not nearly as expensive and “sci-fi-ish” as one might think (or fear).

Tens of thousands of hours of video already exists to train computers to judge events (the same videos that judges, athletes and coaches review in training libraries—libraries even better than robo.watch). Computing time is getting cheaper every year thanks to Moore’s Law and public cloud computing. An abundant number of Open Source libraries for machine learning are available (some companies have opened proprietary libraries; others are offering Machine Learning-as-a-Service). There are now even low-cost LIDAR sensors available for less than $500 that can resolve distances of 1 cm or less (for $2,000 college programs and Tier I competitive venues can get sensors that resolve to 1 mm or less).

Given the millions of dollars poured into these sports (and the billions into transmission rights), it would not require an Apollo Program to build a pilot of this in time for the 2020 Olympics (or even 2018 Winter Olympics). Companies like Google and IBM likely donate some R&D to show off their capabilities. Universities like MIT, Carnegie Mellon, and Stanford are already putting millions of dollars in biomimetics, computer vision, and more. Even companies like ILM and Weta Digital might offer their mo-cap expertise as they would benefit from joint R&D. Thousands of engineers would likely jump in to help out via Kaggle Competitions and Hackathons as this would be really fun to create.

Some Interesting Side Benefits

There are benefits to technology outside of “just” providing more accurate judging and better training tools. This same technology could create amazing television that would enable spectators to better appreciate and understand these amazing sports. Yes, you could also add your Oculus Rift or similar AR technology to create some amazing immersive games (creating new sources of funding for organizations like the US Olympic Team or USA Gymnastics to help pay for athlete training).