Tag Archives: drones

Wine Clones, Drones, and Behavioral Cloning: Heretical IoT Thoughts on Winemaking

In addition to technology, I love cooking and craftsmanship. That naturally leads to an interest and wine, and especially, the winemaking process. Over the past few years, I have been lucky to meet a few great winemakers and discuss how they work their craft. This gave me an opportunity to learn not only how skilled they are, but also the incredible amount of work they and their teams do. They routinely begin their work at 3am and work past sunset. While much of this work is repetitive, the majority of it requires continuous application of expert judgment. The more I spoke to them the engineer in me started to think, “What could make their work easier without sacrificing their expertise?” That led me to this post.

Caveat: This post explores use of automation and IoT to augment (not replace) work by humans. The idea is not to replace people, but rather to take some of backbreaking labor out of their work and give them back more time with their families and friends. It is not intended to be as “heretical” as the idea may initially appear 😉

* * *

If you visit a winery and get to speak with the winemaker, he or she will talk about the process of routinely going out and doing things such as:

  • Inspecting the vines for damage, disease, and general health
  • Pruning grapes to concentrate resources to the very best
  • Pruning leaves to control the amount of sun grape bunches get

Watching someone do this on one vine is amazing (you can see the expertise in action). Watching them repeat alone a single row of vines starts to give the idea of how much work it takes to make great wine. Staying around all day to see this done across acres of grapes (and considering this is done throughout the season) drives home how we should respect everyone who works in a vineyard. It also gives an appreciation for the effect of this labor on their backs, knees, fingers, and eyesight. The goal of IoT is to reduce danger, dirtiness and drudgery. This is where my idea started.

Imagine doing this 10,000 times, year-in, year-out

Imagine this

The team goes out in the morning, before sunrise to tend their grapes. For this day, let’s assume they are pruning bunches of grapes to concentrate resources on the very best. Each person grabs their section of the vineyard and starts his or her work. However, they are each partnered with two drones. The first drone watches what each person does, recording which grapes are pruned. The other drone picks up discarded grapes for retrieval and composting.

After the expert finishes a few rows of grapes, the drones fly back, plug in, and upload their video images. The wait for a machine learning program to complete construction of a new model that they then upload. Then the drones go out and finish the work, based on what they have learned from that wine expert, for that section of grapes, for that location, for that day’s weather and solar conditions. Much backbreaking (and eye-straining) work is saved. Allowing the team to do the umpteen other activities that require their expertise and attention. However, today they may work “only” ten hours instead of sixteen.

This is not replacing people

No jobs are lost. People do an enormous amount of work at wineries. With this technology they might now work “only” ten hours instead of sixteen. Imagine the benefits to their health of this.

Furthermore, human knowledge and expertise are not replaced. Every plot of land and every day bring new variables. Every workday, the human shares expertise that is used to reduce repetitive work for that day. If you remove the human, you remove the expert. Eventually the winemaking would get worse and worse and the drones do not have sufficient expertise. The vineyard would suffer and lose out to others who apply expert knowledge every day. What is reduces is drudgery.

This is not far-fetched

A decade ago this would have been more Star Trek than reality. However, the advances of the last few years in technologies for drone, autonomous vehicle, and machine learning have made this achievable. Let’s look at a few

A photo taken on September 9, 2014 shows a drone flying over vineyards of the Pape Clement castle, belonging to Bordeaux winemaker Bernard Magrez in the southwestern French town of Pessac. Magrez is the first winemaker to have bought last February a drone equipped with a infrared camera to determine the optimal maturity of the domain’s grapes and thus harvest them at different times. AFP PHOTO JEAN PIERRE MULLER.
Drone and Autonomous Tech

Over the last few years, growth in drone technology and its supporting infrastructure has exploded. It will hit $12 billion per year in the next four years (comparison: US wineries sold $34 billion of product last year). Major “Blue Chip” companies are now exploring active programs to embed drones in their supply chain. Major consultancies are now using drones in numerous aspects of farming. Furthermore, regulations are now clear. Two years ago, it cost you over $2,000 to get certified as drone pilot. Today you can do this with $200.

Autonomous technology is growing as fast or faster. There will be 10+ million autonomous vehicles on the road by 2020. Autonomous vehicles are no longer just an idea in Silicon Valley. All “Big Three” US automotive manufacturers have autonomous vehicle programs. Even major shipping companies are now exploring combined drone and autonomous technology for cargo ships.

Computing and Machine Learning (ML) Infrastructure

Thanks to computer gaming, GPU costs have dropped dramatically. You can know get resources to build models on-demand and pay by the minute. The libraries to process imagery (e.g., OpenCV) and to build neural networks (e.g., TensorFlow, Keras) are advancing new major versions every 3-4 months. Some have even wrote that the “Object Detection Problem” is now solved (a great technical, but to technical write here). As one of my friends said to me this week—over a 2012 Tempranillo—“It’s a great time to be in tech.”

However, having the ML infrastructure is only the start. Building models takes lots of time, trial and error, and compute time. Luckily we now have models that recognize images with greater accuracy than humans. Using “Knowledge Transfer” we can start with these basic models and extend them to add new knowledge (here is an example of what Gilt did to extend Microsoft’s ReNet vision recognition model to detect clothing styles). Combine this with “Behavioral Cloning” (an approach widely used to teach autonomous vehicles how to drive; here is one example I have used) we can clone and graft winemaking knowledge to these existing models—just a winemaker grafts a wine clone to his or her vines.


The foundations are in place and costs are coming down drastically. Making this cost-feasible for small business is just a matter of time (and perhaps the startup to focus on it).

Finally, this does not eliminate nuance and individuality

One of the most enjoyable aspects of exploring wine and vineyards is seeing how each winemaker executes his or craft (just as going to new restaurants lets you explore how each chef interprets his or her craft). This technology does not remove the nuance and individuality of winemaking.

Even if every vineyard used this technology and started with the same baseline drones and machine learning models, they would each evolve differently. Thanks to behavioral cloning and knowledge transfer, each vineyards model would evolve weekly as they learn how the winemakers and his or her team applies their expertise day-by-day, wine block-by-wine block, year-by-year. These models really would not even be trade secrets that could be stolen as they would evolve to literally fit specific terriors—just as best practices in winemaking are.

Planning for a post-work future

It’s that time of year where many bloggers make their predictions for next year. Rather than do that, I wanted to look a generation out, when those who entering college today send their children to college, and think about the events of automation on our future. This is not a prediction per se. Instead it is more of a RFI (a request for ideas).

As a caveat, I work in automation (machine intelligence and machine learning, sensor and computer vision, automated controls and planning systems). I also have a prior background in policy—one that is driving me to think about the bigger picture of automation. However, this post is not about the work I am doing now. It is about the near-term “practical realities” I can imagine.

We are at the onset of an “Automaton Renaissance.” Five years ago, most people outside of tech thought about self-driving cars as something from the Jetsons. Last week, the governor of Michigan signed a bill allowing live testing of self-driving cars without human testers. Chatbots are not just the stuff of start-ups. Last month, I attended a conference where Fortune-500, large-cap companies were sharing results of pilots to replace back office help desks and call centers with chatbots. Two weeks ago, I was at a supply chain conference where we discussed initial pilots that a replacing planning and decision-making with machine learning (pilots involving billions of dollars of shipments). Automation is not coming—it is here already, and accelerating. Last week, I was at a conference for advanced manufacturing, we the speakers discussed the current and future impacts (good and bad) on jobs in the US.

So what will life (and work) be like in 20 years? Here are just a few things that we already have the technology to do today (what is left is the less-futuristic problems of mass production, rollout, support and adoption):

  • If you live in the city and suburbs, you will not need to own a car. Instead you can call an on-demand autonomous car to pick you up. No direct insurance costs, no auto loans, less traffic and pollution. In fact the cars will tell Public Works about detected potholes (street light and infrastructure sensors will tell Public Works when maintenance is needed).
  • If you work in a manufacturing plant, you will have fewer workers who are monitoring and coordinated advanced manufacturing (automation + additives). The parts will have higher durability and fewer component suppliers—also a reduction in delays, cost and pollution.
  • If you work on a farm you will demonstrate (supervised learning) to drones how you want plants pruned and picked, holes dug, etc. These drones will reduce back-breaking labor, reduce accidents and automatically provide complete traceability of the food supply chain (likely via Block Chain)
  • If you do data entry or transcription, your work will be replaced with everything from voice recognition-based entry, to Block Chain-secured data exchange, to automated data translation (like the team is doing at Tamr)
  • 95% of call centers will be chatbots. Waiting for an agent will be eliminated (as well as large, power-hungry call centers). The remaining 5% of jobs will be human handling escalation of things the machines cannot.

These are just five examples. They are all “good outcomes” in terms of saving work, increasing quantity and quality of output, reducing cost (and price), and even improving the environment. (If you are worried about the impact of computing on energy, look at what Google is doing with making computing greener.)

However, they will all radically change the jobs picture worldwide. Yes, they will create new, more knowledge-oriented jobs. Nevertheless, they will greatly reduce the number of other jobs. Ultimately, I believe we will have fewer net jobs overall. This is the “post-work future” — actually a “post-labor future”, a term that sounds a bit too political. What do we do about that?

We could ban automation. However any company or country that embraces it will gain such economic advantage that it will force others to eventually adopt automation. The smarter answer is to begin planning for an automation-enhanced future. The way I see it, our potential outcomes fall between the following two extremes:

  1. The “Gene Roddenberry” Outcome: After eliminating the Three D’s (dirt, danger, and drudgery) and using automation to reduce cost and increase quantity, we free up much capacity for people to explore more creative outcomes. We still have knowledge-based jobs (medicine, engineering, planning). However, more people can spend time on art, literature, etc. This is the ideal future.
  2. The “Haves vs. Have Nots” Outcome: Knowledge workers and the affluent do incredibly well. Others are left out. We have the resources (thanks to higher productivity) but we wind up directing this to programs that essentially consign many people to living “on the dole” as it was called when I lived in the UK. While this is humane, it omits the creative ideas and contributions of whole blocks of our population. This is a bad future.

Crafting where we will be in 20 years is not just an exercise in public policy. It will require changes in how we think and talk about education, technology, jobs, entitlement programs, etc. Thinking about this often keeps me up at night. To be successful, we will need to do this across all of society (urban and rural, pre-school through retirement age, across all incomes and education levels, across all countries and political parties).

Regardless of what we do, we need to get started now. Automation is accelerating. Guess how many autonomous vehicles will be on the roads in the US alone by 2020 (virtually three years from now):

10 million

Note: The above image is labeled for re-use by Racontour. Read more on the post-work word at The Atlantic magazine and Aeon magazine.