Tag Archives: eCommerce

If a tree falls in the woods and no one heard it, did it happen? Not in Streaming Analytics

Interest in “Streaming Analytics” has exploded over the past few years. The reasons are two-fold. First, the rise of the Internet of Things has made it possible for the first time ever to get data directly (and automatically) from infrastructure, cars, homes, factories and more—all without a human people ever having to do something. To put this in perspective, last quarter more new automobiles were connected to mobile networks than new cellphones were. Second, the technology is now readily available to implement streaming analytics at massive scales without needing to invent your own frameworks. Not one, but three technology projects (Storm, Spark, and Flink) are available for your choice. One of them, Apache Spark, is now the second-fastest growing open source project in history.

Streaming Analytics is a very fun field to be in (I have been in for 22 years—in the national security arena, eCommerce, med-tech, and now Industrial IoT). Taking in data faster than any human being could examine it and analyzing in near real-time to make split-second decisions creates provide omnipresent knowledge and enormous business value. However, Streaming Analytics presents a new challenge that does not exist in traditional After-the-fact Analytics:

You need to figure out how to make decisions on data that you do not know about yet—and may not ever find out about it time to make it worth your time.

Three real-world examples

To put it in philosophical parlance, how does anyone know if a tree in the forest fell down if no one ever sees (or hears) that it fell? As philosophical as this sounds, it can have multi-million-dollar impacts in the real world. Here are three examples:

Example 1: eCommerce Chatbot

My chatbot is engaged with a new prospective customer who may eligible—based on her mobile number—for our bank’s highest value credit card. Unfortunately, that data is delay in getting to my bot. As a result, at this point in time, I do not know whether the customer is: very valuable, average, or a credit risk. What does my chatbot do?

chartbot_400px

Example 2: Guaranteed Shipping

I have a booking to delivery high-value cargo to a customer site by end of business today. It is now 15 minutes after the day is over. I might be inclined to escalate to my carrier that the container has not arrived. However, at this point in time, I cannot tell if: the container arrived but the signal from the carrier is delayed getting to me or of the container did not arrive. What do I do?

container_3

Example 3: Infrastructure Security Monitoring

I run a cattle farm that is hundreds of thousands of acres. I have equipped all gates in my Smart Ranch with sensors to alert me if any are open (so I can prevent the cattle from getting away). The sensors send updates every 15 minutes. However, one of the gate sensors is a few minutes late. At this point in time, I do not know if the gate is open or closed. Does my system trigger an alert?

140647

What makes Streaming Analytics different

All of these challenges are based on lack of information. Lack of information is typical in analytics (as well as messy data, data gaps, corrupt data, duplicate data and many other issues). However, in the streaming analytics there is one critical difference: you will eventually have the data you need right now to answer your question. However, by the time you receive it, it will be too late to make your decision: the eCommerce customer will be gone; your freight contract will be honored or broken; the cattle will be safe or have gotten away.

What makes this especially different is that all the parties involved with your business will know the answer as well. If my chatbot fails to offer a valuable customer the best credit card, the line of business GM will ask why “it was so stupid.” If I call the customer up to tell them the freight has not arrived and they respond with “but it got here 10 minutes before closing”, I will look stupid. It all boils down to this:

56818465People may not know when After-the-fact Analytics miss a point; however, everyone will know that your Streaming Analytics made a mistake.

 

That can be stressful 😉

What’s a person to do?

The essential thing to remember when designing your Streaming Analytics solution is this:

Close enough and in-time is much more valuable than perfect and too late

This means you need to build your solution to make a decision based on the information available (rather than waiting until the critical moment has passed). The trick is determining what is “close enough”. The answer to that question depends on your business context. Specifically, given your context, is it better to accidentally do something you should not have (a Type I error) or is it better to not doing something you should have done (a Type II error).

Let’s looks at how this works in each of the three examples:

Example 1: eCommerce Chatbot

Our business context determines it is far worse to get a prospective customer excited about an offer that we cannot deliver instead of offering a less valuable package (i.e., we are Type II biased, something typical in ad-tech and eCommerce). We do not make the highest-value offer.

Depending on our Risk Policies we make the normal offer (one for which a majority of customers qualify) or shunt the customer to a slower process (email vs. chat) to wait time for the data to catch up (essentially shifting to batch). Most commerce companies have created default packages that allow the former action, allowing them to make more money in the “80% most likely case”. We could also apply a machine learning algorithm to guess the best alternative offer, maximizing revenue and minimizing the risk of an angry customer (or wasted time).

Example 2: Guaranteed Shipping

Our business context indicates that it does not make sense to alert that we are late if we do not know it (yet)—especially given the likelihood that this could result in some “egg on our face” when the customer asks why we did not know the container arrived 20 minutes ago. As a result, we do not alert we are late at 5:00pm. We make the call when we know for sure that the container was on time vs. late (i.e., when the delivery message actually arrives). This scenario is also Type II biased.

However, we do not want to expose ourselves to a completely irate customer in high-value circumstances. As such, we place a secondary streaming analytic in place: if we do not receive confirmation within more than 60 minutes from scheduled delivery we trigger an alert to reach out to our delivery carrier and find out the real status (i.e., by taking the expensive step of talking to a person vs. a sensor). We determined the “magic number” of 60 minutes by doing After-the-fact Analytics that determined waiting this long will automatically resolve the 80% of false positives while still giving us enough heads up to detect the true issues. If we are even smarter we can have our After-the-fact Analytics system automatically calculate the magic number to delay alerts based on location, time-of-day, day-of-week and other features.

Example 3: Infrastructure Security Monitoring

Our business context indicates that is not good to close the farm door after all the cattle got away. As such, we have programmed our Streaming Analytic system to alert us if the gate is opened (before a human has sent a “I am opening the gate” message) OR if we have not received confirmation that the gate is closed for period of longer than 15 minutes. Essentially we are Type I biased (not uncommon in safety and security situations).

Unfortunately this bias will result in lots of alerts. Essentially any time the sensor message is delay in the cell network our alarm will go off. Luckily, we have some more advanced analytic techniques to help with this. Namely, we can use a Lambda Architecture model that provides self-healing: the initial lack of confirmation that the gate is closed triggers an alert; the arrival of the delayed message that the gate WAS closed then cancels this alert (with a resolution message). This is still a bit chatty. However, it short-circuits false positives and prevents the need to send a worker (or a drone) all the way out to the gate to check if it is open.

Conclusion

Yes, Streaming Analytics is a harder than After-The-Fact Analytics. However, it the near real-time omnipresence (not omniscience) offers tremendous benefits. You just need to think in philosophical terms when designing your analytic rules.

entropy

Drone Commerce, Part 1: Same-day Delivery

As an aerospace engineer-turned-Internet software architect, it was probably only a matter of time before I wrote a post about the expanding use of Unmanned Aerial Vehicles (a.k.a. Drones). Now that we have two eCommerce giants entering the Drone Space makes it a good time as any to explore the practical viability of this from the point of view of someone who has built vehicles that fly, software that controls them, and large-scale eCommerce and data platforms.

In one corner we have Amazon’s exploration of drones for same-day delivery (interesting aside: Jeff Bezos, as owner of Blue Origin is also a commercial space entrepreneur). In the other corner with have Google’s recent purchase of Titan Aerospace for drone-based Internet service provision and Google Earth data capture (interesting aside: Google the ‘K’ in KML stands for Keyhole, something Google got when they bought Keyhole, Inc.—a company with a very interesting provenance to anyone who has worked in satellite technology for the IC).

Are these explorations of drone tech whacky uses of capital buy companies with more cash then they know what to do with or are they viable commercial pursuits with long, complex lead times? In part 1 of this series, I will look at the Amazon’s consideration of drones for same-day delivery. In part 2, I will look at Google’s ideas for Internet service provision and Google Earth data capture.

amazon-air

To use drones (on a repeated basis) for package delivery, you have to overcome many, many challenges, including very big aerospace-related ones:

  • Large payload without large size
  • Safe, efficient flying (and delivery) operation
  • FAA approval to fly in populated places

This leaves out other non-aerospace challenges, ones Amazon has solved very well, such as efficient logistics management, protecting drones against hackers, notifying customers of immediate delivery, and other classic delivery challenges.

Challenge 1: Large Payload without Large Size

To be profitable (from a gross margin perspective), you need to either be able to deliver a good amount of packages (think how many packages a UPS or FedEx truck carries). This requires payload capacity. However, to fly drones to where people live and work, you need small drones. It is not viable to fly (and land) a MQ-9 Reaper-sized drone a 20-meter wingspan on residential street or commercial rooftop. You will most likely need a drone with a 1-2 meter wingspan (or body-span for quadcopter drones). Unfortunately small drones do not have high carrying capacity:

Body Span or Wing Span Payload
1 m 1 – 10 kg
3 m 5 – 20 kg
5 m 20 kg – 100 kg
10 m 100 kg – 400 kg

Payload Capacity by Wingspan with Current Drone Technology

The cause of is simply the laws of physics (specifically flight kinematics). The more you carry, the heavier you are. The heavier you are, the larger your wingspan or propellers need to be to generate sufficient lift. The large heavier you are, and the larger your wingspan—the more fuel (or heavier batteries) you need—compounding the problem. This is a non-trivial problem to solve. It is the major reason we do not have the capability for more than sub-orbital commercial space flight or commercially viable hovercrafts people can ride in residential areas—even 45 years after the Moon landing. It is also a problem that exacerbated by the style of flying needed for repeated delivery of packages throughout the day.

Challenge 2: Safe, Efficient Flying Operation

To be a viable vehicle for delivery, drones have to safely take off, navigate through a complex three-dimensional space, and land—repeatedly throughout the day, day-in and day-out. As this is a big set of challenges, it is easier to look at each individually.

Takeoff and Landing: Takeoff and landing is a lot more challenging than simple level flight. First, take consumes a lot more than level flight—the energy penalty depends on you payload, wingspan and runway size, exacerbating the “payload vs. size” challenge discussed above. Second, when you are flying slowly at low altitudes (i.e., just as you are taking off or landing) you are much higher risk of crashing due to wind shear-induced lost of lift: when this happens at 36,000’ you get turbulence; when this happens at 36’ you count on the training and experience of your pilot to adjust rapidly to keep you from hitting the ground. Unfortunately, drones do not have pilots. As such, the round-trip communication to a remote operator is not fast enough for instant adjustments, drones typically rely on onboard software to make immediate corrections. However, this software is not handling the simple operation of landing a drone on a remote landing strip, it is managing landing in a city (perhaps on a building or perhaps in your driveway). This obstacle can be overcome with better software (and lots of machine learning). Nevertheless you still have the “energy penalty” of taking off and landing over and over again through the day.

Cruising Navigation: Drones that delivery goods are going to have navigate a complex three-dimensional space populated with buildings, power wires, antennae, birds, other drones, and weather (small size and repeated take off and landing are going to require them to fly well-below 12,000’, making them subject rain, hail, snow, lower visibility and much more turbulent airflow than one encounters at altitudes that planes need to reach before pilots let you get up and move around the cabin). After 20 years of military use, we have gotten pretty good at letting drones to this successfully. Unfortunately you are flying at an altitude that is very inefficient. As a result you still have the nagging “energy penalty” cited several times already.

Challenge 3: Getting FAA Approval

This is the one challenge on my list that is based on socio-political systems—rather than flight dynamics and physics. However, it is an especially big barrier to overcome, as Amazon would not be getting approval large-scale, complex drone operation: many drones taking off and landing, many times a day, in populated area instead of drone that cruise at high altitudes or are operate at low scale by hobbyists at parks.

The numbers that drive the scale for profitable operations make this challenge especially difficult. Today, the piloted planes have 9.4 accidents per million flights (statistically very safe). Let’s use this to run some numbers:

  • If drones are as safe as planes and I have only five drones operating in each of the 50 largest cities and they are only doing 10 deliveries each a day (not very efficient vs. UPS), I will see a drone crash every 40 days
  • If drones are a bit less safe (likely as they fly at low altitude, take and land very often in complex environment, and do not the support of air traffic controllers and highly-trained pilots on board), I will see a crash every month.
  • These are not “landed hard and broken the container” crashes. They are “collided with building, power wire, tree or other obstacle”-type and “got flipped over in the crosswind and “dropped several stories into a street or rooftop”-type crashes.

What would this look like in the aviation regulatory space? The first crash would lead to a fleet grounding, NTSB investigation, and perhaps some hearings. A second crash after the first set of issues are addressed would lead to an even longer grounding and likely a change in allowed places of drone operation—especially if a person was hurt. At best, this would make commercial operation low margin; at worst, a big drag on the company’s reputation, stock price and liability.

Conclusion

A hate to be a naysayer—especially as a person who went to school to build planes, rockets and satellites that would give us the ability to travel more places, faster and more conveniently. However, it is hard to be profitable while fighting the laws of physics in complex conditions. Aviation technology can be improved, but generally not at the same rate as Moore’s Law (precisely because you are dealing with objects with much more mass than electrons and photons). As such, I do not see drone-based delivery being profitable—especially given the low-cost and high-efficiency of ground-based delivery (something that is going to get even better as the Internet of Things makes fleet management more efficient and Amazon’s machine learning lets them pre-position items before you even order them).

Jeff Bezos is a very, very smart man. It is my guess that his work in drone technology is not really focused on a better goods delivery mousetrap but instead something else that can scale at lower cost and higher efficiency. If I had to guess, I would say it would be related to streaming content or using peer-to-peer networking to bypassing carrier restrictions. That’s more of a topic or my next post in this series.

Updates: