Tag Archives: disruptive innovation

Why business owners should care about this thing called the Lambda Architecture

Updated on April 19, adding “Mapping this back to…” final section

In the past 25 years I have seen four things that really made me step back and say, “This changes everything.” The first was the browser (before that we got data from the Internet using news groups and anonymous FTP). The second was open source distribution (we could get whole architectures up in hours, not weeks or months). The third was App Stores (Amazon and Apple allowed us to distribute software with zero marginal cost). The most recent was the Lambda Architecture

Yep, it is that big.

If into a business owner or product manager who is into Big Data, data-driven decision-making, iterative A/B testing, machine learning-driven recommendation or any similar analytics application you have probably heard a passing reference about this thing called the Lambda Architecture. However, anyone digging in deeper immediately finds a menagerie of arcane terms that could only appeal to developer: Kafka, Storm, Spark, Cassandra, Elephant DB, Impala, Speed Layer, Batch Layer, Immutable Data Store, etc. This is unfortunate, because it obscures how disruptive of a change the Lambda Architecture represents. As a result, many people with decision-making authority to fund technology changes are missing out on something really big.

Life in the traditional architecture world

Traditional architectures are based on transactions. They force collection of data into formats required to complete a given transaction (i.e., I need to collect N fields of information to process sale of an item). In addition, traditional architectures enable data to be changed: I can update my profile, update my shopping cart, update my order status, etc. This makes perfect sense if your object is to complete a transaction.

But what if I want to understand more about who buys what, who is doing what, or often more importantly what leads something to happen (or not happen)? I cannot get this from the transaction data but instead have to perform “data archaeology” stitching multiple sources of data together to create what happened just before and after the transaction. If I am lucky, I have all this data. However, more often than not I need to engage in development efforts to: collect more data at the time of transaction, log more info, pull it into a data warehouse, change my reports, then dig in to see if I can figure things out. This not only takes much time and effort; it is also a ripe source of errors.

Lambda flips how we view data on its head

The Lambda Architecture starts with an entirely different premise: that it is impossible to understand today all the future uses and interpretations we will need from our data.

This is not just a platitude. It is underlying philosophy that the value of data comes from the ability to ask it to answer as many questions for you that would every want to ask. This drives entirely different approaches to how data is captured, stored, interpreted—and most importantly of all—continuously reinterpreted as you learn and discover more about your company, customers and operations:

  • First data is preserved in its original form and never changed or destroyed. This lets you look at any piece of data at any point in time and factor in changes over time. For example, you could re-segment your customers every year, quarter, or even day as you learn new patterns.
  • Second data is not forced into arbitrary formats (i.e., schemas) but is preserved raw as you may want to go back and gleam different elements. For example you could later realize a variable such as source IP address of a customer visit to your site may entirely change how you measure, interpret and react to customers from this address
  • Third data is engineered to allow it to be easily reinterpreted as you learn more. This does not just focus on making reinterpretation fast; it also makes reinterpretation fault-tolerant (i.e., easy to correct in the event of a bug—without any loss of information)
  • Finally it allows all of this in real-time with two points of view: a just-in-time view and the deep cross-sectional view (both of which are always current). This lets you make decisions quickly without sacrificing the 100% loss-less accuracy needed for important business areas (such as finance, medicine, or mission-critical operations).

Once you have these capabilities, the things you can do with data—quickly and at scale—are pretty amazing. I will share some of these in future posts, as I want to keep this post short.

However, I will close this post out with a simple analogy…

“Think Like I Chef” vs. the Fast Food Menu

Traditional architectures are like fast food menus. You have these options. If you want to change the menu, we can do some market research, see what works and rollout a new menu. If you want to change again (or explore “what if we had done this?”) we can repeat this process.

Lambda architecture is like the pantry of a great chef. You have all these ingredients. If you feel like duck à l’orange, we can make this. If you want a duck confit salad, we can re-purpose the ingredients. If you want really rich potatoes, we can render the fat and cook the potatoes in it. If you want vegan, we can pull other items out of the pantry and make something else. There are so many more options.

Mapping This Back to Things Business People Care About

So what does this mean for your business? Do you remember the last time heard these comments:

  • “You’ll see that report. It will be in our Data Warehouse–tomorrow around 10am.”
  • “Oh, that’s in our warehouse. We can build a program to convert and and load the data into production. It will only take 3 weeks. Can you submit your TPS form to the Steering Committee so we can prioritize this?”
  • “Gee, it’s too bad we did not capture that data. We can start to capture it now. In a few months we can start analyzing it.”

With Lambda, all of these comments–and many more–go away. Data is never thrown away. It is always in production, ready to be used–for analysis or real-time transactions. There is no delay between transactional use and analysis–data flows down both paths as once.

Just imagine what problems you can solve when these limitations go away.

Unboxing Google Glass

After a second try, I finally got into the Google Glass Explorer program. I tried last year but waited too long to apply (about 36 hours after the application process opened). This time, I moved faster.

As not a lot of people have the opportunity to get to use Glass (I am lucky, my employer is paying for me to explore its use for M2M and IoT), I thought I would share my initial experiences getting—and unboxing—Glass to help those considering entering the program later.

The First Step: Registering for the Explorer Program

Registering for and buying Glass is a bit different, so I thought I would start here. It turns out you will need to link Glass to a Gmail account. As such, I strongly recommend using a Gmail account when you apply to the program. I think Google should add these instructions in the registration process, perhaps if it detects your email is not one that they manage at Gmail or Google Apps.

Application Approval

GlassProgramSmallMy application got approved about 10 days later, via email. The email contained a sixteen-digit alpha purchase code that was very obviously place. It also contains a 16-digit numeric unique ID in much smaller font in the email footer. Keep track of this, as you will need to enter it if you call the Glass Help Center.

I made the mistake of using a corporate email account (one not based on Google Apps). This created a bit of a problem for me, one that required a call to the Glass Help Center to resolve. I can say that the Glass Help Center staff are quite friendly and responsive. Working with them is more akin to a call with a Professional Services team than a call to a typical call center or corporate IT help desk.

Purchasing Glass

It turns out that you can only buy Glass with Google Wallet. As such, your experience will be much easier if you 1) apply with a Gmail account and 2) have a Google Wallet for this account set up in advance. If so, you need only click on the Get Glass URL and proceed. You will be prompted to confirm which Gmail account you want to use, then re-authenticate to Wallet to make your purchase. The entire process should take less than five clicks and one password to complete.

I did it a bit backwards. As the Glass Help Center let me know I would need to have Wallet setup, I was able to log into my Gmail, register for Wallet and add a Payment Method before re-starting my purchase. Once I go my purchase reference code reset, I was able to go through this process pretty quickly (it would have been less fun to stop, setup Wallet, then re-start).

I chose to purchase basic Glass (I picked the Shale color). I did have the option of a few Hipster-like frames that could support prescription lenses. However, I do not wear glasses so I went for the minimalist—and least expensive—options. I did get the free–detachable–Active Shades (essentially Terminator-style sunglass shade). Get these. They are incredibly useful if you are looking at the viewer screen in bright sunlight (a rarity in Boston).


Shipping and Delivery

The time from purchase to delivery was amazingly fast: I purchased around 11am, got an email notice that Google was handing off my purchase to UPS around five hours later, and received the package in Boston the next morning by 10am. This next-day shipping was included in the $1,500 price.

What arrives will be a four-pound box about 2x the width and 1.5x the length of a shoebox. Coincidentally, my Glass came on the same day as my new Nexus 7. However as the Nexus was uncharged, I used my iPhone to take all of the following photos:


Unboxing Glass

I waited until the end of the workday to open the box (I admit the engineer in me wanted to start right away).

Upon opening the UPS box and packing I was presented with a white box with a San Serif Glass logo and XE on the side. The packaging was very similar to what you would see with a high-end product (akin to Lytro and first iPad, but a bit nicer). The back of the box is black (so is the inner cover):


The Archive Shades are in their own box (and could be shipped separated based on my initial email receipt). They come in their own “Glass” branded felt sleeve:


Opening the Glass box reveals a translucent paper screen cover (mysterious?!):


This cover easily comes off, revealing the Glass with the only written instructions for use that comes in the packaging:


Lifting this off reveals an interesting felt pouch with an armored base—yes, an armored felt pouch. As the ticket explains, this is intended to protect your Glass when you pack it away (the armored shell makes the pouch 1.75” deep:


Underneath this pouch is a black card with your ear bud:


And underneath this is your USB 3.0 cable with detachable electric plug. The cord appears to be 36” long:


Also included are some replacement nose pads and a funny FAQ. One example:

Q: Can I use Glass while operating a jackhammer?
A: Use caution.


Charging Glass

When I plugged Glass into charge, it automatically booted up without me pressing the ON button. This can also take up to 30 seconds at times. However, it charges rather quickly (about the same speed that a smartphone charges, much faster than a tablet does).

Setting Up Glass

IMG_1875You will need either an Android or iOS phone or tablet to setup Glass as you will need to install the MyGlass App (iTunes version, Google Play version). I chose to use my iPhone as I did not want to walk around with an Android Tablet in my hand and Google Glass on my face. However, I may pair the phone to my Tablet as well as I experiment with Glass a bit more.

I definitely recommend you turn BlueTooth and your Personal HotSpot on BEFORE launching the MyGlass App and starting the pairing process. As I learned first-hand, it will save you the mess of aborting the process, turning these on (I keep them off to save power), and re-starting the process. I would recommend Google improve the App to detect these settings and notify you to exit and turn them before continuing to device pairing (unfortunately, iOS now forbids apps from turning these settings on for you—a now-needed security precaution in today’s world).

Unfortunately I could not take photos through Glass while I was setting it up (not unexpected). You can see videos of the setup process here, on the Glass YouTube Channel. I admit that setting up Glass created the opportunity for me to imitate Fred Armisen’s infamous Glass skit on SNL. I was very glad I could do this in the privacy of my house, doing it in the office would have created more than a few laughs.

Using Glass

I will now spend the next week playing around with Glass to fully understand the UX before I start thinking about how I would designing how an Glass app would work. However, I can say from my first hour of using Glass that it IS a very different experience, one that takes some getting accustomed to. I want to try to remember this experience so I can design applications that will be immediately useable from Day 1. (I did try to see if I could get actions of Glass to trigger IFTTT–alas, there are no Glass triggers yet).

This does not surprise me. I consider Glass on of Clayton Christensen’s classic disruptive innovations. While it is behind in some areas (the camera is not as good as a standard smartphone, usability is still a work in progress), it provides other capabilities nothing else does.

Final Note: Commercial Applicability of Glass

I know many people think Glass is not a commercially viable product. Some cite price point, others appearance, others limited availability. However, I believe that coupling of SDK from the people who brought us the Operating System with fastest adoption in history with a wide range of capabilities (POV-based camera and microphone, hands-free operation and telephony, voice recognition and Internet access) opens the door to some very interesting augment reality-based applications.

I have always thought the most beneficial Glass apps would be those that mapped to real-life activities—but streamlining them by eliminating the need to use your hands to record information and augmenting them by capturing information from your direct POV and combining this with other information. This could be provide enough benefit to justify Glass’ not-insignificant cost in wide range of business situations, from capturing the vision of a contract artisan or craftsperson (my first idea) to a whole new set of ideas I am now exploring.

Glass is a trademark of Google Inc.