Tag Archives: architecture

Why business owners should care about this thing called the Lambda Architecture

Updated on April 19, adding “Mapping this back to…” final section

In the past 25 years I have seen four things that really made me step back and say, “This changes everything.” The first was the browser (before that we got data from the Internet using news groups and anonymous FTP). The second was open source distribution (we could get whole architectures up in hours, not weeks or months). The third was App Stores (Amazon and Apple allowed us to distribute software with zero marginal cost). The most recent was the Lambda Architecture

Yep, it is that big.

If into a business owner or product manager who is into Big Data, data-driven decision-making, iterative A/B testing, machine learning-driven recommendation or any similar analytics application you have probably heard a passing reference about this thing called the Lambda Architecture. However, anyone digging in deeper immediately finds a menagerie of arcane terms that could only appeal to developer: Kafka, Storm, Spark, Cassandra, Elephant DB, Impala, Speed Layer, Batch Layer, Immutable Data Store, etc. This is unfortunate, because it obscures how disruptive of a change the Lambda Architecture represents. As a result, many people with decision-making authority to fund technology changes are missing out on something really big.

Life in the traditional architecture world

Traditional architectures are based on transactions. They force collection of data into formats required to complete a given transaction (i.e., I need to collect N fields of information to process sale of an item). In addition, traditional architectures enable data to be changed: I can update my profile, update my shopping cart, update my order status, etc. This makes perfect sense if your object is to complete a transaction.

But what if I want to understand more about who buys what, who is doing what, or often more importantly what leads something to happen (or not happen)? I cannot get this from the transaction data but instead have to perform “data archaeology” stitching multiple sources of data together to create what happened just before and after the transaction. If I am lucky, I have all this data. However, more often than not I need to engage in development efforts to: collect more data at the time of transaction, log more info, pull it into a data warehouse, change my reports, then dig in to see if I can figure things out. This not only takes much time and effort; it is also a ripe source of errors.

Lambda flips how we view data on its head

The Lambda Architecture starts with an entirely different premise: that it is impossible to understand today all the future uses and interpretations we will need from our data.

This is not just a platitude. It is underlying philosophy that the value of data comes from the ability to ask it to answer as many questions for you that would every want to ask. This drives entirely different approaches to how data is captured, stored, interpreted—and most importantly of all—continuously reinterpreted as you learn and discover more about your company, customers and operations:

  • First data is preserved in its original form and never changed or destroyed. This lets you look at any piece of data at any point in time and factor in changes over time. For example, you could re-segment your customers every year, quarter, or even day as you learn new patterns.
  • Second data is not forced into arbitrary formats (i.e., schemas) but is preserved raw as you may want to go back and gleam different elements. For example you could later realize a variable such as source IP address of a customer visit to your site may entirely change how you measure, interpret and react to customers from this address
  • Third data is engineered to allow it to be easily reinterpreted as you learn more. This does not just focus on making reinterpretation fast; it also makes reinterpretation fault-tolerant (i.e., easy to correct in the event of a bug—without any loss of information)
  • Finally it allows all of this in real-time with two points of view: a just-in-time view and the deep cross-sectional view (both of which are always current). This lets you make decisions quickly without sacrificing the 100% loss-less accuracy needed for important business areas (such as finance, medicine, or mission-critical operations).

Once you have these capabilities, the things you can do with data—quickly and at scale—are pretty amazing. I will share some of these in future posts, as I want to keep this post short.

However, I will close this post out with a simple analogy…

“Think Like I Chef” vs. the Fast Food Menu

Traditional architectures are like fast food menus. You have these options. If you want to change the menu, we can do some market research, see what works and rollout a new menu. If you want to change again (or explore “what if we had done this?”) we can repeat this process.

Lambda architecture is like the pantry of a great chef. You have all these ingredients. If you feel like duck à l’orange, we can make this. If you want a duck confit salad, we can re-purpose the ingredients. If you want really rich potatoes, we can render the fat and cook the potatoes in it. If you want vegan, we can pull other items out of the pantry and make something else. There are so many more options.

Mapping This Back to Things Business People Care About

So what does this mean for your business? Do you remember the last time heard these comments:

  • “You’ll see that report. It will be in our Data Warehouse–tomorrow around 10am.”
  • “Oh, that’s in our warehouse. We can build a program to convert and and load the data into production. It will only take 3 weeks. Can you submit your TPS form to the Steering Committee so we can prioritize this?”
  • “Gee, it’s too bad we did not capture that data. We can start to capture it now. In a few months we can start analyzing it.”

With Lambda, all of these comments–and many more–go away. Data is never thrown away. It is always in production, ready to be used–for analysis or real-time transactions. There is no delay between transactional use and analysis–data flows down both paths as once.

Just imagine what problems you can solve when these limitations go away.

Building a Native Mobile App Experience—Without the App

Note: This post was adapted from the more developer-oriented “Piggybacking on backbone.js for performant mobile web”, originally published on Sawdust Software.

Native mobile apps are great. They provide a rich user interface (UI). They can easily access other core applications (e.g., camera, calendar, message center) on your smartphone to create stickier, more immersive experiences. They routinely provide a much faster user experience (UX) in terms of menu traversal and screen-to-screen navigation.

However these benefits come at a cost. They require lots of work: learning, developing, and testing on an entirely different tech stack (most likely iOS or Android). They bind you to navigate a proprietary app store release and submission process (one far slower than the ability to release software on your own web farms). Worse, these are not one-time costs. They can double the cost of adding new features (as you have to develop these for web and native mobile). The also introduce a new mobile OS compliance cost: keeping your app up-to-date with new mobile OS releases—while supporting users on older mobile OS versions—can routinely add 30% to your costs of mobile development and testing.

In some situations, the benefits of using a native mobile app far outweigh the costs; in others they do not. Luckily, there are many technologies and approaches available when you want to create a mobile web experience and do not require (or want to) development of a native mobile app:

Create mobile web-optimized templates

The responsive web design (RWD) movement has gained much traction in the design and development community. While it is appealing to create universal web templates that can detect and respond to the size of the browser screen, this is not always ideal for mobile web. More often than not, universal web templates can have quite large file sizes. Downloading these over a 3G connection can be quite slow. Worse, they can burn battery life on your customer’s phones, leading them to not use your site on mobile.

When considering functionality for mobile pay attention to size of the pages you need users to download. If the desktop (or ever increasingly Tablet-plus-WiFi) versions of your HTML templates are large, create mobile-only templates that are smaller and optimized for download and display on smartphones. If you are on a standard web framework, you have a range of plug-ins you can add to easily detect then customers are accessing your site with a mobile device and serve these templates (e.g., Django Mobile, User Agent (for Rails), Spring Mobile Device Module). You can even use these to serve up a different experience for mobile tablet (vs. mobile smartphone) users.

Use HTML5 and jQuery Mobile to emulate native mobile UIs

Thanks to the rise of mobile and responsive web design, there are now many libraries available to develop lightweight (i.e., small file size) touch-responsive web interfaces using HTML5. Some great examples include: JQuery mobile (if you are a jQuery purist), jQT (if you have a Sass shop), and Zepto (if your really want super fast performance).

When you combine these with mobile browser detection (see above), you can serve up HTML interfaces that use widgets that are virtually indistinguishable from their native mobile counterparts to 80% of your customers:

Entirely HTML5, Sass and jQuery mobile web--with emulated iOS widgets (www.custommade.com)
Entirely HTML5, Sass and jQuery mobile web–with emulated iOS widgets (www.custommade.com)

It is worth pointing out that using these libraries not only lets you emulate a mobile app experience, using them also keeps your teams on a single technology stack. Developers building web UIs can easily switch over and develop mobile web equivalents (or vice versa). This makes life much more interesting for developers (they can do more, more easily) and management (people can do more and switch to whatever is most important more easily).

Use backbone.js to load navigation control into the browser

One of the reasons that native mobile apps are able to navigate (between menus and moving from screen to screen) so quickly is that they embed a model-view-controller (MVC) architecture directly into the app installed on the mobile device. In an MVC architecture, the Controller directs what Views (screens, menu elements) to display when a user touches (or clicks) on something. On native mobile apps, the Controller is located on the device itself (an is accessible virtually instantly). This is very different than “traditional” browser based web-applications, where the View Controller is hundreds (or thousands) of miles away on a web server—something made even worse when a smartphone is trying to ping a server over a 3G (or worse) mobile connection.

However, over the past 2-3 years, several JavaScript (JS) frameworks have been developed that solve this problem. All of these frameworks were created to simplify development of Single-page browser Applications (SPAs): rich web applications that approximate the interactivity of traditional “thick client” applications. They do this by loading MVC (or MVC-like) frameworks into the browser. This not allows browser-based apps to display updates without action by the end user; it also provides the secondary (and critical) benefit for mobile web apps of moving the View Controller from a remote server into mobile device in the hands of your customer:

Adding a SPA-oriented JS framework like Angular, Backbone or Knockout embeds a MVC architecture in the hands of your customer, just like a native mobile app does.
Adding a SPA-oriented JS framework like Angular, Backbone or Knockout embeds a MVC architecture in the hands of your customer, just like a native mobile app does.

As result web apps on a smartphone, tablet or desktop can be just as fast as native mobile OS apps. Several JS frameworks are available to achieve this. Google first developed Angular.js in 2009. Steve Sanderson of Microsoft developed Knockout.js a year later. Jeremy Ashkenas also developed Backbone.js—a favorite of many eCommerce and consumer web companies—in 2010.

Caveat

This technology stack does not allow for completely offline operation, something that is possible to achieve with a native mobile application. If you need to support offline operation (for extended periods of time) without losing any data, you will still need to build a native mobile app with an embedded SQLite database. However, that is not the use case more the vast majority of mobile apps.

Credits & Disclaimer

This author of this post used to lead technology at CustomMade Ventures where we have used HTML5, Sass, jQuery Mobile, Backbone.js and Django Mobile to create a mobile web experience for ideation and collaboration for our two-sided marketplace. Translation of this architecture into reality is the work of two great members of our Engineering team: Brendan Smith and Mike Manning.