Tag Archives: NoSQL

Why business owners should care about this thing called the Lambda Architecture

Updated on April 19, adding “Mapping this back to…” final section

In the past 25 years I have seen four things that really made me step back and say, “This changes everything.” The first was the browser (before that we got data from the Internet using news groups and anonymous FTP). The second was open source distribution (we could get whole architectures up in hours, not weeks or months). The third was App Stores (Amazon and Apple allowed us to distribute software with zero marginal cost). The most recent was the Lambda Architecture

Yep, it is that big.

If into a business owner or product manager who is into Big Data, data-driven decision-making, iterative A/B testing, machine learning-driven recommendation or any similar analytics application you have probably heard a passing reference about this thing called the Lambda Architecture. However, anyone digging in deeper immediately finds a menagerie of arcane terms that could only appeal to developer: Kafka, Storm, Spark, Cassandra, Elephant DB, Impala, Speed Layer, Batch Layer, Immutable Data Store, etc. This is unfortunate, because it obscures how disruptive of a change the Lambda Architecture represents. As a result, many people with decision-making authority to fund technology changes are missing out on something really big.

Life in the traditional architecture world

Traditional architectures are based on transactions. They force collection of data into formats required to complete a given transaction (i.e., I need to collect N fields of information to process sale of an item). In addition, traditional architectures enable data to be changed: I can update my profile, update my shopping cart, update my order status, etc. This makes perfect sense if your object is to complete a transaction.

But what if I want to understand more about who buys what, who is doing what, or often more importantly what leads something to happen (or not happen)? I cannot get this from the transaction data but instead have to perform “data archaeology” stitching multiple sources of data together to create what happened just before and after the transaction. If I am lucky, I have all this data. However, more often than not I need to engage in development efforts to: collect more data at the time of transaction, log more info, pull it into a data warehouse, change my reports, then dig in to see if I can figure things out. This not only takes much time and effort; it is also a ripe source of errors.

Lambda flips how we view data on its head

The Lambda Architecture starts with an entirely different premise: that it is impossible to understand today all the future uses and interpretations we will need from our data.

This is not just a platitude. It is underlying philosophy that the value of data comes from the ability to ask it to answer as many questions for you that would every want to ask. This drives entirely different approaches to how data is captured, stored, interpreted—and most importantly of all—continuously reinterpreted as you learn and discover more about your company, customers and operations:

  • First data is preserved in its original form and never changed or destroyed. This lets you look at any piece of data at any point in time and factor in changes over time. For example, you could re-segment your customers every year, quarter, or even day as you learn new patterns.
  • Second data is not forced into arbitrary formats (i.e., schemas) but is preserved raw as you may want to go back and gleam different elements. For example you could later realize a variable such as source IP address of a customer visit to your site may entirely change how you measure, interpret and react to customers from this address
  • Third data is engineered to allow it to be easily reinterpreted as you learn more. This does not just focus on making reinterpretation fast; it also makes reinterpretation fault-tolerant (i.e., easy to correct in the event of a bug—without any loss of information)
  • Finally it allows all of this in real-time with two points of view: a just-in-time view and the deep cross-sectional view (both of which are always current). This lets you make decisions quickly without sacrificing the 100% loss-less accuracy needed for important business areas (such as finance, medicine, or mission-critical operations).

Once you have these capabilities, the things you can do with data—quickly and at scale—are pretty amazing. I will share some of these in future posts, as I want to keep this post short.

However, I will close this post out with a simple analogy…

“Think Like I Chef” vs. the Fast Food Menu

Traditional architectures are like fast food menus. You have these options. If you want to change the menu, we can do some market research, see what works and rollout a new menu. If you want to change again (or explore “what if we had done this?”) we can repeat this process.

Lambda architecture is like the pantry of a great chef. You have all these ingredients. If you feel like duck à l’orange, we can make this. If you want a duck confit salad, we can re-purpose the ingredients. If you want really rich potatoes, we can render the fat and cook the potatoes in it. If you want vegan, we can pull other items out of the pantry and make something else. There are so many more options.

Mapping This Back to Things Business People Care About

So what does this mean for your business? Do you remember the last time heard these comments:

  • “You’ll see that report. It will be in our Data Warehouse–tomorrow around 10am.”
  • “Oh, that’s in our warehouse. We can build a program to convert and and load the data into production. It will only take 3 weeks. Can you submit your TPS form to the Steering Committee so we can prioritize this?”
  • “Gee, it’s too bad we did not capture that data. We can start to capture it now. In a few months we can start analyzing it.”

With Lambda, all of these comments–and many more–go away. Data is never thrown away. It is always in production, ready to be used–for analysis or real-time transactions. There is no delay between transactional use and analysis–data flows down both paths as once.

Just imagine what problems you can solve when these limitations go away.

Ten Tech Trends for Your 2012 New Year’s Resolutions List

Article first published as Ten Tech Trends for Your 2012 New Year’s Resolutions List on Technorati

BabyNewYearOne of the most exciting things about working in tech is using it to create new ways to work, play—and even live. We have seen many great technology innovations develop over the past few years. Over 2012, ten of them will complete the jump from “new concept” to “mainstream trend.” How many of them are your ready for?

1. Everything Will Be Portable. The move to portable computing (smartphones, tablets and ultrabooks) will accelerate. Thick laptops and—even worse—desktops will be a relic of the past (except for those with high-power computing needs). If you are not yet mobile- and portable-ready, you better get there very soon.

2. Augmented Reality Will Go Mainstream. Augmented Reality (AR) is no longer a science fiction concept. Smartphones and (especially) tablets are mass-market platforms for everyday augmented reality. We are already seeing the first applications at Tech Meetups, CES and more. At least three innovators will exploit this, gaining mainstream adoption, by the end of 2012.

3. Touch Will Be Ubiquitous. Over the past five years, capacitive touch interfaces have re-programmed how millions of us interact with technology. As more devices are now sold today with touch than without, it is time to begin optimizing your user interface and user experience for touch (instead of a two-button mouse and keyboard).

4. Voice Will Be Next. While the intuitiveness of touch is a leapfrog improvement over mouse-and-keyboard, it still ties up our hands. Voice-based interaction is where we need to go. Apple’s Siri began the move of voice-driven interaction into the mainstream. This year, we’ll see SDKs for iOS and Android that harness the creativity of thousands to explode use of voice.

5. Fat Will Be the New Thin. Over a decade ago, broadband Internet enabled browsers to replace thick client applications. Now, portable computing usage across low power, lossy networks (e.g., mobile, WiFi, Bluetooth) coupled with AppStore Model has brought locally installed apps back in vogue. Building web apps is not enough; you need AppStore apps too.

6. Location-based Privacy Will Be Solved. Over the last two years location-based services became really hot. Unfortunately location-related privacy issues became hot too. The move of these services into mainstream populations of tens of millions will expand anecdotal security scares into weekly news stories, forcing adoption of safer location-based privacy policies.

7. Cloud Will Be the New Norm. Cloud computing is no longer an “edge market.” It is now adopted by big enterprises, public sector agencies—and even consumer tech providers. The cost, convenience and flexibility advantages of cloud computing will make it too hard for everyone not to use—everyday—by the end of this year.

8. …So Will Twitter. While people still love to debate the reasons to use Twitter, everything from the Arab Spring to the Charlie Sheen Meltdown showed that Twitter is now a well-recognized media channel. #Election2012 will accelerate mainstream use of Twitter—with the same overwhelming intensity we have seen for years in “traditional” campaign advertising.

9. ‘Consumerization of IT’ Planned and Budgeted. Consumer tech has become so sophisticated (without sacrificing ease-of-use and intuitiveness) that we began last year to demand its use in the enterprise. 2012—the first year in which most enterprise budgets include planned projects to support the consumerization of IT—will both accelerate and “lock in” this new tech trend.

10. 2012 Will Be Declared the Begin of “The ‘Big Data’ Era.” This year we will see another 40% increase in data we need to manage. This growth, coupled with recent releases of enterprise-ready high-scale NoSQL products will begin adoption of this tech by the entire industry. Looking back, 2012 will represent the start of the global, cross-industry Big Data era.

If you haven’t started embracing these already, now is a great time to add them to your “2012 Technology New Year’s Resolution List.” Sponsor a few pilot projects in your enterprise. Buy one or two Post-CES products to help you work more efficiently at the office. Or—if you want to include the whole family—buy one to use while you shop online, watch TV or manage your household.