Tag Archives: scalability

How to Architect for IoT

Last week I had the pleasure of doing a podcast with Forbe-contributor Mike Kavis on how to architect for the Internet of Things (“IoT”). We originally connected on Twitter regarding a discussion on whether the IoT and sensors are Big Data. That discussion led a podcast on architecture challenges–from device to data to data consumer–created by the onset of millions (or billions) of connected sensors and smart things.

Here in an excerpt of what we discussed

  • Connected devices bring back some classic engineering challenges back into the forefront.  How do you transmit data securely and with low power consumption? How do you handle lossy networks and cut-off transmissions?
  • Not everything is smartphone app transmitting JSON over HTTP (that would be cost prohibitive from both a hardware and bandwidth perspective). How do you handle communication myriad protocols, each of which could be using a near-infinite variety of data encoding formats?
  • IoT data is messy. Devices get cut off in mid-transition (or repeat over and over until they get an acknowledgement). How do you detect this–and clean it up–as data arrives?
  • IoT data is of incredibly high volume. By 2020, we will have 4x more sensor and IoT data than enterprise data. We already get more data today from sensors than we do from PCs. How do we scale to consume and use this. In addition, connected devices are not always smart or fault-tolerant. How do you ensure you are always ready to catch all that data (i.e., you need a zero-downtime IoT utility)
  • IoT and sensor and of itself is not terribly useful. It is rarely in a format that a (business or consumer) analyst would even be able to read. It would be incredibly wasteful to store all this as-is in a business warehouse, DropBox repo, etc.
  • IoT and sensor data needs context. Knowing device Knowing that FE80:0000:0000:0000:0202:B3FF:FE1E:8329 is at GPS location X,Y is of no use. You need to marry it to data about the “things” to get useful insights.
  • IoT data simultaneously “lives” in two points of view: what does this mean right now and what does this imply for the big picture. The Lambda Architecture is an ideal tool to handle this.
  • Finally, while all the attention is on the consumer stories, the real money is the Industrial and Enterprise Internet of Things. It’s also where smart things are far less creepy.

Listen to the podcast to hear more of the details

You can find the full podcast on Cloud Technology Partner’s website and SoundCloud:

I also want to take a moment to extend a big thank you to the folks at Cloud Technology Partners, SYS-CON Media, and Cloud Computing Journal for sharing this podcast.  I also want thanks to all of you on Twitter who retweeted it. I was happily overwhelmed by the sharing and interest!

Cloud Computing: Its not just about access from anywhere

Article first published as Cloud Computing: It’s Not Just About Access From Anywhere on Technorati.

Too many extolling the virtues of cloud computing are ignoring its most transformational benefits

Cloud computing has definitely moved into the mainstream. You now see commercials from Microsoft, Cisco, IBM and others every evening on prime time Cable TV. CNBC has created a Cloud Computing Special Report for investors to learn more about it. Even government agencies are now moving to cloud-based solutions.

Unfortunately, one of the most touted reasons we see for using cloud computing – that it provides universal access to data and applications from the Internet – has nothing to do with what cloud computing actually is. This is simply what web-based applications have been doing since the 1990s. True cloud computing offers a whole lot more.

In October 2009, The National Institute of Standards and Technology (NIST) published an excellent definition of cloud computing that calls out five essential characteristics that separate clouds from simple remotely hosted, web-based computing models:

  1. On-demand self-service
  2. Broad network access
  3. Resource pooling
  4. Rapid elasticity
  5. Measured service

I know, some of these terms are mouthful – especially to those who do “live and breathe” technology. However, they remove so much of the work and complexity that has so frequently made management of computing so painful and costly:

On-demand Self-Service (Think “Now”): With on on-demand self-service, you do not need to ask your provider to execute an “IT project” to enable you to use your application (or update it) to support a new business development. You can do whatever you need, when you need it – without the cost and delay of overhead managing your vendor.

Broad Network Access (Think “Convenience”): This lets you work wherever you need, whenever you need – from your work or home computer, netbook, tablet, or smartphone. Traditionally, this was done through browser, to bypass the need to install local software. However, the rise of (cloud-based) App Stores now allows us to install richer applications to access our data – wherever we are, on-demand.

These first two characteristics are what most people think of when talking about cloud computing. However, it is the next three characteristics that make true clouds stand out:

Resource Pooling (Think “Black Box”): Somewhere far away IT people are managing shared, redundant infrastructure across many data centers. They manage maintenance, business continuity, elimination of failures and bottlenecks, etc. You gain all of the benefit of these large-scale investments in time and resources – but without the need to do any work.

Rapid Elasticity (Think “No Limits”): You never have to worry about capacity planning. If you suddenly get a surge in traffic (due to an emergency or unexpected popularity) the computing resources you need are automatically – and immediately – available. You avoid slow-downs, timeouts and outages that waste time, cause frustration and turn away customers.

Measured Service (Think “Value”): Pay only for what you use – and no more. Rather than paying 100% for servers that you only use at 20% utilization, you pay for the exact number of resources you use, when you use them. The ideal cloud providers charge usage in terms that everyday people – not just IT systems administrators – understand and value.

cloudcomputing-180pxsWhen explaining these cloud computing characteristics to those whose “day jobs” are not in tech, I like to use the electricity analogy. When you buy a new television, you do not call the power company and ask them to initiate a project to set up your television. You simply plug it in and begin using it. If you don’t like where it is in your house, you unplug it, move it to a different room, and plug it in again. At the end of the month, you don’t pay for the power company’s generator and labor investments; you pay for the extra kilowatt-hours your television used.

Services that meet all five of these characteristics are so much more convenient and valuable than legacy computing models. That’s why cloud computing has the potential to be so transformational.