Tag Archives: reliability

Spark Streaming and Expert Systems for the Industrial IoT

This week, at the Washington DC Spark Interactive, Savi Engineering shared some of our work on using Spark Streaming and Expert Systems technology (Drools) to analyze the Industrial IoT in near-real time.

At Savi, we use a hybrid Lambda Architecture (see my post on why Lambda is so important). By “hybrid” we mean that unlike pure Lambda Architectures, we cannot restate the past 100% as we have already notified humans of critical IoT events (e.g., theft, safety risk). We can only enrich and auto-resolve these as more data becomes available. You can find tips on how do this — in general with streaming technologies and specifically with Spark — in the following presentation. You can also learn more about tackling real-world IoT challenges:

In addition, at Savi we combine fully explicit rules with real-time machined learning algorithms to perform risk and performance analytics in near-real time (see my post on the differences in focus areas between our Data Engineers and Data Scientists).  James Nowell of our Engineering team provided a great presentation on how we run Drools inside Spark RDDs (yes–Drools, we do this without performance penalties) to create linear-scale expert systems to analyze all that IoT as if we were an omniscient human. You can find his presentation here:

In future presentations, we will expand on areas such as:

  • The differences in use of Spark (using the same data) between Data Scientists and Engineers
  • How we scale machine learning algorithms for real-time, sub-second execution (thousands of times per second)
  • Creating a DAG that combines hardware device edge intelligence with cloud-based intelligence

If you like what you see here, Savi is hiring. Take a look at here.


Moving from Storm to Spark Streaming: Real-life Results and Analysis

In my last post, I explained why we decided to move the Speed Layer of our  Lambda Architecture from Apache Storm to Apache Spark Streaming in . In this post, I get the “real meat”:

  • How did it go?
  • What did we learn?
  • And most important of all… would we do it again?

This post recounts our detailed experiences moving to a 100%-Spark architecture. While Spark is arguably the most popular open source project in history (currently its only rival in terms of number of contributors is AngularJS), our experience with it was not all wine and roses. Some experiences were great. Others remain frustrating today after nearly nine months in live operation, streaming mission-critical data. Whether you love Spark or Storm, there are some bragging rights for your favorite platform.

Before I get started I should warn you that this post is pretty long. I could have broken it up into different posts,  one on each category of analysis. However, I thought it was more useful as a single blog post.

Our Real-world Environment

This is not one of those simple streaming analytic run-offs using the the canonical “Twitter Word Count” test (Spark version, Storm version). This is a real-life comparison of Storm vs. Spark Streaming after months of live operation in production, analyzing complex, real-life data from many enterprise customers.

We do not use either technology by itself, but instead use it in conjunction with Apache Kafka (Cloudera’s distribution), Apache Cassandra (DataStax’s distribution), and Apache Hadoop (also Cloudera’s distribution, storing data in Apache Parquet format). I am not divulging any trade secrets here, as we list these technologies on our job descriptions for recruiting.

Similarly, we do not simply pass data through a single stage graph (no robust real-world system uses a single-stage DAG). Instead our DAG processing traverses from 3-7 stages, depending on the type of data we receive. At each stage we persist data back to Kafka for durability and recovery.

Obviously, everything we run is clustered (no single servers). Along these lines,  we only use native installations of downloaded distributions. Everything here can be hosted anywhere you like: your own data center, GCE, AWS, Azure, etc. The results are not tied to IaaS solutions like AWS EMR.

This comparison is also not a short-duration test (which would also  be artificial). We run our streaming processing 24×7, without scheduled downtime. Our Lambda Architecture enabled us to stream the same data into Storm and Spark at the same time, allowing a true head-to-head comparison of development, deployment, performance and operations.

Finally, these results are not just based on uniform sample data (e.g., 140-character Tweets). We used a wide range of real-life sensor data, in multiple encoding formats, with messages ranging from 100 Bytes to 110 Megabytes in size (i.e., real-world, multi-tenant complexity). We tested this at data rates exceeding 48 Gbps per node. We have come up with novel ways to stream data larger than the message.max.bytes size in Kafka in real-time along our DAG–disclosing how we do this would be a trade secret 😉

So what did we learn? I will discuss the results from four perspectives:

  1. Developing with each (a.k.a., the software engineering POV)
  2. Head-to-head performance comparison
  3. Using each with other “Big Data” technologies
  4. Managing operations of each (a.k.a., the DevOps POV)

BTW, of course Spark Streaming is a micro-batch architecture while Storm is a true event processing architecture. Storm Trident vs. Spark Streaming would be a true “apple-to-apple” comparison. However, that was not our real-life experience. The move from one-transaction-at-a-time to micro-batches presented some changes in conceptual thinking (especially for “exactly once” processing). I include some learnings from this.

Next Page: Storm vs. Spark Streaming: Developing With Each