L5: Past Lessons

Lagrange Point 5 (L5): Looking at past tech developments to learn, grow and avoid mistakes

After Moving to Slack: Inbox Zero (at least twice per week)

TL,DR Version: Moving my entire Engineering organization to Slack and adopting a ChatOps collaboration mindset has reduced email volume over 95% and now enables us to resolve issues in 1/4th of the time.

I have always been a fan of using automation, web hooks, and chat-assisted operations to streamline work and enable collaboration across different locations. However, this traditionally required some Engineering and Operations (i.e., DevOps) investment in setting up collaboration servers, programming bots, etc. Slack has changed this, virtually eliminating all technical friction of moving to a ChatOps model. Here is our of how Slack enabled us to move to a ChatOps environment with far less email, faster response times, and greater overall productivity.

In late 2013, I joined a new company that—at the time—had no one on staff at the time with a DevOps-style background. After coming from several years of Chat-integrated operations, it felt like hitting the brakes. One night a few weeks later, I saw a Tweet by @mparca on February 9 about this new great service called Slack. I took a quick look and realized I could achieve everything that HipChat and Hubot did—with a simple push-button SaaS service. My initial set up (our organization, some channels, and hooks to Github) took less than 10 minutes. As it was free (for many features), I did not even need to process a Purchase Order (even better). I was off to setting ChatOps at a new place.

Initially, things went pretty slow. At the time, most of our tech stack and tools were hosted on-premise. Our chat tool of choice as a Skype (not a hook-friendly app). I got a few people to move to Slack, but not many.

Over time, as we implemented a full continuous integration and deployment chain, we added more and more hooks into Slack. First came a move to Atlassian (Cloud). Next Jenkins. Next Sentry. Then Ansible. Then Icinga. Then came custom RTM scripts for more complex things, such as letting our Data Scientists know they have left an idle PySpark context running for more than eight hours. What made this so easy was that everything but the custom RTM scripts could be done in less than five mouse-clicks (it is very helpful that so many collaboration and monitoring tools have enabled web hooks).

As we added more hooks, and started to bring more people onboard, I noticed an interesting shift. People joining our team began to just use Slack to communicate. One developer would come across an interesting new open source repo or article and share it with the rest in #general. Developers with DevOps privileges would jump on issues as soon as they saw a Sentry alert in #prod (saving the need to even text or phone the on-call engineer). Some people even answering questions in code review while they were doing things like waiting at the airport to depart on vacation.

Today our Engineering Teams (Product, Hardware, Software, Data, and Ops) all now primarily use Slack for communications. Most even use it in favor of texting. We Slack each other tickets that are ready for work, UAT, or release). We use group chats to have conversations to answer questions about stories, designs, bugs, and more. We use Slack channels integrated with Github for better code reviews. We use Slack to facilitate pair programming (and pair testing). Slack is now our default tool  for issue diagnosis, as sharing log messages, code snippets, and JSON is much clearer thanks to native markdown.

We achieve this with the following channel model:

  • One channel for each environment (so we can let people know if we are about to add nodes, run a load test, etc.). We have our respective Jenkins, Ansible, Icinga and Sentry hooks tied into each environment channel as well.
  • One channel for each code repository (to see PRs and conduct code reviews). We have aligned our JIRA projects with these to integrate tickets as well.
  • We have some basic team channels for more focused group conversations
  • We also have a #rm channel for simple-to-read log of what was released, when

As our organization moved to this model, life and work got easier in some rather visceral ways:

First, I have been able to dial down my notifications to only ping me when four things happen: I get a call, I get a text, I get a direct Slack message, or there is public Slack in a mission-critical channel. I no longer get endless interruptions, making me more productive at work (and more attentive in meetings and at home). If my phone does ping, it means there is something very important—which actually lets me react to these issues faster.

Second, my email volume is down over 95%. The bulk of the emails I now get are related to true business questions (vs. endless status messages and FYIs). As a result, I can answer email faster and now regularly hit “Inbox Zero” at least twice a week—while managing a 24x7xForever SaaS Engineering organization with follow-the-sun development and operations spanning California, Washington, Europe, and Asia.

Our full embrace of Slack did not happen overnight. It organically evolved over a period of about 18 months–a natural rate of adoption for organizational change. Because it was organic, we did have to institute policies  that forced usage. Instead we allowed our teams to naturally adopt Slack in ways that made work easier. I hope more organizations can make this transition as everyone could benefit from less email and fewer interruptions.

Natural adoption of Slack over other forms of communication would have happened if the usability was not as good as it is. One of my favorite features is how well Slack detects when I am no longer at my desk: if I walk down the hall, my phone chirps on key Slack messages; when I sit back down my phone stops and my laptop takes over.  This happens within seconds.

Oh BTW, we do all of this with the baseline free Slack account. That’s one less excuse to not give it a try.

PS – Want to work in an environment like this? Check us out.

Data Scientists vs. Data Engineers: Facts vs. Interpretation

Some of the things we build at work are closed-loop, Internet-scale machine learning micro-services. We have created algorithms that run in milliseconds that we can invoke via REST calls, thousands of times per second. We also have created data pipeline processes that process new (mostly sensor) data and build and publish new models when critical thresholds are reached. This work requires the collaboration of two very in-demand specialists: Data Scientists and Data Engineers.

Contrary to the classic Math vs. Coding vs. Domain Expertise Venn diagram, Data Scientists and Data Engineers share many similarities. Both love data. Both have domain expertise. Both are great functional programmers. Both are good at solving complicate mathematical problems—both discrete and continuous. Both use many similar tools and languages (in our case, Spark, Hadoop, Python and Scala).

However, over the past two years, as we have improved the collaboration between each to build better machine learning services, we have some key differences between each role. These differences are not just based on skill set or disposition. They also include differences areas of responsibility that are essential to creating fast, scalable, and accurate machine learning services.

It is easy to muddle raw data from fully deterministic derived data from algorithmically derived data. Raw data never changes. Rules may change but are easy to manage with clean version controls. However, even the same deterministic algorithms can produce different results (one example: whenever you refit or rebuild a model using new data, your results can change). If you are building algorithmic services you need to keep everything clean and separate. If not, you cannot cleanly “learn” from new data and continuously improve your services.

We have found a very nice separation of responsibility that prevents muddling things:

  • Our Data Engineers are responsible for determinist facts
  • Our Data Scientists are responsible for interpretation of these

This boils down to this: determinist rules are the purview of engineers while algorithmic guesses come from scientists. This is a gross simplification (as both engineers deal in many, many complexities). However, this separate keeps it very clear, not only in determining “who does what” but also preventing errors, guesses, and other unintended consequences that pollute data driven decision-making.

Let’s take Google Now’s “Where you parked” service as an example. Data Engineers are responsible for processing the streaming sensor updates from your phone, combining this with past data, determining motion vs. at rest, factoring out duplicate transmission, geospatial drift, etc. Data Scientists are responsible for coming up with the algorithm to determine whether your detected stop state is a place where you parked (vs. simply being at work, at home, or at a really bad stop light). Essentially, Data Engineers capture and process the data to extract required model features while Data Scientists come up with the algorithm to interpret these features and provide an answer.

Once you have separation down, both teams can collaborate cleanly. Data Scientists experiment and test algorithms while Engineers design how to apply at scale, with sub-second execution. Data Scientists determine what approach is used to build models (and what triggers model optimization, build and re-fitting). Data Engineers build seamless implementation of this. Data Scientists build algorithm prototypes and MVPs; Data Engineers scale these into fast, reliable, services. Data Scientists worry about (and define rules) to exclude outliers that would wreak havoc on F-tests; Data Engineers implement defensive programming and automated test coverage to ensure unplanned data does not wreak havoc on production operation.