Operations

Scalability, Performance, Reliability, and Operations Excellence

What it takes to successfully implement COTS enterprise apps

That dreaded failure number and its root cause

As soon as you start planning for an enterprise program you will almost immediately begin to hear some variation of the following quote:

“You know, X% of [enterprise vendor name’s] implementations fail or are canceled before they ever make it to production.” (Where X% is usually around 50% or 60%.)

I will leave communication of the exact number to groups like Gartner. Nevertheless, many enterprise programs fail. Nearly all exceed their recommended budget and time allowances, even for their scoped-down first phases (I will get into scope control in a whole other series of posts). The single most important reason for this I have seen is the following:

Program sponsors (and their program teams) forget to include much of the work required to implement a successful off the shelf product

As a result, they need to come back to Capital Review Teams or their CFO to ask for extra money to complete their programs. In good times, they get this money (but reduce or eliminate their IRR), in bad times they can see their programs severely cut back, delayed or killed outright. Most companies are not in good times right now.

Buying off the shelf and integrating does not eliminate as many costs as you might think

My last post outlined everything you have to do to built applications in-house. Buying off the shelf and integrating eliminates many of these (in lieu of licensing and integration costs). However, it leaves many tasks that are often underestimated in the traditional “build vs. buy” analysis. It is easiest to highlight these by directly comparing the tasks for each:

Activities by project discipline area (i.e., not a Waterfall plan)

Key: Struck-out = Eliminated by purchasing and integrating an enterprise application. Red = added to successfully integrate the enterprise application. Italics = Clarifying note

  • Scoping (About half the time vs. building it yourself)
    • Negotiating contracts for acquisition
    • Negotiating contracts and Statements of Work for integration
    • Defining Concept of Operations or Vision Document outlining how the application will work at a high-level
    • Negotiating scope and phasing of capabilities over time
    • Estimating cost and time for this
  • Requirements Management (About one third of the time vs. building it yourself)
    • Capturing Use Cases, translating these into technical feature configuration and integration requirements
    • Determining non-functional requirements – performance, reliability, scalability, availability
    • Negotiating sign-off and prioritization by phase
  • Design (About one quarter of the time vs. building it yourself)
    • Designing an top-level architecture
    • Designing the logical data model. Specifying the configurable meta data for the application.
    • Designing the object model.
    • Designing the software architecture (what is internal vs. external, what is the software vs. the database, etc.)
    • Designing interfaces with external systems
    • Designing the User Interface – a huge one from determining style conventions, wireframes, mockups, usability analysis, Section 508 compliance and lots more. Specifying configuration settings for the User Interface (much easier than designing “frem scratch”)
    • Specifying the business and workflow rules to configure in the system
    • Designing the systems architecture – how you will handle maintenance, scaling on-the-fly, backup and recovery, disaster recovery, elimination of SPOFs (Single Points of Failure) – this task is so large that I built a whole portion of my career doing this for utility-class systems (always-on systems with reliabilities exceeding 99.999% and scales exceeding 10,000 transactions per second)
  • Infrastructure Management (Often LONGER than building it yourself as your infrastructure team will have to learn how the application is architected)
    • Capacity planning – how much hardware to you need, where should it be, when do you need it
    • Hardware acquisition – Selecting and ordering your hardware
    • Systems Integration – Installing the operating systems, patches, and base middleware. Integrating hardware into your network (configuring IP settings, adding to your routers and switches, much more)
    • Setting up your monitoring and production support teams and systems
    • Doing this for EVERY environment – development, integration, certification and production (your environments will multiple post-launch as you will have systems under development AND in production)
  • Construction (About one-third of the time vs. building it yourself)
    • Setting up the code repository
    • Setting up your continuous build process
    • Defining your interfaces and configuration management files. Defining exactly how existing systems will integrate with the new system
    • Building your databases
    • Building your database scripts and stored procedures
    • Building software modules
    • Building hooks for monitoring and business intelligence
    • Defining and executing your unit test cases around configured metadata, business rules and work-flows
    • Integrating and verifying the integration of your software modules with your databases and external interfaces the application with the rest of your environment.
  • Certification
    • Tracing your configuration requirements (functional and non-functional) to Test Cases
    • Building Configuring your Test Harnesses
    • Conducting verification testing, sending configuration builds back for repair, integration and certification/regression testing
    • Optional – usually done for highly-integrated mission-critical applications only: Conducting load and performance testing, including analyzing and repairing performance and scale bottlenecks
  • Training & Support
    • Developing training documentation and curriculum (for end users)
    • Setting up your Help Desk – including your knowledge base of how-to resolve expected questions
    • Setting up a Center of Excellence to manage the application after implementation (you will need to build or rent as your internal staff will not have the expertise to do this)
    • Training end users
  • Project Management
    • Developing the plan (tasks, durations, assignments, critical path, etc.) to do all this
    • Managing and resolving all those issues that come up with something this complicated
    • Estailishing governance for decision-making and issue resolution – requiring lots of time from your executives
  • General Management
    • Getting approval for the money to do this – Capital and Expense (often requiring you to in front of your Board of Capital Review Committee)
    • Managing the RFI, RFP and contact negotiation process for your Integration Vendor
    • Getting (or reallocating) the staff to do all of this – usually this is now external staff such as Integration Vendors (pro: you do not divert as many of your staff, con: they are more expensive than internals)
    • Managing the staff for requirements, design, development, integraiton, testing and installation (acquiring resources, approving vacation, backfilling departures, etc.)
    • Managing the capital and expense expenditures, including all the invoices for purchases and services and weekly or monthly reporting to your Finance organization
    • Handing all those items like staff departures (people quitting, not performing, getting sick, etc.) – This is reduced as you have “outsourced” a good fraction of it to your implementation vendor

This is still a lot of work — and a lot of time you could instead focus on the core of your business. Some will be done by internal staff. Much will require you to hire an external integrator (as your internal staff will not have the expertise to do this). The good news is that this frees up more internal staff than building it yourself. The bad news is that external implementation staff often cost 1.5x to 2.5x as much as your internal staff (making schedule overruns very costly). The cost of these overruns increases exponentially based on your degree of under estimate – this is known as Parkinson’s Law. Using a SaaS model avoids many of these costs.

SaaS: Culminating 40 years of software evolution

SaaS represents a culmination of a roughly forty years of journey that brought software acquisition a highly uncertain, large-risk capital investment to an easy-to-plan-for utility.

Let’s take a look at the evolution of software delivery:

  • In the beginning, there were no independent software vendors. If you wanted software, you had to build the infrastructure, hire people to operate it, hire more people to write software and hire more people to manage the process of building and deploying software. (BTW, you also had to hire people to build and deliver training to all of your users). Very expensive and lots of uncertainty–as expected in an immature market.
  • Next came professional software firms. With a contract you could bring in the people to run your hardware, build your software and manage your software and training process. This reduced uncertainty by enabling you to end contract. To combat this, vendors charged high rates and often only signed time-and-materials or cost-plus contracts. This did not eliminate much uncertainty. (BTW, these days were fun for us creative software engineers as we got to reinvent things with custom work all the time.)
  • Next came enterprise software vendors. Now you could buy finished software. However, that was just the beginning of the costs: You had to buy the hardware, hire and integrator (or hire staff to configure and integrate the software) then create something called a Center of Excellence to run the thing when you were done. That was just the first step. Within 18-24 months you would get told you needed to upgrade to retain your “low” 14%-18% maintenance support. This caused you to buy upgrades, modify your configuration, re-certify your application and re-deploy it. (Woe to the enterprise who customized what they bought or changed more than 15% of the enterprise application “out of box.”)
  • Then came SaaS. SaaS is a utility. You pay for what you use. You turn it off when you are done. You have no hardware to buy, no upgrades to manage and benefit from economies of scale from your SaaS provider. As a result, you can spend more time focusing on your enterprise

Think of the applicability of this in today’s economy. You “rent” what you need, when you need it. You have few, if any, fixed costs. You also have no capital investment to get approved by the board. The level of uncertainty on your projects — and all those “hidden risks” that arise in the “last 10%” or your projects disappears. A little less uncertainty is nice in today’s economy, huh?

This gets even better if you are not a technology company. Instead of diverting time and attention to build and manage technology you buy it like you buy electricity, allowing you to focus more attention on your core business.

To really appreciate how much better SaaS is as a model, you need to break down all the costs, risks and extra work required to acquire and deliver software over the competing, prior models:

  1. Build it in-house
  2. Buy off-the-shelf and integrate it, or
  3. Get an application service provider (ASP) and find someone to manage it

I will cover these in my next three posts to highlight how much better SaaS really is.