What it takes to successfully implement COTS enterprise apps

That dreaded failure number and its root cause

As soon as you start planning for an enterprise program you will almost immediately begin to hear some variation of the following quote:

“You know, X% of [enterprise vendor name’s] implementations fail or are canceled before they ever make it to production.” (Where X% is usually around 50% or 60%.)

I will leave communication of the exact number to groups like Gartner. Nevertheless, many enterprise programs fail. Nearly all exceed their recommended budget and time allowances, even for their scoped-down first phases (I will get into scope control in a whole other series of posts). The single most important reason for this I have seen is the following:

Program sponsors (and their program teams) forget to include much of the work required to implement a successful off the shelf product

As a result, they need to come back to Capital Review Teams or their CFO to ask for extra money to complete their programs. In good times, they get this money (but reduce or eliminate their IRR), in bad times they can see their programs severely cut back, delayed or killed outright. Most companies are not in good times right now.

Buying off the shelf and integrating does not eliminate as many costs as you might think

My last post outlined everything you have to do to built applications in-house. Buying off the shelf and integrating eliminates many of these (in lieu of licensing and integration costs). However, it leaves many tasks that are often underestimated in the traditional “build vs. buy” analysis. It is easiest to highlight these by directly comparing the tasks for each:

Activities by project discipline area (i.e., not a Waterfall plan)

Key: Struck-out = Eliminated by purchasing and integrating an enterprise application. Red = added to successfully integrate the enterprise application. Italics = Clarifying note

  • Scoping (About half the time vs. building it yourself)
    • Negotiating contracts for acquisition
    • Negotiating contracts and Statements of Work for integration
    • Defining Concept of Operations or Vision Document outlining how the application will work at a high-level
    • Negotiating scope and phasing of capabilities over time
    • Estimating cost and time for this
  • Requirements Management (About one third of the time vs. building it yourself)
    • Capturing Use Cases, translating these into technical feature configuration and integration requirements
    • Determining non-functional requirements – performance, reliability, scalability, availability
    • Negotiating sign-off and prioritization by phase
  • Design (About one quarter of the time vs. building it yourself)
    • Designing an top-level architecture
    • Designing the logical data model. Specifying the configurable meta data for the application.
    • Designing the object model.
    • Designing the software architecture (what is internal vs. external, what is the software vs. the database, etc.)
    • Designing interfaces with external systems
    • Designing the User Interface – a huge one from determining style conventions, wireframes, mockups, usability analysis, Section 508 compliance and lots more. Specifying configuration settings for the User Interface (much easier than designing “frem scratch”)
    • Specifying the business and workflow rules to configure in the system
    • Designing the systems architecture – how you will handle maintenance, scaling on-the-fly, backup and recovery, disaster recovery, elimination of SPOFs (Single Points of Failure) – this task is so large that I built a whole portion of my career doing this for utility-class systems (always-on systems with reliabilities exceeding 99.999% and scales exceeding 10,000 transactions per second)
  • Infrastructure Management (Often LONGER than building it yourself as your infrastructure team will have to learn how the application is architected)
    • Capacity planning – how much hardware to you need, where should it be, when do you need it
    • Hardware acquisition – Selecting and ordering your hardware
    • Systems Integration – Installing the operating systems, patches, and base middleware. Integrating hardware into your network (configuring IP settings, adding to your routers and switches, much more)
    • Setting up your monitoring and production support teams and systems
    • Doing this for EVERY environment – development, integration, certification and production (your environments will multiple post-launch as you will have systems under development AND in production)
  • Construction (About one-third of the time vs. building it yourself)
    • Setting up the code repository
    • Setting up your continuous build process
    • Defining your interfaces and configuration management files. Defining exactly how existing systems will integrate with the new system
    • Building your databases
    • Building your database scripts and stored procedures
    • Building software modules
    • Building hooks for monitoring and business intelligence
    • Defining and executing your unit test cases around configured metadata, business rules and work-flows
    • Integrating and verifying the integration of your software modules with your databases and external interfaces the application with the rest of your environment.
  • Certification
    • Tracing your configuration requirements (functional and non-functional) to Test Cases
    • Building Configuring your Test Harnesses
    • Conducting verification testing, sending configuration builds back for repair, integration and certification/regression testing
    • Optional – usually done for highly-integrated mission-critical applications only: Conducting load and performance testing, including analyzing and repairing performance and scale bottlenecks
  • Training & Support
    • Developing training documentation and curriculum (for end users)
    • Setting up your Help Desk – including your knowledge base of how-to resolve expected questions
    • Setting up a Center of Excellence to manage the application after implementation (you will need to build or rent as your internal staff will not have the expertise to do this)
    • Training end users
  • Project Management
    • Developing the plan (tasks, durations, assignments, critical path, etc.) to do all this
    • Managing and resolving all those issues that come up with something this complicated
    • Estailishing governance for decision-making and issue resolution – requiring lots of time from your executives
  • General Management
    • Getting approval for the money to do this – Capital and Expense (often requiring you to in front of your Board of Capital Review Committee)
    • Managing the RFI, RFP and contact negotiation process for your Integration Vendor
    • Getting (or reallocating) the staff to do all of this – usually this is now external staff such as Integration Vendors (pro: you do not divert as many of your staff, con: they are more expensive than internals)
    • Managing the staff for requirements, design, development, integraiton, testing and installation (acquiring resources, approving vacation, backfilling departures, etc.)
    • Managing the capital and expense expenditures, including all the invoices for purchases and services and weekly or monthly reporting to your Finance organization
    • Handing all those items like staff departures (people quitting, not performing, getting sick, etc.) – This is reduced as you have “outsourced” a good fraction of it to your implementation vendor

This is still a lot of work — and a lot of time you could instead focus on the core of your business. Some will be done by internal staff. Much will require you to hire an external integrator (as your internal staff will not have the expertise to do this). The good news is that this frees up more internal staff than building it yourself. The bad news is that external implementation staff often cost 1.5x to 2.5x as much as your internal staff (making schedule overruns very costly). The cost of these overruns increases exponentially based on your degree of under estimate – this is known as Parkinson’s Law. Using a SaaS model avoids many of these costs.

Why ‘Build’ is ALWAYS more expensive than ‘Buy’

The situation

The head of Marketing has a great idea for a new approach (process and/or technology). He or she goes to the technology team and proposes buying a leading system to do this. The head of Technology indicates a great interest in helping out, then points out all the problems that will have to be over-come to use or integrate this new vendor. He or she then shows he the Technology team can build or assemble it in-house, meeting all of the enterprises unique needs, faster and for less money. From here, the classic argument ensure…who wins will be based on the power dynamics of the organization.

We have all been here before

I don’t need to ask if this situation sounds familiar as we have all been here — dozens of times. I have sat at both sides of this table (two times more often on the Technology side of it). I have said I could build-it better more times than not. Many times I have been right, some times I have been clearly wrong to better understand this, it is worth looking at two items:

  1. What are the true costs (direct and opportunity) to build it yourself?
  2. When is it worth investing time and money – and diverting staff – to do this?

The true costs of building it yourself?

There is a phenomenon in the industry known by the technical term “Optimistic Bias” (see my “Recommend Reads” for more). It is an estimation term that basically says that engineers typically underestimate what it will take to get something done (due to forgetting items and not factoring risk and issues). Any of us who have every custom built a house or asked a mechanic to repair and old car know this is not just isolated to software. However, it can be more prevalent in software because software is not a physical object you can walk around and touch (and code libraries are so large than no manager can read every line of code)

A way of combatting this is using a technique know as Decomposition-Recomposition. Essentially, the concept is you break down a big project into small tasks (as low as you can go) to ensure you can “think of as much as possible to do.” You will likely never complete all the tasks at the exact proscribed time — but if you plan for this many to do — you will likely hit the major milestones within cost. I have been doing this for about three thousand projects now, with a high rate of success. Let’s do this for building an application (please excuse the long list but there is a lot to do)…

Activities by project discipline area (i.e., not a Waterfall plan)

  • Scoping
    • Defining Concept of Operations or Vision Document outlining how the application will work at a high-level
    • Estimating cost and time from this
    • Negotiating scope and phasing of capabilities over time
  • Requirements Management
    • Capturing Use Cases, translating these into technical feature requirements
    • Determining non-functional requirements – performance, reliability, scalability, availability
    • Negotiating sign-off and prioritization by phase
  • Design
    • Designing an top-level architecture
    • Designing the logical data model
    • Designing the object model
    • Designing the software architecture (what is internal vs. external, what is the software vs. the database, etc.)
    • Designing interfaces with external systems
    • Designing the User Interface – a huge one from determining style conventions, wire frames, mock-ups, usability analysis, Section 508 compliance and lots more
    • Designing the systems architecture – how you will handle maintenance, scaling on-the-fly, backup and recovery, disaster recovery, elimination of SPOFs (Single Points of Failure) – this task is so large that I built a whole portion of my career doing this for utility-class systems (always-on systems with reliabilities exceeding 99.999% and scales exceeding 10,000 transactions per second)
  • Infrastructure Management
    • Capacity planning – how much hardware to you need, where should it be, when do you need it
    • Hardware acquisition – Selecting and ordering your hardware
    • Systems Integration – Installing the operating systems, patches, and base middleware. Integrating hardware into your network (configuring IP settings, adding to your routers and switches, much more)
    • Setting up your monitoring and production support teams and systems
    • Doing this for EVERY environment – development, integration, certification and production (your environments will multiple post-launch as you will have systems under development AND in production)
  • Construction
    • Setting up the code repository
    • Setting up your continuous build process
    • Defining your interfaces and configuration management files
    • Building your databases
    • Building your database scripts and stored procedures
    • Building software modules
    • Building hooks for monitoring and business intelligence
    • Defining and executing your unit test cases
    • Integrating and verifying the integration of your software modules with your databases and external interfaces
  • Certification
    • Tracing your requirements (functional and non-functional) to Test Cases
    • Building your Test Harnesses
    • Conducting verification testing, sending builds back for repair, integration and certification/regression testing
    • Conducting load and performance testing, including analyzing and repairing performance and scale bottlenecks
  • Training & Support
    • Developing training documentation and curriculum (for end users)
    • Setting up your Help Desk – including your knowledge base of how-to resolve expected questions
    • Training end users
  • Project Management
    • Developing the plan (tasks, durations, assignments, critical path, etc.) to do all this
    • Managing and resolving all those issues that come up with something this complicated
    • Estailishin governance for decision-making and issue resolution – requiring lots of time from your executives
  • General Management
    • Getting approval for the money to do this – Capital and Expense (often requiring you to in front of your Board of Capital Review Committee)
    • Getting (or reallocating) the staff to do all of this
    • Managing the staff
    • Managing the capital and expense expenditures, including all the invoices for purchases and services and weekly or monthly reporting to your Finance organization
    • Handing all those items like staff departures (people quitting, not performing, getting sick, etc.)

That’s a whole lot to do – just to get to Release 1.0 of your application. Once you are done, the REAL FUN begins…

  • Fixing bugs
  • Resolving performance issues
  • Fielding complaints and planning for future releases to address them
  • Monitoring the market to keep up with everything else out there
  • Integrating with all the new systems that will come over the next few years
  • Best of all … handing upgrades (such as operating system, database, or middleware upgrades to stay current with your vendor maintenance agreements)

You will have to perform all of the above for the life of the application…

Of course, if this is a new application area for you, you will likely get the planning wrong. What happen then:

  • Budget and time line over-runs
  • Escalations and explanations to your key decision-makers
  • Fending off all those conversations that the thing you could have bought does something different

The cost of these items becomes exponential based on your degree of error – this is known as Parkinson’s Law.

Compare this to Software-as-a-Service (SaaS)

SaaS wraps all of the above into a “Big Black Box” and simply gives you a service that you can turn on or off based on your business need. You pay a month or annual fee and simply use the service like a utility (just like your electric or phone service). If your SaaS vendor is good, they will provide you an on-boarding and training process to make it simple for you to–

  • Tell them what you need in the solution
  • Verify and accept the configured solution based on its need
  • Get training in how to use it
  • Rely on 24×7 monitoring, support and customer service

Whenever I have built services, in-house or as a SaaS provider, I have made sure I did this. (If I did not do so as an in-house provider, my internal customers would go external).

When would you build it yourself?

I did promise I would present this counter-argument. There are times to build something yourself. These times all have one thing in common:

When considering whether to buy or build, only build it if it is intrinsic to the core of your business. If it is not, buy it.

This essentially paraphrases Adam Smith’s “Wealth of Nations.” The opportunity cost of wasting your time on something that is not the core of what you do is too high. As a result:

  • If you are Media company, produce your content (but do not build content management and web systems)
  • If you are an eCommerce platform company, build your eCommerce software (but not your middleware or operating systems)

Take advantage of the expertise of others, the marginal cost of this is the founding principal of the enterprise. SaaS is the best model to do this with software.

5 points where tech balances between life and work