Tag Archives: systems engineering

Four big technology challenges in a Web 2.0 world

Web 2.0 created a REAL change in how we use information

A lot of people call Web 2.0 an empty buzzword (there are some Buzzword Bingo apps out there that play on this. However, the product change that came with Web 2.0 drove real changes in how people use information.

To fully understand this, it makes sense to explain what I mean by Web 2.0. To do this I am going to repeat the explanation I give when asked what Web 2.0 is (those of you who read my posting on Social Collaboration can skip this section with my apologies)

Web 1.0 let organizations publish information that could be accessed easily by all of us when we needed it, at our convenience. This changed entirely how we read the news, looked up movie times and checked stock quotes or the weather. The problem with Web 1.0 was that it was biased towards making it easy for large organizations to share information. CNN could easily setup a web site to share news and opinion. However, if I wanted to share my information with many other people in this fashion, I had to setup my own web site, publish content, figure out how to control access to it, etc. This was too hard for the everyday person (who had non-technology things to worry about in his or her life).

Web 2.0 changed this by making it easy to share my views and information–and to control how I share it. Now I can use facebook to share my vacation pictures with my friends (but limit my contact information to my professional colleagues). I do not have to build a web site, administer it, share the URL with my friends and get them to bookmark and visit it. Instead, I can rely on the fact they will visit facebook as part of their normal life and see my updates there (the network effect). It makes it much easier for me.

Enterprise 2.0 and Government 2.0 do the same — but directly in support of the missions of public and private enterprises. They let stakeholders share information and views regarding how industry and government should work (instead of simply proclaiming, “I feel blue today.”) I think of this as Web 2.0 with a purpose of creating Measurable Enterprise Value (i.e., demonstrable business value or public benefit)

This has created new technology challenges

Web 2.0 effectively shifted the direction in how we need to manage creation, access, update, sharing and archive of information. It moved from a hub model to a true network model:

web1020

This shift is not trivial. It has created new challenges across a broad range of technology areas:

  1. Scaling dynamic data: In the Web 1.0 world we used two different approaches to scaling static vs. dynamic data. These approaches do not work in a world where data can be viewed by everyone can come from everywhere
  2. Moderating user-generated content: In the Web 1.0 world, user-generated data were viewed by relative small populations. In a Web 2.0 world, they can be viewed by everyone. This presents significant moderation challenges and new balances between openness, speed of publishing, control and safety
  3. Intellectual property management: Who owns the data on a social network? Web 2.0 stresses every aspect of intellectual property challenges—from ownership and revenue attribution to control and privacy
  4. Cross-platform media management: We live in a multimedia world. That means you have to manage both upload and download of media across hundreds of platforms (browser and operating system combinations). Anyone who has ever worked with CODECs can appreciate how difficult this is

Addressing these using elegant, high-scale, cost-efficient approaches is not trivial prospect. However it as exciting and stimulating as the challenges I learned when helping to build out the Web 1.0 world). Over my next few posts, I will blog about how I have tackled each of these and what I have learned along the way.

What it takes to successfully implement COTS enterprise apps

That dreaded failure number and its root cause

As soon as you start planning for an enterprise program you will almost immediately begin to hear some variation of the following quote:

“You know, X% of [enterprise vendor name’s] implementations fail or are canceled before they ever make it to production.” (Where X% is usually around 50% or 60%.)

I will leave communication of the exact number to groups like Gartner. Nevertheless, many enterprise programs fail. Nearly all exceed their recommended budget and time allowances, even for their scoped-down first phases (I will get into scope control in a whole other series of posts). The single most important reason for this I have seen is the following:

Program sponsors (and their program teams) forget to include much of the work required to implement a successful off the shelf product

As a result, they need to come back to Capital Review Teams or their CFO to ask for extra money to complete their programs. In good times, they get this money (but reduce or eliminate their IRR), in bad times they can see their programs severely cut back, delayed or killed outright. Most companies are not in good times right now.

Buying off the shelf and integrating does not eliminate as many costs as you might think

My last post outlined everything you have to do to built applications in-house. Buying off the shelf and integrating eliminates many of these (in lieu of licensing and integration costs). However, it leaves many tasks that are often underestimated in the traditional “build vs. buy” analysis. It is easiest to highlight these by directly comparing the tasks for each:

Activities by project discipline area (i.e., not a Waterfall plan)

Key: Struck-out = Eliminated by purchasing and integrating an enterprise application. Red = added to successfully integrate the enterprise application. Italics = Clarifying note

  • Scoping (About half the time vs. building it yourself)
    • Negotiating contracts for acquisition
    • Negotiating contracts and Statements of Work for integration
    • Defining Concept of Operations or Vision Document outlining how the application will work at a high-level
    • Negotiating scope and phasing of capabilities over time
    • Estimating cost and time for this
  • Requirements Management (About one third of the time vs. building it yourself)
    • Capturing Use Cases, translating these into technical feature configuration and integration requirements
    • Determining non-functional requirements – performance, reliability, scalability, availability
    • Negotiating sign-off and prioritization by phase
  • Design (About one quarter of the time vs. building it yourself)
    • Designing an top-level architecture
    • Designing the logical data model. Specifying the configurable meta data for the application.
    • Designing the object model.
    • Designing the software architecture (what is internal vs. external, what is the software vs. the database, etc.)
    • Designing interfaces with external systems
    • Designing the User Interface – a huge one from determining style conventions, wireframes, mockups, usability analysis, Section 508 compliance and lots more. Specifying configuration settings for the User Interface (much easier than designing “frem scratch”)
    • Specifying the business and workflow rules to configure in the system
    • Designing the systems architecture – how you will handle maintenance, scaling on-the-fly, backup and recovery, disaster recovery, elimination of SPOFs (Single Points of Failure) – this task is so large that I built a whole portion of my career doing this for utility-class systems (always-on systems with reliabilities exceeding 99.999% and scales exceeding 10,000 transactions per second)
  • Infrastructure Management (Often LONGER than building it yourself as your infrastructure team will have to learn how the application is architected)
    • Capacity planning – how much hardware to you need, where should it be, when do you need it
    • Hardware acquisition – Selecting and ordering your hardware
    • Systems Integration – Installing the operating systems, patches, and base middleware. Integrating hardware into your network (configuring IP settings, adding to your routers and switches, much more)
    • Setting up your monitoring and production support teams and systems
    • Doing this for EVERY environment – development, integration, certification and production (your environments will multiple post-launch as you will have systems under development AND in production)
  • Construction (About one-third of the time vs. building it yourself)
    • Setting up the code repository
    • Setting up your continuous build process
    • Defining your interfaces and configuration management files. Defining exactly how existing systems will integrate with the new system
    • Building your databases
    • Building your database scripts and stored procedures
    • Building software modules
    • Building hooks for monitoring and business intelligence
    • Defining and executing your unit test cases around configured metadata, business rules and work-flows
    • Integrating and verifying the integration of your software modules with your databases and external interfaces the application with the rest of your environment.
  • Certification
    • Tracing your configuration requirements (functional and non-functional) to Test Cases
    • Building Configuring your Test Harnesses
    • Conducting verification testing, sending configuration builds back for repair, integration and certification/regression testing
    • Optional – usually done for highly-integrated mission-critical applications only: Conducting load and performance testing, including analyzing and repairing performance and scale bottlenecks
  • Training & Support
    • Developing training documentation and curriculum (for end users)
    • Setting up your Help Desk – including your knowledge base of how-to resolve expected questions
    • Setting up a Center of Excellence to manage the application after implementation (you will need to build or rent as your internal staff will not have the expertise to do this)
    • Training end users
  • Project Management
    • Developing the plan (tasks, durations, assignments, critical path, etc.) to do all this
    • Managing and resolving all those issues that come up with something this complicated
    • Estailishing governance for decision-making and issue resolution – requiring lots of time from your executives
  • General Management
    • Getting approval for the money to do this – Capital and Expense (often requiring you to in front of your Board of Capital Review Committee)
    • Managing the RFI, RFP and contact negotiation process for your Integration Vendor
    • Getting (or reallocating) the staff to do all of this – usually this is now external staff such as Integration Vendors (pro: you do not divert as many of your staff, con: they are more expensive than internals)
    • Managing the staff for requirements, design, development, integraiton, testing and installation (acquiring resources, approving vacation, backfilling departures, etc.)
    • Managing the capital and expense expenditures, including all the invoices for purchases and services and weekly or monthly reporting to your Finance organization
    • Handing all those items like staff departures (people quitting, not performing, getting sick, etc.) – This is reduced as you have “outsourced” a good fraction of it to your implementation vendor

This is still a lot of work — and a lot of time you could instead focus on the core of your business. Some will be done by internal staff. Much will require you to hire an external integrator (as your internal staff will not have the expertise to do this). The good news is that this frees up more internal staff than building it yourself. The bad news is that external implementation staff often cost 1.5x to 2.5x as much as your internal staff (making schedule overruns very costly). The cost of these overruns increases exponentially based on your degree of under estimate – this is known as Parkinson’s Law. Using a SaaS model avoids many of these costs.