Tag Archives: COTS

Good Architecture: Isolation of logic by application area (Where I break with many COTS vendors)

Shortcuts lead to long delays

Most of us have been presented with the challenge of rapidly developing a product to beat the competition to market, support a major event or simply please a stakeholder. In these situations it is easy to “give in” and take a few shortcut to get to market sooner. Some of the most common shortcuts I have seen involving placing logic in whichever place has the fastest upfront coding time. The damage these cause is significant

Shortcut Number 1: Placing too much business logic in the user interface

Rational: UI development is rapid and often less technologically complex than back end server development (it can done by less senior engineers and visually checked very rapidly). As a result, many team place too much business logic in the UI (either in work flows or JavaScript).

While this does speed initial time to market, it creates many problems.

Problems that Emerge Immediately: Loading pages with JavaScript makes them slow (a bad user experience). It also makes them error prone, as each browser type can render JavaScript differently and some people turn it off.

Problems that Emerge in Month 2: Business rule-based application logic requirements change often. Now, when you make a change in business rules or application logic you need to change your UI. This creates usability and training problems. It also is expensive, as more testing is needed to certify UI vs. application rule changes. In addition, as time progresses more and more of your application logic spread out across your back end and UI (making your application harder or costlier to maintain).

Problems That Emerge in Quarter 2: Once you want to extend your application to a new client (call center client, mobile client, etc.) you either have to move all your application logic to the backend or re-produce it in your second (and third) UI clients. At this point, your “shortcut” is now a “long one.”

Shortcut Number 2: Hard-coding business logic

Rational: Making rules and settings configurable (and enabling end users to administer these without changing your application) takes time and effort. Building more complex rule engines is even harder. Simply hard-coding your logic is much faster

While this does speed initial time to market, it too creates many problems.

Problems that Emerge Immediately: Unless you get the logic wrong, you rarely see immediate problems. (See the next paragraph for situations when you get the logic wrong.)

Problems that Emerge in Month 2: Business rule-based application logic requirements change often. Now, when you make a change in business rules you need to go deep inside the code to make changes. This requires testing of the change and regression testing of the whole application. This is risky, time-consuming and expensive.

Problems That Emerge in Quarter 2: Eventually you will need to make enough changes that you turn your application in “spaghetti code.” If you keep your original development team, you can manage this. If you switch team members, you will lose much time waiting for the new team to “hunt and peck” through the code base to figure out how to change things. Eventually (usually within 6-9 months) the team will tell you they want to re-build the application. Your “shortcut” has now led to creation of a whole new project.

Shortcut Number 3: Placing too much business logic in the database

Rational: Database Management Systems (DBMSs) come with whole sets of functionality to manage transaction completion and referential integrity. In addition, SQL is a higher-level domain-specific language. As such, building logic with Stored Procedures or Dynamic SQL is easier and faster than building the equivalent in Java, C++, C# or another language.

While this does speed initial time to market, it too creates many problems.

Problems that Emerge Immediately: You add load to the database. In general, it is much more difficult to scale stateful database capacity (see this post) than it is to scale stateless application servers. If you get enough of a spike in traffic, all of your users suffer. You can solve this with a bigger box – but this means more CPUs and less reliability (just compare the MTBF for a 192-CPU server to a 2-CPU server).

Problems that Emerge in Month 2: Business rule-based application logic requirements change often. Now, when you make a change in business rules you are modifying how you manage database information. This requires changes to how you manage your data model (changes that pose significant risk and require high levels of regressing testing). In addition, as time progresses you have more and more of your application logic spread out across your application and database (making your application harder or costlier to maintain).

Problems That Emerge in Quarter 2: Eventually you will need to make enough changes to your logic that you will break your data model (not only adding columns but now creating new entities and relationships). Changing the data model of an operational, large-scale database is akin to changing the engine of an in-flight aircraft. It can be done, but it is risky, costly and requires someone with real-world experience doing this. At this point, your “shortcut” is now a “dead end.”

Depicting this in the constitution of good architecture

Introducing the concept of Isolation of Logic not only circumvents these problems, it also enables you to create world-class applications that can “turn on a dime” in response to business changes. This is that High Repeatability / High Configurability Goal that all CEOs ask their CIOs and CTOs to achieve.

Isolation of Logic by Type was second article I defined for good architecture. Here us how I wrote it in the language of software architecture:

Article. II. Isolation of Logic by Type

Section. 1.

Applications shall only include presentation and navigation logic at the UI layer.

Section. 2.

Applications shall only employ data management systems for to explicit purpose of searching, accessing and updating data.

Section. 3.

Applications shall modularly separate business logic from application logic in order to enable changes in business policy without changes in executable code.

Section. 4.

Applications shall modularly separate business logic from presentation logic to enable rapid, low cost internationalization, localization, and branding.

Section. 5.

Modules shall soft-code all business logic, operational configuration and initialization settings. Soft-coding externalizes all configurable variables to a repository capable of supporting dynamic variable change and incorporation by all referencing modules without the need to “bounce” instances of the application in production or modify, recompile, rebuild, and re-release the underlying application source code base

You will notice that, I did not use specific technologies. This is because these principles hold for regardless of where we are in the technology curve (e.g., ten years ago, we used thick clients to enable back end servers to validate user input without requiring a slow HTTP Post, now we can use AJAX to do this “on the fly.”) In addition, a whole industry of rule and workflow engines developed to enable easy isolation of business logic from transaction and state management.

Why isn’t isolation of logic by type used universally?

The principles outlined above seem like “common sense.” So why isn’t everyone using them? I have seen three top reasons over and over:

Reason #1: Lack of experience

The best way you can learn not to put logic in the wrong place(s) (or to hard-code it) is to get “burned” by doing so. The best way you can learn to make soft coding of logic something that does not need software engineers is to be in an environment of rapid change. I was blessed to be at AOL where, 1) I needed to learn this to survive, and 2) I had access to people who could show me how to do it.

Since learning and repeating these principles at multiple companies I have found that once you expose a team to this approach, they never go back.

Reason #2: “Ready-Fire-Aim” mentality

I have never met a stakeholder that wants to wait longer for a system or application. Almost all of the time, stakeholders challenge teams to cut the time or cost on projects. This leads to short-term shortcuts.

However, no stakeholder – business or technical – wants to create technology that is inflexible or hard to maintain and adapt to evolving business needs. If you start from this premise, and collaborate both on what business needs are critical for launch and how the business will evolve over time, you can almost always come to an agreement that balances speed to market with creation of flexible, adaptable technology. (This is not a theoretical platitude; it is something that I have experienced over and over as both a business and technology stakeholder.)

Reason #3: License-based COTS vendors don’t want you to do this

Many of the commercial-of-the-shelf (COTS) vendors do not follow these principles. The most common violation is putting too much logic in the database. Vendors do this because it lets them adjust their applications more easily to many customers. This is fine during implementation (when there is no load on the system) but can become highly problematic later (when you are adding customizations, adapting to an evolving business model or performing an application upgrade).

The worst case I encountered was at a company that was the second or third largest user in the world (depending on whether you counted transaction volume or data volume) of a CRM package that created a whole business logic programming language around SQL. (What was even worse was the fact that this programming language was evoked from UI—forms, essentially splitting all business logic across the UI and the database—the worst possible scenario).

This company needed to upgrade their CRM package (their current version was no longer supported by the vendor). Unfortunately, the tight coupling of logic to the database would have required a multi-day application and database outage (effectively shutting down their business). As a result, they moved to different vendor (one that split business logic from the UI, database or even transaction processing, via use of rule and workflow engines). This move not only circumvented the problem, it also gave them a better CRM platform.

(Call me if you want the details on this problem and the vendors they used.)

How can you see if you suffer from this?

Take a look at your portfolio of projects over the past 12 months. If you see many tactical projects that required software engineers to make changes in UI, workflow, business rules, etc., you likely have insufficient isolation of logic. If these projects took more than 16-24 hours of work on average you definitely have a problem (if it took less, you may simply want to look at improved process and administration tools to enable business analysts to make these changes).

What it takes to successfully implement COTS enterprise apps

That dreaded failure number and its root cause

As soon as you start planning for an enterprise program you will almost immediately begin to hear some variation of the following quote:

“You know, X% of [enterprise vendor name’s] implementations fail or are canceled before they ever make it to production.” (Where X% is usually around 50% or 60%.)

I will leave communication of the exact number to groups like Gartner. Nevertheless, many enterprise programs fail. Nearly all exceed their recommended budget and time allowances, even for their scoped-down first phases (I will get into scope control in a whole other series of posts). The single most important reason for this I have seen is the following:

Program sponsors (and their program teams) forget to include much of the work required to implement a successful off the shelf product

As a result, they need to come back to Capital Review Teams or their CFO to ask for extra money to complete their programs. In good times, they get this money (but reduce or eliminate their IRR), in bad times they can see their programs severely cut back, delayed or killed outright. Most companies are not in good times right now.

Buying off the shelf and integrating does not eliminate as many costs as you might think

My last post outlined everything you have to do to built applications in-house. Buying off the shelf and integrating eliminates many of these (in lieu of licensing and integration costs). However, it leaves many tasks that are often underestimated in the traditional “build vs. buy” analysis. It is easiest to highlight these by directly comparing the tasks for each:

Activities by project discipline area (i.e., not a Waterfall plan)

Key: Struck-out = Eliminated by purchasing and integrating an enterprise application. Red = added to successfully integrate the enterprise application. Italics = Clarifying note

  • Scoping (About half the time vs. building it yourself)
    • Negotiating contracts for acquisition
    • Negotiating contracts and Statements of Work for integration
    • Defining Concept of Operations or Vision Document outlining how the application will work at a high-level
    • Negotiating scope and phasing of capabilities over time
    • Estimating cost and time for this
  • Requirements Management (About one third of the time vs. building it yourself)
    • Capturing Use Cases, translating these into technical feature configuration and integration requirements
    • Determining non-functional requirements – performance, reliability, scalability, availability
    • Negotiating sign-off and prioritization by phase
  • Design (About one quarter of the time vs. building it yourself)
    • Designing an top-level architecture
    • Designing the logical data model. Specifying the configurable meta data for the application.
    • Designing the object model.
    • Designing the software architecture (what is internal vs. external, what is the software vs. the database, etc.)
    • Designing interfaces with external systems
    • Designing the User Interface – a huge one from determining style conventions, wireframes, mockups, usability analysis, Section 508 compliance and lots more. Specifying configuration settings for the User Interface (much easier than designing “frem scratch”)
    • Specifying the business and workflow rules to configure in the system
    • Designing the systems architecture – how you will handle maintenance, scaling on-the-fly, backup and recovery, disaster recovery, elimination of SPOFs (Single Points of Failure) – this task is so large that I built a whole portion of my career doing this for utility-class systems (always-on systems with reliabilities exceeding 99.999% and scales exceeding 10,000 transactions per second)
  • Infrastructure Management (Often LONGER than building it yourself as your infrastructure team will have to learn how the application is architected)
    • Capacity planning – how much hardware to you need, where should it be, when do you need it
    • Hardware acquisition – Selecting and ordering your hardware
    • Systems Integration – Installing the operating systems, patches, and base middleware. Integrating hardware into your network (configuring IP settings, adding to your routers and switches, much more)
    • Setting up your monitoring and production support teams and systems
    • Doing this for EVERY environment – development, integration, certification and production (your environments will multiple post-launch as you will have systems under development AND in production)
  • Construction (About one-third of the time vs. building it yourself)
    • Setting up the code repository
    • Setting up your continuous build process
    • Defining your interfaces and configuration management files. Defining exactly how existing systems will integrate with the new system
    • Building your databases
    • Building your database scripts and stored procedures
    • Building software modules
    • Building hooks for monitoring and business intelligence
    • Defining and executing your unit test cases around configured metadata, business rules and work-flows
    • Integrating and verifying the integration of your software modules with your databases and external interfaces the application with the rest of your environment.
  • Certification
    • Tracing your configuration requirements (functional and non-functional) to Test Cases
    • Building Configuring your Test Harnesses
    • Conducting verification testing, sending configuration builds back for repair, integration and certification/regression testing
    • Optional – usually done for highly-integrated mission-critical applications only: Conducting load and performance testing, including analyzing and repairing performance and scale bottlenecks
  • Training & Support
    • Developing training documentation and curriculum (for end users)
    • Setting up your Help Desk – including your knowledge base of how-to resolve expected questions
    • Setting up a Center of Excellence to manage the application after implementation (you will need to build or rent as your internal staff will not have the expertise to do this)
    • Training end users
  • Project Management
    • Developing the plan (tasks, durations, assignments, critical path, etc.) to do all this
    • Managing and resolving all those issues that come up with something this complicated
    • Estailishing governance for decision-making and issue resolution – requiring lots of time from your executives
  • General Management
    • Getting approval for the money to do this – Capital and Expense (often requiring you to in front of your Board of Capital Review Committee)
    • Managing the RFI, RFP and contact negotiation process for your Integration Vendor
    • Getting (or reallocating) the staff to do all of this – usually this is now external staff such as Integration Vendors (pro: you do not divert as many of your staff, con: they are more expensive than internals)
    • Managing the staff for requirements, design, development, integraiton, testing and installation (acquiring resources, approving vacation, backfilling departures, etc.)
    • Managing the capital and expense expenditures, including all the invoices for purchases and services and weekly or monthly reporting to your Finance organization
    • Handing all those items like staff departures (people quitting, not performing, getting sick, etc.) – This is reduced as you have “outsourced” a good fraction of it to your implementation vendor

This is still a lot of work — and a lot of time you could instead focus on the core of your business. Some will be done by internal staff. Much will require you to hire an external integrator (as your internal staff will not have the expertise to do this). The good news is that this frees up more internal staff than building it yourself. The bad news is that external implementation staff often cost 1.5x to 2.5x as much as your internal staff (making schedule overruns very costly). The cost of these overruns increases exponentially based on your degree of under estimate – this is known as Parkinson’s Law. Using a SaaS model avoids many of these costs.