Architecture and Planning for Integration

Enterprise Application Integration (EAI) is one of the most critical areas in IT today. EAI technologies available today can greatly ease the task of integrating disparate systems. However, if not backed by a solid architecture and plan before beginning to deploy them, these technologies will not yield the hoped-for results of enabling organizational responsiveness to rapidly changing market conditions. It’s critical that the use of EAI technologies is seen as a decision about creating a platform of flexible, reusable services that support rapid reconfiguration of business processes. The goal must be about enabling enterprise business processes.

In this article, I highlight three common pitfalls that companies fall into when they try to deploy EAI technologies without a solid architecture and a clear plan in place.

Development Over Operations

An integration solution needs to support long-term operability and recoverability. Frequently, these get overlooked, and what happens later isn’t pretty. Architectural deficiencies usually aren’t discovered until well into the later phases of a project, and remediation can be disruptive to both project budgets and timelines.

One example is the need for end-to-end traceability of a transaction. This is a vital capability for recoverability. When integrating applications, best practice calls for the use of Application Programming Interfaces (APIs) or native connectors provided by those applications. This ensures that any required processing rules are taken care of by the API or connector before the incoming data is loaded into the application’s core components. Frequently, these APIs or connectors will natively generate data to be stored with the external data, such as a serially assigned unique transaction identifier, e.g., Purchase Order (PO) numbers. Even if incoming data already had a PO number, systems such as SAP may require the assignment of a new, unique number. If the API processing fails, the original PO number or another transaction identifier could be already lost and difficult to trace if the scenario wasn’t accounted for up-front in the architecture.

In a complex, high-volume environment, these sorts of problems can quickly undermine confidence in the system. Imagine an environment where one front-end application passes information to five different back-end systems, and one of the updates fails. Even if the transaction can be traced and corrected, how can the integration be re-run without duplicate entries or inconsistencies between the original transaction and the corrected transaction?

Operability and recoverability are key capabilities that must be considered up-front and designed into the architecture of the solutions.

Tight-Coupling

To provide flexible support for processes and reconfiguration, the integration solution must allow for the abstraction of business objects and rules. This abstraction requires a clear understanding of the application landscape, which often is heterogeneous. Reliable, flexible integration cannot be achieved without an abstracted model of the business objects that are updated and the methods for updating them.

Organizations need to define Canonical Business Objects (CBOs) to enable loose coupling in the integration. A CBO is a superset of the definition of an object in each of the applications within the landscape that requires that type of object to function. For example, the CBO definition for an invoice object would contain a superset of all data elements of all invoice data types in all affected applications. It would also contain the rules required to convert the canonical object into the Application-specific Business Object (ABO). These ABO definitions would correspond to the API specifications or connector signatures required for the integration to succeed.

Theoretically, it’s fine to populate this central business object, and it should not matter where the data is coming from or going if object methods are used. However, avoid any design that requires bidirectional synchronization; otherwise, the issues of traceability, recoverability, and operability will again threaten to undermine the process integrity. For example, simple rules for master data updates must be developed and followed to avoid the need for bidirectional synchronization. For every master data object, there needs to be a single source of truth for the master information. That single source of truth would then be responsible for propagating data to all systems requiring that information, commonly through a publish and subscribe model. Propagating information from one master source to another system and then from there to a third should be avoided to minimize complexity.

In large, complex environments, there will be multiple places where the same data is needed. For example, many companies use SAP for financial systems and PeopleSoft for human resources (HR) and payroll. SAP requires HR data to drive workflow, messaging, and other aspects of the system. So, a good architecture needs to define explicitly, for each data element, the point of origination and when and how updates are made to this HR data.

Sometimes, the master application won't collect or provide enough information to create a complete ABO in another system. For these cases, the (intermediate) integration could be used to elicit the additional requisite information from other sources in a request-reply fashion.

The use of an abstracted business object model began several years ago as a Service Oriented Architecture (SAO) concept but with the advent of Microservices architectures. The idea has generated much debate among its advocates and opponents alike. Indeed, as data integration requirements grow, CBOs can easily become heavy and complex, making them difficult to manage. The canonical object should therefore be kept lightweight and generic enough, with a lot of optional fields, to allow for various systems to use only a subset of what is relevant to them. It should be dosed with a little practicality, too, and designed at an appropriate scope level, for example, at a functional domain level as opposed to an enterprise-wide level. Lastly, it is important to involve data providers and data consumers in the design and development of CBOs. Their input will be valuable in enriching the CBO, and their participation will significantly increase their willingness to accept the CBO.

Demonstrating ROI

Demonstrating a positive Return on Investment (ROI) or Net Present Value (NPV) can be difficult for EAI solutions. That’s partly because EAI platforms tend to be behind-the-scenes enablers of other, more visible systems. Quantifying their value, therefore, is not as straightforward. Then there is the general mistrust between business and IT. Many IT people view the finance folks as obstacles preventing them from doing the right thing and deploying enterprise solutions. And those responsible for ensuring profitability don't always believe the promises IT people make. They've heard the stories before, and often — the technology didn't meet expectations.

Again, this points to the importance of having a solid architecture and plan in place before launching an EAI effort. It reduces the cost and risk of deploying new technology. Moreover, in the eyes of the business, it reassures them that there is an overall IT plan and that this project fits into that plan. It addresses the issues of trust.

Trust also stems from making clear commitments on cost and savings and then meeting or beating those commitments. Basically, to calculate an ROI, you need to know your upfront and ongoing costs for the new solution and the ongoing costs for the current or legacy solution. A simple formula that considers the time value of money and the cost to borrow or obtain equity financing is used to determine whether a project or technology is viable. There's little room for strategic fit or other soft factors. Many companies won't even allow the use of soft savings such as productivity improvement or resource allocation. They've frequently seen that such soft savings weren't actually realized on IT projects.

So, including the complete current operating cost picture is essential. It's difficult to do and rarely gets the attention it deserves. Benefits get overstated and costs understated. This may fund the current project but dooms future efforts because expectations set in the business case cannot be met. Finance managers will not easily forget a set of numbers promised as cost savings! They will also latch onto the most optimistic numbers they hear, even if they don't make it into the final business case.

Some current costs are obvious (e.g., the cost of maintaining the current tools and the labor needed to maintain current integrations), but others are less obvious and difficult to quantify. These hidden costs include:

• Labor is required to fix bad transactions and repair data repositories that get corrupted as a result of bad transactions. This is probably the largest source of overlooked costs. In the new EAI solution, we've presumably included the costs to create a highly fault-tolerant, operable, and recoverable environment. What were the costs of not having this, or what were the costs in people resources needed to monitor, repair, and rerun the current or legacy system?
• Infrastructure costs are needed to run the current or legacy integration jobs. This requires documenting all jobs related to integration and tracing back the costs to execute them.
• Developer resources are needed to maintain the current or legacy integration. Often, programmers will work on integration and application maintenance chores. You should make the effort required to separately assign a cost to each of these.
• Network and egress data costs for moving or transferring data from storage/source to destination. Many providers charge by the bit or transfer, and do these have a monthly cost?
• Batch processing costs — what did it cost to have batch versus real-time or near real-time integration? Did this result in a slower response to customer requests for inventory availability or slower credit approvals because the data wasn't available?
• Data integrity costs —what's the cost of redundant or incorrect data in the legacy systems? What happens when the information in the payroll system doesn't match the HR system?

Evaluating all these factors and assigning costs to them on the current or legacy side and benefits on the new application will be critical to getting EAI projects funded and approved. Set expectations upfront, then meet them. Often, the right thing for the organization is to employ a tool, but usually, IT managers aren't diligent enough to run down all these hidden costs. This then requires them to promise the moon in benefits to pass the ROI hurdle. Fully exposing the costs of the current integration environment will help deliver a compelling ROI case and set realistic expectations for the new environment.

Conclusion

Introducing EAI technologies into an organization isn't for the faint of heart. However, the promised benefits and cost savings can be achieved, and the use of the tool is justified. A competitive advantage can accrue from developing a comprehensive architecture and a solid plan. The EAI solution should allow rapid reconfiguration of information flow between information systems to support an adaptive business process environment.