Integration Models
Integrating applications may have multiple and differing objectives. For instance, you may want to synchronize the data between disparate applications or provide the user community with a single view of multiple legacy systems that your organization collected over the years. Or you would like to do both.
Experience shows that application integration efforts fall into one of the following three categories (integration models): multi-step process, composite application, or data integration. Differentiating between the integration models is helpful: each of the models has different characteristics in terms of the processing model, appropriate implementation approach and software requirements. You are likely to start your integration by implementing one of the models expecting that this is the only model you’ll implement. That is unlikely. Sooner or later a need for other models will emerge. Therefore, it is my belief that a ‘complete’ integration solution should provide an infrastructure to support all models. At the same time, it is very likely (80/20 rule) that one of the models dominates. This model should drive the key architecture design decisions.
Multi-step process
The multi-step processintegration model (Figure 1) integrates the functional flow of processing between multiple applications.
Figure 1 Multi-step process application integration model
A good example of a multi-step process model is a telecommunication order-to-cash system. An order-to-cash system coordinates activities within multiple business and operations support systems in order to fulfill customer’s order a telecommunications service. The fulfillment of an order typically requires establishing an account in business support systems such as billing and CRM, reservation of network elements that comprise the service in a network inventory management system, and provisioning of the service on the network. Often, the process of provisioning the network elements requires coordination of work orders for engineering personnel; execution of such work orders may take days or weeks and not seconds like it is the case with automated systems. Interactions with humans are not limited to work orders; human interactions may be required for approvals, escalations, etc. Provisioning of complex telecommunication services frequently requires interaction with third-party suppliers. For example, a telecommunication company may outsource the ‘last mile’ connection or on-site installation. As the multi-step process integration often includes interactions with humans, the integrated business processes may require weeks or months to complete. A typical implementation approach relies on a Business Process Manager that coordinates such long-running business processes.
Composite application
The goal of the composite application (Figure 2) integration model is to provide a single virtual application that is composed of separate applications, using an integration platform.
Figure 2 Composite application integration model
A good example of composite application is ‘webbing-up’ of a set of legacy systems. Continuing with the telecommunication example, a telecommunication services provider may expose the systems that participate in multi-step ordering process to the end-customers via common front-end. The provider allows the customers to see a ‘single-pane’ view of their service that combines the business and technical characteristics of the service. The user can see on a single web page the service’s pricing (business characteristics) and implementation details such as the IP addresses or routing (technical characteristics). A common choice for facilitating such a common view is translating the customer’s request for a ‘single pane’ into a coordinated set of interactions with the back-end systems. The front-end system (typically an application server) delegates the ‘single pane’ request to the integration framework. The integration framework de-composes the request into a set of calls to the back-end systems, collects the responses, and provides the re-combined data back to the front-end.
Data integration
The goal of the data integration (Figure 3) integration model is to provide data consistency across multiple applications. This is accomplished by propagating data changes between the integrated applications.
Figure 3 Data integration application integration model
Expanding on the telecommunication services provider examples the need for data integration emerges when the components of an integrated share elements of the integration-wide data model. Perhaps, in an ideal world, there would be no need for data integration as all the systems would be updated as a part of multi-step process or consolidated application model. In reality, constructing such an infrastructure is rarely justified. For example, porting a highly-efficient and complex interface of customer relationship management system to the generic common front-end us unlikely to yield equally efficient interface and the costs of such migration would be prohibitive. Consequently, many applications within the integrated ecosystem are accessed directly through their native user interfaces. If a customer service representative keys-in a change to the customer’s address into the CRM application, then it would be valuable for that change to be reflected in the billing system; hence the need for data integration.
Data integration is typically implemented using event-driven publish-subscribe interactions between the application. An application that accepted changes to a shared data element (e.g CRM) ‘publishes’ an event of change onto the integration framework. Applications that are interested in the updates (e.g. billing or data warehouse applications) ‘subscribe’ to such interesting events and update their internal data stores in response.
Integration models’ implementation characteristics
Integration models have different characteristics in many areas terms of processing model, communication model, and required software. The following provides a few high-level characteristics without delving into the implementation details. See sections [XXX] and [XXX] for details on transactional processing, BPM, and task flow.
Table 1 Processing and distributed transactional model
| Multi-step process |
Multi-step process model comes in two flavors:
Both models require stateful coordination. Long-term stateful coordination is typically implemented by a BPM; the short term is typically implemented by task flow function of an integration broker. Both long-term and short term business processes typically rely on a compensatory-transaction-based business transaction[1] management model. [Refer to the transaction management section] |
|
| Consolidated application |
Consolidated application employs a short term stateful coordination, typically provided either by a task flow mechanism of an integration broker or by code executing in a transactional container of an integration server.
Depending on the implementation approach (application server or integration broker) distributed transactions may be implemented using traditional two-phase commit protocol (common application server model) or a compensatory-transaction-based schema (common integration broker model). |
|
| Data Integration |
Data integration in publish/subscribe mode requires no coordination at all. The processing of messages between the data-integrated applications is stateless and limited to transformation and routing. Publish/subscribe interactions are not centrally coordinated. As such, there is no distributed transaction coordination1. |
Table 2 Communication model
| Multi-step process |
Asynchronous interactions between the business process manager in ‘response eventually needed’ mode.
In case of short-term multi-step, specifically in time-critical environments or environments that require distributed two-phase commit transactions, synchronous[2] processing may be applicable. |
|
| Consolidated application | Consolidated application model creates an interactive application. Synchronous communication model is a typical choice. Note that it is possible that a consolidated application may enact a fire-and-forget or ‘response eventually needed’ semantics. A good example is front-end that allows for a submission of a form. In such cases asynchronous communication may be in order. | |
| Data Integration | Asynchronous publish/subscribe communication model is a natural choice. |
Table 3 Tools
|
All models |
All models require facilities for connectivity, transport, transformation, and routing. |
|
|
Multi-step process |
Long-term business process coordination requires an engine that is capable of persisting it’s state. A BPM product is a typical choice[3]. Short-term process coordination requires an engine that does not need to be able to persist it’s state. Typical solutions include Integration Brokers’ task flow manager or ad-hoc development within an application server. |
|
|
Consolidated application |
Application server or application platform is a common choice for constructing a consolidated application. |
|
|
Data Integration |
Integration broker’s transformation and routing |
Summary
Application integration encompasses three different integration models. Each model exhibits different and often conflicting with other models implementation characteristics.
In a typical scenario, only one of the models is implemented during the initial integration phase. However, as the integration matures, the need for the remaining models inevitably emerges. A durable integration platform must provide facilities for all three models.
——————————————————————————–
[1] The scope of this discussion is limited to distributed transactions that span the integration platform. and multiple applications. Distributed transactional semantics is required in other context as well. See [TRANSACTIONS] for more.
[2] This is not to impose a synchronous underlying communication model. A synchronous semantics may be implemented over asynchronous transport (see [MESSAGING] for more)
[3] Note that the BPM products provide a strong user-facing functionality for managing workflows that require human involvements; such functionality includes user interface, role/privilege management, workflow document management (versioning etc.) and allows for workflow modification by non-technical users. Such functionality is of a very limited applicability in multi-step process environments where the coordination is limited to automated systems.