SOI: marrying EAI and SOA
How do you apply SOA ideas in the EAI world? the following is an outline of Service-oriented Integration architecture: an EAI based on SOA concepts. In order to marry the two, we’ll borrow some SOA concepts, we’ll reject some of them and finally we’ll add a few extensions in order to cover EAI-specific functions.
SOA Defined
SOA is an approach to constructing enterprise systems from components. (services) An SOA system is a collection of interacting services. Each service provides a well-defined collection of coarse grained functionalities.
Figure 1 illustrates components of an SOA system:
- At the core of an SOA system is a service. A service is a discoverable software entity that offers coarse-grained functionality through well-defined, published interface. A service is offered by a service provider – a software entity that implements the service specification.
- A service is accessed by a service requestor. A service requestor may be a client application or a service itself.
- SOA is enabled by an SOA broker. An SOA broker provides the following two components:
Figure 1 SOA / Context diagram
- Service registry is a special service that maintains a database of services (service interfaces and service providers) and provides a lookup facility to this database. Service providers publish service interfaces to the service registry. Service requestors browse the service registry, discover the services and invoke them.
- Service invocation infrastructure allows the service requestors to bind to a service and invoke it.
Applying SOA to EAI
SOA is an attractive architectural model for EAI. SOA recognizes and embraces the distributed nature of most computing systems. The SOA’s concept of service as a collection of functionally related functions is close to the idea of an EAI-integrated system. At the same time SOA takes the concept of integration further: the service is to be reusable in multiple contexts. This requires that the services be properly abstracted for future re-use, as opposed to simply ‘making the applications talk’.
For the purpose of this architecture proposition, we will slightly tweak the SOA to fit the EAI needs. We will reject some of the ideas and expand upon the others.
The ideas we borrow
First the SOA ideas that we borrow to build our EAI architecture: the concept of service, the distributed nature of SAO, and the concept of service broker.
Traditionally, the EAI solutions have been operating on the API level. The typical adapters are level 1 / level 2 adapters that simply expose the application’s API to the middleware layer. API-oriented integration works reasonably well in simple cases but fails to deliver the most important promises of the EAI; the following two are the key reasons:
- The reduction in the topological complexity of an integration solution is limited to the transport layer (Figure 2).
Figure 2 Point-to-point integration on the logical levelThe big promise of EAI – or to be more precise, of the broker-centric architecture – is that it reduces the integration complexity from O(N2)to O(N) where N is the number of integrated applications. In case of the API centric integration this reduction does indeed materialize on the transport level: all applications are connected only once to the common medium. However, on the logical level, this rarely is the case.
First, the broker-integrated applications expose to the broker their native API’s. Each such API expresses application-specific semantics and in the worst case semantics of each broker-integrated application are different. These differences range from simple ones, such as different field names to complex ones, such as different cardinalities of cross-entity relationships between the data model entities. Consequently, for two applications to communicate, their semantic differences must be reconciled. This is achieved by ‘mapping’ of the fields between the messages exchanged by the applications. In a naïve — and unfortunately common – solution, such mapping is done on a point-to-point basis. A standard solution to this problem is the common semantics approach (a.k.a. common business objects, canonical data model). [1]
Second – the API’s not designed for re-use. The applications expose low-level, ad-hoc interfaces that too fine-grained to be easily re-used.
- The API based interfaces are fine-grained and thus fragile. It is not uncommon to see API’s to business applications that include tens if not hundreds of fields. The API methods are frequently very detailed; performing a single business functions, for example creation of an account, may require multiple invocations of a single business method. Such API’s tend to change frequently – as a response to application-specific changes – and thus create ‘ripple effects’ that affect multiple broker-integrated applications in unpredictable ways.
For the purpose of this architecture we will use the SOA’s concept of service as a solution to the problems of API-oriented interfaces. The concept of service as collection of functionally related coarse-grained functions is a natural ‘wrapper’ concept for exposing the functionality of to-be-integrated systems. We will expose the functionality of integrated applications as coarse-grained interfaces. We will achieve this coarse granularity in a bit unorthodox way (the common semantics being the orthodox way): through document-oriented interfaces.
SOA extensions
SOA is a natural fit in composite application problems. The to-be-integrated systems expose their interfaces as services to the broker; the broker — in turn – re-combines them and provides them to the end-users. As this is not the only integration model, we must expand the traditional SOA to meet the requirements of data integration and multi-step process models.
SOA implies request/reply semantics: service requestors invoke a service and receive a response. Such SOA is easiest implemented using synchronous transport; examples of such synchronous implementations of SOA include Web Services over HTTP or CORBA component model. EAI requires not only synchronous but also asynchronous transport. In this architecture we’ll go as far as mandating that asynchronous transport be the only transport used and that synchronous semantics be implemented over the asynchronous transport (see [MESSAGING]).
The data integration and multi-step process integration model require extensions to the basic SOA request/reply semantics. In the data integration model, upon the change of the data, the source-of-truth publishes an event notifying of the data change; the parties that replicate this information subscribeto the respective events and modify their data stores accordingly. As such, there is no service requestor and no service provider in this scenario. Rather, there is an event publisher and an event subscriber and the semantics of the communication model is publish/subscribe[2].
The multi-step process model employs a combination of request/reply and event-driven processing:
- The BPM coordinates long-running multi-step processes; at each step, it invokes the broker-integrated services. As the business processes are typically long running, the request/reply invocation of services operates in “response eventually needed” mode (and not “response needed immediately”). As such, asynchronous request/reply based on reliable messaging is in order, primarily because it affords us temporal de-coupling.
- Frequently, business processes operate in event-driven mode. For example, an event, such as new order, may start a business process.
In addition to the event-driven processing that the infrastructure provides for data integration, the basic SOA must be therefore enhanced to provide semantically asynchronous request/reply based on reliable messaging.
In order to accommodate the above, the proposed architecture extends the basic SOA to provide event-driven asynchronous semantics (Figure 3) as follows: in addition to fulfilling the requests, the services can also originate and consume events. Consumption of an event does not require a service to generate response. For clarity, we’ll use the following terminology to denote these special services:
- Services that consume events without producing a response are referred to as event subscribers.
- Service requestors that generate events are event publisher and not just by service requestor.
Figure 3 SOA
The ideas we reject
At this time it appears that the concept of service registries – such as UDDI – for machine-to-machine service invocation within the boundaries of a firewall is not practical; we reject this concept in our EAI architecture. Note, however, that we retain the concept of service registry as a central repository for service information; it just needs to be machine-readable. This is not to say that service registries and automated discovery of services do not work. It is author’s opinion that automated service discovery may be far better used in situation where a similar function can be performed by multiple services – for example in B2B scenarios. Also, discoverability is great for services that are to be used directly by humans, i.e. services with a human-facing aspect such as GUI or a form. Examples may include portals that contain a service directory or the Internet where the service registry is called Google[3].
Emerging architecture
Figure 4 illustrates an EAI system approached using SOA concepts.
Figure 4 EAI as SOA
A SOA-style EAI system consists of a collection of services that interact through the enabling infrastructure.
The SOA enabling infrastructure includes the integration broker and core services. It is augmented by infrastructure support functions and user interface.
Note on terminology: for the purpose of this architecture we’ll distinguish between services and functions. Both services and function are collections of coarse-grained functionalities; the difference is that services are accessible and invokeable through the broker infrastructure while the functions are not.
The integration broker enables the interaction between services by providing the following functions (look here for an explanation of these functions):
- Transport
- Connectivity
- Transformation
- Routing
- Taskflow
These broker functions are augmented by core services:
- Business Process Management (BPM).
- General Persistence Mechanism
The functionality is delivered by the fulfillment services. These include your CRM, your billing, your accounting, and what else you bought over the years. The fulfillment services may be re-combined into more sophisticated ones using the brokers taskflow and BPM services. Such services — that emerge as a result of recombination — will be referred to as composite services.
If you build a composite application, you’ll need a user interface. User interface is provided by a portal that delivers authentication/authorization, personalization, content management and related functions.
All that wonderful infrastructure must be monitored and managed. Thus the need for the infrastructure functions, such as monitoring or proactive error handling.
We’ll discuss all services and functions in detail in the following sections …
——————————————————————————–
[1] A standard solution to the problem point-to-point translations the common semantics approach (a.k.a. common business objects, canonical data model). The integration broker (or the application adapters) translates the application-specific semantics to the common semantics on the boundaries of the broker. As a result, the applications are not only technologically, but also semantically de-coupled. See the [INTEGRATION PATTERNS] section for more.
[2] Note that the publish/subscribe model pertains here to the cross-service communication over the broker. It does not imply that the underlying communication model employed by the broker is publish/subscribe.
[3]Another nice aspect of Internet as SOA is that it is built using document centric interfaces.