EAI Convergence: IBM WBI & TIBCO
This paper reflects the vendors stacks as of June 2004. Some of the products have been phased out or re-labelled since then. Still, it is important to discuss them because of the historical perspective of the product development and broad install based. Examples of such products include the already phased-out TIBCO’s Message Broker or the no-longer-actively-sold Integration Manager.
This paper is organized as follows:
- Model outlines the convergent functionality that emerges from the comparison.
- TIBCO implementation maps TIBCO’s portfolio to this model; likewise, IBM implementation maps the model onto the IBM stack.
- Finally, both products are compared and final conclusions are drawn in Comparison
Model
Speaking crudely, objective of EAI is to connect disparate systems in a cost-efficient fashion. Such integration is possible by employing a set of enabling techniques such; the key techniques include loose coupling or the reduction of topological by introducing a common medium between the to-be integrated systems (integrate once, re-compose many times).
Specific integration cases aside (such as batch integration, ETL for data warehousing, common database / operational data stores, etc), EAI vendors offers a generic model based on message-oriented middleware (MOM). This model includes the following functions (Figure 1):
- To-be-integrated applications
- Connectivity between applications and transport
- Transport
- Transformation & Routing
- Task Flow
- Business Process Management
Other commonly seen functions, but not discussed here include user interface, operations support, or recently emerging Business Activity Monitoring (BAM).
Figure 1 Integration stack
Connectivity
The Connectivity function encapsulates the transport and target application interface by connecting each integrated application to the transport function. In the EAI world, the connectivity function is provided by adapters and connectors.
Transport
Transport performs two important functions:
- Moves the data across the network.
- Facilitates some of the key aspects loose coupling[1], most importantly temporal and technological de-coupling.
Transformation & Routing
In almost all integration activities, transformation from source format to target format is required. The base transformation process involves format modifications (both syntax and semantic mapping). More sophisticated transformation services offer message augmentation (message boosting) or data cleansing (for example removal of duplicates) that may be required to handle the differences between the to-be-integrated applications.
Except for basic point-to-point integration routing of messages is a key component of an integration solution. These message routing activities encompass:
- Managing the delivery of the messages over communication connections, including protocol conversion, flow control, guaranteed delivery, and connection optimization (for example, connection pooling)
- Multi-point message decomposition/re-composition, enabling one to many and many to one message routing
- Content-based routing with associated directory-based or rules-based routing.
Transformation and routing function are typically provided by the same component of an integration broker and for that reason we group them together.
Task Flow
The Task Flow Management function of the broker coordinates relatively simple, short time activities amongst the integrated systems. Task flow management allows for re-combining applications functionality to yield a more complex functionality.
Frequently, the Business Process Management (see below) function delegates a single business tasks to the Task Flow Management function. The Task Flow Management function service translates the business task into a set of lower-level (often application specific) technical tasks.
Task Flow Management service is also known as: technical process management, micro-workflow, system-level workflow, and message choreography
Business Process Management
The Business Process Management function (BPM) coordinates long-running business processes. In many aspects, BPMS resembles Task Flow Management function. BPM differs from Task Flow Management in that:
- It is geared towards managing tasks that may range from hours to months in duration. BPM persists its state in a database.
- BPM focuses on business-level tasks, frequently tasks that are performed by humans. To support such tasks, BPM support sophisticates access control and authorization models, escalation procedures, task delegation, etc.
Other services
Other integration functions are out of scope for further discussion (non-core services). These include:
- User interface facilitates user’s access to the EAI-integrated systems (‘consolidated application’ integration model). User access implementations include various portal packages.
- Common persistence mechanism for cross-system key mapping or system-wide data storage.
- Operations support functions such as monitoring and proactive error handling.
- Newer integration services such as BAM functions, event matching, rules engines, and other next big things.
TIBCO implementation
TIBCO started its presence in the EAI space started as a message-oriented middleware vendor with it’s flagship product, RendezVous (RV). Over time, TIBCO shifted its focus from MOM to the higher stack levels. Either by acquisitions (InConcert, Staffware, Talarian, BAM platform) or in house development (MessagBroker, IntegrationManager, BusinessWorks), TIBCO has assembled a complete EAI stack (Figure 2).
Figure 2 TIBCO projection
Messaging
TIBCO offers a portfolio of messaging products:
- RendezVous (RV) has been traditionally the dominant product. RV is a distributed UDP/multicast messaging system augmented with ‘certified messaging’ to provide reliable delivery (at least once delivery) and RVTX extension for transactional (once and only once) delivery. Apparently, as of August 2005, RVTX is no longer sold.
- EMS is TIBCO’s own implementation of hub-and-spoke style JMS server database-based, very similar to other pure-play JMS offerings such as Sonic MQ or WebLogic JMS.
- Talarian/SmartSockets high-volume near-real-time best-effort messaging.
Transformation & routing, Taskflow
Collection of T&R products reflects the history of TIBCO focus shift away from messaging and towards higher stack components:
- The original T&R product, message broker, was essentially an add-on to the messaging platform that allowed for stateless message transformation, but no task flow control. This product is now discontinued and replaced by IntegrationManager / BusinessWorks.
- IntegrationManager is essentially the next generation MessageBroker. It adds task flow control, multi-transport capabilities (specifically JMS, CORBA, JDBC) but remains very much a MOM add-on. IntegrationManager is tightly coupled with RV and relies on RV/AE message format as internal data format.
- BusinessWorks represents a shift towards an integration broker model with support for multiple transports, such that RV remains just one of multiple transports to chose from. BusinessWorks uses XML as internal messaging format, thus becoming virtually independent from RV (note, however, that BW still relies in RV for implementing high-availability features).
BPM
TIBCO’s BPM offering has grown through acquisition:
- InConcert — originally developed at [CHECK]– is a general purpose workflow management system with capabilities for managing both technical and ‘human’ processes. InConcert provides an extensive API, a prolog-based rules engine, dynamic process instantiation..
InConcert does not support modern workflow features such as BPEL modeling / execution. InConcerts user and administrative interfaces (at least the Windows-based fact cleint) are very limited and simplistic; at the same time the back-end engine is capable and robust .
Per unconfirmed signals from TIBCO, InConcert will be discontinued.
- Staffware is the recently acquired workflow engine with a strong foot especially in the. European market. [FILL IN – get more info].
Other products
- TIBCO has been one of the pioneers of BAM (both acquisition and in-house development, products include BusinessFactor and OpsFactor).
- Hawk is RV-based monitoring / proactive error handling tooll.
- TIB Portal Builder is a portal package providing user access function.
- TIB BusinessConnect and TIB BusinessPartner (rumoured to be discontinued) provide a B2B interface.
IBM implementation
Like TIBCO, through both acquisition and in house development, IBM assembled a complete integration stack (Figure 3).
IBM’s presence in the integration world started with MQ Series, the original commercial message queue. To this ‘core’, IBM added a message broker solution (MQSI [CHECK-was MQSI acquired from NEON, or just the rules engine was?]), an integration broker (Interchange from CrossWorlds), and a BPM (MQSeries Workflow.). In, parallel IBM developed the WebSphere J2EE application server family [on the basis of Orion?].
Figure 3 IBM projection
Messaging
IBM’s MQSeries, is the pioneer of the MOM market. Because of its market share (60% according to [Gartner - check]) and third-party vendors support (300+ vendors actively develop supporting products), it is de facto the standard amongst the message queueing products.
MQSeries architecture is best described as a multi-node hub-and spoke queueing system providing both reliable transactional delivery of messages across variety of operating systems and network protocol. The basic queueing function is augmented by a multitude of features including high-availability (clustering) and high throughput publish/subscribe (multicast).
Transformation & routing, Taskflow
IBM’s transformation and routing capabilities have been both built and acquired. IBM includes two separate components into their ‘stack’:
- WebSphere Integrator Broker is an in-house build MOM add-on that provides transformation, routing and flow control.
- WebSphere Interchange has been acquired from/with Crossworlds. Interchange is a full fledged transport-agnostic integration broker.
BPM
[Research / FILL IN ]
Other products
IBM’s portfolio includes a multitude of EAI related products:
- WebSphere J2EE application server.
- WebSphere Portal provides user access function.
- WebSphere DataInterchange is a B2B interface supporting standards such as EDI/X.12, HIPAA, or HL7.
Comparison
Initial Impressions
Both stacks grew by both acquisitions and in-house development. At the first glance, the TIBCO stack seems more consistent as it constucted of fewer products. In constrast, IBM constantly re-brands products (most recently, pre-pending all names brands ‘WebSphere’) making it difficult to understand.
Both stacks include products with overlapping functionality. It is not clear from the vendor which product should be used under what circumstances; rather, both vendors attempt to create an impression of perfectly coexisting and augmenting each other components. TIBCO seems to pay more attention to maintaining consistency within its portfolio by eliminating overlapping products (for example phasing out of MB, IM, and InConcert as newer components become available); IBM maintains multiple overlapping products but introduces a common front end for all of them (Eclipse IDE). This product management difference may simply be a result of sheer difference of size between both companies and TIBCO’s inability to maintain/support a ‘fat’ portfolio of products.
Messaging: Multicast versus Hub-and-Spoke
The comparison of TIBCO and IBM is de-facto as comparison of two messaging paradigms:
- Distributed multicast-based publish/subscriber represented by RV. For the purpose this comparison we focus on RV as the base TIBCO’s messaging; not, however, that TIBCO offers its own hub-and-spoke product as well (EMS).
- Hub-and spoke queueing (MQ Series).
In my opinion, the multicast-based publish/subscribe messaging is an excellent solution for near-real-time message dissemination when 1 Ã very-many delivery capabilities matter. RV originated on the trading floors as a vehicle for disseminating financial information such as stock prices. However, in most EAI cases the opposite requirements are true:
- ‘Cardinality’ of message delivery is 1-1, 1-2; 1à -very-many is a rare case.
- With exception of ‘consolidated application’ integration model (near real time request reply with timeout heuristics), reliability of message delivery takes priority over performance. Effectively, in most EAI implementations of RV, RV ends up simulating a queueing system using its ‘Certified Delivery[2]’ mechanism. While this works, it is a flawed solution (see the discussion of Certified Messaging below).
Reliable delivery is a ‘native’ function of hub-and-spoke solutions. The multicast solution must be augmented with local persistence mechanism and re-try mechanism
- While RV offers reliable delivery (queueing) referred to as Certified Messaging, this solution is flawed in that:
- Inherently, reliability of CM is not comparable to the hub-and-spoke topologies as the data is stored in local file systems using non-transactional disk operations, as opposed to centralized database in hub-and-spoke topology. Corruption of RV ‘ledger files’ is not a rare case that leads to loss of data.
[Would be nice to investigate ‘solidness’ of MQ in this regard]
- Temporal de-coupling is not the case. While a message can be queue for a later delivery (in case the target system is unavailable), for a message to be actually delivered both system must be up at the same time.
- RV / CM lacks a basic facility of any queueing system: a queue browser that is frequently required for production support (for example, in order to remove an offending message). In contrast, MQ Series offers an out-of the box queue browser and a host of third party solutions.
- Inherently, reliability of CM is not comparable to the hub-and-spoke topologies as the data is stored in local file systems using non-transactional disk operations, as opposed to centralized database in hub-and-spoke topology. Corruption of RV ‘ledger files’ is not a rare case that leads to loss of data.
- Transactional messaging is difficult to implement in multicast environment. TIBCO offers a transactional augmentation of RV (RVTX) that guarantees a 1-and-onlu-once delivery; however, this solution essentially converts RV into a hub-and-spoke system. Consequently, very few RV implementations are transactional.
Having said that, the shortcomings of hub and spoke include:
- Single point of failure, when the hub is down, everything is down. MQ Series addresses this problem by providing high-reliability clustering.
- Non-native publish/subscriber (publish/subscribe is emulated programmatically ) that results in reduced performance, especially in 1-to-very many delivery
- Overhead of hub management (a need for administering hub objects: queues, channels, etc).
- Inferior performance, especially in 1Ã many publish/subscribe and request/reply cases.
These shortcomings — in my opinion — do not outweigh benefits and for that reason MQ Series / hub-and-spoke solutions constitute a better choice for most EAI problem. Multicast-based publish subscribe is better left to its niches (high volume, high performance, accepter unreliability, 1 Ã very many).
Also, note the following new offering, indicative of product convergence:
- The discussion ignored TIBCO’s newer offering: the EMS hub-and-spoke server. While it is not yet an official position of TIBCO’s, it appears that TIBCO recognizes RV’s liabilities and shifts to EMS as the base transport.
- MQ Series provides a multicast add-on for niche publish/subscribe problems.
Other concerns:
- At the first glance, MQ Series offers an unparalleled presence across operating and programming systems languages.
- MQ is message-format agnostic. RV used to rely on a family of proprietary message formats (Active Enterprise / AE ) only recently shifting to XML.
- While messaging appears to commoditize, both vendors decline that that’s the case stressing their unique capabilities in niche applications.
- MQ Series is conceptually far more complex than its JMS substrate. [ further research: is this extra complexity warranted?}
- Both vendors introduce dependencies on their proprietary messaging into the higher level of the stack. For example, TIBCO’s BusinessWorks integration broker uses RV as internal messaging; IBM’s MQSI is tightly coupled with MQ Series (though the newest version allows for stating processes from non-MQ channels); IBM has introduced MQ dependencies into Crossworlds.
Transformation & Routing, Task Flow
Both stacks include more than one component implementing T&R and task flow functionalities:
IBM:
- WebSphere Integrator
- WebSphere Interchange
TIBCO:
- MessageBroker
- IntegrationManager
- BusinessWorks
For the purpose of this discussion we skip Message Broker from TIBCO as this product has been discontinued (note that Integration Manager, a successor of MB, is also rumoured to be discontinued). Table 1 summarizes the comparison.
IBM
TIBCO
WebSphere Integrator
WebSphere Interchange
Integration Manager
BusinessWorks
Architecture
Model
Transport-centric. A MOM add-on.
Integration broker
Transport centric
Integration broker
Runtime
C/C++
JVM
JVM
JVM
External
Abundance of imaginable formats
Multiple formats, but brokers do not operate directly on data
but require translation to internal formats first
Transport support
MQ-centric
Multi-transport
RV-centric
Multi-transport; RV required
Functionality
Transaformations
& flow control
Any-Any
Java - Java
RV-RV
XSLT
Internal format
Any
Java classes; GBO / ABO
RendezVous / AE
XML
Long processes
No
Yes
Yes
No
Development environment
‘friendliness’
Unfriendly / Eclipse
Friendly
Friendly
Friendly
Maturity of development model
Matiure
Matrue
Immature.
Mature
Back-end
Platform
Custom
Custom
Custom
Custom
Administration
Mature
Mature
Semi-mature
Mature
Transaction management
XA/XOpen
XA/XOpen
Business transaction support!
No
Limited to single source, no, distributed transactions, no
business transactions
High-availabilty
MQ Based
MQ Based [Research]
RV Distributed Queue / Fault Tolerance
Special features
High performance afforded by native non-JVM implementation.
Introduced GBO/ABO
Business transaction support
General forward-thinking in EAI
Long-running flows (checkpointing)
XML/XSLT based translations
Table 1 T&R and task flow comparison
Overall model [3]
Both stacks offer a MOM-add-on (transport centric) and an integration-broker component. In both cases, the integration brokers are the newer components:
- IBM’s MQSI and TIBCO integration manager are transport add-ons.
- MQSI is tightly linked to MQ Series. It requires MQ Series to operate. Only the most recent version of MQSI allows for launching a task flows by non-MQ events.
- Integration Manager is tightly linked to RendezVous. Integration Manager requires RV to execute and relies on RV/ActiveEnterprise data format for internal data exchange.
TIBCO BuseinesWorks and IBM Interchange represent a shift towards the integration broker model.
BusinessWorks continues to rely RV as the base transport and uses RV it to implement high availability features (failover, load balancing). However, BusinessWorks no longer requires RV data format for internal message exchange; it relies on XML instead.
Likewise, IBM Interchange (originally Crossworlds), is a transport independent product that implements integration broker model.
Runtime
All products except MQSI execute within a JVM; I expect MSQI to yield for smaller footprint and produce higher performance than the rest of the products.
Functionality
MQSI is unique in its support for any multiple internal data format and any-to-any translations. This is very much in line with the format-agnostic nature of MQ. The remaining three products rely on uniform internal formats: AE/RV (Integration Manager), XML (BusinessWorks) or Java (Interchange)
Interchange appears the most innovative product in the pack. It stresses the conceptual aspect of integration. For example, Interchange/Crossworlds and was the first product on the market to incorporate the concept of Canonical Data Model into the product; Crossworlds/Interchange institutionalized this pattern and requires programmer to express interactions in terms of general business object model (GBO) versus application-specific objects (ABO). Crossworlds/Interchange forces the user to think about generalizations and to capitalize on commonalities by forcing the user to express interactions as a set of CRUD operations upon the common objects implemented within the applications.
All components provide task flow function (TIBCO’s message broker is the only exception; its function is limited to transformation and routing; all processing is stateless). Interchange and Integration Manager provide functions typically reserved for BPM products – ability to persist flow state and thus coordinate long running business processes. While this function does not obsolete a need for a BPM product, it allows for constructing limited technical long-running flows.
Both products provide a broad coverage of connectivity and format standard. However, IBM products provide a more reach set of functions for developing mission critical applications, specifically in the area of transaction management:
- MQSI allows the flows and the flow-coordinated resources to participate in distributed XA-compliant transaction transactions (it still requires an external transaction monitor).
- Interchange offers ‘business transaction management’ protocol; while this is not a standard protocol, such as WS-Transactions, Interchange provides a framework for compensatory-action based transaction manager. To my knowledge this is very a unique feature amongst the EAI products.
- In contrast, TIBCO products offer minimal support for constructing transactional system. Neither business transactions nor distributed transactions are supported. IntegrationManager offers no transactional support altogether [VERIFY]; BusinessWorks allow for managing transactions but within single data source only. Thus, encompassing – for example – acceptance of a message and database update within a single transaction is impossible.
In general, it appears that:
- Transport-centric technologies are the older ones and the focus shifts to ‘integration broker’ model. This most likely reflects the multi-transport nature of integration recognizing that most enterprises rely on multiple transports. At the same time, it allows enterprises to take advantage of commoditization and standardization while building portfolio-based EAI stacks.
- Both systems offer ‘complete’ functionality for constructing integrations except for a serious gap in the area of transaction management in TIBCO’s stack.
Development model
We’ll look at the development model from two angles:
- Friendliness –easiness of programming, intuitiveness.
- Maturity – easiness engineering software using product: is it ready for integrating with version control, is it designed to capitalize on commonalities?
In general, TIBCO products appear more user friendly than IBM. Both BusinessWorks and IntegrationManager offer relatively intuitive user interfaces. IBM’s MQSI is counterintuitive and difficult to use. Interchange/Crossworlds improves on MQSI and is comparable with TIBCO’s products.
At the same time IBM products appear significantly more mature than TIBCOs:
- Both MQSI and Interchange/Crossworlds [VERIFY Crossworlds] store development artifacts in separate files, allowing for per-component version control. Common Eclipse-based IDE enables integration with popular version control tools.
- In contrast, IntegrationManager stores ALL development artifacts in a single repository/file, thus disabling the ability to put finer-grained artifacts under configuration management; likewise, it is very difficult to share basic commonalities (such as data schemas) between multiple projects. BusinessWorks (and the underlying Designer 5x) stores project components in per-component files and is integrated with popular version control tools, such as Perforce.
- Both IBM products are designed to capture and capitalize on commonalities. MQSI captures translations and flows in libraries that can be re-used across multiple integrations. Interchange – in addition to pioneering GBO/ABO – allows for capturing commonalities in ‘interaction templates’ and allows for re-use of transforms.
This is in a sharp contrast to TIBCO’s products where capitalization on commonalities in behavior requires copy-paste; BusinessWorkers represents a slight improvement in this area as it represents transforms as XSTL sheets, this potentially allowing for building generic constructs.
‘Refactoring suport’ / refactoring browse – [FILL IN]
Summarizing, IBM’s product focus on engineering aspects of integration development, while TIBCO accents user friendliness. This results in a trade-off is between immediate productivity and ease of long-term maintenance.
Back-end
All components execute logic within custom containers. This is especially surprising and disappointing in case of IBM: one would expect that the brokering solutions (especially the java-based one: interconnect) would be built on top of the WebSphere application server thus allowing to unify management and administration.
All components offer reasonably sophisticated administration tools. TIBCO tools has been lacking a proper command-line administration tools but is rapidly improving. BusinessWorks’ command-line tools allows for automating administrative tasks, such as deployment; these tools are out of the box in IBM stack [verify Crossworlds]
BPM
[Reasearch MQ Workflow]
Summary
[FILL IN]
——————————————————————————–
[1] In general, the lower the degree of coupling between the components, the more flexible and robust their combination (software system) is. Therefore, low degree of coupling is – in most cases — a desired property of a software system. A system that exhibits a low degree of coupling is referred to as loosely coupled.
Coupling pertains to multiple aspects of relations between software systems components. Examples of coupling include technology coupling (components must use the same technology in order to interoperate), location coupling (components must know each others location), or logic coupling (components logic is interdependent; components operate through a non-generic fine-grained APIs). queueing), technology coupling, location coupling, network addressing coupling.
Temporal coupling denotes the existence of temporal dependencies between the components of a system. A temporally coupled system imposes time requirements on interoperating components. For example, a client-server system operating in request/reply fashion is tightly coupled, as it requires that the server be available at the time the client originates a request or a timeout occurs. In a temporally decoupled system, the temporal dependencies are reduced. This is typically achieved by some sort of store-and-forward cross-component information exchange mechanism and by asynchronous request processing. For example, an order processing system that retrieves an order entry request message from an MQ Series queue is temporarily decoupled from the asynchronous
[2] Certified Delivery augments RV with ‘at least once delivery’ semantics. Crudely speaking, if delivery of a message is temporarily impossible, RV stores ‘certified’ messages in local ‘ledger files’. RV re-publishes the message once the target system is ready to accept messages again.
[3] For the purpose of this document we divide integration topologies into transport-centric and integration-broker-centric.
In the transport-centric model. messaging constitutes the common platform (i.e. the ‘integration universe’) for cross-system communication. This passive transport core is augmented by active message brokers that provide transformation, routing, and task flow.
Transport-centric
In integration-broker-centric topology, an integration broker is a software component that enables system integration by managing and orchestrating message flows between the broker-integrated systems. An integration broker is transport-agnostic in that it can control message flows between a varieties of transports. The broker itself (and not messaging) constitutes the ‘integration universe’.
Broker-centric