• Verifying applicability of Agile requires an understanding of what Agile is. Detailing the Agile concepts is out of the scope of any single article, is definitely not an intention of this one. Instead, we’ll try to understand the structure of the ideas that comprise Agile. We’ll then create a carte du jour of – sometimes interdependent but oftentimes orthogonal – Agile ideas
• Having dissected the Agile we’ll take a look at integration and its cousins (EAI, SOA, SOI). We’ll evaluate the propositions on the menu of Agile ideas against the realities of application integration. We’ll determine the ideas that are immediately transferable. Then – more interestingly - we’ll discuss the agile practices that are currently not transferable. We’ll determine the nature of the obstacles to transferring them: are these obstacles inherent to the nature of integration or – perhaps – they result from other less obvious problems, such as the immaturity of integration tools?
• Finally, we’ll attempt to identify integration-specific productivity improvement and risk-reduction practices – some of them Agile-spirited, some not.

1. Dissecting Agile Application Development

The agile manifesto (http://agilemanifesto.org/) is probably the best starting point to understanding Agile. It lists the principles of agile development in an unstructured and very software-centric fashion. In order to perform an informed evaluation of Agile’s applicability to the integration, we’ll dissect the agile principles and try to identify a structure behind them. This article proposes the following structure for Agile Software Development (Figure 1 Structure of Agile Software Development Figure 1):

Figure 1 Structure of Agile Software Development

1. Adaptive Project Management is at the heart for Agile practices. Section 3 contrasts the Adaptive Project Management with the more traditional – for the lack of a better term - Plan-driven Project Management (which is not to imply that Agility precludes planning). It then identifies the reasons for Adaptivity’s success in business software development as well as defines the requirements for projects and products to be managed in an adaptive fashion.
2. Agile adopted several Lean Manufacturing ideas that happen to both orthogonal and a closely related Adaptive Project Management. We discuss them in section 5.
3. Finally the agile community promotes a number Software-specific Productivity Improvement Practices (section 5). Some of these practices – especially the ones relates to Sociology of Software Development, are useful not only on agile projects, and not only on software projects. Some, such as Refactoring, Continuous Integration, or Automated Testing are critical techniques that enable the Adaptive and Lean approaches discussed above.

2 Adaptive
Crudely speaking, project management is about (1) planning and (2) executing the plan.

2.1 Plan-driven and adaptive project management
We’ll introduce the adaptive project management by contrasting it to the traditional approach. Of course, there is a whole spectrum of in-between approaches.

Figure 2 Traditional planned approach

Figure 2 illustrates the traditional plan-driven approach. In this approach, planning is long term. The plan is very specific about the anticipated outcome (scope) as well as about the trajectory through time, the money to spend, and intermediate task breakdown that would get us to that outcome. During the execution of the plan, the objective of the project manager is to minimize the ‘delta’ between the planned trajectory (your list of tasks, with dates, and budget) and the actual trajectory (i.e. how late/early or how over budget you are).

While executing the project, you build your product incrementally, piece by piece. Therefore, the product becomes usable – typically – at the very end of the project. Oftentimes, this is when the customers see the product for the very first time. In the software industry, what they see often surprises them.

Figure 3 illustrates adaptive approach. In this approach, the project is executed in frequent (weekly to monthly) iterations. Each iteration yields more-and-more complete approximations of the final product. The project is managed based on the customers’ feedback to the product approximation. The product often becomes usable well before the complete vision is achieved, although it would then provide less features. Changes to the direction are frequent, based on customers’ feedback. The final target is therefore moving, although within the original vision of what is to be delivered.

Figure 3 Adaptive approach

2.2 Comparing planned and adaptive
Both approaches work fine, but for different type of projects.

The following are key differing characteristics of both approaches:

• Adaptive approach provides for an early customer’s feedback on the product. This is critical for new product development where the customer and designers ideas may significantly differ, be very vague, or the kind of product that is being design has not been ever tried before; therefore, having the ability for the customers to ‘touch’ an approximation is very important if we want to build something useful. Feedback is much less important in developing fairly standard and repetitive products, such as houses; a customer can often see existing model houses and have a fairly good feel for what’s being built for them.
• In many ways, the adaptive approach reduces a group of risks stemming from uncertainities about project requirements and employed technologies.
  The feedback on product mentioned above reduces the risk of building a product that does not meet customers’ requirements. The same feedback allows for measuring the progress of delivery against the customer’s expectations (and not against the plan), thus allowing to identify and fail early projects that are unlikely to satisfy user’s needs within the anticipated cost. This – again – is very important for new product development and much less important for repetitive and well understood projects.
• As adaptive approach delivers the product in approximations (and not in partial increments), it is often possible to use the products before project completion.
• The plan-driven approach allows for a tighter and more precise control over the ‘levers’ of the project (schedule, scope, and cost). Therefore it allows for precise management of different group of risks: risks that are associated with broadly understood logistical uncertainity, such slipping schedules, unexpected meteorological events, etc.
• The plan-driven approach allows for optimization of the project trajectory. The trajectory of adaptive approach is always suboptimal, but this is only apparent once the project is complete.

Therefore:

• The planned approach works well for predictable, often repetitive projects, where the anticipated outcome is relatively easy to define. Examples of such projects include construction projects (buildings, highways) or elaborate projects for switching-over oil refining installations:
  • A very important advantage of plan-driven approach for such projects is the ability to tightly control the scope, schedule, and cost, this allowing for precise management of logistical risk. You precisely plan your contingency for delays of supply delivery, or plan your building actions based the probability of rain. Sometimes this may be a critical safety requirement: for example, the oil-installation switchover requires a precise coordination of parallel activities, or severe accidents can occur.
  • The ability to optimize the parameters of the planned-in-advance project tasks (dynamic programming) is another important benefit for such projects.
    At the same time, these predictable projects do not require methods for managing the risks associated with the multiple unknowns of the product development projects: uncertainity of requirements and unknowns about the technology.
• The adaptive approach works well for difficult-to-predict, often non-repetitive projects where the anticipated outcome is difficult to define (business software). Examples include new product design (i.e. new car, new version of iPod) or business software development (custom claim processing system):
  The critical advantage of the repetitive approach is constant feedback. This includes:
  • Customer’s feedback reduces the risk of building a useless product.
  • ‘Subject matter feedback’ (i.e. how fast are progressing with the project using the technology in question, team in question, with a given experience) the allows to constantly re-adjust expectations regarding cost and schedule; this lets us identify and terminate doomed-to-fail projects very early.

  The benefits of adaptivity come at a cost. Logistical optimizations are unattainable. This makes a perfect sense: as it is know from systems theory, the closer to optimum a system is, the less flexible it becomes. That’s why your flexible SUV stands no chance in racing an optimized-for-racetrack F1 bolid, by try driving F1 bolid around town, not to mention off-roads.

2.3 Requirements for adaptive project management
Based on the above guidelines, it is easy to spot domains in which projects look like wonderful candidates for adaptive. Software projects look that way.

Not so fast!

It is not enough that the above ‘inviting’ characteristics of a project (uncertainity about the outcome, uncertainity about technology) are present. Adaptive approach requires that the subject of management be malleable. It is important that the product can be evolved and that the cost of evolving the product is not prohibitive. In order to effectively respond to the customers’ feedback, it must be possible to alter important aspects of the product late during the project, at a low cost. Therefore, adaptive approach is not applicable to building construction, where, after foundations are laid down, it is difficult to alter the building’s width (fortunately, the ‘inviting’ characteristics for adaptivity are not present either, so our loss is minimal).

What about software?

Software developed traditionally is not much more malleable than a building under construction. Changes to early decisions often result in cascades of ripple-effect changes and in huge costs of re-design and re-test.
Fortunately, modern software development techniques such as refactoring and test-driven development make software malleable and therefore possible to develop in an adaptive fashion. We discuss these malleability or evolution-enabling practices in section 5

2.4 Summary
Table 1 Plan-driven and Adaptive: key differentiators

3. Leanness

‘Leanness’ is the second cornerstone of Agile. In general, lean is about eliminating waste. The origins of lean software lie in the engineering concepts of lean production. The Production System Design Laboratory (PSD), Massachusetts Institute of Technology (MIT) http://lean2.mit.edu/ states that ‘Lean production is aimed at the elimination of waste in every area of production including customer relations, product design, supplier networks and factory management. Its goal is to incorporate less human effort, less inventory, less time to develop products, and less space to become highly responsive to customer demand while producing top quality products in the most efficient and economical manner possible.’

The lean production techniques aim at reducing various aspects of waste. Some of examples of waste in manufacturing include:

• Waiting time. Therefore cut waiting time as waiting is money wasted.
• Inventory. Therefore cut inventory ask holding inventory costs and is therefore waste (just-in-time inventory). Also modify scheduling such that the internal customer pull from the system instead of pushing the system (Kanban / http://www.strategosinc.com/kan.htm )
• Communication and coordination inefficiencies and overhead. Therefore work in small socio-technical systems (Workcells; http://www.strategosinc.com/workcell.htm )

The nature of waste in software is similar. As in manufacturing, there are communication and coordination problems. Yet – I believe – the core of waste in software comes from the traditional approaches to project management. Recall from the previous paragraph that the traditional plan-driven project management deals efficiently with logistical risks. Given a project in trouble, the traditional diagnosis is that the project requires better planning and more discipline. The remedies include beaurocracy of various sorts, tighter scope controls, more precise ‘design’ and tighter ‘construction’ controls, etc. These methods would be successful for a construction project, but given the nature of risks in software, these methods yield waste:

• Rigid and centrally controlled organizations move slowly, wasting time and requiring multiple formalities for simple operations. Yet what business software needs, is agility and a quick reaction to change.
• Formality requires bi-products, such as various ‘design’ documents that add little value to final product are generated. Yet in business software, construction IS design.
• Tighter scope controls convert the ‘change control boards’ into change prevention boards. And oftentimes teams deliver ‘to the scope’ and not to the real business need, producing useless products that need substantial time to be augmented with functionality that is actually needed.

The leanness proposed by Agile is perfectly possible, as extra beaurocracy will not remedy the inherent software risk that are not of logistical nature. Some of the lean principles include ( http://agilemanifesto.org ):

• Individuals and interactions over processes and tools.

  The traditional approach is to eliminate the ambiguity as the perceived source of risk, by implementing a formal (and wasteful) set of processes and tools. As ambiguity is inherent to software requirements, the iterative development with working and ‘touchable’ model is a much better remedy for ambiguity. While constructing the model, individual interaction is far more efficient and lean than formal well-documented communication.

• Working software over comprehensive documentation

  The traditional approach is to improve planning by more precise and well-documented software ‘design’ that then can be implemented. Yet this fails because of the inherent technology risk and the resulting fact that in software construction IS design, leaving the customer with binders of documents (waste) and no working software. The Agile approach is lean in that it moves directly to construction, delivering working, verifiable, ‘touchable’ product in a short timeframe.

• Customer collaboration over contract negotiation

  The traditional approach to dealing with change is by implementing various change-prevention mechanisms (a.k.a. known as Change Control Boards, Scope Control) that make changes to the course of project nearly impossible, leading to deliver of unwanted features. This generates waste both in paid-for but unwanted product as well as in cost of opportunities missed by lack of the needed product that was not delivered (‘not delivered instead’).

4. Software-specific Agile Practices

The third group of Agile ideas are the software-specific practices that are commonly employed on Agile projects. Some of these practices are enabling techniques for adaptive project management, some contribute to leanness, and some are generally good practices that are orthogonal to lean / agile altogether. Much has been written on these practices, so for the purpose of this overview, we’ll list briefly only the most important ones, with explanations as to how they contribute to agility, and with references to to proper definition:

• Automated testing and test-driven development(http://www.objectmentor.com/writeUps/TestDrivenDevelopment).
  Automated testing is a critical enabling technique to adaptive project management. First, the automation of regression testing enables short iterations. Second, automated testing contributes to malleability of the product; automated tests provide a safety net for changes, thus making changes safe and cheap.
• Refactoring ( http://www.refactoring.com/ ) – through maintaining the clarity of product’s structure - contributes to malleability of the product. In addition, refactoring browsers reduce cost of refactoring and other changes, thus making evolving products more economically feasible.
• Other technical practices include Continuos Integration (http://www.martinfowler.com/articles/continuousIntegration.html ) and Object Orientation.
• Multiple non-technical social practices (pair programming, collective code ownership, user stories, etc) tend to promote ‘leannes’ of the process, through improving direct communication and knowledge sharing, and reducing the need for heavy frameworks to manage the two.

5. Summary
We have ‘sliced’ the concepts of Agility. We are now ready to evaluate the applicability of the Agile techniques to Integration.

What am I trying to do?

I am trying to publish a message (store-and-forward message) to Message Box and have another orchestration - which may not be active at the time of message origination - be started by ANOTHER MESSAGE (kick-off message) and pick the message in question (store-and-forward message) later on.

Why do I fail

I faile because BizTalk does not support store-and-forward. BizTalk’s messaging is somewhat unusual in that it is ‘push and awake receiver’.

In traditional messaging systems (thing MQ Series), producer pushes a message into a queue and a queue retains a message until consumer pulls it from a queue.

In BizTalk, when a producer pushes a message into a Message Box, BizTalk’s Messaging Agent identifies all orchestrations that subscribe to that message. If there is no orchestration(s) that subscribe(s) to that message, the producer fails with ‘routing failure’ exception.

Two subscription scenarios are possible:

1. Activating subscription. An orchestration has an activating receive shape for a given message type. If a message of this type is published to Message Box, the Messaging Agent creates a new orchestration instance.
2. Non-activating subscription. An orchestration has a non-activating receive shape. For non-activating receive, BizTalk requires that — in addition to the message type (schema) — the orchestration specify a correlation set so that BizTalk can match a message to exactly one target orchestration instance. See discussion about (Convoys) for more.

In order to implement store and forward, we’d need BizTalk tp support non-activating subscriptionx where the correlation set is not required. But this is not supported.

What to do?

There is no good workaround. You could play with Convoys and create a ’singleton’ orchestration that permanently subscribes to both kick-off message and store-and-forward message. I think that you are unlikely to succeed. Even if you did, the resulting solution is at best fragile and - well .. - ugly.

If you really need store and forward, you need more than Message Boxes: ideall a proper queue (MSMQ, MQ) or - if you dont want to spend the money on licenses or putting up with the bloated messaging infrastructure - use a database as a store for the store-and-forward messaging.

Components of a service

Diagram below illustrates functional components of a service. At this moment we don’ allocate these functions to the specific components of the integration stack.

The service interface is a collection of message types that the service accepts. These messages are ten tranlated by the broker into finer-grained messages to the adapter (s) (or service(s) that are part of this virtual service). The application adapter(s) tranlate the messages from broker into API calls to the applications.

The functional components of a service include:

• One or more service implementations. These are the to-be-integrated applications, likely implemented using object technology.
• Service abstraction has three tasks:
  • Inteface. It acts an interface between the applications and the integration universe. This integration universe may be a bus or a broker.
  • Translation. It translates messages from the integration layer into calls to application(s) and translates the outbound application calls into messages to the integration layer.

    In general, that’s all you should do: pass messages between the applications/services; ideally, use coarse-grained document-style messages. You should avoid RPC-style connection-oriented interactions, for reasons described here.

  • Orchestration. It coordinates multiple API calls or service invocations to yield coarser-grained services.

Service in the Bus integration model

The following figure illustrates an implementation of service in the bus model. For simplicity, lets focus on the service provider side and simplify the service consumer by representing it as a single box.

The service abstraction function may be distributed:

• You can allocate translation task to the adapter, to the broker, or have both share this role.
• Likewise, you can allocated orchestration task to the adapter, to the broker, or to both.

How do you chose to allocate these functions? These common sense guidelines may help you:

• A common school of thought is that translation and orchestration is best done by the broker and that the adapters should remain ‘thin’ (i.e. should be just transport adapters). This is fine thinking unless your broker is counterproductive in which case you should take this opportunity and code your translations / orchestrations for the service layer in your programming language of choice.
• If you coordinate services provided by multiple providers, you are likely to need broker.
• You should try to allocate your orchestrations / translations to the same component, to simplify future changes and refactorings. Again, choose the more productive environment (hopefully the broker) to perform these functions.
• Off-the-sheld adapters come in different flavours, ranging from relatively simple transport adapters to complex adapters that perform business functionality, such as tranlation of the fine-grained application APIs to more coarse grained messages. If you happen to own such a complex adapter, it makes sense to use its orchestration / translation.

Service in the Broker integration model

The following diagram illustrates service composition in the broker model.

Unlike in the bus model, there is no natural demarcation line that would isolate the service requestor from the service. In the bus model, transport (MOM) is such a natural demarcaton line. In the broker model, the service is simply an orchestration that can be invoked by another orchestration (very much like intra-program method call). Therefore, you must pay extra care to clearly isolate services from their consumers.

As in the bus model, you have a choice of distributing the service abstraction functions between adapters and broker. Similar trade-offs apply, however in the broker model, you are less likely to allocate translation or orchestration to the adapters. Having all translations / orchestrations in a single domain, it should be easier to make functional changes / refactor the integration solution, assuming that you have a productive development environment.

Variations on the above options

Of course, multiple variations on the above two options are possible. For example, in the bus model, you may eliminate broker from the service side altogether. Or, you can have broker talk to service providers directly, skipping the common transport. These variations make no substantial difference to the construction of service - that’s all they are, variations.

The Good Stuff: Broker Infrastructure

As Value across the integration stack suggests, generally the infrastructure part of the EAI stacks delivers value but there are often problems with the development tools.

ICS infrastructure does deliver value. The ICS is an apparently-solid integration broker with a number of valuable features. Let us briefly review a few of them:

• A broad selection of technology and application adapters. Technology adapters include MQ Series, JMS, IBM Workflow, COM, CORBA, EJB, files, WebServices, HTTP, and others. Application adapters include SAP, Siebel, Ariba, i2, JD Edwards, Oracle Applications, and other. Take a look at the WBI Infocenter for a complete and up-to-date list.
• A broker that provides an environment for executing ‘flows’ (ICS’ term for orchestrations). The ICS broker includes a number of standard and several non-standard useful features:
  • 2PC transaction manager. (Apparently, ICS is built on top of Orion application server. Its just my guess: I think I have seen Orion traces somewhere.) The 2PC transactions span flows and
  • Two monitoring tools (reach client and web-based). You can use them to find in-error flows and resolve them, monitor status of integration components, report performance statistics and generate other useful reports
  • Compensatory transaction management for non-2PC transactions (i.e. long-running business transactions) . You can specify compensatory actions for your flows. ICS will manage their execution in case of failure and - from what I understand - propagate compensatory invocations down to sub-flows.
  • Ability to persist flow state. You can use ICS to manager for long-runing flows.
  • Recovery of in-progress flows in case the ICS server crashes
  • The relationship manager provides the cross-entity mapping functionality. You can map account ID in your shiny new CRM to account ID in your legacy accounting

In general I have a positive impression of the ICS broker. Clearly, reliability has been on the designers’ minds.

The Bad Stuff: the Development Environment

Multiple aspects of ICS development environment make it difficult and ineffective to work with. The following paragraphs go in details over issues such as lengthy feedback loop, little automation/scripting, little IDE validations, poor Java development support, and difficulties testing.

Lengthy Feedback Loop

I use the term ‘feedback loop’ duration to denote the time between a change to an integration artifact and the time when the effect of the change can be observed / tested. Short feedback loop is critical to productivity.

In case of ICS the feedback loop is very long (order of minutes). Testing of integration artifacts (except for maps) requires deployment. The deployment loop is lengthy. Typical change requires an order of minutes before verification. Some changes (for example, updates to connector meta-objects) require ICS re-start.

No Build Automation

ICS does not provide tools to script-generate the deployment artifacts. The deployment artifacts must be generated manually (clicking-through the IDE). This is bad because:

• Build is error-prone not repeatable.
• Building is annoying and slow.
• You cannot put the build under a continous integration using tool such as Cruise Control

You can work around this problem, at least to some extent. I was able to build the deployment artifacts by mimicking a IDE-generated one; it required re-naming a few files, re-arranging them, and jaring them up. However, I was able to do this only with simple projects and only for a few types of artifacts, so I am not sure how viable it is.

Limited Deployment Automation

Although ICS offers repos_copy.bat script to deploy your solution, you cannot not stop/start your solution from script. At least, not easily. And you need to stop flows, maps, and other components before re-deploying them. After the deployment, you need to start some of the components (or at least some of them - some do get started automatically).

Other than being very inconvenient, this prevents you from employing continous integration, as you do need to deploy & start your solution before you can test it.

Fortunately, ICS has an SNMP interface. You can access SNMP from scripts, using tools such as net-snmp

Difficulties with Making Functional Changes and with Refactoring

Refactoring and functional changes are easier if tools support it. ICS offers little or no support for changes. Even the basic and most common change – renaming – is very frequently not auto-propagated through project. Few examples of problems with basic change propagation below:

• You cannot rename a business object (business object is a peculiar to someone familiar with OO name that ICS uses to denote its data structures). If you want to rename it, you need to export it to a file and re-import it under a different name.
• Changes to objects referenced from Java snippets are not reflected in Java code. For example, if you rename a map, and that map happens to be invoked from Java code, the Java code will not be updated.

  It is not unreasonable to request that the Java code be updated. Just take a look at how Eclipse works with pure Java code. Changes to Java are propagated to properties, XML config files, and other.

• Change to signature of a collaboration template is not auto-propagated to the affected collaboration instance. Again, it is not unreasonable to expect that such changes be auto-propagated or at least that the affected objects be marked invalid.

Little validation by the IDE

IDE offers little validations of project configuration. Many errors are not apparent until deployment. Given the long feedback loop, common tasks such as configuring an adapter are is often very time consuming.

For example, references to so-called meta-objects are often not validated. In ICS-talk, meta-object is nothing more but a bundle of properties. You use these meta-objects to configure ICS components, such as connectors or data-handlers. Meta-objects cen refer other meta-objects. Oftentimes, you refer a meta-object by typing-in its name (no, no drop-downs) and that name is not validated. You will find out of a problem only after deployment. And, given the long deployment loop, this becomes time consuming again.

The IDE provides very little support for populating these meta-objects. The meta-object fields are often comma-separated lists of codes (e.g. line such as SOAPAction=”";cw_mo_soap=SOAPConfigMO;cw_mo_http=ProtocolConfigMO), thus not only error prone, but cannot be populated without documentation at hand. In addition, these meta objects frequently rely on naming conventions (for example, MX_????_FOO describes connector FOO) rather than on cross-referencing.

Very poor Java support

As code-free as the ICS claims to be, it still requires Java coding. Surprisingly, browsing through ICS examples reveals that flows are often expressed as directly as Java, instead of using the graphical notation. Perhaps because it is easier to express your flow in Java, than in the colorful UI?

Given that, there there is little Java development support. The Java editor is a simple text editor with syntax highlighting. Common productivity features of contemporary IDEs (auto-completion / intelli-sense, on-the-fly compilation) are not available.

The developer modifies Java code snippets (not entire classes). The ICS wraps code snippets into classes and then compiles them. The compilation errors include line numbers in the class code (and not in the snipped code), therefore making it difficult to determine the line number that contains the error.

And probably the most important aspects: because you are working with code snippets and not with complete Java classes, and - in addition - these snippets can be ran only in the container, you cannot unit test your code using any of the popular testing frameworks, such as JUnit

Team Collaboration

Given the little scripting, how is the team work to be supported? When a project is maintained by multiple developers, how is the project built? The manual click-through build process is error-prone and not repeatable. Also, it cannot be repeated often enough so that continuous integration can be applied.

The private ICS-specific XML representation of the integration artifacts makes it difficult for developers to work in parallel. Merging of developers work requires manual copy-paste (or rather re-create) using the IDEs.

Other Problems

Other minor problems and common grievances regarding EAI tools apply. These include:

• Real estate: even a simple ‘task flow’ takes a substantial chunk for your monitor. Not even close to the efficiency of a FOR loop or IF statement. Perhaps thats why there is so much Java in ICS examples
• Can you store modification history in your artifacts? Can you maintain you TODOs with your ‘code’?
• Can you temporarily comment out a line or a block to test something?
• Can you automatically enforce development standards (such as variable naming conventions)?
• How about a diff list between two versions?

Few positives

Lets finish with a few positive aspects regading tools that set ICS apart from other tools:

• Configuration Management

  ICS projects are broken into fine-grained files corresponding to integration artifacts (maps, flows, etc) and are therefore easy to put under version control. This is in sharp contrast to many competitors. For example, early products from TIBCO would store your entire project in a single file, making version control impossible (TIBCO has improved since).

  The ICS’ Eclipse-based IDE integrates easily with version control tools, such as CVS or ClearCase.

• Excellent IDE foundation

  ICS uses Eclipse as IDE. Eclipse specific functions are delivered as Eclipse plug-ins. As a result, you get to use a first-rate IDE, with a host of useful features, such as intuitive navigation, context help, version control integration, a host of third-party (ofen open source/free) plug-ins, not to mention such basics as the search function (you’d be surprised that some tools won’t let you search a project for text!).

Summary & what do you do?

ICS exemplifies problems with EAI tools. The value of the solid foundation is significantly reduced by the poor tooling and cumbersome development process.

The integration is all about accomodating change. Therefore, it is not about how fast you build a solution, it is to some extent about how fast it runs, but most importantly, it is about how fast do you change it. ICS tooling may be OK to build a simple solution reasonably fast, but it is a very poor toolset if you want to maintain a complex solution over time, especially in multi-developer environment.

What do you do?

I believe that by now that capabilities of a typical EAI stack have crystallised. You need to look past the typical feature-based evaluation criteria to find real differentiators. Quality of development tools is a frequently overlooked aspect, and one that has a big impact on your bottom line. Therefore, o make sure when evaluating your integration tools you do include the tools’ quality as the key evaluation criteria.

This article takes a critical look at the value across an integration stack. The value is distributed very unevenly. Most of the stack components deliver unquestionable value. However, the core of the integration stack - the integration brokers - tend to provide very primitive development environments. While these development environments allow to built simple solution very fast, it becomes virtually impossible to use them to efficiently and reliably construct complex systems.

This article suggests that the brokers development environment is the one of the key differentiators between the EAI/SOA/SOI vendors. It also mentions the key differentiators between the development environments of TIBCO’s Business Works, IBM’s Interchange, and Microsofts BizTalk.

The stack

For the purpose of this article I use the following model of an integration stack. See the definition of integration stack for explanation and take a peek here for examples.

Figure 1: integration stack

Where the stack delivers a value?

• Connectivity.

  Adapters and connectors provide connectivity to technologies and applications. Under normal conditions, you don’t want to build a JMS, Oracle, or a Siebel adapter yourself. These are usually worth paying for. Note also, that with the emergence of Web Services standards, the application adapters are frequently obsolete as many COTSes provide out-of-the-box Web Services interface.

• Asynchronous transport.

  Message-oriented Middleware provides asynchronous reliable transport to the integration infrastructure. Asynchronous messaging with reliability option is a critical component of a robust integration platform. In most cases, it is worth paying for or getting an open source alternative (take a look at BizTalk’s message boxes for an architecture in which you actually can get away without MOM and still benefit from asynchrony).

• BPM.

  The commercial Business Process Managers are mostly worth paying for. You generally dont want to develop a BPM engine yourself unless your processes are trivial. The point here is that the commercial products tend to deliver value without noticeable liabilities.

• Monitoring.

  Buy - or get as open source - but don’t build enterprise monitoring tools, such as BMC Patrol. They look nice on the monitor and come with pre-packaged set of goodies that let you monitor practically everything and often have some extra functionality, such as proactive error handling.

• BAM and the likes

  I don’t know. I suspect that BAM tools provide functionality that is unlikely to be worth replication using custom development.

Broker and IDE

The typical integration broker (TIBCO’s Business Works, IBM’s Interchange, Microsoft’s BizTalk) provide (Figure 2) provides an integrated development environment (IDE) and a runtime. The developer generates integration artifacts (maps, orchestrations, custom Java code) and deploys them to the broker.

Figure 2: quite obvious view on the broker’s runtime and development environment.

The value of the broker runtime (where the problem isn’t)

The concept of a broker as the component that provides transformation, routing, & orchestration thus de-coupling interactions between the integrated applications is valuable.

The commercial broker runtimes are solid, mature and eliminate the need for custom development of commodity support functions that are must-have for running the business, but are by no means competitive differentiators. These functions include:

• Runtime for executing maps and orchestrations
• Runtime for interacting with adapters & connectors
• Health & Activity Monitors that provide for proactive error handling, auditing, SLA reporting, or resolving in-error flows.
• Transaction managers, both 2PC or business transaction managers (i.e. the compensatory transactions schemes in ICS and BizTalk)
• Service lifecycle management
• Support for long-running flows (ICS & BizTalk; last time I checked BusinessWorks 5.1.2 did not support long-running flows and used to delegate this function to the BPM /InConcert or Staffware/).
• And many more …

There is little chance that building a custom broker runtime is going to provide business value, just like you would no write your own SQL server (you’d write your own SQL queries) or J2EE server (you’d write your own EJBs thought /that is if you don’t know about lightweight containers, such as the Spring framework)

Little divergence: the contemporary business software development practices and development environments that support them

Many integration developers have been doing integration long enough that they’ve missed major improvements in the way business software is being developed. The few interesting developments with bearing on the tools are:

• Refactoring

  “Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior. Its heart is a series of small behavior preserving transformations. Each transformation (called a ‘refactoring’) does little, but a sequence of transformations can produce a significant restructuring. Since each refactoring is small, it’s less likely to go wrong. The system is also kept fully working after each small refactoring, reducing the chances that a system can get seriously broken during the restructuring … ” . Definition from refactoring.com. See refactoring.com for more.

  To me, refactoring is important in two ways:

  1. It is a method for maintaining a clean internal structure of the solution. Clean code structure is critical to sustaining productivity. If you dont mainain a clean internal structure, overtime your software becomes a nightmare to maintain (like virtually all integration projects do).
  2. It enables evolutionary design. You dont need to plan much up-front if you know that you have an arsenal of tools that allow you to evolve your solution as you go, while maintaining its internal clarity. And over-designining business software up-front is usually time wasted.

  Modern development environments, such as InelliJ from JetBrains or Eclipse are working refactoring browsers, automating typical refactorings, ranging from simple ones (such as component re-naming or method extraction) to complex ones (such as re-shuffling class structure).

• Test-driven Development (TDD)

  Test-driven Developement is an Extreme Programming (XP) practice in which you write test first, then you wite your code to satisfy the test. The result is that the programmers actually do test their software, and that testing is automated. The quality is built in the software. The test coverage is maintained high. Such tests provide a safety net for changes, thus enabling fear-free refactoring. I can modify my software to reflect a better understanding of problem domain (of cours, I write a test for this new understanding first), run hundreds or thousands of tests that have been written for the system (typically within minutes) and be comfortable that the system still works.

  The modern programming environments enable TDD in a several ways:

  1. Ability to execute tests is built into IDEs
  2. Lightweight containers, such as Spring allow to test the business components without the need to deploy your solution to the container. This dramatically shortens the feedback loop between the time you change software and you test it. Instead of waiting for minutes for deployment of an EJB, you test your Plain Old Java Object (POJO) instantly.
• Continuous Integration

  Continuous Integration is a practice of continuous building of software developed by multiple developers. Each time a developer commits a code change, the build process is executed. As the software has been built using test-driven development approach, software is self-testing.

  Continuous integration lets you avoid (code) integration problems. An integration problem is spotted instantly, be it a compilation problem of functional problem that is detected by the built-in automated unit tests.

  Continuous integration is relatively easy to implement with the traditional development tools (i.e. Java, C, C++, C#, …) , especially is out-of-container testing is available. This is NOT the case with EAI tools, as you will see it below.

To sum up, the modern leading-edge development tools provide an enabling environment for an evolutionary design. This reduces the time required for up front design or eliminates the need for up-front design altogether.
If it is as easy to re-shuffle your class structure in the code, as it is a design tool — such as Rose — why use the spend time designing? Coding is the design. You get a working product early, thanks to the safety net of testing you can alter its structure without a fear that you break something, thanks to the IDE’s refactoring support you change your structure instantly, and thanks to the lightweight container, the feedback loop is extremely short, so you can build/test the solution hundreds of times a day.

Another little divergence: should integration tools be like the development tools?

Yes. While the integration vendors continuously announce the first development tool that a business analyst can use, this is simply not the case. The transformation, routing, and orchestration remain the domain of developers. Therefore, the integration tools should focus on providing a productive development environment rather than faking a business-analyst-usable tools.

So what are the problems with a typical integration development environments?

The following is a laundry list of common problems with the integration tools:

• No or little support for test-driven development. As most of the tools require that integration artifacts be deployed to an integration broker to be tested, fine-grained out-of container unit testing is difficult. In addition, no tools that I know of provide out-of-the-box unit testing facility. However, there are numerous open source integration unit testing tools.

  BizTalk is a clear leader in this area with tools such as BizUnit or BTSUnit. There are also tools for other vendors, such as RvTest for TIBCO/RV or GH Tester - a generic JMS testing tool.

• -No or little support for refactoring. Frequently as simple change as a variable re-naming is not auto-propagated through the project (example: IBM ICS). This, plus lack of automated testing, prevents integration developers from refactoring their solutions. This quickly leads to messy and non-maintainable solutions.
• Long development feedback loop. Because of the need for deployent, it takes minutes between the time you make a change and the time you see test results. In BizTalk and ICS this takes an order of minutes. TIBCO’s products do allow for out-of-container testing reducing this the feedback time to seconds.
• Poor build support.. Often, there are no script tools for automating build and development. The solutions are build manually using UI tools. This is time consuming, non-repeatable, risky, and does not allow for implementing contiunous integration:
  1. BizTalk does provide command-line build (devenv.exe can be run from command line) and is fully integrated with Nant.
  2. IBM’s ICS does not allow for command-line build. You must use UI to build your solution. It is possible to hand-craft a deployment unit by reverse engineering ICS’ .jar structures but thats just a workaround.
  3. BizTalk 5.1.2 does not provide command line build. However, it is possible to simply jar the project (a workaround again).
• Poor configuration management and teamwork support. For example, in case of development conflicts, merges must be manual. The file structures often create hot-sports: various ‘project’ or ‘binding’ files tend to change each time any component changes, thus effectively preventing parallel development.

  Extreme example: older versions of TIBCO tools used a single XML repository file, effectively preventing concurrent development. The common workaround was to manually break and merge the XML repository file.

• Any custom coding (say Java code in TIBCO) has often zero support form IDE. If you get used to intelli-sense in advanced IDEs, you realize how counterproductive it is to type-up your Java in notepad-style editor with no support for JUnit, no auto-completion, etc. And often, you must use Java/C# to augment the IDE. Again, BizTalk that uses Visual Studio, is a bright spot.

• Few other minor but representative problems:

  • Did you ever try to find out which components publish on topic XXX.YYY.ZZZ? Something a simple UNIX ‘grep’ or an IDE search would do but does your tool have a proper button?
  • Real estate: even a simple ‘task flow’ takes a substantial chunk for your monitor. Not even close to the efficiency of a FOR or CASE statement. In order to develop a complicated flow, do you wish your had a 70 inch monitor?
  • Can you comment your development artifacts, ideally on the screen? Can you store modification history? Can you maintain you TODOs with your ‘code’?
  • Can you temporarily comment out a line or a block to test something?
  • Can you automatically enforce development standards (such as variable naming conventions)?
  • How about a diff list between two versions?

Reasons: how did we get here?

I think that there are at least two categories of problems and two categories of reasons:

• Inherent problems related to visual development, and,
• Poor quality of some visual tools.

Inherent problems include real estate limitations, the constraints of what had been built-in into menu, and –to some extent - difficulties in capitalizing on commonalities. I may very well be wrong on it and the next generation of visual tools may solve these problems. Intentional Programming concept seems especially interesting. However, until that materializes, I feel strongly about a need for text-based source code. This code can very well be generated and manipulated through a visual tool, but the ability to operate on human-readable text addresses many of the shortcomings of visual tools, providing that it is done through a modern IDE, with support for modern design techniques, such as refactoring and test-driven development.

Poor quality problems result from the history of integration brokers. I think that there are at least three categories of integration broker origins:

• Messaging add-on. As messaging becomes commodity, messaging vendors, such as TIBCO, move up the EAI stack to avoid selling commodity products. Such vendors add integration broker to the mix and other elements of ‘EAI stack’ such as BPM or Business Activity Monitoring (BAM).
• Application server add-on. As application servers and EAI offerings converge in functionality, the application server vendors add EAI functions. Good examples include BEA and Oracle. Traditionally application server vendors lag in functionality behind the specialist (such as TIBCO), however they tend to offer more robust infrastructure.
• Integration-broker vendors, such as Crossworlds (now owned by IBM), that designed messaging vendor-agnostic (or sometimes vendor-specific, such as NEON for MQ)

In some of these cases, it is natural that the vendors have – at least initially – difficulties with producing quality development environments:

• Messaging vendors needed to shift their attention from back-end development (messaging frameworks) to development tools development; this requires a completely new skill-set and may result in considerable growing pains. For example, TIBCO is delivering already third generation of its integration broker (after Message Broker and Integration Manager); each of these three generation is a almost complete re-write of the previous one and reflects shifting focus from messaging add-on to standalone, messaging-agnostic product.
• Application server vendors (generalists) are in excellent position to provide back-end infrastructure; frequently, they have substantial experience and capabilities in the area of development tools (IBM, BEA). They are in excellent position to deliver solid integration brokers, however, they traditionally lag in functionality behind the more innovative (in the area of EAI) specialist; this is especially visible when it comes to support of more exotic standards (various HL7 or HIPAAs) or ‘higher levels’ of EAI stacks, such as BAM. Perhaps generalists, being frequently very large companies, are not so hard pressed by commoditization from the bottom of the stack that we force them to escape ‘up’.
• Pure-play Integration-broker vendors, such as Crossworlds (now owned by IBM), that designed messaging vendor-agnostic (or sometimes vendor-specific, such as NEON for MQ) tend to focus their development on their visual tools. The lack of experience (startups) works against them. Given the relatively small number of such vendors, it is hard to say how it works out.

In addition, I think that some solutions are showing signs of overshooting, i.e. providing functionality that customer does not need. This is especially likely in the case of integration specialist that – being pushed by commoditization from the bottom of the stack – try to compete on the integration broker levels. Thus the push towards prettier GUIs and various ‘zero-coding’ and ‘business analyst friendly’ environments.

What do you do about it?

Be careful choosing your integration broker solution. When you see the demo, you may be under the impression that it takes no time to integrate two systems. Just drag, drop, even your business analyst can do it. This is true in trivial integration cases that need not to change (a demo to the executive team is a perfect example).

The more complex and the longer lasting the integration project is, the more important the changebility of your architecture. The ’speed of development’ delivered by graphical integration tools may be deceiving. The importance of more traditional and modern software engineering techniques such as configuration management, re-factoring, test-driven development, reuse of commonalities etc. may very well outweigh the speed at which you can drag-and-drop data elements from left-to-right.

Therefore, look behind the pretty GUIs – make sure that some ‘engineering’ is possible with the tool you are about to buy. Look for configuration management, refactoring, test-driven development. Recognize tools productivity as a critical aspect of your tool’s evaluation process. This should be relatively easy as the other aspects of the stack tend to commoditize.

This article looks at both models and identifies the repercussions of this choice.

Why do I even care: the one-sentence intro to EAI

The key feature of modern integration approaches (EAI, SOI) is the reduction of integration complexity by replacing the point-to-point ad-hoc integration (Figure 1) with systematic integration through an integration platform (ever heard the integrate once, recompose many times mantra? It does not work all that well, but the idea is on Figure 2).

Figure 1 Ad-hoc integration. Mess. If you integrate <3 applications, go for it.

Two common models of implementing systematic integration approach are bus model and broker model.

Figure 2 Systematic integration. Better. Look here to see why.

Bus

In bus model (Figure 3) the transport (MQ, JMS, or other MOM) is the ‘integration universe’. Any cross-system interaction involves the messaging bus. The applications are connected to the bus by adapters. Cross-system transformation and coordination of tasks is enacted by one or more integration brokers.

Figure 3: Bus Model

Broker

In broker model (Figure 4) the broker is the ‘integration universe’. Every cross-system interaction involves broker but there is no requirement for a common transport. Conceivably, you can even implement robust integration solution without message-oriented middleware.

Figure 4: Broker Model

Do I have a choice?

First, you may have very little choice. To a large extent, the choice is imposed upon you by your integration tool vendor. Therefore, consider the model you looking for at the vendor selection time.

Most of the early EAI solutions were transport add-ons, effectively leading to a bus model. Examples include the archaic TIBCO’s Message Broker or the still-alive IBM’s MQSI. However, with the recent popularity of the Enterprise Service Bus (ESB), from vendors such as Sonic, the bus architectures are all but fogotten.

The more recent offerings of EAI vendors tend to implement the broker model. Examples include TIBCO’s Business Works, IBM’s Crossworlds / ICS, or Microsoft’s BizTalk.

Characteristics and consequences

Let’s dicuss the key consequences of the bus-broker choice:

• Interoperability Vendor lock-in

  Using broker, you are more likely to develop a strong dependency on the broker vendor.

  The bus forces you to think about messages, and if messaging is done right and is standard-based, it is broker-agnostic. You can easily mix multiple broker in a single architecture. You can easily add transport adapters from multiple vendors and have them interoperate.

  The broker model is not that flexible. Adapters need to be broker-specific. Exchanged data format are likely to be broker-specific as well.

• Asynchrony and temporal coupling

  I feel strongly about the asynchronous messaging as a foundation for integration. The bus model implies asynchrony. The broker model may lead you to the dark side of RPC. Fortunately, some integration brokers are inherently asynchronous.

• Performance

  Transport-based solution requires few extra hops for each interactions. Theoretically then, it should be slower. In practice, the brokers tend to be the bottleneck and not messaging. So, no clear winner here.

• Up front design cost

  The bus model is likely to cost you significantly more up-front design. Bus model requires some structure for message addressing and routing. Broker can rely on addressing/routing implied by orchestrations.

• Testing

  Bus is easier to test. You can send stimuli to any component using a single transport. It is easy to intercept intermediate messages from a single transport to monitor / evaluate test progress. You can use generic or open-source/free JMS/MQ/TIBCO testing tools for doing so.

• Operations & Management

The Broker model is easier to monitor. It is a single component from a single vendor.

Components of bus are inherently distributed. You need a monitoring / operations support tool that can handle a distributed environment, such as TIBCO’s Hawk or BMC Patrol.

• Complexity

The broker solution is simpler. It includes less components. The bus solution not only includes more components, and not only are the components distributed, but it also requires more design work (messaging approach /envelope/, addressing, routing, etc).

What do I chose?

The broker model is a good solution for small-to-midsize integration problems. It is simpler, cheaper, requires little up-front design. Use it to solve small enterprise or single-or-few-departments integration problems. When you do so, make sure that you keep your solution asynchronous in nature.

As your solution grows, it is likely to evolve into bus model. You are likely to find connectivity requirements that require adapters from ISVs. Or get another broker because of extra transformation requirements (or just because it came bundled with a J2EE app server and developers think its cool). The larger the solution, the less likely that a single broker will suffice; plus there is value in having a portfolio of brokers, if not for having some leverage over vendors, then to protect yourself from consequences of some product going off the market. Such portfolio should interoperate using a standard-based messages and a common bus.

Therefore, rely on a bus for large scale integration solutions.

Why asynchrony?

Synchronous communication implies temporal coupling. Both the client and the server must be up for the
interaction to occur, or an error occurs. This is fine for situations where a response is immediately needed (for example, UI). For other requirements (response eventually needed, fire-and-forget, i.e. long running workflow or data integration) temporally-coupled systems are more fragile than systems built on the basis of reliable asynchronous communication. This is especially significant in distributed network-connected environments.

Traditional solution: MOM

The traditional solution is simple: put MOM in between the interacting parties. In a fire-and-forget scenario (diagram below), the publisher deposits a message to MOM queue. The subscriber retrieves message from queue when the subscriber is ready. Nothing fails the subscriber is down at the time publishes originated the message.

Is there a problem with MOM

Not really. It is an excellent foundation for an EAI solution. However, sometimes it may offer too much functionality and therefore be more expensive and more complex than it needs to be. However, as I asserted in the beginning, you do need asynchrony.

Message-box: a poor (wise?) man’s MOM

A month ago I started working with BizTalk. I like BizTalk for many reasons [TODO: link to BizTalk versus ICS comparison] BizTalk provides a somewhat unique concept of a message box that I illustrated on the following diagram:

In BizTalk, all cross-broker interactions involve Message Boxes. Adapters deposit inbound events / messages in Message Boxes and retrieve outbound messages from message boxes, before delivering them to target applications.
Effectively, all interactions are asynchronous and temporally de-coupled. If the subscriber in the example above is down, there is no problem. The to-be-delivered message is stored in the message box until the adapter can deliver the message, so once the subscriber is up.

Truly, message boxes are not all that innovative if at all. The concept has been around forever, for example in operating systems. However, BizTalk – to my knowledge – is unique with this concept ( IBM/Crossworlds ICS provides a similar functionality)

Message-box versus MOM

I love the concept of Message Box. I believe that it can replace MOM for small and medium installations.
What a Message Box is not?

• It does not solve your transport problem like MOM does, i.e. it does not eliminate the distance from the equation. If you need to distribute information, especially over wide area networks, there is nothing like MQ and the likes.
• It does not give you the fancier stuff, like UDP-based message dissemination.
• Since it is database-based, it will not perform like MQ or TIBCO EMS. It will be more like WebLogic’s database-based JMS implementation.

However, in certain cases, Message Boxes can replace MOM and yield a more elegant, simpler, cheaper, and more manageable solution. Assume that you’ve built a multi-transport broker-type [TODO: link-to broker-centric versus transport-centric] EAI (diagram below):

If your integration is limited distance-wise, and you have a broker with Message Boxes, this can be effectively accomplished without MOM, with quiet a few benefits:

• Simplicity. Just the broker, no MOM.
• Asynchrony and temporal de-coupling is there. Notice that all transports between broker and integrated applications are synchronous and commodity. No extra cost.
• It is easy to monitor and audit. Just run reports against the database.

Summary

Message Box may completely eliminate a need for MOM. In fact, it really is a simplified MOM, with lesser performance and no ‘remoting’ / transport. It will do fine, if you don’t have fancy or hight transport requirements. It works if you build a broker-style integration – for obvious reasons, it will not work in a bus model.

So, before you start evaluating the MOMs to buy, take a look at BizTalk. Maybe you don’t need MOM at all.

This paper reflects the vendors stacks as of June 2004. Some of the products have been phased out or re-labelled since then. Still, it is important to discuss them because of the historical perspective of the product development and broad install based. Examples of such products include the already phased-out TIBCO’s Message Broker or the no-longer-actively-sold Integration Manager.

This paper is organized as follows:

• Model outlines the convergent functionality that emerges from the comparison.
• TIBCO implementation maps TIBCO’s portfolio to this model; likewise, IBM implementation maps the model onto the IBM stack.
• Finally, both products are compared and final conclusions are drawn in Comparison

Model

Speaking crudely, objective of EAI is to connect disparate systems in a cost-efficient fashion. Such integration is possible by employing a set of enabling techniques such; the key techniques include loose coupling or the reduction of topological by introducing a common medium between the to-be integrated systems (integrate once, re-compose many times).

Specific integration cases aside (such as batch integration, ETL for data warehousing, common database / operational data stores, etc), EAI vendors offers a generic model based on message-oriented middleware (MOM). This model includes the following functions (Figure 1):

• To-be-integrated applications
• Connectivity between applications and transport
• Transport
• Transformation & Routing
• Task Flow
• Business Process Management

Other commonly seen functions, but not discussed here include user interface, operations support, or recently emerging Business Activity Monitoring (BAM).

Figure 1 Integration stack

Connectivity
The Connectivity function encapsulates the transport and target application interface by connecting each integrated application to the transport function. In the EAI world, the connectivity function is provided by adapters and connectors.

Transport
Transport performs two important functions:

• Moves the data across the network.
• Facilitates some of the key aspects loose coupling[1], most importantly temporal and technological de-coupling.

Transformation & Routing

In almost all integration activities, transformation from source format to target format is required. The base transformation process involves format modifications (both syntax and semantic mapping). More sophisticated transformation services offer message augmentation (message boosting) or data cleansing (for example removal of duplicates) that may be required to handle the differences between the to-be-integrated applications.

Except for basic point-to-point integration routing of messages is a key component of an integration solution. These message routing activities encompass:

• Managing the delivery of the messages over communication connections, including protocol conversion, flow control, guaranteed delivery, and connection optimization (for example, connection pooling)
• Multi-point message decomposition/re-composition, enabling one to many and many to one message routing
• Content-based routing with associated directory-based or rules-based routing.

Transformation and routing function are typically provided by the same component of an integration broker and for that reason we group them together.

Task Flow

The Task Flow Management function of the broker coordinates relatively simple, short time activities amongst the integrated systems. Task flow management allows for re-combining applications functionality to yield a more complex functionality.

Frequently, the Business Process Management (see below) function delegates a single business tasks to the Task Flow Management function. The Task Flow Management function service translates the business task into a set of lower-level (often application specific) technical tasks.

Task Flow Management service is also known as: technical process management, micro-workflow, system-level workflow, and message choreography

Business Process Management

The Business Process Management function (BPM) coordinates long-running business processes. In many aspects, BPMS resembles Task Flow Management function. BPM differs from Task Flow Management in that:

• It is geared towards managing tasks that may range from hours to months in duration. BPM persists its state in a database.
• BPM focuses on business-level tasks, frequently tasks that are performed by humans. To support such tasks, BPM support sophisticates access control and authorization models, escalation procedures, task delegation, etc.

Other services

Other integration functions are out of scope for further discussion (non-core services). These include:

• User interface facilitates user’s access to the EAI-integrated systems (‘consolidated application’ integration model). User access implementations include various portal packages.
• Common persistence mechanism for cross-system key mapping or system-wide data storage.
• Operations support functions such as monitoring and proactive error handling.
• Newer integration services such as BAM functions, event matching, rules engines, and other next big things.

TIBCO implementation

TIBCO started its presence in the EAI space started as a message-oriented middleware vendor with it’s flagship product, RendezVous (RV). Over time, TIBCO shifted its focus from MOM to the higher stack levels. Either by acquisitions (InConcert, Staffware, Talarian, BAM platform) or in house development (MessagBroker, IntegrationManager, BusinessWorks), TIBCO has assembled a complete EAI stack (Figure 2).

Figure 2 TIBCO projection

Messaging

TIBCO offers a portfolio of messaging products:

• RendezVous (RV) has been traditionally the dominant product. RV is a distributed UDP/multicast messaging system augmented with ‘certified messaging’ to provide reliable delivery (at least once delivery) and RVTX extension for transactional (once and only once) delivery. Apparently, as of August 2005, RVTX is no longer sold.
• EMS is TIBCO’s own implementation of hub-and-spoke style JMS server database-based, very similar to other pure-play JMS offerings such as Sonic MQ or WebLogic JMS.
• Talarian/SmartSockets high-volume near-real-time best-effort messaging.

Transformation & routing, Taskflow
Collection of T&R products reflects the history of TIBCO focus shift away from messaging and towards higher stack components:

• The original T&R product, message broker, was essentially an add-on to the messaging platform that allowed for stateless message transformation, but no task flow control. This product is now discontinued and replaced by IntegrationManager / BusinessWorks.
• IntegrationManager is essentially the next generation MessageBroker. It adds task flow control, multi-transport capabilities (specifically JMS, CORBA, JDBC) but remains very much a MOM add-on. IntegrationManager is tightly coupled with RV and relies on RV/AE message format as internal data format.
• BusinessWorks represents a shift towards an integration broker model with support for multiple transports, such that RV remains just one of multiple transports to chose from. BusinessWorks uses XML as internal messaging format, thus becoming virtually independent from RV (note, however, that BW still relies in RV for implementing high-availability features).

BPM

TIBCO’s BPM offering has grown through acquisition:

• InConcert — originally developed at [CHECK]– is a general purpose workflow management system with capabilities for managing both technical and ‘human’ processes. InConcert provides an extensive API, a prolog-based rules engine, dynamic process instantiation..

  InConcert does not support modern workflow features such as BPEL modeling / execution. InConcerts user and administrative interfaces (at least the Windows-based fact cleint) are very limited and simplistic; at the same time the back-end engine is capable and robust .

  Per unconfirmed signals from TIBCO, InConcert will be discontinued.

• Staffware is the recently acquired workflow engine with a strong foot especially in the. European market. [FILL IN – get more info].

Other products

• TIBCO has been one of the pioneers of BAM (both acquisition and in-house development, products include BusinessFactor and OpsFactor).
• Hawk is RV-based monitoring / proactive error handling tooll.
• TIB Portal Builder is a portal package providing user access function.
• TIB BusinessConnect and TIB BusinessPartner (rumoured to be discontinued) provide a B2B interface.

IBM implementation

Like TIBCO, through both acquisition and in house development, IBM assembled a complete integration stack (Figure 3).

IBM’s presence in the integration world started with MQ Series, the original commercial message queue. To this ‘core’, IBM added a message broker solution (MQSI [CHECK-was MQSI acquired from NEON, or just the rules engine was?]), an integration broker (Interchange from CrossWorlds), and a BPM (MQSeries Workflow.). In, parallel IBM developed the WebSphere J2EE application server family [on the basis of Orion?].

Figure 3 IBM projection

Messaging

IBM’s MQSeries, is the pioneer of the MOM market. Because of its market share (60% according to [Gartner - check]) and third-party vendors support (300+ vendors actively develop supporting products), it is de facto the standard amongst the message queueing products.

MQSeries architecture is best described as a multi-node hub-and spoke queueing system providing both reliable transactional delivery of messages across variety of operating systems and network protocol. The basic queueing function is augmented by a multitude of features including high-availability (clustering) and high throughput publish/subscribe (multicast).

Transformation & routing, Taskflow

IBM’s transformation and routing capabilities have been both built and acquired. IBM includes two separate components into their ‘stack’:

• WebSphere Integrator Broker is an in-house build MOM add-on that provides transformation, routing and flow control.
• WebSphere Interchange has been acquired from/with Crossworlds. Interchange is a full fledged transport-agnostic integration broker.

BPM

[Research / FILL IN ]

Other products

IBM’s portfolio includes a multitude of EAI related products:

• WebSphere J2EE application server.
• WebSphere Portal provides user access function.
• WebSphere DataInterchange is a B2B interface supporting standards such as EDI/X.12, HIPAA, or HL7.

Comparison

Initial Impressions
Both stacks grew by both acquisitions and in-house development. At the first glance, the TIBCO stack seems more consistent as it constucted of fewer products. In constrast, IBM constantly re-brands products (most recently, pre-pending all names brands ‘WebSphere’) making it difficult to understand.

Both stacks include products with overlapping functionality. It is not clear from the vendor which product should be used under what circumstances; rather, both vendors attempt to create an impression of perfectly coexisting and augmenting each other components. TIBCO seems to pay more attention to maintaining consistency within its portfolio by eliminating overlapping products (for example phasing out of MB, IM, and InConcert as newer components become available); IBM maintains multiple overlapping products but introduces a common front end for all of them (Eclipse IDE). This product management difference may simply be a result of sheer difference of size between both companies and TIBCO’s inability to maintain/support a ‘fat’ portfolio of products.

Messaging: Multicast versus Hub-and-Spoke

The comparison of TIBCO and IBM is de-facto as comparison of two messaging paradigms:

• Distributed multicast-based publish/subscriber represented by RV. For the purpose this comparison we focus on RV as the base TIBCO’s messaging; not, however, that TIBCO offers its own hub-and-spoke product as well (EMS).
• Hub-and spoke queueing (MQ Series).

In my opinion, the multicast-based publish/subscribe messaging is an excellent solution for near-real-time message dissemination when 1 à very-many delivery capabilities matter. RV originated on the trading floors as a vehicle for disseminating financial information such as stock prices. However, in most EAI cases the opposite requirements are true:

• ‘Cardinality’ of message delivery is 1-1, 1-2; 1à-very-many is a rare case.
• With exception of ‘consolidated application’ integration model (near real time request reply with timeout heuristics), reliability of message delivery takes priority over performance. Effectively, in most EAI implementations of RV, RV ends up simulating a queueing system using its ‘Certified Delivery[2]’ mechanism. While this works, it is a flawed solution (see the discussion of Certified Messaging below).

  Reliable delivery is a ‘native’ function of hub-and-spoke solutions. The multicast solution must be augmented with local persistence mechanism and re-try mechanism

• While RV offers reliable delivery (queueing) referred to as Certified Messaging, this solution is flawed in that:
  • Inherently, reliability of CM is not comparable to the hub-and-spoke topologies as the data is stored in local file systems using non-transactional disk operations, as opposed to centralized database in hub-and-spoke topology. Corruption of RV ‘ledger files’ is not a rare case that leads to loss of data.

    [Would be nice to investigate ‘solidness’ of MQ in this regard]

  • Temporal de-coupling is not the case. While a message can be queue for a later delivery (in case the target system is unavailable), for a message to be actually delivered both system must be up at the same time.
  • RV / CM lacks a basic facility of any queueing system: a queue browser that is frequently required for production support (for example, in order to remove an offending message). In contrast, MQ Series offers an out-of the box queue browser and a host of third party solutions.
• Transactional messaging is difficult to implement in multicast environment. TIBCO offers a transactional augmentation of RV (RVTX) that guarantees a 1-and-onlu-once delivery; however, this solution essentially converts RV into a hub-and-spoke system. Consequently, very few RV implementations are transactional.

Having said that, the shortcomings of hub and spoke include:

• Single point of failure, when the hub is down, everything is down. MQ Series addresses this problem by providing high-reliability clustering.
• Non-native publish/subscriber (publish/subscribe is emulated programmatically ) that results in reduced performance, especially in 1-to-very many delivery
• Overhead of hub management (a need for administering hub objects: queues, channels, etc).
• Inferior performance, especially in 1àmany publish/subscribe and request/reply cases.

These shortcomings — in my opinion — do not outweigh benefits and for that reason MQ Series / hub-and-spoke solutions constitute a better choice for most EAI problem. Multicast-based publish subscribe is better left to its niches (high volume, high performance, accepter unreliability, 1 à very many).

Also, note the following new offering, indicative of product convergence:

• The discussion ignored TIBCO’s newer offering: the EMS hub-and-spoke server. While it is not yet an official position of TIBCO’s, it appears that TIBCO recognizes RV’s liabilities and shifts to EMS as the base transport.
• MQ Series provides a multicast add-on for niche publish/subscribe problems.

Other concerns:

• At the first glance, MQ Series offers an unparalleled presence across operating and programming systems languages.
• MQ is message-format agnostic. RV used to rely on a family of proprietary message formats (Active Enterprise / AE ) only recently shifting to XML.
• While messaging appears to commoditize, both vendors decline that that’s the case stressing their unique capabilities in niche applications.
• MQ Series is conceptually far more complex than its JMS substrate. [ further research: is this extra complexity warranted?}
• Both vendors introduce dependencies on their proprietary messaging into the higher level of the stack. For example, TIBCO’s BusinessWorks integration broker uses RV as internal messaging; IBM’s MQSI is tightly coupled with MQ Series (though the newest version allows for stating processes from non-MQ channels); IBM has introduced MQ dependencies into Crossworlds.

Transformation & Routing, Task Flow

Both stacks include more than one component implementing T&R and task flow functionalities:

IBM:

• WebSphere Integrator
• WebSphere Interchange

TIBCO:

• MessageBroker
• IntegrationManager
• BusinessWorks

For the purpose of this discussion we skip Message Broker from TIBCO as this product has been discontinued (note that Integration Manager, a successor of MB, is also rumoured to be discontinued). Table 1 summarizes the comparison.

Table 1 T&R and task flow comparison

Overall model [3]

Both stacks offer a MOM-add-on (transport centric) and an integration-broker component. In both cases, the integration brokers are the newer components:

• IBM’s MQSI and TIBCO integration manager are transport add-ons.
• MQSI is tightly linked to MQ Series. It requires MQ Series to operate. Only the most recent version of MQSI allows for launching a task flows by non-MQ events.
• Integration Manager is tightly linked to RendezVous. Integration Manager requires RV to execute and relies on RV/ActiveEnterprise data format for internal data exchange.

TIBCO BuseinesWorks and IBM Interchange represent a shift towards the integration broker model.

BusinessWorks continues to rely RV as the base transport and uses RV it to implement high availability features (failover, load balancing). However, BusinessWorks no longer requires RV data format for internal message exchange; it relies on XML instead.

Likewise, IBM Interchange (originally Crossworlds), is a transport independent product that implements integration broker model.

Runtime

All products except MQSI execute within a JVM; I expect MSQI to yield for smaller footprint and produce higher performance than the rest of the products.

Functionality

MQSI is unique in its support for any multiple internal data format and any-to-any translations. This is very much in line with the format-agnostic nature of MQ. The remaining three products rely on uniform internal formats: AE/RV (Integration Manager), XML (BusinessWorks) or Java (Interchange)

Interchange appears the most innovative product in the pack. It stresses the conceptual aspect of integration. For example, Interchange/Crossworlds and was the first product on the market to incorporate the concept of Canonical Data Model into the product; Crossworlds/Interchange institutionalized this pattern and requires programmer to express interactions in terms of general business object model (GBO) versus application-specific objects (ABO). Crossworlds/Interchange forces the user to think about generalizations and to capitalize on commonalities by forcing the user to express interactions as a set of CRUD operations upon the common objects implemented within the applications.

All components provide task flow function (TIBCO’s message broker is the only exception; its function is limited to transformation and routing; all processing is stateless). Interchange and Integration Manager provide functions typically reserved for BPM products – ability to persist flow state and thus coordinate long running business processes. While this function does not obsolete a need for a BPM product, it allows for constructing limited technical long-running flows.

Both products provide a broad coverage of connectivity and format standard. However, IBM products provide a more reach set of functions for developing mission critical applications, specifically in the area of transaction management:

• MQSI allows the flows and the flow-coordinated resources to participate in distributed XA-compliant transaction transactions (it still requires an external transaction monitor).
• Interchange offers ‘business transaction management’ protocol; while this is not a standard protocol, such as WS-Transactions, Interchange provides a framework for compensatory-action based transaction manager. To my knowledge this is very a unique feature amongst the EAI products.
• In contrast, TIBCO products offer minimal support for constructing transactional system. Neither business transactions nor distributed transactions are supported. IntegrationManager offers no transactional support altogether [VERIFY]; BusinessWorks allow for managing transactions but within single data source only. Thus, encompassing – for example – acceptance of a message and database update within a single transaction is impossible.

In general, it appears that:

• Transport-centric technologies are the older ones and the focus shifts to ‘integration broker’ model. This most likely reflects the multi-transport nature of integration recognizing that most enterprises rely on multiple transports. At the same time, it allows enterprises to take advantage of commoditization and standardization while building portfolio-based EAI stacks.
• Both systems offer ‘complete’ functionality for constructing integrations except for a serious gap in the area of transaction management in TIBCO’s stack.

Development model

We’ll look at the development model from two angles:

• Friendliness –easiness of programming, intuitiveness.
• Maturity – easiness engineering software using product: is it ready for integrating with version control, is it designed to capitalize on commonalities?

In general, TIBCO products appear more user friendly than IBM. Both BusinessWorks and IntegrationManager offer relatively intuitive user interfaces. IBM’s MQSI is counterintuitive and difficult to use. Interchange/Crossworlds improves on MQSI and is comparable with TIBCO’s products.

At the same time IBM products appear significantly more mature than TIBCOs:

• Both MQSI and Interchange/Crossworlds [VERIFY Crossworlds] store development artifacts in separate files, allowing for per-component version control. Common Eclipse-based IDE enables integration with popular version control tools.
• In contrast, IntegrationManager stores ALL development artifacts in a single repository/file, thus disabling the ability to put finer-grained artifacts under configuration management; likewise, it is very difficult to share basic commonalities (such as data schemas) between multiple projects. BusinessWorks (and the underlying Designer 5x) stores project components in per-component files and is integrated with popular version control tools, such as Perforce.
• Both IBM products are designed to capture and capitalize on commonalities. MQSI captures translations and flows in libraries that can be re-used across multiple integrations. Interchange – in addition to pioneering GBO/ABO – allows for capturing commonalities in ‘interaction templates’ and allows for re-use of transforms.

  This is in a sharp contrast to TIBCO’s products where capitalization on commonalities in behavior requires copy-paste; BusinessWorkers represents a slight improvement in this area as it represents transforms as XSTL sheets, this potentially allowing for building generic constructs.

‘Refactoring suport’ / refactoring browse – [FILL IN]

Summarizing, IBM’s product focus on engineering aspects of integration development, while TIBCO accents user friendliness. This results in a trade-off is between immediate productivity and ease of long-term maintenance.

Back-end
All components execute logic within custom containers. This is especially surprising and disappointing in case of IBM: one would expect that the brokering solutions (especially the java-based one: interconnect) would be built on top of the WebSphere application server thus allowing to unify management and administration.

All components offer reasonably sophisticated administration tools. TIBCO tools has been lacking a proper command-line administration tools but is rapidly improving. BusinessWorks’ command-line tools allows for automating administrative tasks, such as deployment; these tools are out of the box in IBM stack [verify Crossworlds]

BPM
[Reasearch MQ Workflow]

Summary
[FILL IN]

——————————————————————————–

[1] In general, the lower the degree of coupling between the components, the more flexible and robust their combination (software system) is. Therefore, low degree of coupling is – in most cases — a desired property of a software system. A system that exhibits a low degree of coupling is referred to as loosely coupled.

Coupling pertains to multiple aspects of relations between software systems components. Examples of coupling include technology coupling (components must use the same technology in order to interoperate), location coupling (components must know each others location), or logic coupling (components logic is interdependent; components operate through a non-generic fine-grained APIs). queueing), technology coupling, location coupling, network addressing coupling.

Temporal coupling denotes the existence of temporal dependencies between the components of a system. A temporally coupled system imposes time requirements on interoperating components. For example, a client-server system operating in request/reply fashion is tightly coupled, as it requires that the server be available at the time the client originates a request or a timeout occurs. In a temporally decoupled system, the temporal dependencies are reduced. This is typically achieved by some sort of store-and-forward cross-component information exchange mechanism and by asynchronous request processing. For example, an order processing system that retrieves an order entry request message from an MQ Series queue is temporarily decoupled from the asynchronous

[2] Certified Delivery augments RV with ‘at least once delivery’ semantics. Crudely speaking, if delivery of a message is temporarily impossible, RV stores ‘certified’ messages in local ‘ledger files’. RV re-publishes the message once the target system is ready to accept messages again.

[3] For the purpose of this document we divide integration topologies into transport-centric and integration-broker-centric.

In the transport-centric model. messaging constitutes the common platform (i.e. the ‘integration universe’) for cross-system communication. This passive transport core is augmented by active message brokers that provide transformation, routing, and task flow.

Transport-centric

In integration-broker-centric topology, an integration broker is a software component that enables system integration by managing and orchestrating message flows between the broker-integrated systems. An integration broker is transport-agnostic in that it can control message flows between a varieties of transports. The broker itself (and not messaging) constitutes the ‘integration universe’.

Broker-centric

How do you apply SOA ideas in the EAI world? the following is an outline of Service-oriented Integration architecture: an EAI based on SOA concepts. In order to marry the two, we’ll borrow some SOA concepts, we’ll reject some of them and finally we’ll add a few extensions in order to cover EAI-specific functions.

SOA Defined

SOA is an approach to constructing enterprise systems from components. (services) An SOA system is a collection of interacting services. Each service provides a well-defined collection of coarse grained functionalities.

Figure 1 illustrates components of an SOA system:

• At the core of an SOA system is a service. A service is a discoverable software entity that offers coarse-grained functionality through well-defined, published interface. A service is offered by a service provider – a software entity that implements the service specification.
• A service is accessed by a service requestor. A service requestor may be a client application or a service itself.
• SOA is enabled by an SOA broker. An SOA broker provides the following two components:

  

  Figure 1 SOA / Context diagram

• Service registry is a special service that maintains a database of services (service interfaces and service providers) and provides a lookup facility to this database. Service providers publish service interfaces to the service registry. Service requestors browse the service registry, discover the services and invoke them.
• Service invocation infrastructure allows the service requestors to bind to a service and invoke it.

Applying SOA to EAI

SOA is an attractive architectural model for EAI. SOA recognizes and embraces the distributed nature of most computing systems. The SOA’s concept of service as a collection of functionally related functions is close to the idea of an EAI-integrated system. At the same time SOA takes the concept of integration further: the service is to be reusable in multiple contexts. This requires that the services be properly abstracted for future re-use, as opposed to simply ‘making the applications talk’.

For the purpose of this architecture proposition, we will slightly tweak the SOA to fit the EAI needs. We will reject some of the ideas and expand upon the others.

The ideas we borrow

First the SOA ideas that we borrow to build our EAI architecture: the concept of service, the distributed nature of SAO, and the concept of service broker.

Traditionally, the EAI solutions have been operating on the API level. The typical adapters are level 1 / level 2 adapters that simply expose the application’s API to the middleware layer. API-oriented integration works reasonably well in simple cases but fails to deliver the most important promises of the EAI; the following two are the key reasons:

1. The reduction in the topological complexity of an integration solution is limited to the transport layer (Figure 2).

   
   Figure 2 Point-to-point integration on the logical level

   The big promise of EAI – or to be more precise, of the broker-centric architecture – is that it reduces the integration complexity from O(N2)to O(N) where N is the number of integrated applications. In case of the API centric integration this reduction does indeed materialize on the transport level: all applications are connected only once to the common medium. However, on the logical level, this rarely is the case.

   First, the broker-integrated applications expose to the broker their native API’s. Each such API expresses application-specific semantics and in the worst case semantics of each broker-integrated application are different. These differences range from simple ones, such as different field names to complex ones, such as different cardinalities of cross-entity relationships between the data model entities. Consequently, for two applications to communicate, their semantic differences must be reconciled. This is achieved by ‘mapping’ of the fields between the messages exchanged by the applications. In a naïve — and unfortunately common – solution, such mapping is done on a point-to-point basis. A standard solution to this problem is the common semantics approach (a.k.a. common business objects, canonical data model). [1]

   Second – the API’s not designed for re-use. The applications expose low-level, ad-hoc interfaces that too fine-grained to be easily re-used.

2. The API based interfaces are fine-grained and thus fragile. It is not uncommon to see API’s to business applications that include tens if not hundreds of fields. The API methods are frequently very detailed; performing a single business functions, for example creation of an account, may require multiple invocations of a single business method. Such API’s tend to change frequently – as a response to application-specific changes – and thus create ‘ripple effects’ that affect multiple broker-integrated applications in unpredictable ways.

   For the purpose of this architecture we will use the SOA’s concept of service as a solution to the problems of API-oriented interfaces. The concept of service as collection of functionally related coarse-grained functions is a natural ‘wrapper’ concept for exposing the functionality of to-be-integrated systems. We will expose the functionality of integrated applications as coarse-grained interfaces. We will achieve this coarse granularity in a bit unorthodox way (the common semantics being the orthodox way): through document-oriented interfaces.

SOA extensions

SOA is a natural fit in composite application problems. The to-be-integrated systems expose their interfaces as services to the broker; the broker — in turn – re-combines them and provides them to the end-users. As this is not the only integration model, we must expand the traditional SOA to meet the requirements of data integration and multi-step process models.

SOA implies request/reply semantics: service requestors invoke a service and receive a response. Such SOA is easiest implemented using synchronous transport; examples of such synchronous implementations of SOA include Web Services over HTTP or CORBA component model. EAI requires not only synchronous but also asynchronous transport. In this architecture we’ll go as far as mandating that asynchronous transport be the only transport used and that synchronous semantics be implemented over the asynchronous transport (see [MESSAGING]).

The data integration and multi-step process integration model require extensions to the basic SOA request/reply semantics. In the data integration model, upon the change of the data, the source-of-truth publishes an event notifying of the data change; the parties that replicate this information subscribeto the respective events and modify their data stores accordingly. As such, there is no service requestor and no service provider in this scenario. Rather, there is an event publisher and an event subscriber and the semantics of the communication model is publish/subscribe[2].

The multi-step process model employs a combination of request/reply and event-driven processing:

• The BPM coordinates long-running multi-step processes; at each step, it invokes the broker-integrated services. As the business processes are typically long running, the request/reply invocation of services operates in “response eventually needed” mode (and not “response needed immediately”). As such, asynchronous request/reply based on reliable messaging is in order, primarily because it affords us temporal de-coupling.
• Frequently, business processes operate in event-driven mode. For example, an event, such as new order, may start a business process.

In addition to the event-driven processing that the infrastructure provides for data integration, the basic SOA must be therefore enhanced to provide semantically asynchronous request/reply based on reliable messaging.

In order to accommodate the above, the proposed architecture extends the basic SOA to provide event-driven asynchronous semantics (Figure 3) as follows: in addition to fulfilling the requests, the services can also originate and consume events. Consumption of an event does not require a service to generate response. For clarity, we’ll use the following terminology to denote these special services:

• Services that consume events without producing a response are referred to as event subscribers.
• Service requestors that generate events are event publisher and not just by service requestor.

Figure 3 SOA

The ideas we reject

At this time it appears that the concept of service registries – such as UDDI – for machine-to-machine service invocation within the boundaries of a firewall is not practical; we reject this concept in our EAI architecture. Note, however, that we retain the concept of service registry as a central repository for service information; it just needs to be machine-readable. This is not to say that service registries and automated discovery of services do not work. It is author’s opinion that automated service discovery may be far better used in situation where a similar function can be performed by multiple services – for example in B2B scenarios. Also, discoverability is great for services that are to be used directly by humans, i.e. services with a human-facing aspect such as GUI or a form. Examples may include portals that contain a service directory or the Internet where the service registry is called Google[3].

Emerging architecture

Figure 4 illustrates an EAI system approached using SOA concepts.

Figure 4 EAI as SOA

A SOA-style EAI system consists of a collection of services that interact through the enabling infrastructure.

The SOA enabling infrastructure includes the integration broker and core services. It is augmented by infrastructure support functions and user interface.

Note on terminology: for the purpose of this architecture we’ll distinguish between services and functions. Both services and function are collections of coarse-grained functionalities; the difference is that services are accessible and invokeable through the broker infrastructure while the functions are not.

The integration broker enables the interaction between services by providing the following functions (look here for an explanation of these functions):

• Transport
• Connectivity
• Transformation
• Routing
• Taskflow

These broker functions are augmented by core services:

• Business Process Management (BPM).
• General Persistence Mechanism

The functionality is delivered by the fulfillment services. These include your CRM, your billing, your accounting, and what else you bought over the years. The fulfillment services may be re-combined into more sophisticated ones using the brokers taskflow and BPM services. Such services — that emerge as a result of recombination — will be referred to as composite services.

If you build a composite application, you’ll need a user interface. User interface is provided by a portal that delivers authentication/authorization, personalization, content management and related functions.

All that wonderful infrastructure must be monitored and managed. Thus the need for the infrastructure functions, such as monitoring or proactive error handling.

We’ll discuss all services and functions in detail in the following sections …

——————————————————————————–

[1] A standard solution to the problem point-to-point translations the common semantics approach (a.k.a. common business objects, canonical data model). The integration broker (or the application adapters) translates the application-specific semantics to the common semantics on the boundaries of the broker. As a result, the applications are not only technologically, but also semantically de-coupled. See the [INTEGRATION PATTERNS] section for more.

[2] Note that the publish/subscribe model pertains here to the cross-service communication over the broker. It does not imply that the underlying communication model employed by the broker is publish/subscribe.

[3]Another nice aspect of Internet as SOA is that it is built using document centric interfaces.

Experience shows that application integration efforts fall into one of the following three categories (integration models): multi-step process, composite application, or data integration. Differentiating between the integration models is helpful: each of the models has different characteristics in terms of the processing model, appropriate implementation approach and software requirements. You are likely to start your integration by implementing one of the models expecting that this is the only model you’ll implement. That is unlikely. Sooner or later a need for other models will emerge. Therefore, it is my belief that a ‘complete’ integration solution should provide an infrastructure to support all models. At the same time, it is very likely (80/20 rule) that one of the models dominates. This model should drive the key architecture design decisions.

Multi-step process

The multi-step processintegration model (Figure 1) integrates the functional flow of processing between multiple applications.

Figure 1 Multi-step process application integration model

A good example of a multi-step process model is a telecommunication order-to-cash system. An order-to-cash system coordinates activities within multiple business and operations support systems in order to fulfill customer’s order a telecommunications service. The fulfillment of an order typically requires establishing an account in business support systems such as billing and CRM, reservation of network elements that comprise the service in a network inventory management system, and provisioning of the service on the network. Often, the process of provisioning the network elements requires coordination of work orders for engineering personnel; execution of such work orders may take days or weeks and not seconds like it is the case with automated systems. Interactions with humans are not limited to work orders; human interactions may be required for approvals, escalations, etc. Provisioning of complex telecommunication services frequently requires interaction with third-party suppliers. For example, a telecommunication company may outsource the ‘last mile’ connection or on-site installation. As the multi-step process integration often includes interactions with humans, the integrated business processes may require weeks or months to complete. A typical implementation approach relies on a Business Process Manager that coordinates such long-running business processes.

Composite application

The goal of the composite application (Figure 2) integration model is to provide a single virtual application that is composed of separate applications, using an integration platform.

Figure 2 Composite application integration model

A good example of composite application is ‘webbing-up’ of a set of legacy systems. Continuing with the telecommunication example, a telecommunication services provider may expose the systems that participate in multi-step ordering process to the end-customers via common front-end. The provider allows the customers to see a ‘single-pane’ view of their service that combines the business and technical characteristics of the service. The user can see on a single web page the service’s pricing (business characteristics) and implementation details such as the IP addresses or routing (technical characteristics). A common choice for facilitating such a common view is translating the customer’s request for a ‘single pane’ into a coordinated set of interactions with the back-end systems. The front-end system (typically an application server) delegates the ‘single pane’ request to the integration framework. The integration framework de-composes the request into a set of calls to the back-end systems, collects the responses, and provides the re-combined data back to the front-end.

Data integration

The goal of the data integration (Figure 3) integration model is to provide data consistency across multiple applications. This is accomplished by propagating data changes between the integrated applications.

Figure 3 Data integration application integration model

Expanding on the telecommunication services provider examples the need for data integration emerges when the components of an integrated share elements of the integration-wide data model. Perhaps, in an ideal world, there would be no need for data integration as all the systems would be updated as a part of multi-step process or consolidated application model. In reality, constructing such an infrastructure is rarely justified. For example, porting a highly-efficient and complex interface of customer relationship management system to the generic common front-end us unlikely to yield equally efficient interface and the costs of such migration would be prohibitive. Consequently, many applications within the integrated ecosystem are accessed directly through their native user interfaces. If a customer service representative keys-in a change to the customer’s address into the CRM application, then it would be valuable for that change to be reflected in the billing system; hence the need for data integration.

Data integration is typically implemented using event-driven publish-subscribe interactions between the application. An application that accepted changes to a shared data element (e.g CRM) ‘publishes’ an event of change onto the integration framework. Applications that are interested in the updates (e.g. billing or data warehouse applications) ‘subscribe’ to such interesting events and update their internal data stores in response.

Integration models’ implementation characteristics

Integration models have different characteristics in many areas terms of processing model, communication model, and required software. The following provides a few high-level characteristics without delving into the implementation details. See sections [XXX] and [XXX] for details on transactional processing, BPM, and task flow.

Table 1 Processing and distributed transactional model

Table 2 Communication model

Table 3 Tools

Summary
Application integration encompasses three different integration models. Each model exhibits different and often conflicting with other models implementation characteristics.

In a typical scenario, only one of the models is implemented during the initial integration phase. However, as the integration matures, the need for the remaining models inevitably emerges. A durable integration platform must provide facilities for all three models.

——————————————————————————–

[1] The scope of this discussion is limited to distributed transactions that span the integration platform. and multiple applications. Distributed transactional semantics is required in other context as well. See [TRANSACTIONS] for more.

[2] This is not to impose a synchronous underlying communication model. A synchronous semantics may be implemented over asynchronous transport (see [MESSAGING] for more)

[3] Note that the BPM products provide a strong user-facing functionality for managing workflows that require human involvements; such functionality includes user interface, role/privilege management, workflow document management (versioning etc.) and allows for workflow modification by non-technical users. Such functionality is of a very limited applicability in multi-step process environments where the coordination is limited to automated systems.