Value across the integration stack: does your broker pull its weight?
This article takes a critical look at the value across an integration stack. The value is distributed very unevenly. Most of the stack components deliver unquestionable value. However, the core of the integration stack - the integration brokers - tend to provide very primitive development environments. While these development environments allow to built simple solution very fast, it becomes virtually impossible to use them to efficiently and reliably construct complex systems.
This article suggests that the brokers development environment is the one of the key differentiators between the EAI/SOA/SOI vendors. It also mentions the key differentiators between the development environments of TIBCO’s Business Works, IBM’s Interchange, and Microsofts BizTalk.
The stack
For the purpose of this article I use the following model of an integration stack. See the definition of integration stack for explanation and take a peek here for examples.
Figure 1: integration stack
Where the stack delivers a value?
- Connectivity.
Adapters and connectors provide connectivity to technologies and applications. Under normal conditions, you don’t want to build a JMS, Oracle, or a Siebel adapter yourself. These are usually worth paying for. Note also, that with the emergence of Web Services standards, the application adapters are frequently obsolete as many COTSes provide out-of-the-box Web Services interface.
- Asynchronous transport.
Message-oriented Middleware provides asynchronous reliable transport to the integration infrastructure. Asynchronous messaging with reliability option is a critical component of a robust integration platform. In most cases, it is worth paying for or getting an open source alternative (take a look at BizTalk’s message boxes for an architecture in which you actually can get away without MOM and still benefit from asynchrony).
- BPM.
The commercial Business Process Managers are mostly worth paying for. You generally dont want to develop a BPM engine yourself unless your processes are trivial. The point here is that the commercial products tend to deliver value without noticeable liabilities.
- Monitoring.
Buy - or get as open source - but don’t build enterprise monitoring tools, such as BMC Patrol. They look nice on the monitor and come with pre-packaged set of goodies that let you monitor practically everything and often have some extra functionality, such as proactive error handling.
- BAM and the likes
I don’t know. I suspect that BAM tools provide functionality that is unlikely to be worth replication using custom development.
Broker and IDE
The typical integration broker (TIBCO’s Business Works, IBM’s Interchange, Microsoft’s BizTalk) provide (Figure 2) provides an integrated development environment (IDE) and a runtime. The developer generates integration artifacts (maps, orchestrations, custom Java code) and deploys them to the broker.
Figure 2: quite obvious view on the broker’s runtime and development environment.
The value of the broker runtime (where the problem isn’t)
The concept of a broker as the component that provides transformation, routing, & orchestration thus de-coupling interactions between the integrated applications is valuable.
The commercial broker runtimes are solid, mature and eliminate the need for custom development of commodity support functions that are must-have for running the business, but are by no means competitive differentiators. These functions include:
- Runtime for executing maps and orchestrations
- Runtime for interacting with adapters & connectors
- Health & Activity Monitors that provide for proactive error handling, auditing, SLA reporting, or resolving in-error flows.
- Transaction managers, both 2PC or business transaction managers (i.e. the compensatory transactions schemes in ICS and BizTalk)
- Service lifecycle management
- Support for long-running flows (ICS & BizTalk; last time I checked BusinessWorks 5.1.2 did not support long-running flows and used to delegate this function to the BPM /InConcert or Staffware/).
- And many more …
There is little chance that building a custom broker runtime is going to provide business value, just like you would no write your own SQL server (you’d write your own SQL queries) or J2EE server (you’d write your own EJBs thought /that is if you don’t know about lightweight containers, such as the Spring framework)
Little divergence: the contemporary business software development practices and development environments that support them
Many integration developers have been doing integration long enough that they’ve missed major improvements in the way business software is being developed. The few interesting developments with bearing on the tools are:
- Refactoring
“Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior. Its heart is a series of small behavior preserving transformations. Each transformation (called a ‘refactoring’) does little, but a sequence of transformations can produce a significant restructuring. Since each refactoring is small, it’s less likely to go wrong. The system is also kept fully working after each small refactoring, reducing the chances that a system can get seriously broken during the restructuring … ” . Definition from refactoring.com. See refactoring.com for more.
To me, refactoring is important in two ways:
- It is a method for maintaining a clean internal structure of the solution. Clean code structure is critical to sustaining productivity. If you dont mainain a clean internal structure, overtime your software becomes a nightmare to maintain (like virtually all integration projects do).
- It enables evolutionary design. You dont need to plan much up-front if you know that you have an arsenal of tools that allow you to evolve your solution as you go, while maintaining its internal clarity. And over-designining business software up-front is usually time wasted.
Modern development environments, such as InelliJ from JetBrains or Eclipse are working refactoring browsers, automating typical refactorings, ranging from simple ones (such as component re-naming or method extraction) to complex ones (such as re-shuffling class structure).
- Test-driven Development (TDD)
Test-driven Developement is an Extreme Programming (XP) practice in which you write test first, then you wite your code to satisfy the test. The result is that the programmers actually do test their software, and that testing is automated. The quality is built in the software. The test coverage is maintained high. Such tests provide a safety net for changes, thus enabling fear-free refactoring. I can modify my software to reflect a better understanding of problem domain (of cours, I write a test for this new understanding first), run hundreds or thousands of tests that have been written for the system (typically within minutes) and be comfortable that the system still works.
The modern programming environments enable TDD in a several ways:
- Ability to execute tests is built into IDEs
- Lightweight containers, such as Spring allow to test the business components without the need to deploy your solution to the container. This dramatically shortens the feedback loop between the time you change software and you test it. Instead of waiting for minutes for deployment of an EJB, you test your Plain Old Java Object (POJO) instantly.
- Continuous Integration
Continuous Integration is a practice of continuous building of software developed by multiple developers. Each time a developer commits a code change, the build process is executed. As the software has been built using test-driven development approach, software is self-testing.
Continuous integration lets you avoid (code) integration problems. An integration problem is spotted instantly, be it a compilation problem of functional problem that is detected by the built-in automated unit tests.
Continuous integration is relatively easy to implement with the traditional development tools (i.e. Java, C, C++, C#, …) , especially is out-of-container testing is available. This is NOT the case with EAI tools, as you will see it below.
To sum up, the modern leading-edge development tools provide an enabling environment for an evolutionary design. This reduces the time required for up front design or eliminates the need for up-front design altogether.
If it is as easy to re-shuffle your class structure in the code, as it is a design tool — such as Rose — why use the spend time designing? Coding is the design. You get a working product early, thanks to the safety net of testing you can alter its structure without a fear that you break something, thanks to the IDE’s refactoring support you change your structure instantly, and thanks to the lightweight container, the feedback loop is extremely short, so you can build/test the solution hundreds of times a day.
Another little divergence: should integration tools be like the development tools?
Yes. While the integration vendors continuously announce the first development tool that a business analyst can use, this is simply not the case. The transformation, routing, and orchestration remain the domain of developers. Therefore, the integration tools should focus on providing a productive development environment rather than faking a business-analyst-usable tools.
So what are the problems with a typical integration development environments?
The following is a laundry list of common problems with the integration tools:
- No or little support for test-driven development. As most of the tools require that integration artifacts be deployed to an integration broker to be tested, fine-grained out-of container unit testing is difficult. In addition, no tools that I know of provide out-of-the-box unit testing facility. However, there are numerous open source integration unit testing tools.
BizTalk is a clear leader in this area with tools such as BizUnit or BTSUnit. There are also tools for other vendors, such as RvTest for TIBCO/RV or GH Tester - a generic JMS testing tool.
- -No or little support for refactoring. Frequently as simple change as a variable re-naming is not auto-propagated through the project (example: IBM ICS). This, plus lack of automated testing, prevents integration developers from refactoring their solutions. This quickly leads to messy and non-maintainable solutions.
- Long development feedback loop. Because of the need for deployent, it takes minutes between the time you make a change and the time you see test results. In BizTalk and ICS this takes an order of minutes. TIBCO’s products do allow for out-of-container testing reducing this the feedback time to seconds.
- Poor build support.. Often, there are no script tools for automating build and development. The solutions are build manually using UI tools. This is time consuming, non-repeatable, risky, and does not allow for implementing contiunous integration:
- BizTalk does provide command-line build (devenv.exe can be run from command line) and is fully integrated with Nant.
- IBM’s ICS does not allow for command-line build. You must use UI to build your solution. It is possible to hand-craft a deployment unit by reverse engineering ICS’ .jar structures but thats just a workaround.
- BizTalk 5.1.2 does not provide command line build. However, it is possible to simply jar the project (a workaround again).
- Poor configuration management and teamwork support. For example, in case of development conflicts, merges must be manual. The file structures often create hot-sports: various ‘project’ or ‘binding’ files tend to change each time any component changes, thus effectively preventing parallel development.
Extreme example: older versions of TIBCO tools used a single XML repository file, effectively preventing concurrent development. The common workaround was to manually break and merge the XML repository file.
- Any custom coding (say Java code in TIBCO) has often zero support form IDE. If you get used to intelli-sense in advanced IDEs, you realize how counterproductive it is to type-up your Java in notepad-style editor with no support for JUnit, no auto-completion, etc. And often, you must use Java/C# to augment the IDE. Again, BizTalk that uses Visual Studio, is a bright spot.
-
Few other minor but representative problems:
- Did you ever try to find out which components publish on topic XXX.YYY.ZZZ? Something a simple UNIX ‘grep’ or an IDE search would do but does your tool have a proper button?
- Real estate: even a simple ‘task flow’ takes a substantial chunk for your monitor. Not even close to the efficiency of a FOR or CASE statement. In order to develop a complicated flow, do you wish your had a 70 inch monitor?
- Can you comment your development artifacts, ideally on the screen? Can you store modification history? Can you maintain you TODOs with your ‘code’?
- Can you temporarily comment out a line or a block to test something?
- Can you automatically enforce development standards (such as variable naming conventions)?
- How about a diff list between two versions?
Reasons: how did we get here?
I think that there are at least two categories of problems and two categories of reasons:
- Inherent problems related to visual development, and,
- Poor quality of some visual tools.
Inherent problems include real estate limitations, the constraints of what had been built-in into menu, and –to some extent - difficulties in capitalizing on commonalities. I may very well be wrong on it and the next generation of visual tools may solve these problems. Intentional Programming concept seems especially interesting. However, until that materializes, I feel strongly about a need for text-based source code. This code can very well be generated and manipulated through a visual tool, but the ability to operate on human-readable text addresses many of the shortcomings of visual tools, providing that it is done through a modern IDE, with support for modern design techniques, such as refactoring and test-driven development.
Poor quality problems result from the history of integration brokers. I think that there are at least three categories of integration broker origins:
- Messaging add-on. As messaging becomes commodity, messaging vendors, such as TIBCO, move up the EAI stack to avoid selling commodity products. Such vendors add integration broker to the mix and other elements of ‘EAI stack’ such as BPM or Business Activity Monitoring (BAM).
- Application server add-on. As application servers and EAI offerings converge in functionality, the application server vendors add EAI functions. Good examples include BEA and Oracle. Traditionally application server vendors lag in functionality behind the specialist (such as TIBCO), however they tend to offer more robust infrastructure.
- Integration-broker vendors, such as Crossworlds (now owned by IBM), that designed messaging vendor-agnostic (or sometimes vendor-specific, such as NEON for MQ)
In some of these cases, it is natural that the vendors have – at least initially – difficulties with producing quality development environments:
- Messaging vendors needed to shift their attention from back-end development (messaging frameworks) to development tools development; this requires a completely new skill-set and may result in considerable growing pains. For example, TIBCO is delivering already third generation of its integration broker (after Message Broker and Integration Manager); each of these three generation is a almost complete re-write of the previous one and reflects shifting focus from messaging add-on to standalone, messaging-agnostic product.
- Application server vendors (generalists) are in excellent position to provide back-end infrastructure; frequently, they have substantial experience and capabilities in the area of development tools (IBM, BEA). They are in excellent position to deliver solid integration brokers, however, they traditionally lag in functionality behind the more innovative (in the area of EAI) specialist; this is especially visible when it comes to support of more exotic standards (various HL7 or HIPAAs) or ‘higher levels’ of EAI stacks, such as BAM. Perhaps generalists, being frequently very large companies, are not so hard pressed by commoditization from the bottom of the stack that we force them to escape ‘up’.
- Pure-play Integration-broker vendors, such as Crossworlds (now owned by IBM), that designed messaging vendor-agnostic (or sometimes vendor-specific, such as NEON for MQ) tend to focus their development on their visual tools. The lack of experience (startups) works against them. Given the relatively small number of such vendors, it is hard to say how it works out.
In addition, I think that some solutions are showing signs of overshooting, i.e. providing functionality that customer does not need. This is especially likely in the case of integration specialist that – being pushed by commoditization from the bottom of the stack – try to compete on the integration broker levels. Thus the push towards prettier GUIs and various ‘zero-coding’ and ‘business analyst friendly’ environments.
What do you do about it?
Be careful choosing your integration broker solution. When you see the demo, you may be under the impression that it takes no time to integrate two systems. Just drag, drop, even your business analyst can do it. This is true in trivial integration cases that need not to change (a demo to the executive team is a perfect example).
The more complex and the longer lasting the integration project is, the more important the changebility of your architecture. The ’speed of development’ delivered by graphical integration tools may be deceiving. The importance of more traditional and modern software engineering techniques such as configuration management, re-factoring, test-driven development, reuse of commonalities etc. may very well outweigh the speed at which you can drag-and-drop data elements from left-to-right.
Therefore, look behind the pretty GUIs – make sure that some ‘engineering’ is possible with the tool you are about to buy. Look for configuration management, refactoring, test-driven development. Recognize tools productivity as a critical aspect of your tool’s evaluation process. This should be relatively easy as the other aspects of the stack tend to commoditize.