IBM Interchange: an example of not-so-productive EAI tool
The Good Stuff: Broker Infrastructure
As Value across the integration stack suggests, generally the infrastructure part of the EAI stacks delivers value but there are often problems with the development tools.
ICS infrastructure does deliver value. The ICS is an apparently-solid integration broker with a number of valuable features. Let us briefly review a few of them:
- A broad selection of technology and application adapters. Technology adapters include MQ Series, JMS, IBM Workflow, COM, CORBA, EJB, files, WebServices, HTTP, and others. Application adapters include SAP, Siebel, Ariba, i2, JD Edwards, Oracle Applications, and other. Take a look at the WBI Infocenter for a complete and up-to-date list.
- A broker that provides an environment for executing ‘flows’ (ICS’ term for orchestrations). The ICS broker includes a number of standard and several non-standard useful features:
- 2PC transaction manager. (Apparently, ICS is built on top of Orion application server. Its just my guess: I think I have seen Orion traces somewhere.) The 2PC transactions span flows and
- Two monitoring tools (reach client and web-based). You can use them to find in-error flows and resolve them, monitor status of integration components, report performance statistics and generate other useful reports
- Compensatory transaction management for non-2PC transactions (i.e. long-running business transactions) . You can specify compensatory actions for your flows. ICS will manage their execution in case of failure and - from what I understand - propagate compensatory invocations down to sub-flows.
- Ability to persist flow state. You can use ICS to manager for long-runing flows.
- Recovery of in-progress flows in case the ICS server crashes
- The relationship manager provides the cross-entity mapping functionality. You can map account ID in your shiny new CRM to account ID in your legacy accounting
In general I have a positive impression of the ICS broker. Clearly, reliability has been on the designers’ minds.
The Bad Stuff: the Development Environment
Multiple aspects of ICS development environment make it difficult and ineffective to work with. The following paragraphs go in details over issues such as lengthy feedback loop, little automation/scripting, little IDE validations, poor Java development support, and difficulties testing.
Lengthy Feedback Loop
I use the term ‘feedback loop’ duration to denote the time between a change to an integration artifact and the time when the effect of the change can be observed / tested. Short feedback loop is critical to productivity.
In case of ICS the feedback loop is very long (order of minutes). Testing of integration artifacts (except for maps) requires deployment. The deployment loop is lengthy. Typical change requires an order of minutes before verification. Some changes (for example, updates to connector meta-objects) require ICS re-start.
No Build Automation
ICS does not provide tools to script-generate the deployment artifacts. The deployment artifacts must be generated manually (clicking-through the IDE). This is bad because:
- Build is error-prone not repeatable.
- Building is annoying and slow.
- You cannot put the build under a continous integration using tool such as Cruise Control
You can work around this problem, at least to some extent. I was able to build the deployment artifacts by mimicking a IDE-generated one; it required re-naming a few files, re-arranging them, and jaring them up. However, I was able to do this only with simple projects and only for a few types of artifacts, so I am not sure how viable it is.
Limited Deployment Automation
Although ICS offers repos_copy.bat script to deploy your solution, you cannot not stop/start your solution from script. At least, not easily. And you need to stop flows, maps, and other components before re-deploying them. After the deployment, you need to start some of the components (or at least some of them - some do get started automatically).
Other than being very inconvenient, this prevents you from employing continous integration, as you do need to deploy & start your solution before you can test it.
Fortunately, ICS has an SNMP interface. You can access SNMP from scripts, using tools such as net-snmp
Difficulties with Making Functional Changes and with Refactoring
Refactoring and functional changes are easier if tools support it. ICS offers little or no support for changes. Even the basic and most common change – renaming – is very frequently not auto-propagated through project. Few examples of problems with basic change propagation below:
- You cannot rename a business object (business object is a peculiar to someone familiar with OO name that ICS uses to denote its data structures). If you want to rename it, you need to export it to a file and re-import it under a different name.
- Changes to objects referenced from Java snippets are not reflected in Java code. For example, if you rename a map, and that map happens to be invoked from Java code, the Java code will not be updated.
It is not unreasonable to request that the Java code be updated. Just take a look at how Eclipse works with pure Java code. Changes to Java are propagated to properties, XML config files, and other.
- Change to signature of a collaboration template is not auto-propagated to the affected collaboration instance. Again, it is not unreasonable to expect that such changes be auto-propagated or at least that the affected objects be marked invalid.
Little validation by the IDE
IDE offers little validations of project configuration. Many errors are not apparent until deployment. Given the long feedback loop, common tasks such as configuring an adapter are is often very time consuming.
For example, references to so-called meta-objects are often not validated. In ICS-talk, meta-object is nothing more but a bundle of properties. You use these meta-objects to configure ICS components, such as connectors or data-handlers. Meta-objects cen refer other meta-objects. Oftentimes, you refer a meta-object by typing-in its name (no, no drop-downs) and that name is not validated. You will find out of a problem only after deployment. And, given the long deployment loop, this becomes time consuming again.
The IDE provides very little support for populating these meta-objects. The meta-object fields are often comma-separated lists of codes (e.g. line such as SOAPAction=”";cw_mo_soap=SOAPConfigMO;cw_mo_http=ProtocolConfigMO), thus not only error prone, but cannot be populated without documentation at hand. In addition, these meta objects frequently rely on naming conventions (for example, MX_????_FOO describes connector FOO) rather than on cross-referencing.
Very poor Java support
As code-free as the ICS claims to be, it still requires Java coding. Surprisingly, browsing through ICS examples reveals that flows are often expressed as directly as Java, instead of using the graphical notation. Perhaps because it is easier to express your flow in Java, than in the colorful UI?
Given that, there there is little Java development support. The Java editor is a simple text editor with syntax highlighting. Common productivity features of contemporary IDEs (auto-completion / intelli-sense, on-the-fly compilation) are not available.
The developer modifies Java code snippets (not entire classes). The ICS wraps code snippets into classes and then compiles them. The compilation errors include line numbers in the class code (and not in the snipped code), therefore making it difficult to determine the line number that contains the error.
And probably the most important aspects: because you are working with code snippets and not with complete Java classes, and - in addition - these snippets can be ran only in the container, you cannot unit test your code using any of the popular testing frameworks, such as JUnit
Team Collaboration
Given the little scripting, how is the team work to be supported? When a project is maintained by multiple developers, how is the project built? The manual click-through build process is error-prone and not repeatable. Also, it cannot be repeated often enough so that continuous integration can be applied.
The private ICS-specific XML representation of the integration artifacts makes it difficult for developers to work in parallel. Merging of developers work requires manual copy-paste (or rather re-create) using the IDEs.
Other Problems
Other minor problems and common grievances regarding EAI tools apply. These include:
- Real estate: even a simple ‘task flow’ takes a substantial chunk for your monitor. Not even close to the efficiency of a FOR loop or IF statement. Perhaps thats why there is so much Java in ICS examples
- Can you store modification history in your artifacts? Can you maintain you TODOs with your ‘code’?
- Can you temporarily comment out a line or a block to test something?
- Can you automatically enforce development standards (such as variable naming conventions)?
- How about a diff list between two versions?
Few positives
Lets finish with a few positive aspects regading tools that set ICS apart from other tools:
- Configuration Management
ICS projects are broken into fine-grained files corresponding to integration artifacts (maps, flows, etc) and are therefore easy to put under version control. This is in sharp contrast to many competitors. For example, early products from TIBCO would store your entire project in a single file, making version control impossible (TIBCO has improved since).
The ICS’ Eclipse-based IDE integrates easily with version control tools, such as CVS or ClearCase.
- Excellent IDE foundation
ICS uses Eclipse as IDE. Eclipse specific functions are delivered as Eclipse plug-ins. As a result, you get to use a first-rate IDE, with a host of useful features, such as intuitive navigation, context help, version control integration, a host of third-party (ofen open source/free) plug-ins, not to mention such basics as the search function (you’d be surprised that some tools won’t let you search a project for text!).
Summary & what do you do?
ICS exemplifies problems with EAI tools. The value of the solid foundation is significantly reduced by the poor tooling and cumbersome development process.
The integration is all about accomodating change. Therefore, it is not about how fast you build a solution, it is to some extent about how fast it runs, but most importantly, it is about how fast do you change it. ICS tooling may be OK to build a simple solution reasonably fast, but it is a very poor toolset if you want to maintain a complex solution over time, especially in multi-developer environment.
What do you do?
I believe that by now that capabilities of a typical EAI stack have crystallised. You need to look past the typical feature-based evaluation criteria to find real differentiators. Quality of development tools is a frequently overlooked aspect, and one that has a big impact on your bottom line. Therefore, o make sure when evaluating your integration tools you do include the tools’ quality as the key evaluation criteria.