Application Integration and Agility (1): Understanding Agility
This article looks at the ideas behind the agile application development and at the realities of application integration to determine which ideas – if any – are ‘portable’. This write-up consists of two parts:
- Verifying applicability of Agile requires an understanding of what Agile is. Detailing the Agile concepts is out of the scope of any single article, is definitely not an intention of this one. Instead, we’ll try to understand the structure of the ideas that comprise Agile. We’ll then create a carte du jour of – sometimes interdependent but oftentimes orthogonal – Agile ideas
- Having dissected the Agile we’ll take a look at integration and its cousins (EAI, SOA, SOI). We’ll evaluate the propositions on the menu of Agile ideas against the realities of application integration. We’ll determine the ideas that are immediately transferable. Then – more interestingly - we’ll discuss the agile practices that are currently not transferable. We’ll determine the nature of the obstacles to transferring them: are these obstacles inherent to the nature of integration or – perhaps – they result from other less obvious problems, such as the immaturity of integration tools?
- Finally, we’ll attempt to identify integration-specific productivity improvement and risk-reduction practices – some of them Agile-spirited, some not.
1. Dissecting Agile Application Development
The agile manifesto (http://agilemanifesto.org/) is probably the best starting point to understanding Agile. It lists the principles of agile development in an unstructured and very software-centric fashion. In order to perform an informed evaluation of Agile’s applicability to the integration, we’ll dissect the agile principles and try to identify a structure behind them. This article proposes the following structure for Agile Software Development (Figure 1 Structure of Agile Software Development Figure 1):
Figure 1 Structure of Agile Software Development
- Adaptive Project Management is at the heart for Agile practices. Section 3 contrasts the Adaptive Project Management with the more traditional – for the lack of a better term - Plan-driven Project Management (which is not to imply that Agility precludes planning). It then identifies the reasons for Adaptivity’s success in business software development as well as defines the requirements for projects and products to be managed in an adaptive fashion.
- Agile adopted several Lean Manufacturing ideas that happen to both orthogonal and a closely related Adaptive Project Management. We discuss them in section 5.
- Finally the agile community promotes a number Software-specific Productivity Improvement Practices (section 5). Some of these practices – especially the ones relates to Sociology of Software Development, are useful not only on agile projects, and not only on software projects. Some, such as Refactoring, Continuous Integration, or Automated Testing are critical techniques that enable the Adaptive and Lean approaches discussed above.
2 Adaptive
Crudely speaking, project management is about (1) planning and (2) executing the plan.
2.1 Plan-driven and adaptive project management
We’ll introduce the adaptive project management by contrasting it to the traditional approach. Of course, there is a whole spectrum of in-between approaches.
Figure 2 Traditional planned approach
Figure 2 illustrates the traditional plan-driven approach. In this approach, planning is long term. The plan is very specific about the anticipated outcome (scope) as well as about the trajectory through time, the money to spend, and intermediate task breakdown that would get us to that outcome. During the execution of the plan, the objective of the project manager is to minimize the ‘delta’ between the planned trajectory (your list of tasks, with dates, and budget) and the actual trajectory (i.e. how late/early or how over budget you are).
While executing the project, you build your product incrementally, piece by piece. Therefore, the product becomes usable – typically – at the very end of the project. Oftentimes, this is when the customers see the product for the very first time. In the software industry, what they see often surprises them.
Figure 3 illustrates adaptive approach. In this approach, the project is executed in frequent (weekly to monthly) iterations. Each iteration yields more-and-more complete approximations of the final product. The project is managed based on the customers’ feedback to the product approximation. The product often becomes usable well before the complete vision is achieved, although it would then provide less features. Changes to the direction are frequent, based on customers’ feedback. The final target is therefore moving, although within the original vision of what is to be delivered.
Figure 3 Adaptive approach
2.2 Comparing planned and adaptive
Both approaches work fine, but for different type of projects.
The following are key differing characteristics of both approaches:
- Adaptive approach provides for an early customer’s feedback on the product. This is critical for new product development where the customer and designers ideas may significantly differ, be very vague, or the kind of product that is being design has not been ever tried before; therefore, having the ability for the customers to ‘touch’ an approximation is very important if we want to build something useful. Feedback is much less important in developing fairly standard and repetitive products, such as houses; a customer can often see existing model houses and have a fairly good feel for what’s being built for them.
- In many ways, the adaptive approach reduces a group of risks stemming from uncertainities about project requirements and employed technologies.
The feedback on product mentioned above reduces the risk of building a product that does not meet customers’ requirements. The same feedback allows for measuring the progress of delivery against the customer’s expectations (and not against the plan), thus allowing to identify and fail early projects that are unlikely to satisfy user’s needs within the anticipated cost. This – again – is very important for new product development and much less important for repetitive and well understood projects. - As adaptive approach delivers the product in approximations (and not in partial increments), it is often possible to use the products before project completion.
- The plan-driven approach allows for a tighter and more precise control over the ‘levers’ of the project (schedule, scope, and cost). Therefore it allows for precise management of different group of risks: risks that are associated with broadly understood logistical uncertainity, such slipping schedules, unexpected meteorological events, etc.
- The plan-driven approach allows for optimization of the project trajectory. The trajectory of adaptive approach is always suboptimal, but this is only apparent once the project is complete.
Therefore:
- The planned approach works well for predictable, often repetitive projects, where the anticipated outcome is relatively easy to define. Examples of such projects include construction projects (buildings, highways) or elaborate projects for switching-over oil refining installations:
- A very important advantage of plan-driven approach for such projects is the ability to tightly control the scope, schedule, and cost, this allowing for precise management of logistical risk. You precisely plan your contingency for delays of supply delivery, or plan your building actions based the probability of rain. Sometimes this may be a critical safety requirement: for example, the oil-installation switchover requires a precise coordination of parallel activities, or severe accidents can occur.
- The ability to optimize the parameters of the planned-in-advance project tasks (dynamic programming) is another important benefit for such projects.
At the same time, these predictable projects do not require methods for managing the risks associated with the multiple unknowns of the product development projects: uncertainity of requirements and unknowns about the technology.
- The adaptive approach works well for difficult-to-predict, often non-repetitive projects where the anticipated outcome is difficult to define (business software). Examples include new product design (i.e. new car, new version of iPod) or business software development (custom claim processing system):
The critical advantage of the repetitive approach is constant feedback. This includes:- Customer’s feedback reduces the risk of building a useless product.
- ‘Subject matter feedback’ (i.e. how fast are progressing with the project using the technology in question, team in question, with a given experience) the allows to constantly re-adjust expectations regarding cost and schedule; this lets us identify and terminate doomed-to-fail projects very early.
The benefits of adaptivity come at a cost. Logistical optimizations are unattainable. This makes a perfect sense: as it is know from systems theory, the closer to optimum a system is, the less flexible it becomes. That’s why your flexible SUV stands no chance in racing an optimized-for-racetrack F1 bolid, by try driving F1 bolid around town, not to mention off-roads.
2.3 Requirements for adaptive project management
Based on the above guidelines, it is easy to spot domains in which projects look like wonderful candidates for adaptive. Software projects look that way.
Not so fast!
It is not enough that the above ‘inviting’ characteristics of a project (uncertainity about the outcome, uncertainity about technology) are present. Adaptive approach requires that the subject of management be malleable. It is important that the product can be evolved and that the cost of evolving the product is not prohibitive. In order to effectively respond to the customers’ feedback, it must be possible to alter important aspects of the product late during the project, at a low cost. Therefore, adaptive approach is not applicable to building construction, where, after foundations are laid down, it is difficult to alter the building’s width (fortunately, the ‘inviting’ characteristics for adaptivity are not present either, so our loss is minimal).
What about software?
Software developed traditionally is not much more malleable than a building under construction. Changes to early decisions often result in cascades of ripple-effect changes and in huge costs of re-design and re-test.
Fortunately, modern software development techniques such as refactoring and test-driven development make software malleable and therefore possible to develop in an adaptive fashion. We discuss these malleability or evolution-enabling practices in section 5
2.4 Summary
Table 1 Plan-driven and Adaptive: key differentiators
| Plan Driven | Adaptive | Efficiency |
High Long term planning allows for employing logistical optimization. |
Low Short-term planning horizon disables efficient logistical optimization. |
| Mitigated Risk | Effectively mitigates logistical risks. For example, schedule slips due to statistically quantifiable ‘higher power caused’ events (for example, bad weather). | Effectively mitigates risks due to uncertainity of requirements and unknowns of technology. |
| Malleability Requirements |
Very little needed Risk of changes to the scope is minimal and mitigated by intense planning, that – in turn – is possible because of the predictability of the project. There is therefore no need for the ‘subject matter’ to be malleable. |
Malleability is critical Cost of changes must be minimal as significant product structure changes in the course of the project are anticipated. Especially, the cost of late changes must be reasonable |
| Ideal Project Profile |
Warehouse construction.
Real-time Embedded Software for an Airline Jet |
New custom business system, constructed with support of test-driven development, refactoring, and continuous integration. |
3. Leanness
‘Leanness’ is the second cornerstone of Agile. In general, lean is about eliminating waste. The origins of lean software lie in the engineering concepts of lean production. The Production System Design Laboratory (PSD), Massachusetts Institute of Technology (MIT) http://lean2.mit.edu/ states that ‘Lean production is aimed at the elimination of waste in every area of production including customer relations, product design, supplier networks and factory management. Its goal is to incorporate less human effort, less inventory, less time to develop products, and less space to become highly responsive to customer demand while producing top quality products in the most efficient and economical manner possible.’
The lean production techniques aim at reducing various aspects of waste. Some of examples of waste in manufacturing include:
- Waiting time. Therefore cut waiting time as waiting is money wasted.
- Inventory. Therefore cut inventory ask holding inventory costs and is therefore waste (just-in-time inventory). Also modify scheduling such that the internal customer pull from the system instead of pushing the system (Kanban / http://www.strategosinc.com/kan.htm )
- Communication and coordination inefficiencies and overhead. Therefore work in small socio-technical systems (Workcells; http://www.strategosinc.com/workcell.htm )
The nature of waste in software is similar. As in manufacturing, there are communication and coordination problems. Yet – I believe – the core of waste in software comes from the traditional approaches to project management. Recall from the previous paragraph that the traditional plan-driven project management deals efficiently with logistical risks. Given a project in trouble, the traditional diagnosis is that the project requires better planning and more discipline. The remedies include beaurocracy of various sorts, tighter scope controls, more precise ‘design’ and tighter ‘construction’ controls, etc. These methods would be successful for a construction project, but given the nature of risks in software, these methods yield waste:
- Rigid and centrally controlled organizations move slowly, wasting time and requiring multiple formalities for simple operations. Yet what business software needs, is agility and a quick reaction to change.
- Formality requires bi-products, such as various ‘design’ documents that add little value to final product are generated. Yet in business software, construction IS design.
- Tighter scope controls convert the ‘change control boards’ into change prevention boards. And oftentimes teams deliver ‘to the scope’ and not to the real business need, producing useless products that need substantial time to be augmented with functionality that is actually needed.
The leanness proposed by Agile is perfectly possible, as extra beaurocracy will not remedy the inherent software risk that are not of logistical nature. Some of the lean principles include ( http://agilemanifesto.org ):
- Individuals and interactions over processes and tools.
The traditional approach is to eliminate the ambiguity as the perceived source of risk, by implementing a formal (and wasteful) set of processes and tools. As ambiguity is inherent to software requirements, the iterative development with working and ‘touchable’ model is a much better remedy for ambiguity. While constructing the model, individual interaction is far more efficient and lean than formal well-documented communication.
- Working software over comprehensive documentation
The traditional approach is to improve planning by more precise and well-documented software ‘design’ that then can be implemented. Yet this fails because of the inherent technology risk and the resulting fact that in software construction IS design, leaving the customer with binders of documents (waste) and no working software. The Agile approach is lean in that it moves directly to construction, delivering working, verifiable, ‘touchable’ product in a short timeframe.
- Customer collaboration over contract negotiation
The traditional approach to dealing with change is by implementing various change-prevention mechanisms (a.k.a. known as Change Control Boards, Scope Control) that make changes to the course of project nearly impossible, leading to deliver of unwanted features. This generates waste both in paid-for but unwanted product as well as in cost of opportunities missed by lack of the needed product that was not delivered (‘not delivered instead’).
4. Software-specific Agile Practices
The third group of Agile ideas are the software-specific practices that are commonly employed on Agile projects. Some of these practices are enabling techniques for adaptive project management, some contribute to leanness, and some are generally good practices that are orthogonal to lean / agile altogether. Much has been written on these practices, so for the purpose of this overview, we’ll list briefly only the most important ones, with explanations as to how they contribute to agility, and with references to to proper definition:
- Automated testing and test-driven development(http://www.objectmentor.com/writeUps/TestDrivenDevelopment).
Automated testing is a critical enabling technique to adaptive project management. First, the automation of regression testing enables short iterations. Second, automated testing contributes to malleability of the product; automated tests provide a safety net for changes, thus making changes safe and cheap. - Refactoring ( http://www.refactoring.com/ ) – through maintaining the clarity of product’s structure - contributes to malleability of the product. In addition, refactoring browsers reduce cost of refactoring and other changes, thus making evolving products more economically feasible.
- Other technical practices include Continuos Integration (http://www.martinfowler.com/articles/continuousIntegration.html ) and Object Orientation.
- Multiple non-technical social practices (pair programming, collective code ownership, user stories, etc) tend to promote ‘leannes’ of the process, through improving direct communication and knowledge sharing, and reducing the need for heavy frameworks to manage the two.
5. Summary
We have ‘sliced’ the concepts of Agility. We are now ready to evaluate the applicability of the Agile techniques to Integration.