Take Charge of 
Application Integration Chaos

REPRINTED FROM ARCHITECTURE & GOVERNANCE MAGAZINE (2010-09-15)

As I continued to serve as the Enterprise Architect at Aer Lingus (Dublin, Ireland), I am collecting references from EA journals, magazines and general articles which mirror my experience in shaping the IT Strategy, Enterprise Architecture and transforming People, Process and Technology as part of a 3-5 year programme of work.

One key capability that I’m introducing relates to Business Process Management (BPM) and Enterprise Application Integration (EAI) as part of an effort to consolidate, rationalise and retire the application landscape which previously extended to 242 applications that will be reduced to 52 core applications (in Year 1 – 2011). The article below is pertinent to my current experiences and hits the spot. Recall, in my “past life” I was a TIBCO and IBM consultant and have over 15 years of experience carrying out integration of applications (EAI) and information (EII) which typically extends to Information management strategies …

Want to know more about what I’m doing to transform Business/IT capabilities at Aer Lingus? Leave your contact details and any other comments/feedback and I’ll be in touch …

Enjoy!


At a high level, application integration means leveraging technical infrastructure to make diverse applications communicate with each other to achieve a business objective. The integration needs to be seamless and reliable, regardless of platform and geographical location of these applications.

The move toward service-oriented architecture, business process management, and software as a service has accelerated the recognition that application integration can increase business efficiency. To ensure that the integration is both beneficial and feasible, one should closely examine the business processes that are being supported before focusing on the systems and technologies themselves. Understanding this is the key to determining how to select the most suitable integration technologies.

This article examines the application integration challenges while considering the enterprise diverse technology landscape and architectural concerns. The presented “scenario-driven approach” describes how to successfully implement application integration standards at the enterprise level, leveraging the TOGAF methodology, and, ultimately, supporting the business capabilities.

Challenges of Application Integration

Making suboptimal or incorrect choices when selecting your technology toolset can lead to complex architectural issues, which in turn lead to tightly coupled systems and support and maintenance problems. Moreover, guidelines and best practices espoused by architecture groups are not consistently written down, nor are they consistently followed by application development teams at large. Without clear standards on when to use which technology, and for what purpose, one runs the risk of creating an unnecessarily complex technology environment.

Many enterprises have also suffered from organic growth and now have every integration technology of the last two decades in place: JDBC/ODBC, database links, remote method invocation (RMI), enterprise Java beans (EJB), Web services, Java message services (JMS), MQSeries, and flat files abound. This technology diversity has resulted in increased support costs and has adversely affected system performance.

Additionally, there are organizational barriers that may impact your application integration efforts as well. These issues typically arise from the fact that enterprise IT systems span multiple departments in the organization. Different development teams in the enterprise, if not properly guided or monitored, may tend to choose the path of least resistance or resort to technologies that they are familiar with, which can add to integration complexity.

Organizational issues in application integration can be tricky in a larger organization. However, these companies often have the greatest need for an effective application integration environment. Therefore, it is particularly important to clearly define standards, so that development groups can focus on business needs.

Levels of Application Integration

Application integration can occur at many different levels of a system’s architecture, including the data layer, the application layer, the service layer, and the presentation layer.

Integrating applications at the data layer can sometimes be the quickest implementation approach, due to the simplicity and power of data layer integration technologies. Data layer integration includes the use of database links, shared database catalogs, and direct database queries. However, this approach leaves many internal application details exposed, and upstream and downstream application changes result in significant impact, rework, and testing.

With application style integration, low-level implementation details may or may not be exposed or accessible from other systems. Application layer integration includes the use of flat files, message queues, and remote procedure calls. An “API” of some level is leveraged. Although this approach gives a better degree of separation, there are still problems with this approach.

Message-based technologies such as JMS and MQSeries are examples of popular queuing solutions but are based on proprietary implementations. Error handling can be problematic as messages can be lost on “undeliverable mail” queues. The level of application details that is needed to expose for interfacing applications is one of the major disadvantages of integration using remote procedure calls (RMI and RPC style of integration). In short, integrating applications at the application layer is preferable to data layer integration, but still has its own issues.

Service-based integration includes the use of Web services. The advantage of Web services integration is interoperability, even though the integration can still be point to point. Each Web service client still has the responsibility of knowing which service to call, so the addition of new endpoints will result in additional coding.

An enterprise service bus (ESB) combines the strengths of existing service-based integration technologies but provides more abstraction and interoperability. Application integration using an ESB combines message-oriented processing and Web services, which is the foundation for an event-driven SOA. In our opinion, services-based integration, especially when paired with an ESB, is the preferred method for all application integration in an enterprise.

Now What?

To get your application integration under control, we recommend that you work within your existing landscape of systems and technologies and leverage the TOGAF methodology, instead of kicking off a large “EAI project.”

We use a “scenario-driven approach” that focuses on aligning business scenarios with supporting technologies. This approach consists of the following steps:

  1. Leverage TOGAF architecture principles.
  2. Identify technology standards and create building blocks.
  3. Identify usage scenarios and map scenarios to the technology standards.

Leverage TOGAF Architecture Principles

The main reason to integrate applications is to support a business process. Taking a technology-first approach can lead to inflexible solutions that can be costlier to maintain as the business environment changes.

Architecture groups do not typically create technology standards in a vacuum for idealistic purposes (although some development teams may have differing opinions). When developing standards, you need to make sure there is direct traceability to enterprise architecture principles such as:

  • Creating loosely coupled interfaces.
  • Setting platform independent, open standards.
  • Developing reusable, shared services.
  • Minimizing application impact.
  • Promoting data consistency.
  • Recognizing that business logic is a strategic business asset and should not be placed in closed vendor solutions.

These principles are based on the default TOGAF enterprise architecture principles; however, they may be too vague for some project teams and leave much room for interpretation. To address this vagueness, the architecture group needs to identify acceptable integration technologies and map usage scenarios to those technologies to control the complexity of the integration environment.

Identify Technology Standards and Create Building Blocks

Based on organizational strategy, solutions in place, and staffing and skill levels, you should then identify the technology standards that are preferred at your company. Determine if your strategy is to be a custom Java shop, a Microsoft shop, or an SAP enterprise. Regardless of the technology, set clear standards for technology usage and avoid allowing developers to choose their favorite technology to use for integration.

TOGAF promotes the use of building blocks to support effective enterprise architecture. Building blocks are simply a package of functionality defined to meet business needs. For example, an architecture building block is a high-level, abstract architectural pattern. A solution building block is a specific instance of a technology or product. Every organization must decide for itself what arrangements of building blocks work best for it.

For each architecture building block, create the corresponding and solution building block(s). For example, ETL is an architecture building block; it can be realized by a solution building block consisting of products such as Informatica PowerCenter or Business Objects Data Integrator.

A well-specified catalog of building blocks will lead to improvements in application integration, facilitate interoperability, control technical diversity, and provide flexibility in the creation of new systems and applications.

Table 1 lists a sample of the approved architecture and corresponding solution building blocks.

File 931

Map Scenarios to Standards

As a final step, map the integration technologies to various usage scenarios. You will need to develop usage scenarios that can be used to identify types of application interaction.

Our set of usage scenarios includes:

  • Perform domain entity validation.
  • Synchronize business information in multiple systems.
  • Notify of state change between applications.
  • Notify other systems of event of interest.

Notice how our usage scenarios are not technology focused—the scenario is not “put a message on an application queue”; the scenario is “Notify other systems of event of interest.”

We use a matrix to map the preferred integration standards to various usage scenarios at various layers, i.e. data, business logic, and presentation. The goal of this matrix is to enable consistent architecture between system interfaces by establishing a common set of prescribed integration and usage patterns.

For example, if a usage scenario is to synchronize information in multiple systems, the preferred integration standard would be to use message bus or ESB as suggested by the matrix. Any deviations from preferred integration techniques would require an exception process to be followed.

The preferred technology for a particular usage scenario is indicated on the matrix with a + sign; a – sign indicates that this technology is acceptable but not preferred. Also, some technologies such as FTP and DB links are not included, meaning that they are not permitted and their usage would require an exception approval.

The matrix acts as a guide to architecture and solution delivery teams. Basing decisions on this matrix minimizes the decision variability across teams and provides a base for service orientation that ensures interoperability and true integration in a heterogeneous enterprise landscape.

Conclusion

The key to successful services-based integration is to focus on why applications need to exchange information. Our scenario-driven approach enables the application teams to use the matrix as a guide to identify approved integration technologies based on business requirements. This approach helps to control technical diversity as well as fosters consistent integration standards across the enterprise.

Figure 1

File 932


By Carrin Tunney, an enterprise architect at DTE Energy with more than 16 years in application development and distributed systems, TOGAF certified as well as a Sun Certified Enterprise Architect, and Srini Sastry, a technical architect at DTE Energy with more than 15 years of application development experience and also holds TOGAF certification and Sun Certified Enterprise Architect certification.

The Case for 
Application 
Modernization: A How-To Guide for Modernizing the Application Portfolio

Reprinted from Architecture & Governance Magazine (2010-09-15)

As I continued to serve as the Enterprise Architect at Aer Lingus (Dublin, Ireland), I am collecting references from EA journals, magazines and general articles which mirror my experience in shaping the IT Strategy, Enterprise Architecture and transforming People, Process and Technology as part of a 3-5 year programme of work.

Enjoy!


The goal of many businesses is simple: Have the business and IT departments work together to drive growth and innovation, at an affordable cost. Applications play a critical role, and modernizing them is the key to achieving long-term sustainable results. Luckily, by applying a comprehensive and strategic approach to application modernization, organizations can create an application environment that is efficient as well as cost effective for the business. The first step in the process is to assess all of the applications currently in the environment in order to effectively modernize the application portfolio.

The Mess We Get In

Cost: Organizations are spending upwards of 70–80 percent of their technology budget on maintenance and keeping the lights on.1 Whether these costs are infrastructure, application licenses, or support staff, they inhibit the flexibility as well as the ability to innovate and grow the business.

Complexity: Much of the code found in applications today is redundant, and it is common to see multiple instances of redundant applications. In many cases, the applications have been “patched” many times, becoming so bloated that changing them is a risk that threatens business continuity. In addition, the accumulation of applications through growth and acquisitions steadily increases the total number of applications in the portfolio and the level of complexity.

Security: As we move to a world that allows information to flow through many different devices and techniques, providing a security layer to match is difficult, especially if you have to change the applications to implement security. Tactical attempts to modernize applications have served as Band-aids that add to complexity, which only decreases overall security. New regulations on data security exacerbate the problems.

The combination of these issues makes it a struggle to keep applications aligned to the business. The good news is steps can be taken, which offer minimal risk and compelling IT and business benefits.

The Mess Can Be Cleaned Up, But . . .

What’s ironic is that most organizations acknowledge that a formal application modernization program would benefit their organization. In fact, recent HP-commissioned research with more than 200 global organizations conducted by Forrester Consulting showed that 78 percent of respondents believe their organization would greatly benefit from an application rationalization effort. But in contrast, only 56 percent plan to do so. That’s not the end of the contradictions.2

The study also addressed how applications should be modernized versus how they will be modernized. Eighteen percent of respondents believe that 21–40 percent of their applications should be migrated to new technology platforms, but only 14 percent will do so. Forty-seven percent of respondents believe that they should add new features but not modernize up to 20 percent of their applications. Yet 44 percent will modernize. Forty-three percent of respondents believe that their applications should be maintained but not modernized up to 20 percent of the time. Yet 34 percent will modernize.3

Further, respondents were asked to rate their perception of modernization risk before and after employing various techniques. The percentage of respondents that perceived high/very high risk ranged from a low of 39 percent to a high of 50 percent—noting worry about requirements for external help, but most worried about the risk of cost overruns, business process impact, staffing, scheduling overruns, and technical issues. The good news is many organizations overestimated their exposure to risk. After modernization, 25–30 percent of survey respondents actually experienced less risk than they feared.4

The research further found that the top barriers preventing strategic modernization programs were cost, risk, business buy-in, and subject matter expertise. The study also asked respondents, “What would accelerate your plans to modernize/replace your applications?” The number one accelerator by a substantial margin is the ability to bring all parties to the table to reach agreement, followed in turn by increased staff/funding, the ability to make a proper business case, and the ability to lessen risk.

Why is that significant? It ties together many of the points—risks are actually less of an issue than they seem, modernization plans must be relevant to organizational plans, and cost is an ever-present issue. Still, enterprise leaders who do not understand our reasoning for modernization will not support it.

Why Modernize Now

The most compelling reason for a formal application modernization approach is that other organizations, including competitors, are modernizing and gaining a competitive edge.
Secondly, there are many benefits that provide significant outcomes from modernization. Organizations see results that include:

  • Better alignment of business and technology
  • Increased agility
  • Improved security
  • Improved business continuity
  • More balanced technology maintenance spend to 50 percent or less of the IT budget
  • Improved understanding of how business processes are implemented in the applications

A Comprehensive Approach

Transforming the applications portfolio requires a comprehensive approach that treats each application appropriately based on its value to the business. The approach should also address the applications management processes to ensure that the applications portfolio will support the needs of the organization as they change and evolve over time.

Modernizing the applications portfolio itself can be achieved through assessment and a program road map designed to maximize benefits, accelerate ROI, minimize risk, all while identifying self-funding options. HP advocates an approach that consists of three phases: assess, modernize, and manage.

Assess: Assess the complete technology environment including applications, data, and infrastructure. Discover the assets available today through automation, inspections, and code samples. Analyze that information in comparison to business goals as well as priorities, and determine the strategic value of each application to the business. Look for redundancies in applications and identify opportunities for modernization. Finally, decide the best approach to reach the ideal future state and accompanying modernization road map.

Modernize: This phase includes realigning the applications portfolio so that it works for the business. It is designed to improve quality, security, information management, and performance. Multiple strategies are required to modernize an environment. Three basic approaches can help: keep, change, and retire. A modernization road map consisting of all three approaches will maximize benefits of the organization, while reducing costs.

  • Keep: The focus is on making as little direct impact on the application as possible. We would use this approach for performing actions such as extracting business logic from existing applications or simply extracting knowledge for the purposes of documenting the application. It could be used for rehosting applications to higher performance, lowering cost platforms, or wrapping existing applications as services to expose their functions to other systems. This approach is focused on cost savings.
  • Change: This is where we will make fundamental changes in the structures of the applications so they meet the new demands of the business. Examples include replacing legacy applications with packaged ERP or industry applications. It could also be modernizing from legacy languages to new .Net or J2EE based languages. Where is makes sense, it could include going to SOA or cloud computing models. This approach is focused on increasing agility.
  • Retire: For applications that perform duplicate business functions or no longer provide business value, this involves rationalizing the application’s role and capturing and archiving data from the retired application. This process is critical to minimize disruption to existing users and for future business reporting and regulatory needs.

Manage: The third phase is all about gaining the greatest degree of visibility and control to maintain the optimized environment. By understanding the current investments in the application portfolio, IT can calibrate those investments for the greatest impact.
For companies that choose to not manage the application portfolio internally, one viable option to consider is outsourcing to an experienced service provider. Whether an organization chooses to outsource the management of the entire portfolio or only select applications, this phase will need to include the following:

  • Clearly defined standardized service offerings based on ITIL framework
  • Right-size support based on the criticality of each application by choosing from service package tiers
  • Scalable, global delivery capabilities representing the highest industry standards backed by service level commitments
  • Fixed, predictive spending levels that can adjust as the applications portfolio changes
  • The ability to tune investments without renegotiation for quick and easy business alignment

How Modernization Pays Off

Each organization, including its process and benefits, is unique. By taking the steps to complete application modernization, businesses will be able to deliver impressive outcomes. Some of the specific benefits that applications modernization provides include:

  • Increased responsiveness to business priorities and changing demands by eliminating the complexity inherent to legacy systems and server sprawl.
  • Improved productivity for employees and reduced maintenance costs by streamlining complex, legacy systems to more modern, efficient technologies.
  • Faster speed to market with modernized infrastructure and applications.

It is clear that organizations need to take action and truly understand the infrastructure in order to garner the business results that they desire. Applications modernization is one way for organizations to take control of the data center and uncover important information that would not have been made available without the process. Without applications modernization, organizations cannot effectively leverage their portfolios, which slows down innovation and business growth for companies all over the globe.

Notes

  1. 2009 InformationWeek Analytics Survey of InformationWeek 500 executive.
  2. Source: A commissioned study conducted by Forrester Consulting on behalf of HP, May 2010.
  3. Ibid.
  4. Ibid.

File 934

by Larry Acklin, product marketing manager for HP Enterprise Services. He is responsible for global Application Modernization Services.


How to Initiate an Enterprise Architecture Effort in an Austere Environment

Reprinted from Architecture & Governance Magazine (2010-12-10)

As I continued to serve as the Enterprise Architect at Aer Lingus (Dublin, Ireland), I am collecting references from EA journals, magazines and general articles which mirror my experience in shaping the IT Strategy, Enterprise Architecture and transforming People, Process and Technology as part of a 3-5 year programme of work.

Enjoy!


Imagine yourself driving but not knowing where you are going, why you are going there, or how much it is going to cost. Taking a vacation in this manner may not result in the desired outcome. To ensure that your vacation is successful, many factors need to be considered first, such as desired goals and budget limits. Like planning a personal vacation, the federal government and Department of Defense (DoD) must strategically plan their business services, operations, and information systems to achieve their end goals.

Background of Enterprise Architecture Mandates

Enterprise architecture (EA) is currently mandated for federal government organizations. In 1996, the Clinger-Cohen Act reformed the federal government’s approach to information systems acquisition. The intention of this act was to assist in controlling risks, minimizing costs, and improving the performance of federal enterprises. To build upon the Clinger-Cohen Act, the DoD adopted a similar approach in 2003 known as the Joint Capabilities Integration and Development System (JCIDS). Both approaches require the creation of disciplined architecture products to objectively provide key and accurate information for the architecture decision-making and assessment process.

The true intention of these mandates is often lost because federal organizations will sometimes focus on simple mandate compliance rather than the actual goals of the Clinger-Cohen Act and the JCIDS process. For example, EA products are often created to pass reviews, but there is no intent to use them as part of the decision-making process. The EA products should instead be tightly integrated into the strategic planning and acquisition planning activities to be effective and truly meet the objectives of the federal mandates.

Common Problem: 
Translating the Mandates into Practical Use

Because EA products are mandated and may utilize an unfamiliar approach, many organizations become overwhelmed when deciding how to begin their efforts. Enterprises within federal government and DoD agencies are often very complex and usually quite large, as are their private sector counterparts. Tackling a structured, objective enterprise assessment with such complexity and magnitude can be quite intimidating. To add to the strategic planning and EA challenges, organizations often have limited funding to perform the full scope of EA analysis. With the intimidating size of the enterprise architectures and the challenge of limited resources creating an austere environment, organizations often find themselves quite overwhelmed and struggle with how to begin EA efforts.

Key Startup Activities 
for a Successful EA Effort

To overcome an architecture’s complexity and potentially limited resources, MorganFranklin recommends several key startup activities that have proven to be essential to effective EA efforts. In the vacation analogy, in order to not drive aimlessly, the destination, arrival method, and many other factors need to be determined before traveling. In our past experience, there have been organizations that spend large amounts of money to ramp up architecture teams without strongly defining the desired goals and visions of their enterprises. To ensure architecture assessment consistency and overall success, we recommend three fundamental steps to successfully begin an EA endeavor:

  1. Establish and prioritize strategic business goals. What does the organization want to be as it evolves? What business services is it providing to the rest of the community and who are its users? What discriminators or strengths does it want to build or maintain? What overarching policies is it going to embrace?
  2. Create high-level capability and concept views for common vision. Once the goals have been established, how will they be achieved? What is the common vision for the concept of operations? What options are there for new transformational concepts of operations and which concepts currently work successfully?
  3. Capture a catalog of “as-is” processes, systems, applications, infrastructure, services, and standards. Too often the current systems, applications, and infrastructure are completely ignored. It is essential to have a validated, up-to-date catalog of all resources that currently support the organization’s mission set.

How to Realize Immediate Value-Add While Overarching Vision Is Being Set

In an ideal, theoretical world, an organization can wait until the entire “to-be” enterprise architecture has been described and vetted before analyzing and describing a strategic plan forward. However, if organizations were to wait, they would lose valuable architecture assessments that could be completed in the meantime. Organizations can quickly start realizing value from architecture engineering efforts if they are focused on narrowly defined and near-term solutions. While the key startup activities should continue to be carried out by a set of undistracted team members, experience has shown that organizations can have small-scale architecture development successes. The practical and tangible results can also provide the necessary traction and advocacy often needed to continue support and funding for overarching EA efforts.

What are some of these focused architecture analyses? Our experience has shown that an organization can focus on the top-priority systems being acquired and transitioned within the next year. These architecture efforts can make certain that the systems are defined well enough to ensure integration into the greater enterprise. Another focused effort can involve saving operations and maintenance (O&M) costs by analyzing legacy systems and determining what type of redundancies exist. Additionally, a federal organization can explore what potential solutions could be used by obtaining the lessons learned from other organizations’ successes and failures. While the vision is being set, the organization can begin to adopt successful and applicable common practices and technologies.

What If a Formal EA Tool Is Unavailable?

Would you delay your vacation if you owned a Corolla but always dreamed of traveling in a Cadillac? If you did not have the money to buy a new vehicle, you would most likely still take your vacation. For architecture product creation, organizations do not need to wait until they bring in a robust architecture tool before capturing the necessary information. Architecture analysis can occur without a formal architecture tool. The effort can begin by simply using standard desktop commercial off-the-shelf (COTS) products, such as Microsoft Word, Excel, Visio, and PowerPoint. Basic architectural analysis and data collection are essential in strategic and acquisition planning and can successfully occur without a robust tool.

It is important to keep in mind that EA analysis is an ongoing, long-term endeavor. While the architectural assessments should continue despite tool constraints, bringing in a more formal EA tool should be considered for the long-term solution. EA tools are built to support information sharing and expedite architecture product creation. In the long run, a more robust tool will save time and improve collaboration among a large group of architects.

Conclusion

To achieve the vacation of your dreams, capturing the desired goals, objectives, budget, and timeline is essential to successfully arriving at your ultimate destination. EA should be treated no differently. While the ultimate destination is being determined, one can take mini-trips or complete short-term architecture projects that will realize immediate value and potentially provide insight into the end goal. Lastly, the type of transportation, or architecture tool, should not be the primary focus or a limiting factor at the start of an EA effort. Following these guidelines will help any organization get closer to achieving its vision.


Ruth Burgess serves as program manager for IT strategic planning projects at MorganFranklin. She possesses more than fifteen years of experience specializing in systems integration and engineering, with specific experience leading teams developing comprehensive and practical architecture products for several critical government enterprises.

John Forte recently served as technical director at MorganFranklin, specializing in enterprise architecture, information systems modernization, and technology insertion. He currently serves as program manager for national command programs at the Johns Hopkins University Applied Physics Laboratory.

Sharon Stran and Daniel Hughes also contributed to this article.