An SOA Vendor Evaluation Methodology
Join the DZone community and get the full member experience.Join For Free
This article describes a methodology for evaluating SOA vendors that, when applied to analytical sources such as Gartner Group or Forrester reports, can help shortlist appropriate vendors in an objective way. This objectivity is ensured by a ranking and scoring system based on a set of evaluation criteria within the corresponding evaluation models. Toward the end of this article a final short list is revealed based on the correlation between preceding short lists obtained from each model.
The following vendor evaluation methodology is intended to be only an initial step of an actual evaluation process that you may want to carry out in order to filter vendors. The approach is based on the analysis of results from studies carried out by the Gartner Group and Forrester Research identifying the "Best-of-Breed" vendors available in the SOA market.
The procedure that we will follow in the evaluation process consists of the following steps:
1. selection of appropriate sources from Gartner and Forrester
2. ranking vendors based on the results in the Gartner and Forrester sources
3. short-listing of vendors
4. correlation of the short lists
5. review of the evaluation methodology results
Gartner Group Reports
Three major reports published by the Gartner Group in 2007 are being included in our assessment:
Magic Quadrant for Application Infrastructure for Back-End Application Integration Projects, 2Q07
This report puts the focus on integration with EIS and legacy systems relevant in the current IT landscape. The selection of vendors in relation to appropriate integration frameworks is considered critical. In this context the predominant types of software employed for back-end application integration usage are integration suites and ESBs.
Magic Quadrant for Application Infrastructure for Composite-Application Projects, 2Q07
Composite applications are perceived to be the next technological step on many enterprise SOA roadmaps. It essentially involves building a new style applications based on compositions of services that provide full end-to-end business process automation and reuse of exiting business processes and services. Gartner has identified a composite applications infrastructure market as one that offers integrated suites of application infrastructure components with end-to-end functionalities of user interface management, composition and orchestration, integration and adapters, interoperability and messaging protocols, container technology, etc.
Magic Quadrant for Application Infrastructure for New Service-Oriented Business Application Projects, 2Q07
The topic of this report represents a further step in a typical SOA implementation by addressing the actual construction of service-oriented applications from scratch. The Gartner Group characterizes this market segment by focusing on service orchestration with less of an emphasis on integration. The major product categories expected to constitute the market are application servers, horizontal portals, design, development, modelling and governance tools, SOA infrastructure tools and metadata repositories, business process management (BPM) and orchestration tools, various user-interaction tools and high-end performance enhancement tools.
The Gartner Group methodology is based on “Magic Quadrants” that position and categorize vendors with respect to their ability to execute and their completeness of vision.
Each of the three major categories represented by the preceding reports is assumed to have an equal weight in the overall evaluation and the same critical impact for the future state of integration and application of services.
Conceptually, this reflects the connections between the past, present and future that corresponds to the evolution path from “Back-End Application Integration projects” to “Composite-Application Projects” and subsequently to “New Service-Oriented Business Application Projects”. The importance of these categories is addressed by the Enterprise SOA Maturity Reference Model [REF-1], which also describes the business requirements, key challenges, exit criteria and success factors pertinent to each category.
How We Use Gartner Results
The ranking methodology employed by this study is based on the following rules:
|•||Only vendors scoring and represented in each of Gartner’s three categories are considered.|
|•||Only the first six vendors identified in the “Leaders”, “Challengers” or “Niche Players” magical quadrants are considered.|
|•||Vendors are ranked by their position within Gartner’s magic quadrants in both the “Ability to Execute” and “Completeness of Vision” dimensions.|
|•||A ranking score is assigned to each vendor in each dimension as determined by the rank position. The score from the “Challengers” magical quadrant is increased by one, and the score from the “Niche Players” magical quadrant is increased by two.|
|•||The total score is assigned for each of the three categories calculated as the sum of the scores in the “Ability to Execute” and “Completeness of Vision” dimensions.|
|•||The final ranking is produced by gradation of grand totals of the totals of each vendor across each investigated category.|
|•||Vendors not scoring in all categories are dropped from the evaluation.|
|•||Vendors with lighter scores are considered superior.|
Table1 displays the vendor ranking carried out in accordance with the evaluation rules defined previously and based on the results provided in the Gartner reports. Table 2 summarizes the results from Table 1 and shows the total scores for each vendor. (Note that vendor scores are increased by 1 or 2 when the vendors are from the “Challengers” and “Niche Players” Magic Quadrants respectively.)
**Vendor is positioned in the “Niche Players” Magic Quadrant
The “Combined Ranking” column in Table 1 is produced by adding the ranks of each vendor in respect to a given category. The lower the score, the higher is the rank in the classification ladder. For example, Microsoft is ranked first with respect to “Ability to Execute” criteria and sixth with respect to “Completeness of Vision” criteria based on Gartner’s “Back-End Application Integration Projects” category. The overall score for Microsoft is 7, which positions them second in the “Combined” ranking column.
Forrester Research Results
Forrester Research Inc. has published a report that analyses the positioning of integration-centric business process management suites (IC-BPMS), a product concept that addresses the link between SOA, legacy integration and BPM. This linkage is reflected in the ranking of the products evaluated. The Forrester Wave Model [REF-5] provides three dimensions used as evaluation criteria:
|•||Current Offering – The vendors’ capabilities in the areas of connection, BPM, and SOA.|
|•||Strategy – The vendors’ vision and product roadmap, as well as associated solution costs and strategic alliances.|
|•||Market Presence – The vendors’ installed customer base, new customers in the past quarter and year, and the level of global sales and implementation support.|
How We Use Forrester Results
This study incorporates the results from Forrester presented in the report [REF-5] and applies the evaluation methodology based on the assumption that all the dimensions (criteria) of the model have equal weightings.
The ranking methodology employed by this study is based on the following rules:
|•||Vendors are ranked by their position in the Forrester Wave Model evaluation dimensions: Current Offering, Strategy and Market Presence.|
|•||Only the top six ranked vendors are considered.|
|•||The total score is assigned to each vendor as a sum of the three categories rank values.|
|•||The final ranking is produced by gradation of grand totals of each vendor across each investigated category.|
Table 3 shows the vendors ranking carried out in accordance with the evaluation rules defined previously.
Table 4 summarizes the results from Table 3 and displays the scores of each vendor subject to evaluation.
Correlation of the Short Lists from Gartner and Forrester
Now it’s finally time to see where all of this research was heading. Table 5 gives a summary of the results of the evaluations carried out in the previous two sections. Only vendors which are represented on both short lists are considered in order to ensure a high level of synergy between the Gartner and Forrester Evaluation Models. Further ranking of the short-listed vendors is achieved by ordering the sums of the scores listed in Table 6.
The objectivity of this assessment is ensured by a vendor ranking system that determines scores based on a set of common evaluation criteria from two industry-recognized sources (Gartner or Forrester). The final short list is the result of a correlation between the short lists produced by these models and reflects.
Provided in Table 7 is a recap of the criteria from both the Gartner and Forrester assessment models.
[REF-1] SOA Practitioners’ Guide Part 2: SOA Reference Architecture, 2006
[REF-2] Magic Quadrant for Application Infrastructure for Back-End Application Integration Projects, 2Q07
[REF-3] Magic Quadrant for Application Infrastructure for Composite-Application Projects, 2Q07
[REF-4] Magic Quadrant for Application Infrastructure for New Service-Oriented Business Application Projects, 2Q07
[REF-5] The Forrester Wave: Integration-Centric Business Process Management Suites, Q4 2006
Opinions expressed by DZone contributors are their own.