EMA APM 2012 RadarReport Summary

EMA Radar for Application Performance Management (APM) for Cloud Services: Q1 2012 Report Summary
18 Cloud-Ready APM Solutions Available TODAY

By Julie Craig, Research Director Enterprise Management Associates (EMA) January 2012
Table of Contents
Executive Summary .............................................................................................................................................1 Introduction and Methodology .........................................................................................................................2 Vendors Covered by this Report .......................................................................................................................4 APM: Evolving with the Cloud .........................................................................................................................4 The Problem Set: Cloud-specific APM Challenges .......................................................................................5 Management versus Monitoring ........................................................................................................................6 Criteria ....................................................................................................................................................................8 Minimum Criteria ..........................................................................................................................................8 Ideal Criteria ....................................................................................................................................................8 Scoring ................................................................................................................................................................. 10 Cost Efficiency ................................................................................................................................................. 11 Cost Advantage ........................................................................................................................................... 12 Product Strength .............................................................................................................................................. 12 Architecture and Integration..................................................................................................................... 12 Functionality................................................................................................................................................. 14 Vendor Strength ............................................................................................................................................... 15 EMA Radar Maps for Application Performance Management ............................................................... 15 Multi Component/Suite Solutions................................................................................................................. 16 Vendors and Solutions Profiled................................................................................................................ 16 General Findings ......................................................................................................................................... 17 Value Leaders ............................................................................................................................................... 18 Strong Value ................................................................................................................................................. 19 Point Solutions ................................................................................................................................................... 20 Vendors and Solutions Profiled................................................................................................................ 21 General Findings ......................................................................................................................................... 21 Value Leaders ............................................................................................................................................... 22 Strong Value ................................................................................................................................................. 22 Exceptional Characteristics: Awards .............................................................................................................. 23 AppDynamics: Best Infrastructure as a Service (IaaS) Automation ................................................ 23 Aternity: Most Comprehensive Endpoint Diagnostics ....................................................................... 24 eG Innovations: Best Private Cloud Coverage ...................................................................................... 24 HP: Most Comprehensive and Real-Time Service Model .................................................................. 25 IBM: Best Cloud Vision and Design ....................................................................................................... 25 OPNET: Best Overall Automation ......................................................................................................... 26
I
2012 Enterprise Management Associates, Inc. All Rights Reserved.
Executive Summary
Over the past ten to fifteen years, the complexities involved in Application Performance Management (APM) have multiplied exponentially. As applications have moved off monolithic hosting platforms and on to increasingly tiered and integrated systems, monitoring and managing end-to-end execution have become increasingly problematic. Widespread adoption of Cloud computing introduces additional complexity and signals a paradigm shift in enterprise computing that is impacting vendors and customers alike. Cloud is changing the structure of IT organizations and making inroads on traditional on-premise application hosting models. This, in turn, shapes design decisions and product investments by APM vendors. Consider the following facts:1
Cloud is changing the structure of IT organizations and making inroads on traditional on-premise application hosting models. This, in turn, shapes design decisions and product investments by APM vendors.
The majority of medium-sized to large companies have already embraced private Cloud as a viable delivery option for business-critical workloads. Infrastructure as a Service (IaaS), a natural extension of private Cloud, is gaining momentum, with 66% of companies either already using IaaS or planning to do so during the next 12 months. Software as a Service (SaaS) has become a viable alternative as well. While 44% of companies are already utilizing at least one SaaS service, another 33% plan to do so over the next 12 months. Companies are deploying Cloud services in some very sophisticated ways. Almost 50% are running tiered transactions/services that span both Cloud and on-premise. Thirty-five percent have integrated or are in the process of integrating multiple SaaS applications. Cloud readiness is becoming a factor in developing Requests for Proposals (RFPs). Today, over 40% of companies surveyed are re-assessing the Cloud-readiness of their management tools investments for SaaS (35%), Platform as a Service (PaaS- 29%), and IaaS (29%) Clearly, Cloud adoption is challenging traditional application management models and the APM solutions that support them. Widespread adoption of private Cloud technology drives new requirements for visibility to virtual environments. Public Cloud presents unique challenges because, while IT still has responsibility for service delivery, it has virtually no control over the hosting platform. And while IaaS offers almost unlimited capacity on demand, most IT organizations lack management solutions capable of end-to-end management and automated remediation across on-premise and off-premise Cloud environments. Finally, while the purpose of enterprise management solutions is to deliver visibility and control, the bottom line impact of Cloud is to obscure visibility and limit the customers control. This EMA Radar explores these challenges in more depth. It focuses on the issues IT organizations are encountering in terms of managing Cloud environmentspublic, private, and hybrid and provides in-depth analysis of solutions that can help solve them.
1
EMA Research on APM for Cloud Services
1
EMA sees this as an early-stage market with vendors adding Cloud-focused products and features at a furious pace. In developing this Radar Report, EMA engaged eighteen top providers of APM solutions in a detailed analysis of the scope and capabilities of their offerings. These solutions represent a rich cross-section of the IT management tools landscape, ranging from small to very large, from pure software to appliance- and SaaS- based, and from point products to extensive multi-component/ multi-function suites. EMAs intent in producing this report is to give businesses seeking Cloud-ready APM solutions a starting point from which to develop a short list of potential solutions, based on their Because nearly thirty own specific requirements. vendors were offered the Because nearly thirty vendors were offered the opportunity to participate in this rigorous study, EMA sees the eighteen vendors who did participate as the most Cloud-ready in the industry. Although this is still an early market, these vendors all have distinctive capabilities for managing private, public, or hybrid Cloud environments which can deliver significant value to companies of virtually any size or in any industry.
opportunity to participate in this rigorous study, EMA sees the eighteen vendors who did participate as the most Cloudready in the industry.
Introduction and Methodology
Each vendor completed a comprehensive questionnaire consisting of nearly 100 questions and over 500 data points. The survey questions covered the five key functions common to all EMA Radar Reports, which include Architecture, Functionality, Deployment and Administration, Vendor Strength, and Cost Advantage. EMA also conducted lengthy interviews and demos with each vendor to clarify product capabilities and vendor direction. Finally, EMA conducted more than 40 interviews with customers using the products. While all vendors provided some level of access to their customers (from one to three customers), availability of reference customers was viewed favorably in assessing and validating vendor portfolios. The degree to which customers were readily provided and available for dialog was one of the many indicators used for validating APM solutions. Finally, EMA leveraged ongoing industry dialogs and extensive existing knowledge of the APM space to evaluate, consider, and validate each vendors strengths and limitations. These evaluations focus on providing balanced, consistent insights across all vendors and solutions. EMA has produced a companion piece which is targeted at presenting and explaining Radar Reports in general, entitled How to Use the EMA Radar Report.2 The goal is to use a combined approach to quantitatively and qualitatively evaluate providers of solutions in a particular IT management functional area and presenting their relative differences in a clear, graphical format.
2
Available at: http://www.enterprisemanagement.com/research/asset.php/1715/How-To-Use-The-EMA-Radar-Report, EMA, April 2010.
2
Figure 1: The EMA Radar is optimized to show how vendor solutions cluster in terms of two primary axes: Vendor Strength (architecture, integration, functionality) and Cost Efficiency (ease of administration, deployment, support & services, costs advantage)
Quoting from How to Use the EMA Radar Report, No analysis of this type can tell you which vendor is best for you. The data collected for an EMA Radar Report can certainly be used to make that determination, but it must be applied to the specifics of your current environment, level of maturity, and goals and priorities. Since the authors of any given Radar Report do not have your unique specifics, the Radar Report can only be a starting place and a guideline. It can inform you of the market and short-cut your process to developing a short list.
Functionality Functionality
Architecture & Integration
Deployment & Administration
Architecture & Integration
Vendor Strength
Cost Advantage
Vendor Strength
Cost Advantage
Strong Product Higher Price
Specialized Product Low Cost
Figure 2: Radars for each vendor solution are included in the full report and show a fiveaxis contrast between the average profile and the vendor in question.
3
Vendors Covered by this Report
APM solution providers covered in the report include AppDynamics, AppFirst, Aternity, CA, Compuware, Correlsense, eG Innovations, HP, IBM, INETCO, Nastel, Netuitive, New Relic, OPNET, OpTier, Quest, SolarWinds and Splunk. These solutions are clearly different in multiple respects, and represent a variety of capabilities, technology vantage points, and form factors. While this differentiation makes a strict apples to apples comparison virtually impossible, it is possible to compare and highlight features, costs, and overall value proposition relevant to the APM discipline. In the interest of fairness and for reader clarity, final scoring has been divided into two broad product groups based on general architecture. Both groups answered the same questions and were scored similarly. However, the two groups are charted and averaged separately for this report, depending on whether the solution is built around a single component or around a suite of pre-integrated components. The net impact is to highlight the strengths of both groups while comparing breadth of functionality and value proposition against a more comparable group of solutions. Solution groupings are as follows: Point Solutions: AppDynamics, AppFirst, Aternity, INETCO, Netuitive, New Relic, and Splunk. Multi Component/Suite solutions: CA, Compuware, Correlsense, eG Innovations, HP, IBM, Nastel, OPNET, OpTier, Quest, and SolarWinds. Point Solutions: In general, the point solutions profiled here address a targeted functional APM capability or set of capabilities. Multi-Component/Suite Solutions: These solutions vary considerably in terms of capabilities, with CA, Compuware, HP, IBM, Nastel, OPNET, OpTier, and Quest each delivering distinctive takes on integrated APM toolsets. In contrast, Correlsense and eG Innovations both deliver depth of visibility addressing specific APM problems via a multi-component form factor. Finally, SolarWinds delivers a cost-effective platform for mid-market APM. EMA awards a limited number of citations to vendors with particularly noteworthy capabilities in one or more functional areas. While the vendors participating in this particular study were all strong and the choice was difficult, only six solutions received awards. Four were chosen from the MultiComponent group, and two from the Point Solutions group, commensurate with the number of participants in each.
APM: Evolving with the Cloud
From multiple perspectives, APM solutions for Cloud services are still evolving. The ultimate answers to many of the questions related to managing Cloud applications are still being addressed. For example, interviews conducted as part of this study indicate that many public Cloud providers do not yet offer monitoring agents or APIs, a factor that hampers the capabilities management vendors can build into APM solutions. Similarly, virtualization vendors have exposed varying levels of insight into private Cloud execution with some stronger than others. VMware, for example, is a poster child for providing exceptional manageability capabilities with vCenter and related APIs.
4
While neither public Cloud nor APM vendors have all the pieces in place at present, virtually every vendor is aware of the necessity for developing such capabilities. In fact, many of the vendors included in this study announced new products and upgrades to existing solutions during the short time span required to produce this report. There have also been significant developments with non-participating vendors during this time frame. While BMC declined to participate because product development was in-process, it has since announced enhancements to its APM solutions. In short, although this market is still evolving, it is extremely dynamic and constantly improving. Meanwhile, the management capabilities which are available are already quite impressive.
Although this market is still evolving, it is extremely dynamic and constantly improving. Meanwhile, the management capabilities which are available are already quite impressive.
The Problem Set: Cloud-specific APM Challenges
Todays applications and transactions can be delivered in a wide variety of hosting environments They can be hosted on-premise via traditional tiered hosting or virtualized private Cloud. They can be componentized such that elements of the execution software are hosted on-premise and in the public Cloud. They can access partner and provider Clouds, be delivered via SaaS, and can even encompass SaaS-to SaaS integrations. While this diversity offers a wide variety of deployment options, it also introduces a bewildering array of budgeting, management, and tooling considerations. Some of the most challenging include: New requirements driving APM purchasing: Many companies are finding that Cloud-related APM requirements are beyond the capabilities of incumbent solutions, and are re-examining budgets (and tools strategy) accordingly. In other words, existing APM investments arent Cloud ready, and cannot support this diversity of hosting environments. From this perspective, Cloud computing is driving new requirements which are, in turn, impacting budget line items and purchasing decisions. End-to-End visibility becomes a must have: Companies that are less operationally mature are still transitioning from monitoring infrastructure to monitoring/managing applications. They lack top down monitoring solutions that deliver insight into end-to-end performance. This can put them at a disadvantage in terms of competitive differentiation. Exponential increase in metrics: Companies leveraging Cloud tell EMA that it introduces new requirements for metrics and big data processing. Outlier measurements such as TCP thresholds, rates, and queue limits have become important elements of the APM equation, and are pushing the processing and analytical limits of existing solutions. As an example, one customer interviewed as part of this research reports that monitoring systems are processing over one billion metrics daily. Real time management becomes the new normal: As application systems become increasingly dynamic, real time analytics capabilities become a must have. This requirement is becoming increasingly pressing as more companies begin to leverage dynamic capabilities such as VMware vMotions and Cloud bursting to IaaS. It permeates every aspect of APM from discovery/ dependency mapping to topology visualization and End User Experience Management.
5
Troubleshooting is more complex: This is probably the biggest challenge, as both public and private Cloud deployments increase the number and complexity of moving parts comprising the application ecosystem. To compound the problem, most public Cloud providers are still in the early stages of developing management-level instrumentation that enables them to share key performance information with APM solutions. At the same time, ITs responsibility is still to the user, and it is up to IT to diagnose performance problems regardless of where (and how) the application is hosted. For example, while poor SaaS performance can definitely be on the public Cloud provider side, it can also be impacted by a multiplicity of additional factors such as: Lack of bandwidth on the Wide Area or Corporate network Contention on the users desktop Slow system calls to internal or external integrations Failure of optimization or load-balancing components Contention with other applications or users Database issues Location-based issues Etc. Without adequate tooling, knowing who to call is a best guess exercise.
Without adequate tooling, knowing who to call is a best guess exercise.
Management versus Monitoring
While the focus of this research is on Application Performance Management, the term means different things to different constituencies. This section clarifies EMAs definitions, which in turn clarifies how vendors participating in this study were assessed. These definitions should also help enable IT organizations to hone in on the products that best meet specific internal monitoring and management requirements. Application Management: The role of Application Management is to deliver visibility and control for transactions, applications, and services. While Application Management covers a very broad range of capabilities required to monitor AND manage applications (see Figure 3), Application Performance Monitoring and Application Performance Management are subsets with varying levels of visibility: Application Performance Monitoring: Assesses performance and availability of a transaction, application or service Application Performance Management (APM): Encompasses the functions of performance monitoring but also supports troubleshooting, problem determination, and root cause analysis. Ideally, and this is an emerging capability, APM also includes the ability to recognize (and potentially resolve) performance and availability problems BEFORE users are impacted. In addition, APM supports and links to other IT functions such as Service Level Management and Configuration Management.
6
Such distinctions are particularly important when it comes to Cloud. While multiple solutions may claim to support applications running in public IaaS, SaaS, or PaaS, their levels of granularity can vary radically. For example, while running synthetic transactions every five minutes monitors performance and availability (intermittently), it does not provide the granularity necessary to troubleshoot and diagnose performance problems, particularly in complex multi-tier environments. Vendors have approached these challenges from multiple perspectives. So called top down solutions such as synthetic transactions deliver a point-in-time perspective which is broad versus deep. Bottom up solutions provide deep visibility but lack the cross-technology, end-to-end perspective that is necessary to manage applications versus infrastructure. Clearly, a meld of the two is the holy grail of APM, and APM software vendors have made significant strides in delivering on this vision during the last one to two years.
Figure 3: EMA End-to-End Application Management Semantic Model
Ultimately, it is essential for Cloud and enterprise management vendors alike to become Cloud-ready. In support of this idea, EMA has added a Cloud Analytics layer to its End-to-End Application Management Semantic Model. This model depicts the management capabilities required to gather and analyze the information necessary to track and manage applications from a 360-degree, end-to-end perspective. It includes metrics gathering and supporting analytics functionality. Although solutions monitoring each of the blue layers in the diagram can be deployed separately (and often are), the collective value of the stack far exceeds that of each separate capability. Solutions capable of gathering and analyzing metrics from multiple layers yield a uniquely comprehensive perspective of the application ecosystem. This is critical to managing, versus simply monitoring, application environments in the Cloud-extended enterprise which is a reason why solutions with breadth of coverage across multiple layers of the model achieved higher scores for this Radar. 7
Criteria
To clarify vendor placement in this report, its important to understand the models used to include and evaluate this very heterogeneous group of vendors. This section specifies both the minimum criteria for inclusion and the ideal combination of capabilities used to evaluate all solutions. These criteria were used as the basis for developing and scoring the survey-based portion of this assessment.
Minimum Criteria
1. Product fits within at least one of the layers/categories identified in the Semantic Model or is a multi-component solution covering multiple categories 2. Product also supports monitoring/management of at least one of: a) Public Cloud b) Private Cloud c) Hybrid Cloud 3. All elements of the solution were generally available for purchase (GA) at the time the survey was conducted (October-November, 2011) 4. At least one customer available for interview.
Ideal Criteria
1. Meets minimal criteria listed above 2. Automated versus manual operation: In general, higher levels of automation and minimal reliance on manual tasks received better scores. Examples include optimal (versus excessive) dependence on scripts or scripting, auto-generated and maintained baselines /thresholds, single agent versus multiple agents, self-learning capabilities, etc. 3. Topology models (or service models): Products supporting auto-generated topology models via application discovery, execution analytics, and dependency mapping were favored over those requiring manual topology modeling. 4. End to End transaction/application visibility spanning on-premise and public Cloud 5. CMDB/CMS support: Although some of the vendors participating in this research contested this requirement, EMA research has found that Cloud computing makes CMDB/CMS systems even more important. Ultimately, keeping track of where application components are hosted requires federated or API-level integration of APM solutions with CMDB/CMS systems. Ensuring that CMS systems are accurate requires automated dependency mapping. For these reasons, products with native integration or API support with CMDB/CMS systems (and with dependency mapping functionality) were favored over those that did not. 6. Open Integration: Products with native support for integration with third party solutions received higher scores than those that do not. Examples include trouble ticket generation, metrics assimilation and correlation, and visibility to a variety of heterogeneous platforms and solutions. 7. Visibility into public Cloud platforms: The ideal APM solution demonstrates broad and deep coverage for AND formalized management partnerships with public Cloud IaaS, SaaS, and/ or PaaS vendors. Vendors with actual versus black box visibility into public Cloud platforms, typically via API or similar integration technology, received higher scores.
8
8. Visibility into virtualization platforms: Support for private Cloud and IaaS also requires broad and deep coverage of virtualization platforms. Again, this depth of visibility requires integration with management capabilities delivered by the virtualization vendor. 9. Above-average investments in Research and Development 10. Support for hybrid Cloud, including visibility to integration points 11. Customer Support: Multiple customer feedback options, support options, and interaction methodologies (user groups, online forums, conferences, training, etc.) enabling customers to easily communicate with one another and with the vendor. 12. Fast time to value: While time to value is generally easier to achieve with point solutions versus suite solutions, vendors of all sizes are investing in making this process as easy as possible. Distributing solutions as pre-loaded hard or soft appliances, delivering analytics functions as SaaS, and making application templates and fingerprints available are examples of vendor efforts that improve time to value. 13. Price commensurate with capabilities: While price is certainly a concern, higher value solutions require higher levels of vendor investment and therefore cost more. EMA focuses on Cost Efficiency versus cost alone, a measurement that takes overall value, product and deployment costs, training and support costs, and ongoing administration and consulting costs into account. 14. Proven scalability 15. Customer interviews: While at least one customer interview was a requirement for each vendor, additional customers positively impacted overall scoring.
9
Scoring
For all Radar Reports, EMA evaluates solutions based on five key areas represented by the five sides of a hexagram. They include Deployment and Administration, Cost Advantage, Architecture and Integration, Functionality, and Vendor Strength. The last category, perhaps the only one thats not self-explanatory, is focused on market and industry presence, vision, and the financial stability of the vendor. Within each of these five evaluation areas, EMA creates a superset of capabilities spanning the known solutions in the marketplace. For each superset, specific Radar Reports add questions about new and emerging areas (in this case virtualization and Cloud), and balance the results with standard comparators used across all EMA Radars. The evaluation model used for this Report is presented as Figure 4. The following section details each evaluation area, along with its scope and rating priorities.
Figure 4: Assessment Model for APM/Cloud Radar
10
Cost Efficiency
This is the first set of measures within the EMA Radar Report framework and one of the two major axes of the Radar distribution diagram. It consists of two major sub-sections: Deployment & Administration and Cost Advantage. Each of these, along with the breakout capabilities assessed for each, is described below.

Deploying and administering is the first side of the hexagram. These scores include measures that indicate how easy or difficult it is to deploy a given APM solution into the production environment, as well as initial time to value. The sub-areas assessed include: 1. Ease of Deployment: Resources required to initially deploy the product. High marks go to fast time to value, vendor-provided quick start aids (such as templates), minimal consulting requirements, and support for a variety of databases and database vendors. Simplified packaging such as a SaaS form factor also wins points. a) Time to deploy: Vendors were asked to gauge their answers for a hypothetical customer with two multi-tiered enterprise applications. Within this context, specific questions addressed the average time required to deploy, begin data collection, and deliver functional, actionable performance data. Additional questions addressed the professional services requirements and overall costs associated with deployment. b) Packaging requirements and options: Operating environment requirements and available form factors. Scoring was based on whether the solution includes hardware and software as part of the quoted cost, is turnkey or requires add-ons, is available as SaaS (since SaaS-based products are obviously easier to deploy than on-premise solutions). c) Staff training: The vendors training offerings. Scoring favored breadth of options (i.e., online, partner led, on-premise, etc.) and lower overall cost. It also factored in the average number of training hours necessary for basic proficiency and administrator level efficiency, since training costs for multiple personnel can add significant costs. d) Disruption minimization: The impact to normal IT operations that occurs during product deployment and/or upgrades. 2. Support and Services: Assesses the vendors customer support, professional services, and product maintenance options. High marks go to vendors with a wide variety of options to support customers of different sizes and with different budgets. a) Customer Support: Vendors are scored on the breadth of support options (online, business hours, 24 X 7, etc.). Also takes into account support levels and vendor-provided vehicles for customer product input (product suggestions and status for public review), as well as availability of user interaction options (online, conferences, etc.). b) Professional Services: Professional services options, whether vendor or third-party delivered. Higher scores go to vendors that include professional services as part of the quoted solution cost (to eliminate unpleasant surprises) and to solutions that require minimal professional services to deploy. c) Version Maintenance: Frequency of full releases and software upgrades.
11
3. Ease of Administration: Assesses the product and personnel costs necessary to configure, administer, and maintain the solution on an ongoing basis. a) Ongoing administration: The manpower and time required to manage the solution. b) Update Process: The operational impact and time requirements for upgrades, as well as ease-of-use features that support it. c) Testing/Migration: License fee costs for licenses utilized for product upgrade pre-testing.
Cost Advantage
Cost Advantage is the second side of the hexagram, and assesses overall cost factors related to Pricing, Licensing and ongoing maintenance. 1. Price: Average license cost for a typical deployment, including whether additional for pay modules are necessary, and whether consulting and/or training hours are included in the quoted cost. Also assesses the time required for the average customer to recoup full ROI. 2. Licensing Models: Breadth of licensing models and form factors. Higher scores go to solutions available via multiple licensing models and form factors, which makes solutions more readily available to a wide variety of company sizes and budgets. 3. Maintenance Costs: Average maintenance fees for highest level of service. Industry average is approximately 1922%; higher scores went to solutions with lower maintenance fees.
Product Strength
Product strength is the second major axis of the Radar distribution diagram, and is comprised of two sub-areas, Architecture/Integration and overall Functionality.
Architecture and Integration
The first of the two major product strength categories is Architecture & Integration. It is the third side of the hexagram, and assesses the enabling technology underlying the APM functionality. 1. Architecture: Assesses design elements contributing to overall functionality and usability. a) Design: Design characteristics such as deployment form factors (SaaS, SaaS with on-premise collector, VM, etc.) and metrics collected. Examples include: i. Data collection methodologies (Synthetic transactions, EUE, RUM, tracking/stitching, instrumentation points, platform coverage)
ii. Supplemental data sources monitored (Protocols, network flow, third-party agents, CMDB/CMS, discovery sources, etc.) b) Scalability: Scalability is a key consideration, particularly for rapidly growing companies and large enterprises. Specific questions measured the products ability to scale across large, geographically distributed environments and the largest deployment to date.
12
c) Breadth of Environments Supported i. Support for public, private, and hybrid Cloud environments ii. Support for a variety of tiered transaction environments, i.e. on-premise/public Cloud, on-premise distributed/mainframe (often part of Cloud deployments), SaaS-SaaS integrations, etc. d) Breadth of Applications Supported: i. Application deployment types supported, i.e. Web-based, fat client, Web 2.0. SOA, Mobile, VoIP, etc.
ii. Technologies supported, i.e., ESB/EAI, API-based integrations, Middleware, Storage, CDN. iii. SaaS and IaaS partnerships, which indicate the depth of support for public Cloud vendors (SaaS vendors such as Salesforce.com and OpSource and for IaaS vendors such as Amazon and Rackspace). Higher scores went to APM vendors who have built visibility to execution within public Clouds, such as EC2 and Salesforce.com, versus those simply leveraging synthetic transactions as the primary Cloud monitoring technology OR those monitoring public Cloud as a black box. iv. Specific support for and partnerships with virtualization vendors supporting private Cloud. Higher scores went to APM vendors who have built depth of visibility to execution within private Clouds, and those who have incorporated technologies such as vMotions and vCenter integration into end-to-end monitoring and management dashboards and correlation. 2. Integration/Interoperability: Virtually every IT organization has a variety of enterprise management solutions and, sooner or later, will need to integrate. This section assesses the ability of the solution to integrate with other products within the vendors portfolio, as well as with third party solutions. a) Open Integrations: Integrations to generic events such as SNMP traps as well as to vendordelivered APIs for event, data, and Cloud integration. b) Event Management Integration: The most common fault/event management platforms supported, and whether APIs that support these integrations are provided. c) CMDB/DMS Integration: Several participating vendors questioned whether CMDB/CMS integration is important for APM solutions. EMAs position is based on survey-based data, which indicates that IT organizations are starting to question how Cloud-based services can be managed and tracked. They see CMDB/CMS solutions as the ultimate answer for tracking component sources for increasingly complex and interconnected services.
13
Functionality
The second product strength category is Functionality. It is the fourth side of the hexagram, and covers product features and ease of use. 1. Features: Features are the heart of any solution, and the overall value proposition ultimately boils down to what a solution can actually do. Features were comprehensively assessed across seven key categories and scoring favored features supporting automation versus manual configuration, modeling, and analytics. Key areas assessed include: a) Application discovery/recognition: One key feature EMA sees as being fundamental to managing complex applications is a visual service topology model, ideally generated in real time and requiring minimal (or no) manual input. Application discovery and transaction tracing are key aspects of generating such a model. Assessment questions covered the breadth of elements leveraged to identify a given application (i.e. port, protocol, IP address, fingerprints or templates, packet analysis, etc.) and available discovery methodologies. b) Metrics and measurements: Metrics and measurements delivered natively and via integrated third party solutions. Examples include performance by tier, performance juxtaposed with infrastructure (topology), performance by user/group, completed versus uncompleted transactions, etc. c) Cross-technology visibility: Functionality related to tracing transactions/applications/ services across heterogeneous tiers and across hybrid on-premise/public Cloud environments. d) Alerting/alarming: Automated alerting/alarming features and triggers. Examples include trouble ticket generation/clearing, dynamic baselines, automatic threshold setting, and dynamic variable thresholds by hour, day of the week, etc. e) Troubleshooting: Features related to improving Mean Time to Resolution/Recovery (MTTR) such as visual triage, probable cause analyses and rankings, time-based metrics comparisons, performance comparisons by users/groups/geographies, etc. Assessment covers general APM, private Cloud, SaaS applications, and IaaS applications/components. f) Security and user management: Security-related features such as single sign-on and LDAP integration, user role support, and access control.
g) Analytics/advanced analytics: In this context, advanced analytics generally covers features specifically designed to maximize automation and minimize manual interaction with the tool itself. Specific features assessed include self learning of the application environment, self configuration of the solution based on environmental awareness, rolling baselines that self-update based on ongoing trending, self learning of normal performance across hops, automated real-time service modeling, and visibility across variable transaction execution paths. 2. Ease of Use: Ease of use can make the difference between a product that is fully utilized and one that becomes shelfware. If a product is easy to use, it will require less training and less consulting over time and can be used by personnel with varying levels and types of skills. a) Visualization/reporting: Mapping, grouping, visualization, and reporting capabilities. The ideal solution will include a mix of frequently used canned reports, support for ad hoc reports, and a facility for data sharing (export and import) with tools such as Microsoft Excel. b) Roles supported: Assesses most common IT roles utilizing the solution. 14
Vendor Strength
This is the fifth and final side of the hexagram and covers the multiplicity of factors underlying overall vendor strength. Collectively, the measurements in this section determine the size of the bubble on the chart. These factors are meant to gauge the overall viability of an APM provider, as well as the quality of their vision, strategy, product execution, and market voice. It consists of the following specific categories: 1. Vision: Vision becomes innovation, and visionary vendors create products that enable customers to more easily assimilate changing technologies and application architectures. For this assessment, vendors were asked to describe their Cloud-related product strategy and vision for APM in 500 words or less. More visionary vendors received higher scores. 2. Strategy: Strategy enables vendors to achieve their vision. Again, vendors were asked to describe their longer-term strategy for advancing APM capabilities. 3. Financial Strength: Components of this score include the vendors own assessment of financial position, and estimated year over year revenue growth for the APM solution. For public companies, it also incorporates data included in public documents, such as cash available and forward P/E estimates. 4. Research and Development: Vendors were asked to disclose R&D investments via a multiple choice question with five potential answers: 0 5%, 6 10%, 11 15%, 16 20%, and 21 % +. This answer was then ranked against the answers of other vendors in the study and entered as a score on a sliding scale between 1 and 10. 5. Market Credibility: While features and functions are critical, vendor credibility is another key aspect of product selection which goes far beyond marketing. In fact, for assessments of this nature, marketing hype often does more harm than good, as true product capabilities, strengths, and weaknesses can be obscured by market speak geared toward misinformation versus reality. In evaluating credibility, EMA examined a number of measures, including in-depth interviews with customers. In general, vendors providing two or more customer references were viewed more positively than those providing fewer. Other relevant factors include: how focused the vendor is on the APM space, how straightforward the vendor was in terms of the assessment, how transparent the vendor is to the analyst community, and treatment by other credible industry voices.
EMA Radar Maps for Application Performance Management

As a review, the vendors in the two groups include:
As indicated above, final scoring has been divided into two broad groups by basic high-level architecture. The results for the two groups are presented separately, beginning with the Multi Component group. Multi Component/Suite solutions: CA, Compuware, Correlsense, eG Innovations, HP, IBM, Nastel, OPNET, OpTier, Quest, and SolarWinds Point solutions: AppDynamics, AppFirst, Aternity, INETCO, Netuitive, New Relic, and Splunk
15
Multi Component/Suite Solutions
Figure 5: EMA Radar Map for APM: Multi Component/Suite Solutions
Vendors and Solutions Profiled

CA: CA APM Compuware: Compuware Gomez Application Performance Management Correlsense: Correlsense SharePath eG Innovations: eG Enterprise Suite HP: HP APM IBM: IBM Tivoli Application Performance Management Solution Nastel: Nastel AutoPilot OPNET: OPNET APM Xpert Suite OpTier: OpTier CloudFirst (part of OpTier BTM suite) Quest: Quest Foglight SolarWinds: SolarWinds Application Performance Monitor, SolarWinds Synthetic End User Monitor
16
The value of a solution is defined by comparing its strength with its cost effectiveness. Figure 5 provides a graphical representation of the industry leaders evaluated by this study, positioned in relation to both critical axes. Scores on the x axis reflect Cost Efficiency, which EMA considers to be a measure of overall value. Products shown towards the right on this axis tend to be less expensive or have strong feature/function in relation to overall cost. The y axis reflects the vendors coverage of the specific functionality assessedin this case, APM for Cloud services. The size of the bubble represents a relative measure of Vendor Strength.
General Findings
All of the vendors in this category scored as Strong Value or better. This indicates that all exceeded EMAs rigorous scoring criteria. Although this may appear to show an overly rosy judgment on EMAs part, the truth is rooted in the phenomenon of self-selection. Vendors not prepared to invest the resources required to traverse the Radar process, those daunted by the rigor of the survey and dialogue, and those less certain of product strength declined to participate. From this perspective, those who did participate can be considered the APM elite those providers invested in delivering APM solutions with substantial Cloud-focused value to application management practitioners. HP and IBM were top scorers in the Value Leader quadrant, although a surprising outcome of this study is that four additional vendors are positioned in this quadrant as well. IBM and HP have both focused on closing the pricing gap in recent years, HP with increasing focus on a SaaS form factor for some capabilities and IBM with the introduction of hard and soft appliances (and, most recently, a SaaS option as well). eG Innovations and Nastel are both known as lower cost APM alternatives, with Nastel in particular focused on delivering rich APM functionality at less cost than traditional Big Four competitors. eGs strong suit is private Cloud management, with very strong support for a wide variety of virtualization vendors. Compuware, of course, has been a leading APM vendor for quite some time, and the acquisition of Gomez cemented its place as a high-value APM solution. Finally, OPNET is a more recent entry into the APM market, led by high levels of automation and deep visibility to flow-based messaging. CA, Quest, OpTier, Correlsense, and SolarWinds all placed in the Strong Value category. CAs mainframe support and vendor-agnostic stance make it a very attractive option for companies seeking to consolidate metrics from multiple third party solutions and thereby maximize existing investments. Quest has a very strong contender in Foglight, with a loyal customer following based on ease of installation and maintenance as well as breadth of functionality3. OpTier is in a class of its own in terms of Business Transaction Management, and has recently announced Always-on APM, a methodology for delivering ongoing transaction tracing with minimal impact to production platforms. Correlsense offers an exceptional End User Experience Management solution, which is distinctive in that it can be used to monitor both internal and external users. Finally, SolarWinds, known primarily as an enterprise management platform for the mid-market, continues to add application-focused features that make it increasingly competitive in the APM space. In-depth profiles for all vendors in this report are listed in alphabetical order in the APM Vendor Profiles section.
3
EMA has documented significant Return on Investment by Quest customers in related studies
17
Value Leaders
Compuware: Compuware has invested heavily in Cloud management technology, and the Gomez Application Performance Management (APM) solution bundle covers a broad array of on-premises and SaaS-delivered solutions. Perhaps best known for its flagship Compuware Gomez Data Center Real User Monitoring (RUM) product (formerly Vantage), Compuwares Cloud solutions are a result of its acquisition and successful assimilation of two key technology companies. The acquisition of Gomez in 2009 brought with it rich synthetic and real user monitoring capabilities for Internet-delivered services from the outside in via a global network of monitoring stations. The acquisition of dynaTrace in 2011 added the inside out, code level perspective. These acquisitions substantially improved Compuwares Cloud (and APM) coverage, which now combines metrics from both external and internal monitoring with cross-technology visibility via the Infrastructure and Component Analysis solution. eG Innovations: While eG Innovations eG Enterprise Suite is a well rounded transaction monitoring and management solution, eG is particularly strong in terms of its private Cloud coverage. eG has invested heavily in building support for virtualization, with its trademark In-N-Out virtual infrastructure monitoring supporting performance troubleshooting with both depth and breadth of application visibility. A key differentiator is eGs single monitoring agent architecture (covering a wide variety of heterogeneous technologies), which can considerably decrease administrative overhead over time. HP: HP APM is a robust solution, which is particularly noteworthy for its troubleshooting capabilities. HP has invested heavily in developing a Run Time Service Model (RTSM) as an overlay to the HP Universal Configuration Management Database (UCMDB). Leveraging metrics harvested from across the execution ecosystem, HTTP\RMI tagging, and cross-VM correlations, HP discovers relationships among application components and reports them to RTSM. This delivers a central, up-to-date model of the application run time architecture, which in turn provides a data source for a Federated CMDB/ CMS. EMA sees this as a visionary approach to delivering run-time topology, which is essential to managing, versus monitoring, applications in Cloud environments. IBM: IBM Tivoli Application Performance Management Solution includes five component solutions with deep and broad coverage for public, private, and hybrid Cloud. Since IBM relies on its partner channel to sell APM solutions to the mid-market, IBMs customers for this solution are primarily within enterprise and Service Provider markets. IBM Softwares Tivoli solutions encompass enterprise management for production IT (and extended business technology) monitoring. One very noteworthy differentiator is the fact that IBM has not only tightly integrated its pre-deployment (i.e., Rational) and post-deployment (Tivoli) solutions, it has actively reused components across the two lines. This delivers two primary benefits to customers. By controlling development costs, IBM can pass on savings. In addition, products leveraged by different IT constituencies have a common look and feel. This smooths the transition to cross-functional DevOps by reducing the learning curve required for DevOps teams. Longer term, it also paves the way for extended DevOps, as applications increasingly span public and private Cloud. IBM Tivoli APM focuses on predictive analysis, capacity management, and proactive problem isolation. The packaging strategy is to deliver a comprehensive set of capabilities as modular extensions on a single, scalable and comprehensive product infrastructure while preserving the ability for customers to easily integrate third party applications and data. Tivoli specifically addresses public Cloud by providing integration capabilities to securely monitor and provision across hybrid Cloud environments.
18
Nastel: Nastel AutoPilot is essentially a top level data-agnostic aggregator, capable of assimilating and analyzing data from Nastels own collectors as well as from a wide variety of third party sources and business data. The Nastel Complex Event Processing (CEP) engine forms the core platform of AutoPilot and is the heart of this solution and a distinctive differentiator. Nastel is exceptional in its coverage of data-center-based technologies, and previous EMA research attests to impressive ROI in such environments. AutoPilots functionality is driven by extensive self-learning capabilities and sophisticated data aggregation analytics. OPNET: OPNET APM Xpert Suite is a comprehensive solution for managing the performance of applications in physical, virtual, and Cloud environments. OPNETs High Definition Performance Management approach combines broad visibility across domains with granular drill-down within each domain. It heavily leverages analytics to automate troubleshooting and root cause diagnosis, minimizing the manual configuration, topology modeling, and scripting overhead necessary for many competing APM solutions. EMA sees OPNET as a good example of a new breed of solutions which automate the process of knitting together multiple information sources, including network flow, into real time models of application execution-- with minimal manual intervention.
Strong Value
CA Technologies: CA Technologies CA APM suite monitors user transactions in real time as they traverse the hybrid-Cloud infrastructure (physical, virtual, public Cloud) and is designed to proactively identify, diagnose, and resolve application-related issues before end users are affected. CAs vendor agnostic design supports integration with a wide variety of third party solutions. In addition, excellent virtualization support gives customers a multiplicity of private Cloud options, as CA is one of few vendors covering Microsoft, Citrix, Red Hat, VMware, and Oracle. This solution also boasts strong security and reporting features. Correlsense: Correlsense is squarely positioned in the End User Experience (EUE) market, with fully integrated capabilities supporting Application Performance and Business Transaction Management. Correlsense SharePath uses a lightweight, no touch approach to accurately trace transactions across complex, multi-tier dynamic architectures without modifying application code or tagging payloads. This provides in-depth visibility and analytics for transaction performance in cloud environments. In addition, Correlsense SharePath RUM gathers deep dive user experience metrics via agents and a small snippet of code downloaded into the user browser. Because of this design, it can monitor End User Experience for either internal or external users, a core strength that lends itself to a broad range of use cases. Correlsense integrates with CMDB/CMS systems and is one of few vendors with specific support for SaaS vendors, including Salesforce.com, SugarCRM, and Google. OpTier: OpTiers flagship solution, OpTier BTM, is a highly automated transaction management solution. OpTier cut its teeth in the financial services vertical to monitor mission-critical trading and financial transactions. One distinctive differentiator is OpTiers ability not only to monitor applicationrelated problems, but to take action to mitigate them as well. OpTier CloudFirst is a fully-integrated component of OpTier BTM designed for business-centric Cloud transaction monitoring. CloudFirst delivers visibility into transactions that flow in and out of public and private Clouds, and includes a set of templates to help customers assess the Cloud-readiness of applications before deployment. OpTier is exceptional in its coverage of on-premise-hosted technologies, and previous EMA research attests to impressive ROI in managing large-scale data center environments.
19
Quest: Traditionally known for its richly functional database management solutions, Quest successfully transitioned to the Transaction Management market with its acquisition of Foglight Software in 1999. During the 2006-2007 timeframe, Quest re-engineered Foglight as a modular, hub and spoke solution featuring an analytics and processing core and multiple snap-in components. This model has enabled Quest to adapt to industry disruptions such as Service Oriented Architecture (SOA) and Cloud more nimbly than some competitors. To address Cloud, Quest combines a variety of modular products addressing EUE, infrastructure, database, server, virtualization, and network, all of which seamlessly integrate to the Foglight Performance Monitoring platform. SolarWinds: SolarWinds Application Performance Monitor (APM) and Synthetic End User Monitor (SEUM) solutions offer a wide range of affordable APM capabilities. SolarWinds is popular with IT professionals who regard it as a cost-effective and scalable enterprise management option. SolarWinds is investing in building its APM functionality with coverage for an array of application types including web-based, fat client, Web 2.0, SOA and VoIP (with add-on software modules). It is also noteworthy for its comprehensive security capabilities, such as single sign on, custom or out-of-box user credentials by role, and the ability to issue credentials via software.
Point Solutions
Figure 6: EMA Radar Map for APM: Point Solutions
20
Vendors and Solutions Profiled
AppDynamics: AppDynamics Pro AppFirst: AppFirst Aternity: Aternity Frontline Performance Intelligence Platform Version 5.0 INETCO: INETCO Insight Netuitive: Netuitive 5.5 New Relic: New Relic Pro Splunk: Splunk Enterprise The value of a solution is defined by comparing its strength with its cost effectiveness. Figure 6 provides a graphical representation of the industry leaders evaluated, positioned in relation to both critical axes. Scores on the x axis reflect Cost Efficiency, which EMA considers to be a measure of overall value. Products shown towards the right on this axis tend to be less expensive or have strong feature/function in relation to overall cost. The y axis reflects the vendors coverage of the specific functionality assessedin this case, APM for Cloud services. The size of the bubble represents a relative measure of Vendor Strength.
General Findings
All of the vendors in this category scored as Strong Value or better, which indicates that all exceeded EMAs rigorous basic scoring criteria. As with the Suite solutions profiled above, this does not indicate an overly rosy judgment on EMAs part, but is rather a manifestation of the phenomenon of self-selection. Vendors not prepared to invest the resources required to traverse the Radar process, those daunted by the rigor of the survey and dialogue, and those less certain of product strength declined to participate. From this perspective, those who did participate can thus be considered the APM elite those providers who have invested in delivering complete solutions with substantial value to application management practitioners. Netuitive, AppDynamics, and Aternity all scored in the Value Leader quadrant despite the fact that these are very different solutions. Netuitive is a top-level correlation solution, while AppDynamics is a SaaS-based APM solution focused on code-level analytics. Aternity is an EUE solution focused on the endpoint, with rich capabilities supporting performance management, management of hosted desktop environments, and security compliance. AppFirst, New Relic, Splunk, and INETCO all placed in the Strong Value category. Again, these are very different solutions as well, but all support various elements of APM and related troubleshooting capabilities. AppFirst and New Relic are SaaS-based, one focusing on Application Infrastructure Management (AppFirst) and the other focusing on transaction tracing down to specific lines of code (New Relic). Splunk is a machine data aggregator and analyzer, while INETCO is well established in the payments processing industry and in the process of extending capabilities to address the general APM market. In-depth profiles for all vendors in this report are listed in alphabetical order in the APM Vendor Profiles section.
21
Value Leaders
AppDynamics: AppDynamics Pro is specifically designed for modern (complex, distributed) application architectures in both the Cloud and the data center. Focused on deep transaction monitoring at the application server level, AppDynamics supports both Java Virtual Machines (JVMs) and Microsoft .NET Common Language Runtime (CLRs). AppDynamics also features a variety of leading-edge automation capabilities and its IaaS automation is particularly noteworthy. It has proven support for provisioning and de-provisioning large-scale, IaaS-based application server environments. Aternity: With its Frontline Performance Intelligence Platform, Aternity has developed a distinctive approach to monitoring both Cloud and on-premise service performance, including that of hosted desktop environments. By monitoring the endpoint and the next hop from the endpoint, Aternity delivers a level of visibility to the experience of internal users that is difficult to match with other EUE technologies. This technology can also be applied to a multiplicity of additional monitoring requirements, such as identification of training issues, illicit usage, regulatory compliance, and levels of services delivered by public Cloud vendors. Key differentiators include The Aternity FPI Platforms support for hosted desktop environments and deep visibility to processes, keystrokes, and virtually all other events taking place at the user workstation. Netuitive: Netuitive is a top-level, software-based analytics product that gathers, analyzes, and correlates metrics collected from enterprise source systems (sub-systems). It integrates with third party management solutions by gathering/forwarding performance data to Netuitives Behavior Learning Engine. The engine applies patented analytics to each data stream to learn normal behavior, identifying anomalies, in many cases proactively, based on time-based trends such as day of the week, month of the year, or season. Netuitives single pane of glass design delivers first-rate troubleshooting information and also extends the effective lifespan of management solutions already in place.
Strong Value
AppFirst: AppFirst is a SaaS-based management solution focusing on application infrastructure performance management. It is primarily an infrastructure monitoring solution with some top-level correlation capabilities, supporting log file analytics, for example, and the ability to correlate metrics from a variety of sources. AppFirsts data collection technology allows continuous collection of metrics from every process running on production servers. As a cost-effective product that is easy to deploy, this can be an excellent solution for smaller companies seeking basic application monitoring. INETCO: INETCO has been a respected vendor in the payments industry since the 1980s, initially known for its protocol conversion software. For the past five years INETCO has built its business around Business Transaction Management (BTM) for complex tiered financial applications. Eighty percent of INETCOs customers are service providers, primarily payment systems processors running complex back end credit card processing systems. INETCO recently announced INETCO Insight 5, a new solution intended to extend its business beyond the financials vertical to cross-platform APM. This is a distinctive offering that uses transaction tracing, network capture, and sophisticated transaction templates to capture and identify transactions as they execute across tiered systems. INETCOs CloudTrails technology sees transaction performance across multiple off-premise tiers, with some visibility to whats happening IN the public Cloud as well. While INETCOs APM-specific functionality is still in the process of being built out, EMA welcomes INETCO to this new market.
22
New Relic: New Relic is part of a new breed of APM solutions founded by industry veterans with a wealth of experience in the APM market. New Relic Pro is a SaaS-based solution delivered as a single, all-inclusive product for development, application deployment, and IT operations teams. New Relic Pro is optimized for application support professionals with some knowledge of application code, as it enables them to track application performance through servers and down to specific lines of code. It delivers capabilities such as application performance analytics, Real User Monitoring, availability monitoring, Server Monitoring, SQL performance, and web transaction tracing for application environments including Ruby, Java, .NET, PHP and Python. Splunk: Splunk Enterprise is a multi-purpose machine data aggregation and assimilation solution that can be used to troubleshoot application problems, monitor application and system performance, provide visibility across the IT infrastructure, and investigate security incidents in both real-time and historical time buckets. Splunk describes its offering as the engine for machine data and indicates that customers can question, search, monitor and report on virtually any type of data (logs, events, metrics, status, alerts, traps, APIs, file system changes, diagnostic commands, etc.), and perform advanced analytics on this data to provide operational intelligence. Splunk uniquely delivers a wide range of features that lend themselves to APM monitoring and troubleshooting. It includes a flexible, natural language-like search function supporting analysis, trending, summarization, outlier detection etc. In working with Splunk customers, EMA has also seen significant ROI from Splunks data management functionality, which allows retention of large quantities of data with appropriate role-based access controls and reduces storage requirements. Splunk can be deployed on premise, virtualized, or in the Cloud (AWS, Rackspace, etc).
Exceptional Characteristics: Awards
EMA awards a limited number of citations to vendors with particularly noteworthy capabilities in one or more functional areas. While the vendors participating in this particular study were all very strong and the choice was difficult, only six solutions received awards. Four were chosen from the Suite/ Multi-Component group and two from the Point Solution group, commensurate with the number of participants in each. These participants are worthy of special recognition for specific areas of strength and/or unique areas of innovation.
AppDynamics: Best Infrastructure as a Service (IaaS) Automation
AppDynamics Pro is a single component application performance management solution specifically designed for modern (complex, distributed) application architectures in both the Cloud and the data center. Focused primarily on application servers, AppDynamics has visibility to both Java Virtual Machines (JVMs) and Microsoft .NET Common Language Runtime (CLRs). EMA is recognizing AppDynamics for its IaaS automation based on its support for provisioning and de-provisioning large-scale, IaaS-based application server environments. One of AppDynamics key customers is a Cloud-based media vendor who is, in turn, one of the largest customers of Amazons EC2 IaaS platform. The customer routinely creates 1,000 EC2 virtual machines over 30-60 minutes to push new features as application code. Once the new environments are in production, the vendor then deletes the 1,000 machines previously running that service.
23
While few, if any, enterprise IT organizations are leveraging dynamic EC2 environments to this extent, they may well do so in the future. From this perspective, AppDynamics support for public Cloud is noteworthy and will likely be of interest to companies considering dynamic capacity management and Cloud bursting.
Aternity: Most Comprehensive Endpoint Diagnostics
Because of its distinctive approach to End User Experience Management, EMA is pleased to cite Aternity as the award winner for the Most Comprehensive Endpoint Diagnostics. EMA research has repeatedly found that companies leveraging Cloud services see EUE as a primary monitoring methodology for such environments. Monitoring the User experience is a fail safe way to determine the levels of overall service delivered to the desktop. Aternity has developed a distinctive approach to monitoring both Cloud and on-premise service performance, including that of hosted desktop environments. By monitoring the endpoint and the next hop from the endpoint, Aternity delivers a level of visibility to the experience of internal users that is difficult to match with other EUE technologies. This technology can also be applied to a multiplicity of related requirements, such as identification of training issues, illicit usage, regulatory compliance, and levels of services delivered by public Cloud vendors.
eG Innovations: Best Private Cloud Coverage
EMA sees eG Innovations heavy investments supporting virtualized environments as worthy of an achievement award for Best Private Cloud Coverage. This award is based on multiple functional capabilities, including: Comprehensive Virtualization Coverage: VMware vSphere, Citrix XenServer, Microsoft Hyper-V, Microsoft Virtual Server 2005, Solaris LDOMs, Solaris Containers, AIX LPARs, RHEL, others TBD. Patented Virtualization/Cloud-aware Correlation: eG Innovations correlation supports the new sets of dynamic inter-dependencies, such as VMware vMotions, that come with private Cloud. Universal Agent Technology contributes to cost and administration efficiency. A single agent is capable of monitoring over one hundred fifty (150) applications, ten (10) different operating systems, and seven (7) different virtualization platforms. In-N-Out virtual infrastructure monitoring supports performance troubleshooting by providing both depth and breadth of application visibility. It shows an outside view of a VM (the physical resources the VM is consuming) in context to an inside view that indicates which application(s) inside the VM are causing the resource constraint. Both views are obtained using a single monitoring agent.
24
HP: Most Comprehensive and Real-Time Service Model
A primary characteristic of leading-edge APM solutions is that they not only report application performance, they support troubleshooting when performance degrades. For this reason, EMA sees real-time topology models as key differentiators for companies leveraging Cloud services. HP is particularly noteworthy in this regard, having invested heavily in developing a Run Time Service Model (RTSM) as an overlay to the HP Universal Configuration Management Database (UCMDB). HPs RTSM correlates service models from multiple sources in the application stack to build a full end- to- end view of the application operating environment. Leveraging metrics harvested from across the execution ecosystem, HTTP\RMI tagging, and cross-VM correlations, HP discovers relationships among application components and reports them to RTSM. This delivers a central, up-to-date model of the application run time architecture, which in turn provides a data source for a Federated CMDB/ CMS. EMA sees this as a visionary approach to delivering run-time topology, which is essential to managing, versus monitoring, applications in Cloud environments.
IBM: Best Cloud Vision and Design
EMA is pleased to award the Best Cloud Vision and Design award to the IBM Tivoli Application Performance Management Solution. IBM scored above the norm on all five axes measured. In addition to garnering the highest scores of any vendor, IBM has demonstrated an in-depth understanding of the complex factors that support Cloud management. For example, IBM is one of few vendors paying attention to the CMDB/CMS implications of Cloud. As application components increasingly span on-premise, public Cloud, and partner Clouds, configuration tracking will be a critical capability for performance management and application troubleshooting. IBM is also actively addressing Cloud capacity planning within IBM Smart Cloud Monitoring. Capabilities include performance trending to proactively predict resource bottlenecks, what-if analysis to determine the number of workloads that can be added to existing resources, and optimization of workload placement. IBM has good coverage of virtualization platforms, notably including Red Hat, z, and Solaris, which are missing in most competing solutions. IBM is one of few vendors who have thought through monitoring and management of complex Cloud transactions such as SaaS to SaaS integrations. IBM is also noteworthy in terms of its overall focus on automation. For example, although IBM has synthetic transaction technology, this is not the centerpiece of its Cloud APM strategy, as it is for some competing vendors. Instead, IBM has focused on analytics, including tracing and stitching capabilities that reduce requirements for ongoing hands-on administration. In terms of cost efficiency, IBM has also added capabilities to make its productiongrade products more easily consumable by the mid-market.
25
OPNET: Best Overall Automation
OPNET APM Xpert Suite is a comprehensive solution for managing the performance of applications in physical, virtual, and Cloud environments. OPNETs High Definition Performance Management approach combines broad visibility across domains with granular drill-down within each domain. It heavily leverages analytics to automate troubleshooting and root cause diagnosis. As the hallmark of modern applications is their integrated nature, EMA analysts see the network, supported by leading edge analytics, as the Rosetta Stone reconciling top-down, flow-based application/transaction insight with the bottom-up view of infrastructure topology. Without near real-time correlation of both in context to one another, Application Performance Management (versus Application Performance Monitoring) cannot be automated. OPNET has evolved its solution set based on a similar vision and has continued to add features accordingly. OPNETs Federated Analytics provide a good example of a new breed of solutions capable of knitting together multiple information sources, including transaction flow, into real time models of application execution. Such automation supports troubleshooting and fast Mean Time to Resolve (MTTR) in the short term. In the longer term, it builds a foundation for next generation APM capabilities that will someday enable lights out enterprise management to become a reality. OPNET views Cloud as a continuum, with hybrid as the final answer and is evolving the toolset accordingly.
26
About Enterprise Management Associates, Inc.

Founded in 1996, Enterprise Management Associates (EMA) is a leading industry analyst firm that provides deep insight across the full spectrum of IT and data management technologies. EMA analysts leverage a unique combination of practical experience, insight into industry best practices, and in-depth knowledge of current and planned vendor solutions to help its clients achieve their goals. Learn more about EMA research, analysis, and consulting services for enterprise line of business users, IT professionals and IT vendors at www.enterprisemanagement.com or blogs.enterprisemanagement.com. You can also follow EMA on Twitter or Facebook. This report in whole or in part may not be duplicated, reproduced, stored in a retrieval system or retransmitted without prior written permission of Enterprise Management Associates, Inc. All opinions and estimates herein constitute our judgement as of this date and are subject to change without notice. Product names mentioned herein may be trademarks and/or registered trademarks of their respective companies. EMA and Enterprise Management Associates are trademarks of Enterprise Management Associates, Inc. in the United States and other countries. 2012 Enterprise Management Associates, Inc. All Rights Reserved. EMA, ENTERPRISE MANAGEMENT ASSOCIATES, and the mobius symbol are registered trademarks or common-law trademarks of Enterprise Management Associates, Inc. Corporate Headquarters: 5777 Central Avenue, Suite 105 Boulder, CO 80301 Phone: +1 303.543.9500 Fax: +1 303.543.7687 www.enterprisemanagement.com
2396-Summary.01712

EMA APM 2012 RadarReport Summary

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

EMA APM 2012 RadarReport Summary

Uploaded by

Copyright:

Available Formats

EMA Radar for Application Performance Management (APM) for Cloud Services: Q1 2012 Report Summary

18 Cloud-Ready APM Solutions Available TODAY

EMA Research on APM for Cloud Services

Introduction and Methodology

Available at: http://www.enterprisemanagement.com/research/asset.php/1715/How-To-Use-The-EMA-Radar-Report, EMA, April 2010.

Architecture & Integration

Deployment & Administration

Architecture & Integration

Deployment & Administration

Strong Product Higher Price

Specialized Product Low Cost

APM: Evolving with the Cloud

The Problem Set: Cloud-specific APM Challenges

Without adequate tooling, knowing who to call is a best guess exercise.

Management versus Monitoring

Figure 3: EMA End-to-End Application Management Semantic Model

Figure 4: Assessment Model for APM/Cloud Radar

Deployment & Administration

Architecture and Integration

EMA Radar Maps for Application Performance Management

Figure 5: EMA Radar Map for APM: Multi Component/Suite Solutions

Vendors and Solutions Profiled

Figure 6: EMA Radar Map for APM: Point Solutions

Exceptional Characteristics: Awards

AppDynamics: Best Infrastructure as a Service (IaaS) Automation

Aternity: Most Comprehensive Endpoint Diagnostics

eG Innovations: Best Private Cloud Coverage

IBM: Best Cloud Vision and Design

About Enterprise Management Associates, Inc.

You might also like