How to Build a Successful Performance CoE for Any Organization

The purpose of this article is to assist performance engineers and IT organizations to estimate their current situation and successfully build and maintain their Performance Center of Excellence. These guidelines are based on my experience with many IT companies over the years and in-depth research using real-life best practices and expertise from many sources, including

• Performance Architects
• Many customer testing organizations
• Performance CoE Managers
• Performance Engineers
• QA Managers
• Development Managers
• Application Architects and 
• Project Managers

What Is Performance CoE (Center of Excellence)

A Performance CoE is an internal competition that aims to improve and validate the performance of applications to meet business requirements. Performance Center Of Excellence is a discipline that leverages products, practices, efforts, people, and processes to minimize risks during deployment and load testing of the applications with upgrades or patches.

Performance testing, engineering, and monitoring are about applying production workloads to pre and post-deployment systems while simultaneously measuring system performance and end-user experience with multiple users. A well-designed performance test will address questions such as

• Is the application responding quickly enough to the intended users?
• Will the application handle the expected user load and beyond?
• Will the application process/handle the number of transactions required by the business?
• Is the application stable under expected and unexpected user loads?
• Are you sure that the users will have a positive experience on the day they go live?
• Is the application universally available without any downtime?
• What is the maximum handling capacity of your application/system?

By answering all the above questions, performance testing and engineering aim to measure the impact of a change in business terms and provides the percentage of performance improvements for applications that are tested for performance. This, in practice, clearly indicates the risk of deployment following the load test execution. An effective performance testing and engineering process help you make more informed release decisions and prevents system downtime and availability problems. 

One of the approaches for achieving the maximum value of performance testing is to build a Performance Center of Excellence. Many IT organizations have the Center of Excellence (CoE) model as a practical way to consistently and continuously improve IT operations. In the CoE model, the QA infrastructure, processes, and best practices are consolidated and centralized.

The Performance CoE helps the management, leadership consulting, support services, and companies to optimize the application quality as per the SLA's. It also provides the entire organization with visibility into standardized quality metrics and key performance indicators (KPIs) and keeps all stakeholders informed, applications aligned with business objectives.

Why Performance CoE (Center of Excellence)

The simple answer to this question is because a Performance Center of Excellence delivers valuable and efficient results. The performance requirements have become increasingly more difficult for several organizations from day to day to manage.

Unfortunately, most of the performance engineers fail to meet SLA's because of the following reasons.

• Not having a complete understanding of what stakeholders want to know
• Bad at editing out information that is not necessary or relevant for performance testing 
• Unable to translate technical problems into important business impacts
• Bad at gathering information (NFR gathering) in a way that is easy to digest 
• Not being aware of which tools to use for performance testing, monitoring, and engineering
• Wrong understanding of the overall picture in terms of goals or vision for conducting a load test
• Not validating and monitoring the tests completely
• Unable to simulate the test with the finalized workload characteristics
• Poor to build a performance testing environment for performance testing

Performance CoE eradicates the use of dissimilar tools, inefficient performance testing, and incorrect analysis. A Performance CoE will also provide a central platform for performance management for the entire organization that will provide insight into key performance indicators of the delivered application, keeping everyone updated and keeping applications consistent with business goals. 

Multiple expert reports relate the importance of Performance CoE to the significantly enhanced ability of an organization to meet or exceed the objectives that the CoE supports. The key motivation for creating any Performance CoE should be the need to meet expectations and deliver high-quality results. Without this, the organization will not be able to achieve its full potential.

Why Normal/Traditional Performance Testing is Not Enough

Performance testing is a specialized set of knowledge and skills. It requires an understanding of applications, software/hardware, and third-party systems, an architectural understanding of application behavior from an architectural point of view, modeling skills, and a systematic approach. Not all performance testers have these qualities and it takes years to completely build the right skillset. If performance testers are working in separate project test groups, it is hard to verify that all applications are adequately tested to the same level. 

Also, when new technologies are introduced, not all of these different groups will have the necessary expertise to support those new technologies. In many IT organizations, each project team develops its performance optimization strategies and techniques will use whatever tools that are recommended by individual team members. The result is an unmanageable mix of workflows, practices, tools, and techniques not well documented, not well integrated, and not transferable to other performance teams and result in miscommunication lead to more than inefficiency. 

There are many ways for doing normal/traditional performance testing but establishing an efficient, centralized, practices based and process based Performance CoE in any organization will help every performance testing team to avoid all these errors, predicts risks, and greatly improves the business outcomes in a shorter time.

Performance CoE (Center of Excellence) — Advantages

Any organization's main objective is to achieve the execution of efficient processes that help to achieve business goals and to provide customers with satisfactory service delivery. Performance CoE helps companies to develop customer focus and success across multiple business units/projects and regions. There are several advantages of having a Performance CoE in every organization. 

Centralized Infrastructure and Expertise

Test labs, tools, staff, and best practices are integrated and conveniently accessible to all performance teams from a central entity, eliminating the need to replicate expensive resources (in fact, total headcounts and system licenses could be reduced). A centralized team facilitates higher test quality and improves the efficiency of running and analyzing the tests.

Quick and Improved Processes

Standardization of tools and techniques allows the reliable, cost-effective, and rapid implementation of enhanced processes. Instead of maintaining all scripts, monitor profiles, and scenarios on individual test machines or separate test groups, you can centralize all data to make it easier to share and reuse. This reduces the time it takes to run tests and helps to make testing more efficient.

Business Focus

The Performance CoE model measures performance from a business and end-user perspective, not just systems or components. When the testing systems/applications are centralized, it is easier to use the systems more effectively by ensuring less downtime per application and allowing 24x7 monitoring.  

Realistic and Achievable Targets

Building a reliable Performance CoE is an achievable target. You can start small, maximize existing resources, and grow as the value is proven. Over time, resources, services, and capabilities can be scaled up recursively as management and IT staff start recognizing the value. As long as the Performance CoE is correctly implemented, it will prove its value.

Develop the Business-to-IT Relationship

There are several reasons why performance testing teams may have a poor relationship with their business stakeholders, but a poor engagement model is a frequent culprit. Performance CoE provides a framework for policies, procedures, best practices, and processes that promote improved interactions between all stakeholders, clearly because they come from a cross-functional, business perspective, rather than just an IT perspective.

Time-Saving

The well-applied Performance CoE can improve the process of project design and execution as well as the better management of the skills and abilities that are part of the organization. Also, the final result is usually higher than that of a project that has not been managed by the Performance CoE. As a result, business processes are streamlined and reduce the time needed to perform tasks and achieve performance goals. 

How to Build Performance CoE

Performance testing and engineering are about measuring the overall architecture performance that is under load. The resources that should be used within the performance testing and the overall capacity of your organization to perform this type of testing depend on the answers to the following questions:

• How often does performance testing occur?
• How rigorous are the performance tests and results?
• How much risk and/or cost will be incurred when there are performance issues in production?
• How many technical resources are assigned to performance testing and how often?

If the responses to these questions indicate that the application team or company is making significant ongoing efforts to conduct performance testing and this effort will consume many resources within your organization, you should seriously consider setting up a Performance Center of Excellence.

The Performance CoE model is easily attainable and practical. Initially, it needs to start small, with minimal incremental capital investment. Then, over time, you can incrementally scale up resources, infrastructure, and skills as Management and IT teams have proven their value. The success rate can become recognizable for as long as it is implemented correctly. 

It will develop over time from the consolidation of hardware and software assets to the performance authority phase, regardless of who initiated the Performance CoE. As each application passes through these phases before it goes into production, specialized skills are maintained and shared, and best performance testing practices are documented and followed by every performance team across the organization.

Approaches for Building a Performance CoE

Organizations usually recognize the need to create a Performance CoE based on two-way pressures

1. A top-down approach, where management makes the task of creating a center of excellence mandatory
2. A bottom-up approach, where the urgency of testing consolidation is raised and communicated by the testing community itself

Top-Down Approach

In the top-down approach, the vice president of application development, or a similar role, informs the testing group to come up with a proposal to set up a process-oriented center to support and concentrate testing resources and knowledge. Sometimes the proposal can result from an external consultant believing that a CoE will improve customer satisfaction, exploit acquired expertise, and offer better hardware and software resources ROI. 

Whatever the reason, management must effectively communicate the Performance CoE goals, their initial coverage, and the anticipated go-live date. The Performance teams usually asked to come up with a comprehensive plan including responsibilities, timelines, reporting procedures, and expected challenges, and benefits. The attention that the management team gives to the process of creating the proposal and later its discussion and implementation is probably the most important factor in determining the project's success. 

Bottom-Up Approach

The strong testing group in many organizations will raise the issue of creating a Performance CoE after identifying uncoordinated performance efforts in multiple development teams, lack of expertise in business lines, or inefficient use of limited hardware and software resources. This bottom-up approach usually includes designing the business case, identifying an executive sponsor, and setting acceptance criteria. The establishment of  Performance CoE using this approach will define expectations, the current performance testing model, develop implementation plans, setup Performance CoE services, evaluate results, best practices, and methodologies and their quality assurance needs.

Performance CoE Outcome

The business goal of establishing an effective Performance Center of Excellence is achieved by answering the question "What are the main pain points in the area of performance validation? " The answer to this question should drive the immediate actions, define the scope of changes, and specify the metrics to measure the progress. Some of the goals that are achieved by most of the companies with Performance CoE establishment are listed below 

• Centralized group for knowledge and best practices within the organization
• Centralized Performance tools team that will provide more efficient license management
• Leveraging the tools, resources, and equipment across the organization
• Load Testing more and more applications within the organization
• Minimize risks that provide greater confidence to stakeholders and project teams

Communication

Communication in Performance CoE maximizes success and minimizes risks. To improve the widespread attitude towards a Performance Center of Excellence and communicate its value effectively, a serious effort will be made to create information for numerous stakeholders within the company. Every performance engineer from each project team has to communicate the risks and challenges clearly to the Performance CoE

• Report problems that could potentially delay the go-live date
• Report problems that can occur after the go-live
• Report risks to the project timeline
• Report risks to the project budget
• Report any issues that can prevent acceptance of business
• Report status of all planned completed tasks
• Present project milestones and dependencies on other teams
• The present correct testing approach
• Report if your findings that potentially invalidate system or functional design, or hardware requirements 
• Report about the application’s ability to meet its functional and performance objectives 

Get the Right People for Performance CoE Success

Based on the size of the company, the organization structure, the maturity level, and also the industry the company operates, the Performance Center of Excellence can be divided into several departments and teams. For most situations, the Performance CoE has the following core roles:

Performance CoE Manager

The role of the Performance CoE manager is to effectively lead the performance testing/engineering teams. The CoE manager must be qualified to manage and implement or maintain an effective performance testing and engineering process. He has to promote Performance CoE, escalate performance issues for resolution, provide appropriate training, manage the budget and the CoE manager has to let every project team know the importance of the Performance CoE role that plays within the organizational structure.

Performance Architect

The role of a Performance architect is to determine and maintain the test architectural vision for the Performance CoE. This is the most critical role within any center of excellence and should be undertaken by senior performance architects who have proven experience across several domains, companies in the fields of project management, CoE practice leadership, or performance testing/engineering. 

Performance Testing/Engineering Practice Lead

The role of the performance testing/engineering practice lead is to provide the processes, procedures, and templates that support effective design and testing. These should cover test design, test construction, test execution, capturing test results, defect reporting procedures, and test coverage analysis.  The performance practice lead has to continuously work with multiple performance teams and engineers to improve the performance testing/engineering methodologies

Performance Engineer

The role of a performance engineer is to design, execute, analyze, and provide effective test results. To fulfill this role, the performance engineer applies appropriate performance technologies to meet the short- and long-term goals of the organization. The objective is to test the performance as much of the testing effort as possible using a minimum set of code/scripts. The focus should be on optimizing the performance to meet the SLA's 

Business Analyst

The BA must be closely aligned with the Performance CoE to ensure that the business processes to be tested are clearly understood. If the Performance CoE is an external outsourced entity, then the Performance CoE should include BAs who collaborate with the client and serve as a point of contact within the Performance CoE to ensure that the business goals are transparent and that testing meets the customer requirements

Operations Manager

The task of the Operations Manager is to plan the use of Performance CoE resources, including software, hardware, and human resources. The Operations Manager works closely with the Performance CoE manager, Performance Architects and performance Lead’s to ensure that the schedule can handle any planned or unplanned resource overlaps. It is an implementation of a resource matrix, which all Performance CoE members in all business lines and IT teams must see and recognize.

Conclusion

Setting up and running a Performance CoE increases both short and long-term returns on any investment. With a structure in place, the performance testing and engineering process can now become a well-established process within the organization. The benefits of a successful Performance Center of Excellence include the following 

• This offers a way of leveraging environmental resources, knowledge, and human resources across projects 
• This provides performance testing consistency and efficiency across the projects
• Enables reusability of project artifacts
• Increases the performance of applications/products
• It allows the establishment of key metrics and their analysis for process improvements. 
• It makes testing efforts predictable and accurate. 
• Increased confidence in success when rolling out new code, new software, and new hardware configurations by allowing you to identify capacity constraints, concurrency problems, and other performance bottlenecks before going to production. 
• Improve the performance of existing code through application tuning
• Improve ongoing capacity planning/management by gaining a better understanding of the performance profile of your architecture and its scalability characteristics.

Thanks for reading the article. Happy Learning:)