Performance

This article is meant to be a starting point for sharing observations, experiences, and ideas to begin and build a conversation, exchange experiences, discuss, and shape a product vision we can work towards as a community. Our initial product vision and definition reads as follows: “Score is an open source project that provides a developer-centric and platform-agnostic workload specification to improve developer productivity and experience. It eliminates configuration inconsistencies between local and remote environments.” A lot of words, I know—stick with me as we take a step back to understand why and how we got here. We’ll keep referring to the terms I just mentioned, so at the end, you’ll be able to paint the full picture, and we can understand what Score stands for together. Infrastructure Centric Development The Enemy Is Complexity Above, we talk about “configuration inconsistencies between local and remote environments.” To understand what we mean by that, we have to understand infrastructure-centric development: the source of all evil. In an infrastructure-centric workflow, developers are concerned with the platforms and tooling their workloads run on. Locally, this will be a lightweight tool such as Docker Compose. So far, so good. Remotely, however, you’ll likely be confronted with tooling such as Helm, Kustomize, Terraform, Argo CD, or similar. Developers having to promote their workload from their local setup to production environments that rely on a different set of tech and tools will run into issues surrounding: Environment-specific configuration: Your workload might have many environment variables configured—some of which are environment-specific, and others should remain unchanged. For complex applications, it is often not clear if CPX_API_KEY or SRD_MEM_CAP should be updated for the test environment. What if you miss the update of DBURL, and it is pointing at the development database? Platform-specific configuration: A workload with a dependency on a database might point to a Postgres Docker image or mock server in lower environments. On its way to production, however, a database has to be provisioned and allocated via Terraform. How do you ensure that everything is configured and provisioned correctly? The fact that production environments require specialized knowledge and operational expertise increases the risk of wrongly specified or inconsistent configurations. The question of how things are reflected appropriately between environments is answered differently in teams. There might be a few super helpful people that end up being the “go-to people” for helping others get unstuck. There might be a page on the internal wiki site explaining how to get help or a full-blown ticketing system to track and route requests. The course of action also depends on the complexity of the task at hand. A variable change is easier to manage than satisfying a dependency on a database across various environments. Workload Centric Development Compartmentalizing Complexity Above, we say that Score attempts to “improve developer productivity and experience.” To understand what we mean, we must understand workload-centric development: the solution to all of your problems. By embracing a workload-centric approach to software development, developers no longer have to worry about environment-specific implementation details when promoting their workloads towards production. In a workload-centric world, developers declare what their workload requires to run successfully and independently of a platform or environment. For example: “My workload has one container, which should be made available on a TCP port and relies on a Postgres database.” The following are questions that come with this response: What is the image pull secret? What is the TCP port number? What is the exact number of replicas? What is the provisioning and allocation of the database? The platform in the target environment is responsible for answering these questions and ensuring that everything is configured and injected accordingly. This offers the potential of a reality in which developers can focus on local development without worrying about different configuration constructs, rules, and values in remote environments. Workload-centric development allows developers to focus on their workload’s architecture rather than the tech stack in the target environment. It establishes a clear contract between Dev and Ops. The operations team is provided with a comprehensive set of configurational requirements, which—if met—ensure that the workload runs as intended. Code is passed through the fence rather than being thrown over it. Score Principles So far, we talked about the issue of developers struggling with “configuration inconsistencies between local and remote environments” and how “to improve developer productivity and experience” in this context. We’ll now look at how Score attempts to put this into practice by providing “a developer-centric and platform-agnostic workload specification.” A while ago, the Score team formulated a set of workload-centric development principles based on which the Score specification was eventually developed: Establish a single source of truth for workload configuration that served as a main point of reference for development teams when trying to understand a workload’s runtime requirement and allows them to apply changes in a one-directional and standardized manner. To qualify as an SSOT, it has to be platform agnostic—meaning it’s not tied to an orchestrator, platform, or tool such as Kubernetes, Google Cloud Run, or Nomad—and environment agnostic—meaning it captures the configuration that should be the same across environments. Provide a tightly scoped workload spec that shields developers from the configurational complexity of container orchestrators and tooling. Container orchestration systems such as Kubernetes provide a massive number of options that can be configured just for a workload. Developers may find it challenging to understand which properties are needed, optional, or can be disregarded. By keeping a tightly scoped workload spec that only exposes core application constructs, developers can keep their focus. Implement a declarative approach for infrastructure management that allows developers to describe a workload’s resource dependencies without worrying about whom, when, and how it will be provisioned in the target environment. Looking at our opening statement of it being “developer-centric” and “platform-agnostic,” we can add to the list of words without sounding overwhelming: “environment-agnostic,” “declarative,” and “tightly scoped.” A less wordy explanation offers the name itself. The Score specification, just like the musical notation, makes developers the conductor of their workload, which is deployed across an orchestra of tech and tools. How Is It Implemented? Our story doesn’t end with the Score specification. Its counterpart is the Score implementation, a CLI tool which the Score specification file (score.yaml) is executed against. It is tied to a platform such as Docker Compose (score-compose) or Helm (score-helm) and allows it to generate a platform-specific configuration file such as docker-compose.yaml or a helm values.yaml file. With Score, the developers’ workflow stays largely uninterrupted. The score.yaml file is saved alongside the source code and generates the required configuration with a single command. Example In this example, we are working with a simple Docker Compose service that is based on a busybox image. The score.yaml we created for it looks as follows: apiVersion: score.dev/v1b1 metadata: name: hello-world containers: hello: image: busybox command: ["/bin/sh"] args: ["-c", "while true; do echo Hello World!; sleep 5; done"] To convert the score.yaml file into an executable compose.yaml file, we simply need to run the score-compose CLI tool: $ score-compose run -f ./score.yaml -o ./compose.yaml The generated compose.yaml will contain a single service definition: services: hello-world: command: - -c - 'while true; do echo Hello World!; sleep 5; done'entrypoint: - /bin/sh image: busybox The service can now be run with docker-compose as usual: $ docker-compose -f ./compose.yaml up hello-world [+] Running 2/2 ⠿ Network compose_default Created 0.0s ⠿ Container compose-hello-world-1 Created 0.1s Attaching to compose-hello-world-1 compose-hello-world-1 | Hello World! compose-hello-world-1 | Hello World! The same score.yaml can be run with any other Score implementation CLI. With all platform configuration files being generated from the same specification, the risk of configuration inconsistencies between environments significantly decreases. A change to score.yaml will automatically be reflected in all environments without the developer needing to manually intervene. Check out our current suite of available CLI tools here. Conclusion Score wants to make developers’ lives easier by allowing them to focus on developing features instead of fighting with infrastructure. It simplifies the promotion of workloads from local to remote development environments by automating repetitive configuration work. We’re curious to hear: What experience did you have working with complex microservice applications? Do you have an idea on how to improve the Score specification? Or would you like to support us in building the next generation of Score implementations? Get involved here! Thanks for reading!

This is an article from DZone's 2022 Performance and Site Reliability Trend Report.For more: Read the Report Distributed tracing, as the name suggests, is a method of tracking requests as it flows through distributed applications. Along with logs and metrics, distributed tracing makes up the three pillars of observability. While all three signals are important to determine the health of the overall system, distributed tracing has seen significant growth and adoption in recent years. That's because traces are a powerful diagnostic tool to paint how requests propagate between services and uncover issues along the boundaries. As the number of microservices grows, the complexity in observing the entire lifespan of requests inevitably increases as well. Logs and metrics can certainly help with debugging issues stemming from a single service, but distributed tracing will tie contextual information from all the services and surface the underlying issue. Instrumenting for observability is an ongoing challenge for any enterprise as the software landscape continues to evolve. Fortunately, distributed tracing provides the visibility companies need to operate in a growing microservice ecosystem. In this article, we'll dive deep into the components of distributed traces, reasons to use distributed tracing, considerations for implementing it, as well as an overview of the popular tools in the market today. Components of Distributed Tracing Distributed tracing breaks down into the following components: Spans – smallest unit of work captured in observing a request (e.g., API call, database query) Traces – a collection of one or more spans Tags – metadata associated with a span (e.g., userId, resourceName) To illustrate, let's walk through a distributed tracing scenario for a system with a front end, simple web server, and a database. Tracing begins when a request is initiated (e.g., clicking a button, submitting a form, etc.). This creates a new trace with a unique ID and the top-level span. As the request propagates to a new service, a child span is created. In our example, this would happen as the request hits the web server and when a query to the database is made. At each step, various metadata is also logged and tied to the span as well as the top-level trace. Once all the work is complete for the corresponding request, all of the spans are aggregated with associated tags to assemble the trace. This provides a view of the system, following the lifecycle of a request. This aggregated data is usually presented as a flame graph with nested spans over time. Figure 1: Flame graph of traces Visualizing traces this way helps to reveal performance bottlenecks (i.e., longest span in the trace) as well as map out each interaction with the microservices. Why Use Distributed Tracing For legacy applications largely running in a monolithic manner, logs and metrics were often sufficient for observability. Detailed logging provides a point-in-time snapshot of that service, leaving a record of all the code execution. Metrics gather statistical information about the system and expose the general health of that service. For monolithic applications, combining the two provided the necessary visibility. However, in a microservices world, problems can occur not just inside a single application (which logs and metrics can reveal), but also at the boundaries of those services. To respond to an incident or to debug a performance degradation, it's important to understand how the requests are flowing through one service to another. With that in mind, the benefits of distributed tracing include: Visualizing service relationships – By inspecting the spans within a trace from the flame graph, developers can map out all the service calls and their request flow. This helps to paint a global picture of the system, providing contextual data to identify bottlenecks or ramifications from design changes. Pinpointing issues faster – When the engineer on-call is paged from an incident, traces can quickly surface the issue and lead to reduced mean time to detect (MTTD) and repair (MTTR). This is a big win for the developer experience while maintaining SLA commitments. Isolating specific requests – Since traces document the entire lifecycle of a request, this information can be used to isolate specific actions such as user behavior or business logic to investigate. Despite these benefits, adoption numbers for distributed tracing pale in comparison to logging and metrics as distributed tracing comes with its fair share of challenges. First off, distributed tracing is only useful for the components that it touches. Some tracing tools or frameworks don't support automatic injection or some languages (especially front-end components). This would result in missing data and added work to piece together the details. Also, depending on the application, tracing can generate a significant amount of data. Dealing with the scale and surfacing the important signals can be a challenge. Considerations for Implementation To maximize the benefits from distributed tracing, several factors must be considered: Automatic instrumentation – Most modern tracing tools support automatic injection of tracing capabilities without significant modifications to the underlying codebase. Some languages or frameworks may not be fully supported in some cases but opt for using automated tooling instead of wasting valuable developer time. Scalable data capture – To deal with massive amounts of tracing data, some tools opt to downsample, which may result in missing or unrepresentative data. Choose tools that can handle the volume and intelligently surface important signals. Integrations – Traces are one part of the observability stack. Traces will be more useful if they can be easily tied to existing logs or metrics for a comprehensive overview. The goal should be to leverage the power of tracing, alongside other signals, to get to actionable insights and proactive solutions rather than collect data for retroactive analysis only. Popular Tools The original infrastructure for supporting internet-scale distributed tracing can be attributed to Dapper, Google's internal tool announced in 2010. Since then, there's been a proliferation of both open-source and enterprise-grade SaaS tools in the market. Open-Source Tools The open-source ecosystem for distributed tracing is fairly mature with a lot of the projects backed by large tech companies. Each tool listed below supports most programming languages and flexible deployment options: Zipkin – one of the oldest and popular tools open-sourced by Twitter Jaeger – a Cloud Native Computing Foundation (CNCF) project donated by Uber that builds on ideas from Dapper and Zipkin OpenTelemetry – an industry-leading observability framework developed by the CNCF that aims to standardize how to instrument and export telemetry data, including traces Commercial Tools If enterprise-grade support is required, commercial tools from hyperscalers and observability platforms are also readily available. The benefit of choosing a commercial tool would be easier integrations with existing tooling and infrastructure. For example, hyperscalers such as AWS and Google provide their own flavor of tracing solutions such as AWS X-Ray and Cloud Trace. Conclusion Distributed tracing, when implemented properly with logs and metrics, can provide tremendous value in surfacing how requests move in a complex, microservices-based system. Traces uncover performance bottlenecks and errors as requests bounce from one service to another, mapping out a global view of the application. As the number of services grows alongside the complexity that follows with it, a good distributed tracing system will become a necessity for any organization looking to upgrade their observability platform. While implementing tracing requires some planning, with a growing number of robust open-source and commercial tools available, organizations can now easily adopt tracing without a significant engineering overhaul. Invest in a good distributed tracing infrastructure to reduce MTTD/MTTR and improve the developer experience at your organization. This is an article from DZone's 2022 Performance and Site Reliability Trend Report.For more: Read the Report

Distributed systems are all the rage these days. Whenever I visit a tech publication on the internet, I usually find a flurry of posts going on about the benefits of distributed systems. Everyone seems to be enthralled with the general idea of distributed systems and the seeming advantages they bring to the table. While there is no harm in creating informational content that can help people learn, I find that many times distributed applications are projected as easy-to-build stuff. The reality, however, is quite different. Building and operating distributed systems is hard. Don't let someone tell you otherwise. The task of creating a distributed system is fraught with challenges. Ironically, many of the challenges stem from the benefits that make these systems attractive in the first place. You should not ignore these challenges when trying to build a distributed system. If you do so, you'd end up in a world of trouble. However, if you embrace these challenges and factor them into your design, you can reap the true benefits of distributed systems. Let's look at these challenges one by one. Communication A distributed system is distributed. Multiple nodes. Probably separated geographically. No distributed system can function without communication between its various nodes. Even the seemingly simple task of browsing a website in your web browser requires a significant amount of communication between different processes. When we visit a URL, our browser contacts the DNS to resolve the server address of the URL. Once it gets hold of the address, it sends an HTTP request to the server over the network. The server processes the request and sends back a response. System designers should ask a few important questions regarding the communication aspects: How are request and response messages represented over the wire? What happens in the case of a network outage? How do we guarantee security from snooping? We can handle many of these communication challenges by falling back on abstractions such as TCP and HTTPS. However, abstractions can leak. For example, TCP attempts to provide a complete abstraction of the underlying unreliable network. But TCP cannot do much if the network cable is cut or overloaded. When this happens, the challenge falls back on the system designer. Coordination Imagine there are two generals with their respective armies. They need to coordinate with each other to attack a city at the same time. Only if they attack at the same time, they can capture the city. For that, they need to agree on the time of the attack. Since the armies are geographically separated, the generals can communicate only by sending a messenger. Unfortunately, the messengers have to go through enemy territory and may get captured. How can the two generals agree about the time of the attack? One of them could propose a time to the other by sending a messenger and waiting for the response. But what if no response arrives? Could the messenger have been captured? Could the messenger be injured and take longer than expected? Should the general send another messenger? The problem is not trivial. No matter how many messengers are sent, neither general can be sure that the other army will attack the city at the correct time. Sending more messengers can increase the chances of successful coordination, but the chances can never reach 100%. In the context of distributed systems, this problem is known as the two-general problem. Generals are like nodes in a distributed system. To make the system work, the nodes should coordinate with each other. However, a node can fail at any time due to a fault. While starting, every developer feels they will build a fault-free system. I used to think the same way. Of course, it was a naive pursuit that was bound to fail. You cannot build a system completely free from faults. Larger the distributed systems, the higher the probability of faults. A fault-tolerant system can continue to operate despite the presence of one or more faults. The trick is to make the nodes in the system coordinate with each other in the presence of failures. Scalability Scalability often commands the greatest attention from system designers when building distributed systems. At a basic level, scalability is a measure of a system’s performance with increasing load. But how do we measure the performance and load of a distributed system? For performance, we can use two excellent parameters: throughput and response time. Throughput denotes the number of operations processed per second. Response time is the total time elapsed between a client request and response. Measurement of system load is more specific to the system use case. For example, system load can be measured as the number of concurrent users, communication links, or the ratio of writes to reads. Performance and load are inherently tied to each other. As the load increases, it will eventually reach the system’s capacity. The capacity may depend on the physical limitations of the system such as: The memory size or the clock cycle of the node Bandwidth and latency of the network links When the load reaches capacity, the system’s performance either plateaus or worsens. If the load on the system continues to grow beyond the capacity, it will eventually hit a point where most operations fail or time out. Throughput goes down and response times rocket upwards. At this point, the system is no longer scalable. See the graph below that shows this situation. How Do We Make the System Scalable? If the load exceeding capacity is the cause of performance degradation, we can make a system scalable by increasing the capacity. A quick and easy way to increase capacity is by buying more expensive hardware with better performance metrics. This approach is known as scaling up. Though it sounds good on paper, this approach will hit a brick wall sooner or later. The more sustainable approach is scaling out by adding more machines to the system. In case you are interested, I also have a more detailed view of scalability in distributed systems. Resiliency Failures are pretty common in a distributed system. There are several reasons: Firstly, scaling out increases the probability of failure. Since every component in the system has an inherent probability of failing, the addition of more pieces compounds the overall chances of failure. Second, more components mean a higher number of operations. This increases the absolute number of failures in the system. Lastly, failures are not independent. Failure of one component increases the probability of failures in other components. When a system is operating at scale, any failure that can happen will eventually happen. An ideal distributed system must embrace failures and behave in a resilient manner. A system is considered resilient when it continues to do its job even when failures occur. We can use several techniques such as redundancy and self-healing mechanisms to increase a system's resiliency. However, this is not a zero-sum game. No distributed system can be 100% resilient. At some point, failures are going to put a dent in the system’s availability. Availability is an important metric. It is the amount of time the application can serve requests divided by the duration of the period measured. Organizations often market their system’s availability in percentage terms using the concept of nines. Three nines (99.9%) are mostly considered acceptable for a large number of systems. Anything above four nines (99.99%) is highly available. An even higher number of nines may be needed for mission-critical systems. Here’s a handy chart that demonstrates the availability percentages in terms of actual downtime. AVAILABILITY % DOWNTIME PER DAY 90% (one nine) 2.40 hours 99% (two nines) 14.40 minutes 99.9% (three nines) 1.44 minutes 99.99% (four nines) 8.64 seconds 99.999% (five nines) 864 milliseconds Operations Of all the other challenges, operating a distributed system is probably the most difficult. But I have also seen the most significant innovations happening in this area. Just like any other software system, distributed systems also need to go through the software development lifecycle of development, testing, deployment, and operations. However, gone are the days when developing a piece of software and operating it used to be done by two separate teams or departments. The complexity of distributed systems has given rise to DevOps. Now, the same team that designs and develops a system is expected to operate it in a production environment. This creates several difficulties for the development team that they ignored earlier. New deployments need to be rolled out continuously in a safe manner. Systems need to be observable. Alerts need to be fired when service-level objectives are at risk of being breached. There is a positive side to this change as well. There is no better way for developers to find out where a system is falling short than by providing on-call support for it. Just make sure to get properly paid for ruining your weekends! Conclusion Building distributed systems is not an easy task. It is fraught with multiple challenges that revolve around the areas of communication, coordination, scalability, resiliency, and operations. But solving these challenges is rewarding. When a distributed system works as per plan, it creates a special dance of components that is wonderful to watch and admire. Have you seen this dance? What challenges did you overcome to reach the point where your system worked according to plan? Do share your thoughts in the comments section.

In this seventh installment of the series covering my journey into the world of cloud-native observability, I'm continuing to explore an open-source project called Perses. After laying out the groundwork for this series in the initial article, I spent some time in the second article sharing who the observability players are. I also discussed the teams that these players are on in this world of cloud-native o11y. For the third article, I looked at the ongoing discussion around monitoring pillars versus phases. In the fourth article, I talked about keeping your options open with open-source standards. In my last installment, the fifth article in this series, I talked about bringing monolithic applications into the cloud-native o11y world. In my sixth and last article, I provided you with an introduction to a new open-source dashboard and visualization project and shared how to install the project on your local developer machine. Being a developer from my early days in IT, it's been very interesting to explore the complexities of cloud-native o11y. Monitoring applications goes way beyond just writing and deploying code, especially in the cloud-native world. One thing remains the same: maintaining your organization's architecture always requires both a vigilant outlook and an understanding of available open standards. In this seventh article, we'll be exploring the Perses application programming interface (API) and tooling. This is a preview of the third lab developed for my hands-on workshop dedicated to exploring dashboards and visualization. This article is part of my ongoing effort to get practical hands-on experience in the cloud-native o11y world. Today you'll be learning about the available tooling in the Perses project and how its API works on your local development machine. This is one lab in the more extensive free online workshop. Feel free to start from the very beginning of this workshop here if you missed anything previously. Now let's dive into the Perses project API and tooling lab. Note this article is only a short summary, so please see the complete lab found online (Lab 3) to work through the API and tooling yourself: The following is a short overview of what is in this specific lab of the Perses workshop. API and Tooling This lab explores the Perses API and how to use the Perses command line tooling to gain insights into the currently available projects, dashboards, and data sources. After installing Perses in the previous lab, you have the dashboard open in your browser. There is an application programming interface (API) available for you to gain insights into the configuration applied to your Perses server. You can find out more about the projects, dashboards, and data sources that have been created. This can be handy to start with existing resources and modify them for your own versions. This is how we get started exploring some of the provided API operations for handling a Perses resource, which can be one of the following: Project Dashboard Datasource There are five API operations that can be applied to any resource, each providing us the ability to manipulate resources: Creating a resource Updating a resource Deleting a resource Retrieving a single resource Retrieving a list of resources In this lab, I share how to leverage this API from a browser and from an API client so that the raw browser output is then formatted in a nicely readable format. This section of the lab ends with a pointer to the official Perses documentation. Next up, this lab dives into the tooling, which is the Perses command line executable known as the percli. If you built the project from the source code, you have built this executable, but if you are using the container image installation then you will find an executable percli in the Perses Easy Install project in the support directory at support/bin/percli. The rest of this lab is dedicated to exploring how to use this tooling and we explore a few of the interesting commands: login: Log in to an instance of the Perses API. get: Request a response from the Perses API. project: Select a project to be used as the default. describe: Request details for a specific resource. delete: Delete a specific resource. apply: Create or update existing resources using JSON or YAML file. You learn by using each and every one of these commands through the percli tooling and are provided with both the before and after views of your system as you work through it all. You're now one step closer to creating your own dashboards and visualizations! More To Come Next up, I take you farther into the Perses project with more workshop materials to share. Stay tuned for more insights into a real practical experience as my cloud-native o11y journey continues.

Most existing applications/projects are moving to cloud platforms/on-premise using microservices architectures. Assessment of the existing application is critical concerning application complexity and risk. An appropriate strategy has to define to analyze the current application to determine the risk and complexity of the existing application landscape. The assessment considers factors like overall architecture, presentation layer, business layer, data layer, security, deployment, technology, infrastructure, performance, and monitoring for existing applications. Collect the information from the different parameters and decide which is the best option for migration strategies like rehost, refactor, rearchitect, and rebuild. For each parameter, the architect fills in the information by studying the existing application/collecting the information from the application stack holders. Application assessment helps the organization to understand the following: Understanding of the existing application architecture and critical findings. Integration points across applications (upstream and downstream). Infrastructure and software usage. Pain areas related to business users. Road map to target system platform using any one of the migration strategies. There are a number of assessment criteria that information will need to be gathered for to provide an assessment of the applications as below: Business and technical value contributions like; reliability, scalability, and flexibility. Risk exposures like; strategic, operations, and technology risks. Cost of operations with respect to application, project, and business. The attractiveness of system, business, and delivery. Each and every application assessment team create a template to prepare and identify the appropriate questionnaire related to the above criteria. Based on the assessment report, the application owner can decide to migrate the application to the cloud/on-premise. The below diagram shows a very high-level process flow for application assessment. The below table describes questions for each of the assessment parameters listed above. The application architect's score must be based on his experience. For each and every question, the architect has to assign some score and finally decide whether the assessment application can be moved to the target architecture or not. The application architect has to answer the below questions before proposing the target architecture in the assessment phase. Assessment Parameter Questions to Ask as Part of the Assessment for Migration Application Architecture Is it distributed architecture? Follow the layered architectural model. Loosely coupled. Required level of abstraction without any vendor lock-in at any layer. Appropriate integration techniques between different systems for synchronous and asynchronous operations. Does the architecture follow SOA / microservices design principles? Presentation Layer GUI screens are more appealing in look and feel and easy to navigate with context-sensitive help enabled. The website is a portal based on standard products with out-of-box portal features. GUI design is based on templates such as Apache tiles which can be applied across the GUI screens for faster screen building and roll-out. GUI uses reusable render components. Supports cross-browser compatibility. Supports mobile-enabled services? View exercises any caching technique to cache the static content to improve performance. Business Layer Is the business layer loosely coupled with the presentation layer? Does it use any separation of concerns and loose coupling? Does the business layer invoke any remote objects / APIs running in a distributed environment? What is the mode of invoking? Does the business layer use any workflow? Does the business layer integrate with external systems? What is the mode of integration? Does the business layer consider the network latency and bandwidth while invoking remote objects? Does the business layer use any ESB for service orchestration or message transformation? Data Layer Does the architecture use any DAO pattern? Does this layer leverage the DB connection pool out-of-box feature of the application server? Does this layer use the ORM tool to persist Java objects? Does this layer use application server transaction management? Does this layer use JPA for portability? Does the application use Data base per business / single database for all microservices? Is there any downtime while deploying application changes to the production environment? Security Does the system enable a Single sign-on capability? Does the wire communication enable SSL encryption? Does user role administration control by a separate administration module? Does the system ensure Role-based access to various screens and menus of the system? Deployment Are firewalls protected? Does the system use a load balancer? Do the application servers launch in a cluster configuration? Does the system deploy in a container? Please elaborate The conclusion is that the questionnaire is not limited to the above. The questions can be extended, and assign values for each question to arrive appropriate assessment of the existing application. Then, based on the report, the application owner can make proper migration decisions. Finally, provide the application migration conclusions and recommendations as part of the migration road map document.

PHP is a popular programming language used for developing web applications. However, as with any programming language, it is important to write code that is optimized for better performance. In this article, we will explore some tips and best practices for optimizing PHP code to improve the performance of your web application. 1. Use PHP’s Built-In Functions and Libraries PHP has a number of built-in functions and libraries that can be used to perform common tasks, such as string manipulation, array manipulation, and database interactions. Using these functions and libraries can often be more efficient than writing your own custom code to perform these tasks. For example, consider the following code snippet that uses a custom function to reverse a string: PHP function reverseString($str) { $reversed = ""; for ($i = strlen($str) - 1; $i >= 0; $i--) { $reversed .= $str[$i]; } return $reversed; } This function works as expected, but it is not very efficient. A more efficient way to reverse a string in PHP is to use the strrev() function, which is built into PHP: PHP $reversed = strrev($str); Using built-in functions and libraries can also help reduce the size of your codebase, making it easier to maintain and debug. 2. Use Prepared Statements for Database Interactions When interacting with a database in PHP, it is important to use prepared statements to protect against SQL injection attacks and improve performance. Prepared statements allow you to separate the SQL query from the values that are passed to it, which can help prevent SQL injection attacks and can also improve performance by allowing the database server to optimize the query. Here is an example of how to use a prepared statement in PHP: PHP $pdo = new PDO("mysql:host=localhost;dbname=testdb", "username", "password"); $stmt = $pdo->prepare("SELECT * FROM users WHERE username = :username AND password = :password"); $stmt->bindValue(":username", $username); $stmt->bindValue(":password", $password); $stmt->execute(); 3. Use a PHP Accelerator A PHP accelerator is a tool that speeds up the execution of PHP scripts by caching the compiled bytecode of PHP files. This can lead to significant performance improvements, particularly for applications that rely on a large number of PHP files or have a high volume of traffic. There are several popular PHP accelerators available, including APC (Alternative PHP Cache), XCache, and OPcache (which is built into PHP 5.5 and later). 4. Enable Output Buffering Output buffering is a feature of PHP that allows you to buffer the output of a script and send it to the browser all at once, rather than sending it piece by piece as it is generated. This can be useful for optimizing the performance of scripts that generate a large amount of output, as it reduces the number of times the script has to send data to the browser. To enable output buffering, you can use the ob_start() function at the beginning of your script and the ob_end_flush() function at the end: PHP ob_start(); // script code goes here ob_end_flush(); 5. Use a Content Delivery Network (CDN) A content delivery network (CDN) is a network of servers that are used to deliver web content to users based on their geographic location. Using a CDN can improve the performance of your web application by reducing the distance that data has to travel between the server and the user, as well as distributing the load across multiple servers. To use a CDN with your PHP application, you can store static assets (such as images, JavaScript files, and CSS files) on the CDN and reference them in your HTML code. For example, instead of linking to an image on your server like this: HTML <img src="/images/logo.png"> You can link to the image on the CDN like this: HTML <img src="https://cdn.example.com/images/logo.png"> There are many CDN providers to choose from, including Cloudflare, Akamai, and Amazon Web Services (AWS). 6. Use an Opcode Cache An opcode cache is a tool that speeds up the execution of PHP scripts by caching the compiled version of the PHP code (the opcode). This can lead to significant performance improvements, particularly for applications that have a large number of PHP files or that are accessed frequently.One popular opcode cache for PHP is OPcache, which is built into PHP 5.5 and later. To enable OPcache, you can add the following line to your php.ini file: PHP zend_extension=opcache.so 7. Use a PHP Framework A PHP framework is a collection of libraries and tools that provide a set of conventions and best practices for developing PHP applications. Using a framework can help you write more organized, maintainable, and scalable code, and many frameworks also include built-in support for caching and other performance optimizations. Some popular PHP frameworks include Laravel, CodeIgniter, and Symfony. 8. Optimize Your Database Schema and Queries The performance of your PHP application can also be affected by the design of your database schema and the efficiency of your SQL queries. Here are a few tips for optimizing your database: Use appropriate data types for each column. Use indexes to improve the performance of queries that filter or sort data. Avoid using SELECT in your queries, and instead, specify only the columns that you need. Use EXPLAIN to analyze the performance of SELECT queries and identify any potential issues. 9. Use a Profiler To Identify Performance Bottlenecks A profiler is a tool that allows you to analyze the performance of your PHP code and identify any bottlenecks or areas for improvement. There are several profilers available for PHP, including Xdebug and Blackfire. To use a profiler, you will need to install and configure it, and then run it against your PHP code. The profiler will generate a report that shows you details about the performance of your code, including the time taken to execute each function and the number of times each function was called. 10. Use a Load Balancer A load balancer is a tool that distributes incoming traffic across multiple servers in order to improve the performance and availability of a web application. Using a load balancer can help to scale your application and improve its overall performance, particularly if you have a high volume of traffic or if your application is running on a single server that is unable to handle the load. There are many load balancer options available, including hardware load balancers and software load balancers. Some popular software load balancers for PHP applications include Nginx and HAProxy. Conclusion In this article, we have explored several tips and best practices for optimizing PHP code for better performance. By following these guidelines, you can improve the performance of your PHP application and provide a better experience for your users.

Building performant services and systems is at the core of every business. Tons of technologies emerge daily, promising capabilities that help you surpass your performance benchmarks. However, production environments are chaotic landscapes that exact a heavy performance toll when not maintained and monitored. Although Kubernetes is the defacto choice for container orchestration, many organizations fail to implement it. Growing organizations, in the process of upscaling their services, unintentionally introduce complexities into the system. Knowing how the infrastructure is set up and how clusters operate and communicate are crucial. Most of the infrastructural setup is tranched into a network of systems to communicate and share the workloads. If only we could visually see how the systems are connected and the underlying factors. Mapping the network using an efficient tool for visualization and assessment is essential for monitoring and maintaining services. Introduction To Visual Network Mapping Network mapping is the process of identifying and cataloging all the devices and connections within a network. A visual network map is a graphical representation of the network that displays the devices and the links between them. Visual network maps can provide a comprehensive understanding of a network's topology and identify potential problems or bottlenecks, allowing for modifications and expansion plans that can significantly improve troubleshooting, planning, analysis, and monitoring. Open-source security tools, such as OpenVAS, Nmap, and Nessus, can be used to conduct network mapping and generate visual network maps. These tools are freely available, making them a cost-effective option for organizations looking to improve their network security. Furthermore, many open-source security tools also offer active community support, enabling users to share knowledge, tips, and best practices for using the tool to its full potential. Benefits of Using Visual Network Maps An effective tool for planning and developing new networks, expanding or modernizing existing networks, and analyzing network problems or issues is a visual network map. A proper setup of visual network maps can exponentially augment the monitoring, tracking, and remediation capabilities. It can give you a clear and complete picture of the network, enabling you to pinpoint the issue’s potential source and resolve it then and there, or it can assist you in real-time network monitoring and notify you of any changes or problems beforehand. Introduction to Caretta and Grafana Caretta is an open-source network visualization and monitoring tool that enables real-time network viewing and monitoring. Grafana is an open-source data visualization and monitoring platform that enables you to create customized dashboards and alerts as well as examine and analyze data. An effective solution for comprehending and managing your network can be created by combining Caretta and Grafana. How Caretta Uses eBPF and Grafana Caretta’s reason for existence is to help you understand the topology and the relationships between devices in distributed environments. It offers various capabilities such as device discovery, real-time monitoring, alerts, notifications, and reporting. It uses Victoria Metrics to gather and publish its metrics, and any Prometheus-compatible dashboard can use the results. Carreta makes it possible to accept typical control-plane node annotations by enabling tolerations. It gathers network information, such as device and connection details, using the eBPF (extended Berkeley Packet Filter) kernel functionality and then uses the Grafana platform to present the information in a visual map. Grafana’s Role in Visualizing Caretta’s Network Maps Grafana is designed to be a modular and flexible tool that integrates and onboards a wide range of data sources and custom applications with simplicity. Due to its customizable capabilities, you can modify how the network map is presented using the Grafana dashboard. Additionally, you can pick from several visualization options to present the gathered data in an understandable and helpful way. Grafana is crucial for both showing the network data that Caretta has gathered and giving users a complete picture of the network. Using Caretta and Grafana To Create a Visual Network Map To use Caretta and Grafana for creating a visual network map, you must set up, incorporate, and configure them. The main configuration item is the Caretta daemonset. You must deploy the Caretta daemonset to the cluster of choice that will collect the network metrics into a database and set up the Grafana data source to point to the Caretta database to see the network map. Prerequisites and Requirements for Using Caretta and Grafana Caretta is a modern tool integrated with advanced features. It relies on Linux kernel version >= 4.16 and x64 bit system helm chart. Let's dive in and see how to install and configure this brilliant tool combination. Steps for Installing and Configuring Caretta and Grafana With an already pre-configured helm chart, installing Caretta is just a few calls away. The recommendation is to install Caretta in a new, unique namespace. helm repo add groundcover https://helm.groundcover.com/ helm repo update helm install caretta --namespace caretta --create-namespace groundcover/caretta The same can be applied to installing Grafana. helm install --name my-grafana --set "adminPassword=secret" \n --namespace monitoring -f custom-values.yaml stable/grafana Our custom-values.yaml will look something like below: ## Grafana configuration grafana.ini: ## server server: protocol: http http_addr: 0.0.0.0 http_port: 3000 domain: grafana.local ## security security: admin_user: admin admin_password: password login_remember_days: 1 cookie_username: grafana_admin cookie_remember_name: grafana_admin secret_key: hidden ## database database: type: rds host: mydb.us-west-2.rds.amazonaws.com ## session session: provider: memory provider_config: "" cookie_name: grafana_session cookie_secure: true session_life_time: 600 ## Grafana data persistence: enabled: true storageClass: "-" accessModes: - ReadWriteOnce size: 1Gi Configuration You can configure Caretta using helm values. Values in Helm are a chart’s setup choices. When the chart is installed, you can change the values listed in a file called values.yaml, which is part of the chart package, and customize the configurations based on the requirement at hand. An example of configuration overwriting default values is shown below: pollIntervalSeconds: 15 # set metrics polling interval tolerations: # set any desired tolerations - key: node-role.kubernetes.io/control-plane operator: Exists effect: NoSchedule config: customSetting1: custom-value1 customSetting2: custom-value2 victoria-metrics-single: server: persistentVolume: enabled: true # set to true to use persistent volume ebpf: enabled: true # set to true to enable eBPF config: someOption: ebpf_options The pollIntervalSeconds sets the interval at which metrics are polled. In our case, we have set it to poll every 15 seconds. The tolerations section allows specifying tolerations for the pods. In the shown example, pods are allowed only to run on nodes that have the node-role.kubernetes.io/control-plane label and exist with the effect NoSchedule. The config section allows us to specify custom configuration options for the application. The victoria-metrics-single section allows us to configure the Victoria-metrics-single server. Here, it is configuring the persistent volume as enabled. The eBPF section allows us to enable eBPF and configure its options. Creating a Visual Network Map With Caretta and Grafana Caretta consists of two parts: the “Caretta Agent” and the “Caretta Server.” Every node in the cluster runs the Caretta Agent Kubernetes DaemonSet, which collects information about the cluster’s status. You will need to include the data gathered by Caretta in Grafana in order to view it as a network map and generate a visual network map. apiVersion: apps/v1 kind: DaemonSet metadata: name: caretta-depoy-test namespace: caretta-depoy-test spec: selector: matchLabels: app: caretta-depoy-test template: metadata: labels: app: caretta-depoy-test spec: containers: - name: caretta-depoy-test image: groundcover/caretta:latest command: ["/caretta"] args: ["-c", "/caretta/caretta.yaml"] volumeMounts: - name: config-volume mountPath: /caretta volumes: - name: config-volume configMap: name: caretta-config Data from the Caretta Agent is received by the Caretta Server, a Kubernetes StatefulSet, which then saves it in a database. apiVersion: apps/v1 kind: StatefulSet metadata: name: caretta-depoy-test labels: app: caretta-depoy-test spec: serviceName: caretta-depoy-test replicas: 1 selector: matchLabels: app: caretta-depoy-test template: metadata: labels: app: caretta-depoy-test spec: containers: - name: caretta-depoy-test image: groundcover/caretta:latest env: - name: DATABASE_URL value: mydb.us-west-2.rds.amazonaws.com ports: - containerPort: 80 name: http To accomplish this, you will need to create a custom data source plugin in Grafana to connect to Caretta’s data and then develop visualizations in Grafana to show that data. [datasources] [datasources.caretta] name = caretta-deploy-test type = rds url = mydb.us-west-2.rds.amazonaws.com access = proxy isDefault = true Customization Options for the Network Map and How to Access Them The network map that Caretta and Grafana produced can be customized in a variety of ways. We can customize the following: Display options: With display customization options, you have control over the layout of the map, the thickness, and the color of the connections and devices. Data options: With data options, you may select which information, including warnings, performance metrics, and details about your device and connection, is shown on the map. Alerting options: With alerting options, you can be informed of any network changes or problems, such as heavy traffic, sluggish performance, or connectivity problems. Visualization options: With visualization options, you can present the gathered data in an understandable and useful way. Usually, you’ll need to use the Grafana dashboard to access these and other customization options. Depending on the version of Caretta and Grafana you are running and your particular setup and needs, you will have access to different options and settings. Interpreting and Using the Visual Network Map The primary goals of a visual network map made with Caretta and Grafana are aiding in network topology comprehension, the identification of possible bottlenecks or problems, and the planning and troubleshooting of network problems. You must comprehend the various components of the map and what they stand for in order to interpret and use the visual network map. Some of the types of information that may be displayed on the map are: Devices: The network’s endpoints, including servers, switches, and routers, are presented on the map. Connections: The connections between devices, such as network cables, wireless connectivity, or virtual connections, and sometime the connectivity type may be depicted on the map. Data: Performance indicators, alarms, and configuration information will be displayed on the maps. Tips for Using the Network Map To Assess Performance in Your K8s Cluster Creating a curated, informative and scalable network map is more challenging than it sounds. But with a proper tool set, this becomes manageable. We have seen what we can accomplish using Caretta and Grafana together. Now, let's see what we need to consider for using network maps that showcase the performance metrics of your Kubernetes clusters. First and foremost, understand the network topology of the cluster, including the physical and virtual networks that your services run on. Next, ensure that the network plugin that you are using is compatible with your application. Finally, define network policies to secure communication between pods, control ingress, and egress traffic, monitor, and troubleshoot. Understand pod-to-pod communication and pod networking is happening. Conclusion Breaking down large systems into microservices, making systems distributed, and orchestrating them is the most followed approach to boost performance and uptime. Kubernetes and Docker are the market leaders here. As performant as it is, observability is a concern in large-scale distributed systems. We need to consider all the influencing outliers and anomalies to monitor and enhance the overall system with optimal performance in mind. New technologies make innovations and advancements easy but introduce unknown impediments to the system. You need an observability tool that can track all the network operations and present them in an efficient and informative way. Grafana is the leading tool in the monitoring space. By combining Caratta, an open-source network visualization, and monitoring tool, with Grafana, we can unlock the true value of our infrastructure.

This is an article from DZone's 2022 Performance and Site Reliability Trend Report.For more: Read the Report Software testing is straightforward — every input => known output. However, historically, a great deal of testing has been guesswork. We create user journeys, estimate load and think time, run tests, and compare the current result with the baseline. If we don't spot regressions, the build gets a thumbs up, and we move on. If there is a regression, back it goes. Most times, we already know the output even though it needs to be better defined — less ambiguous with clear boundaries of where a regression falls. Here is where machine learning (ML) systems and predictive analytics enter: to end ambiguity. After tests finish, performance engineers do more than look at the result averages and means; they will look at percentages. For example, 10 percent of the slowest requests are caused by a system bug that creates a condition that always impacts speed. We could manually correlate the properties available in the data; nevertheless, ML will link data properties quicker than you probably would. After determining the conditions that caused the 10 percent of bad requests, performance engineers can build test scenarios to reproduce the behavior. Running the test before and after the fix will assert that it's corrected. Figure 1: Overall confidence in performance metrics Source: Data from TechBeacon Performance With Machine Learning and Data Science Machine learning helps software development evolve, making technology sturdier and better to meet users' needs in different domains and industries. We can expose cause-effect patterns by feeding data from the pipeline and environments into deep learning algorithms. Predictive analytics algorithms, paired with performance engineering methodologies, allow more efficient and faster throughput, offering insight into how end users will use the software in the wild and helping you reduce the probability of defects reaching production. By identifying issues and their causes early, you can make course corrections early in the development lifecycle and prevent an impact on production. You can draw on predictive analytics to improve your application performance in the following ways. Identify root causes. You can focus on other areas needing attention using machine learning techniques to identify root causes for availability or performance problems. Predictive analytics can then analyze each cluster's various features, providing insights into the changes we need to make to reach the ideal performance and avoid bottlenecks. Monitor application health. Performing real-time application monitoring using machine-learning techniques allows organizations to catch and respond to degradation promptly. Most applications rely on multiple services to get the complete application's status; predictive analytics models will correlate and analyze the data when the application is healthy to identify whether incoming data is an outlier. Predict user load. We have relied on peak user traffic to size our infrastructure for the number of users accessing the application in the future. This approach has limitations as it does not consider changes or other unknown factors. Predictive analytics can help indicate the user load and better prepare to handle it, helping teams plan their infrastructure requirements and capacity utilization. Predict outages before it's too late. Predicting application downtime or outages before they happen helps to take preventive action. The predictive analytics model will follow the previous outage breadcrumbs and continue monitoring for similar circumstances to predict future failures. Stop looking at thresholds and start analyzing data. Observability and monitoring generate large amounts of data that can take up to several hundred megabytes a week. Even with modern analytic tools, you must know what you're looking for in advance. This leads to teams not looking directly at the data but instead setting thresholds as triggers for action. Even mature teams look for exceptions rather than diving into their data. To mitigate this, we integrate models with the available data sources. The models will then sift through the data and calculate the thresholds over time. Using this technique, where models are fed and aggregate historical data, provides thresholds based on seasonality rather than set by humans. Algorithm-set thresholds trigger fewer alerts; however, these are far more actionable and valuable. Analyze and correlate across datasets. Your data is mostly time series, making it easier to look at a single variable over time. Many trends come from the interactions of multiple measures. For example, response time may drop only when various transactions are made simultaneously with the same target. For a human, that's almost impossible, but properly trained algorithms will spot these correlations. The Importance of Data in Predictive Analytics "Big data" often refers to datasets that are, well, big, come in at a fast pace, and are highly variable in content. Their analysis requires specialized methods so that we can extract patterns and information from them. Recently, improvements in storage, processors, parallelization of processes, and algorithm design enabled the processing of large quantities of data in a reasonable time, allowing wider use of these methods. And to get meaningful results, you must ensure the data is consistent. For example, each project must use the same ranking system, so if one project uses 1 as critical and another uses 5 — like when people use DEFCON 5 when they mean DEFCON 1 — the values must be normalized before processing. Predictive algorithms are composed of the algorithm and the data it's fed, and software development generates immense amounts of data that, until recently, sat idle, waiting to be deleted. However, predictive analytics algorithms can process those files, for patterns we can't detect, to ask and answer questions based on that data, such as: Are we wasting time testing scenarios that aren't used? How do performance improvements correlate with user happiness? How long will it take to fix a specific defect? These questions and their answers are what predictive analytics is used for — to better understand what is likely to happen. The Algorithms The other main component in predictive analysis is the algorithm; you'll want to select or implement it carefully. Starting simple is vital as models tend to grow in complexity, becoming more sensitive to changes in the input data and distorting predictions. They can solve two categories of problems: classification and regression (see Figure 2). Classification is used to forecast the result of a set by classifying it into categories starting by trying to infer labels from the input data like "down" or "up." Regression is used to forecast the result of a set when the output variable is a set of real values. It will process input data to predict, for example, the amount of memory used, the lines of code written by a developer, etc. The most used prediction models are neural networks, decision trees, and linear and logistic regression. Figure 2: Classification vs. regression Neural Networks Neural networks learn by example and solve problems using historical and present data to forecast future values. Their architecture allows them to identify intricate relations lurking in the data in a way that replicates how our brain detects patterns. They contain many layers that accept data, compute predictions, and provide output as a single prediction. Decision Trees A decision tree is an analytics method that presents the results in a series of if/then choices to forecast specific options' potential risks and benefits. It can solve all classification problems and answer complex issues. As shown in Figure 3, decision trees resemble an upsidedown tree produced by algorithms identifying various ways of splitting data into branch-like segments that illustrate a future decision and help to identify the decision path. One branch in the tree might be users who abandoned the cart if it took more than three seconds to load. Below that one, another branch might indicate whether they identify as female. A "yes" answer would raise the risk as analytics show that females are more prone to impulse buys, and the delay creates a pause for pondering. Figure 3: Decision tree example Linear and Logistic Regression Regression is one of the most popular statistical methods. It is crucial when estimating numerical numbers, such as how many resources per service we will need to add during Black Friday. Many regression algorithms are designed to estimate the relationship among variables, finding key patterns in big and mixed datasets and how they relate. It ranges from simple linear regression models, which calculate a straight-line function that fits the data, to logistic regression, which calculates a curve (Figure 4). OVERVIEW OF LINEAR AND LOGISTIC REGRESSION Linear Regression Logistic Regression Used to define a value on a continuous range, such as the risk of user traffic peaks in the following months. It's a statistical method where the parameters are predicted based on older sets. It best suits binary classification: datasets where y = 0 or 1, where 1 represents the default class. Its name derives from its transformation function being a logistic function. It's expressed as y = a + bx, where x is an input set used to determine the output y. Coefficients a and b are used to quantify the relation between x and y, where a is the intercept and b is the slope of the line. It's expressed by the logistic function:where β0 is the intercept and β1 is the rate. It uses training data to calculate the coefficients, minimizing the error between the predicted and actual outcomes. The goal is to fit a line nearest to most points, reducing the distance or error between y and the line. It forms an S-shaped curve where a threshold is applied to transform the probability into a binary classification. Figure 4: Linear regression vs. logistic regression These are supervised learning methods, as the algorithm solves for a specific property. Unsupervised learning is used when you don't have a particular outcome in mind but want to identify possible patterns or trends. In this case, the model will analyze as many combinations of features as possible to find correlations from which humans can act. Figure 5: Supervised vs. unsupervised learning Shifting Left in Performance Engineering Using the previous algorithms to gauge consumer sentiment on products and applications makes performance engineering more consumer-centric. After all the information is collected, it must be stored and analyzed through appropriate tools and algorithms. This data can include error logs, test cases, test results, production incidents, application log files, project documentation, event logs, tracing, and more. We can then apply it to the data to get various insights to: Analyze defects in environments Estimate the impact on customer experience Identify issue patterns Create more accurate test scenarios, and much more This technique supports the shift-left approach in quality, allowing you to predict how long it will take to do performance testing, how many defects you are likely to identify, and how many defects might make it to production, achieving better coverage from performance tests and creating realistic user journeys. Issues such as usability, compatibility, performance, and security are prevented and corrected without impacting users. Here are some examples of information that will improve quality: Type of defect In what phase was the defect identified What the root cause of the defect is Whether the defect is reproducible Once you understand this, you can make changes and create tests to prevent similar issues sooner. Conclusion Software engineers have made hundreds and thousands of assumptions since the dawn of programming. But digital users are now more aware and have a lower tolerance for bugs and failures. Businesses are also competing to deliver a more engaging and flawless user experience through tailored services and complex software that is becoming more difficult to test. Today, everything needs to work seamlessly and support all popular browsers, mobile devices, and apps. A crash of even a few minutes can cause a loss of thousands or millions of dollars. To prevent issues, teams must incorporate observability solutions and user experience throughout the software lifecycle. Managing the quality and performance of complex systems requires more than simply executing test cases and running load tests. Trends help you tell if a situation is under control, getting better, or worsening — and how fast it improves or worsens. Machine learning techniques can help predict performance problems, allowing teams to course correct. To quote Benjamin Franklin, "An ounce of prevention is worth a pound of cure." This is an article from DZone's 2022 Performance and Site Reliability Trend Report.For more: Read the Report

This is an article from DZone's 2022 Performance and Site Reliability Trend Report.For more: Read the Report In the past few years, the complexity of systems architectures drastically increased, especially in distributed, microservices-based architectures. It is extremely hard and, in most cases, inefficient to debug and watch logs, particularly when we have hundreds or even thousands of microservices or modules. In this article, I will describe what observability and monitoring systems, the patterns of a good observability platform, and the observability subsystem may look like. Observability vs. Monitoring Before we jump directly to the point, let's describe what observability is, what components it includes, and how it differs from monitoring. Observability allows us to have a clear overview of what happens in the system without knowing the details or domain model. Moreover, observability lets us efficiently provide information about: The overall system, separate service failures, and outages The behavior of the general system and services The overall security and alerts We know what functions should cover the observability system. Below we can see what information should be gathered to properly design an observability and monitoring platform. Metrics – Data collection allows us to understand the application and infrastructure states — for example, latency and the usage of CPU, memory, and storage. Distributed traces – Allows us to investigate the event or issue flow from one service to another Logs – This is a message with a timestamp that contains information about application- or service-level errors, exceptions, and information. Alerting – When an outage occurs or something goes wrong with one or several services, alerts notify these problems via emails, SMS, chats, or calls to operators. This allows for quick action to fix the issue. Availability – Ensures that all services are up and running. The monitoring platform sends prob messages to some service or component (to the HTTP API endpoint) to check if it responds. If not, then the observability system generates an alert (see the bullet point for alerting). Also, some observability and monitoring platforms may include user experience monitoring, such as heat maps and user action recording. Observability and monitoring follow the same principles and patterns and rely primarily on toolsets, so in my opinion, the differentiation between the two is made for marketing purposes. There is no clear definition of how observability differs from monitoring; all definitions are different and high-level. Observability Patterns All complex systems based on microservices have recommendations and patterns. This allows us to build a reliable system without reinventing the wheel. Observability systems also have some essential patterns. The following sections discuss five of the most important patterns. Log Aggregation Pattern In distributed systems, logging can be difficult. Each microservice can produce a lot of logs, and it can be a nightmare to find and analyze errors or other log messages of each microservice. Therefore, the log aggregation pattern helps us here. It contains the central log aggregation service as a central log storage. Also, the service provides options to label, index, categorize, search, and analyze all logs. There are a few examples of log aggregation platforms like Grafana Loki, Splunk, Fluentd, and the ELK stack. Figure 1: Log aggregation pattern Health Check Pattern Imagine you have multiple services or microservices, and you need to know their current state. Of course, you can go to the logging aggregation service and check logs. But services may not produce logs in a starting state. Also, it may be the case where logging in is unavailable when the services fail. In all these instances, you need to implement health check patterns. You just need to create a health (or ping) endpoint in your service and point your log aggregation system to check and collect the checks of each service. You can also set up notifications or alerting when the service is unavailable — it saves a lot of time recognizing what service failed to start or went down. Figure 2: Health check pattern Distributed Tracing Pattern Imagine this scenario: you have multiple components, modules, and libraries in one or several microservices. You need to check the whole history of component execution or send the request to one microservice, and you need to check the execution history from one service component list to another. To do this, you need to have some distributed system that will collect and analyze all tracing data. Some open-source services allow you to do so, such as Jaeger, OpenTelemetry, and OpenCensus. Check out the Istio documentation for an example that demonstrates distributed tracing in action. Figure 3: Distributed tracing pattern Application Metrics Pattern Having distributed logging and tracing is essential; however, without application metrics, your observability system will not be complete. You may need to collect infra- and application-level metrics, such as: CPU Memory Disc use Services requests/response time Latency Collecting these metrics will not only help you understand what infrastructure size you need but will also help you save money on cloud providers. It also helps you to quickly mitigate outages caused by a lack of CPU or memory resources. Below is an example of the service that has a proxy agent. The proxy agent aggregates and sends telemetry data to the observability platform. Figure 4: Application metrics pattern Service Mesh for Observability A service mesh not only provides a central management control plane for microservices architecture but also provides a single observability subsystem. Instead of installing a separate tool for gathering metrics, distributed traces, and logs, we can just use one. For example, Azure provides an integrated service mesh add-on that can be set up in a minute. There is also an option to use Istio service mesh, which contains all features required for a proper observability subsystem. Moreover, it can gather metrics, logs, and traces for the control plane. For example, when we set up Grafana, Loki, or other tools, we also need to enable observability for them, as they may also fail while working or during the deployment process; therefore, we need to troubleshoot. Figure 5: Service mesh as observability Observability Architecture for Microservices As an example of the observability architecture, I'm going to use a smart heating system. Smart heating is an essential part of each home (or even smart home) that allows owners to: Manually manage heating in the apartment with an application. Automatically adjust heating depending on time and the temperature outside and inside. In addition, the system can do the following actions to help the owner: Turn on/off the heating when people are about to arrive at the apartment. Notify, alert, or just ask if something requires human attention or if something is wrong. Figure 6: Microservices architecture with an observability subsystem In Figure 6, you can see an architecture that is based on the microservices pattern, since it serves best and represents all system components. It contains main and observability subsystems. Each microservice is based on Azure Functions and deployed to the Azure Kubernetes Cluster. We deploy functions to Kubernetes using the KEDA framework. KEDA is an open-source, Kubernetes-based event autoscaling that allows us to automatically deploy and scale our microservices functions. Also, KEDA provides the tools to wrap functions to the Docker containers. We can also deploy microservices functions directly without KEDA and Kubernetes if we don't have a massive load and don't need the scaling options. The architecture contains the following components that represent the main subsystem: Azure operating as a microservice Azure Service Bus (or Azure IoT Hub) as a central messaging bus that microservices use to communicate Azure API Apps providing an API for mobile/desktop applications The essential part here is an observability subsystem. A variety of components and tools represent it. I've described all components in Table 1 below: COMPONENTS OF THE OBSERVABILITY SYSTEM Tool Description Prometheus Prometheus is an open-source framework to collect and store logs and telemetry as time series data. Also, it provides alerting logic. Prometheus proxy or sidecar integrates with each microservice to collect all logs, telemetry, and tracing data. Grafana Loki Grafana Loki is an open-source distributed log aggregation service. It's based on a labeling algorithm. It's not indexing the logs; rather, it's assigning labels to each log domain, subsystem, or category. Jaeger Jaeger is an open-source framework for distributed tracing in microservices-based systems. It also provides search and data visualization options. Some of the high-level use cases of Jaeger include: Performance and latency optimization Distributed transaction monitoring Service dependency analysis Distributed context propagation Root cause analysis Grafana (Azure Managed Grafana) Grafana is also an open-source data visualization and analytics system. It allows the collection of traces, logs, and other telemetry data from different sources. We are using Grafana as a primary UI "control plane" to build and visualize data dashboards that will come from Prometheus, Loki, and Grafana Loki sources. Let's summarize our observability architecture. Logging is covered by Prometheus and Grafana Loki, and distributed tracing is covered by Jaeger. All these components report to Grafana, which provides UI data dashboards, analytics, and alerts. We can also use OpenTelemetry (OTel) framework. OTel is an open-source framework that was created, developed, and supported by the Cloud Native Computing Foundation (CNCF). The idea is to create a standardized vendor-free observability language specification, API, and tool. It is intended to collect, transform, and export telemetry data. Our architecture is based on the Azure cloud, and we can enable OpenTelemetry for our infrastructure and application components. Below you can see how our architecture can change with OpenTelemetry. Figure 7: Smart heating with an observability subsystem and OpenTelemetry It is also worth mentioning that we do not necessarily need to add OTel, as it may add additional complexity to the system. In the figure above, you can see that we need to forward all logs from Prometheus to OTel. Also, we can use Jaeger as a backend service for OTel. Grafana Loki and Grafana will get data from OTel. Conclusion In this article, we demystified observability and monitoring terms, and we walked through examples of microservices architecture with observability subsystems that can be used not only with Azure but also with other cloud providers. Also, we defined the main difference between monitoring and observability, and we walked through essential monitoring and observability patterns and toolsets. Developers and architects should understand that an observability/monitoring platform is a tooling or a technical solution that allows teams to actively debug their system. This is an article from DZone's 2022 Performance and Site Reliability Trend Report.For more: Read the Report

The arrays of produced software are growing exponentially, which means that the amount of test coverage required to guarantee quality is increasing to the same extent. This also leads to that kind of causal relationship — more lines of code, more defects. So, in addition, to prepare the QA team for test automation and optimization, you also need to think about implementing an effective bug reporting tool. However, even with an effective web-based test case management tool in place, it is necessary to understand what progress metrics in software testing need to be applied in order to get a clearer picture of the efficient efforts of quality assurance. In this article, we will analyze what test metrics and measurements are considered essential, which of them are premier in the agile methodology, and which KPI is necessary to track the performance of the team. How to Understand What Quality Assurance Metrics to Use Before you start evaluating your own testing process, you need to determine what specific types of metrics in software testing you might need for this. The correct solution here is to make sure you have the answers to the following questions: How long will it take to test? How much will it cost to test? Is it reasonable to use low-cost tests? What is the severity of the bugs? What is the status of each bug — closed, reopened, postponed? How many bugs have to be discovered in perspective? How much of the software is tested? Can the test be run on specified time frames, and is it possible to fit more in the same time period? Will it require more test efforts? Once the questions are answered, you can go further and choose QA testing metrics that can meet your requirements. However, you should remember that metrics are not universal — different businesses need different metrics and measurements in software testing. QA Effectiveness Metrics Absolute Numbers Absolute metrics are a great way to get a general idea of how the current testing processes are built. And their presence is recommended for all types of development. Total number of test cases Number of test cases passed Number of test cases failed Number of test cases blocked Number of defects found Number of defects accepted Number of defects rejected Number of defects deferred Number of critical defects Number of planned test hours Number of actual test hours Number of bugs found after shipping Test Execution and Bug Fixing Test metrics in software testing showing the correlation between completed tasks from the total number of functions allow the whole team to understand which errors in which modules disrupt the product and should be addressed primarily: Test completion status Test execution completion status Defect density Defect priority/defect severity Defect acceptance/defect rejection Test case execution productivity Test Design Test design coverage evaluates the correlation between test cases and the number of requirements, while test design performance evaluates the number of test cases generated per day. It is done to find out gaps in functionality on the end-user side: Test design coverage; Test design productivity. Test Coverage Test coverage evaluates test effort and gives the idea of what percentage of the application has already been tested. User Acceptance Testing This metric is supposed to discover missed issues that might occur due to gaps in the testing strategy: Defect leakage Product Exploitation and Support This metric is used for strategy improvements to boost testing performance. It also evaluates testing effectiveness by showing the number of undiscovered issues that need to be addressed before production deployment: Defect removal efficiency Test Economics Metrics The cost of testing consists of infrastructure, tools, and workforce. This metric evaluates how much needs to be spent to finish the project and how much is spent: Total Allocated Cost Actual Cost Budget Variance Time Variance Cost Per Bug Fix Cost of Not Testing Test Execution Status This metric is better represented in a chart to display the total executions organized as passed, failed, blocked, incomplete, and unexecuted. Defects Created vs Defects Resolved Chart This metric is supposed to control defect removal processes and understand testing effectiveness metrics. Overall Metrics Overall testing metrics measure the effectiveness of your test strategy in general identifying needed improvements: Effort variance/schedule variance Rework effort ratio Requirements creep Properly use of the metric for software quality is possible to get the desired results from testing. Their presence in modern development processes such as agile helps managers accurately define smaller goals for each sprint. Using benchmarks and KPIs as a navigator, testers understand what result they should get and what numbers they should focus on. In case of deviation from these test efficiency metrics, we can talk about a change in trends. Such deviation can indicate a critical mistake that could jeopardize the success of the project. This way, managers can pre-check and refocus their team without waiting for the end and thus prevent additional development costs. Given all the above, I can unequivocally say that quality metrics in software testing in the development process can improve its quality and prevent unnecessary risks.