Performance Resources

DZone's Best Performance Resources

Performance Engineering Powered by Machine Learning

By Joana Carvalho

This is an article from DZone's 2022 Performance and Site Reliability Trend Report.For more: Read the Report Software testing is straightforward — every input => known output. However, historically, a great deal of testing has been guesswork. We create user journeys, estimate load and think time, run tests, and compare the current result with the baseline. If we don't spot regressions, the build gets a thumbs up, and we move on. If there is a regression, back it goes. Most times, we already know the output even though it needs to be better defined — less ambiguous with clear boundaries of where a regression falls. Here is where machine learning (ML) systems and predictive analytics enter: to end ambiguity. After tests finish, performance engineers do more than look at the result averages and means; they will look at percentages. For example, 10 percent of the slowest requests are caused by a system bug that creates a condition that always impacts speed. We could manually correlate the properties available in the data; nevertheless, ML will link data properties quicker than you probably would. After determining the conditions that caused the 10 percent of bad requests, performance engineers can build test scenarios to reproduce the behavior. Running the test before and after the fix will assert that it's corrected. Figure 1: Overall confidence in performance metrics Source: Data from TechBeacon Performance With Machine Learning and Data Science Machine learning helps software development evolve, making technology sturdier and better to meet users' needs in different domains and industries. We can expose cause-effect patterns by feeding data from the pipeline and environments into deep learning algorithms. Predictive analytics algorithms, paired with performance engineering methodologies, allow more efficient and faster throughput, offering insight into how end users will use the software in the wild and helping you reduce the probability of defects reaching production. By identifying issues and their causes early, you can make course corrections early in the development lifecycle and prevent an impact on production. You can draw on predictive analytics to improve your application performance in the following ways. Identify root causes. You can focus on other areas needing attention using machine learning techniques to identify root causes for availability or performance problems. Predictive analytics can then analyze each cluster's various features, providing insights into the changes we need to make to reach the ideal performance and avoid bottlenecks. Monitor application health. Performing real-time application monitoring using machine-learning techniques allows organizations to catch and respond to degradation promptly. Most applications rely on multiple services to get the complete application's status; predictive analytics models will correlate and analyze the data when the application is healthy to identify whether incoming data is an outlier. Predict user load. We have relied on peak user traffic to size our infrastructure for the number of users accessing the application in the future. This approach has limitations as it does not consider changes or other unknown factors. Predictive analytics can help indicate the user load and better prepare to handle it, helping teams plan their infrastructure requirements and capacity utilization. Predict outages before it's too late. Predicting application downtime or outages before they happen helps to take preventive action. The predictive analytics model will follow the previous outage breadcrumbs and continue monitoring for similar circumstances to predict future failures. Stop looking at thresholds and start analyzing data. Observability and monitoring generate large amounts of data that can take up to several hundred megabytes a week. Even with modern analytic tools, you must know what you're looking for in advance. This leads to teams not looking directly at the data but instead setting thresholds as triggers for action. Even mature teams look for exceptions rather than diving into their data. To mitigate this, we integrate models with the available data sources. The models will then sift through the data and calculate the thresholds over time. Using this technique, where models are fed and aggregate historical data, provides thresholds based on seasonality rather than set by humans. Algorithm-set thresholds trigger fewer alerts; however, these are far more actionable and valuable. Analyze and correlate across datasets. Your data is mostly time series, making it easier to look at a single variable over time. Many trends come from the interactions of multiple measures. For example, response time may drop only when various transactions are made simultaneously with the same target. For a human, that's almost impossible, but properly trained algorithms will spot these correlations. The Importance of Data in Predictive Analytics "Big data" often refers to datasets that are, well, big, come in at a fast pace, and are highly variable in content. Their analysis requires specialized methods so that we can extract patterns and information from them. Recently, improvements in storage, processors, parallelization of processes, and algorithm design enabled the processing of large quantities of data in a reasonable time, allowing wider use of these methods. And to get meaningful results, you must ensure the data is consistent. For example, each project must use the same ranking system, so if one project uses 1 as critical and another uses 5 — like when people use DEFCON 5 when they mean DEFCON 1 — the values must be normalized before processing. Predictive algorithms are composed of the algorithm and the data it's fed, and software development generates immense amounts of data that, until recently, sat idle, waiting to be deleted. However, predictive analytics algorithms can process those files, for patterns we can't detect, to ask and answer questions based on that data, such as: Are we wasting time testing scenarios that aren't used? How do performance improvements correlate with user happiness? How long will it take to fix a specific defect? These questions and their answers are what predictive analytics is used for — to better understand what is likely to happen. The Algorithms The other main component in predictive analysis is the algorithm; you'll want to select or implement it carefully. Starting simple is vital as models tend to grow in complexity, becoming more sensitive to changes in the input data and distorting predictions. They can solve two categories of problems: classification and regression (see Figure 2). Classification is used to forecast the result of a set by classifying it into categories starting by trying to infer labels from the input data like "down" or "up." Regression is used to forecast the result of a set when the output variable is a set of real values. It will process input data to predict, for example, the amount of memory used, the lines of code written by a developer, etc. The most used prediction models are neural networks, decision trees, and linear and logistic regression. Figure 2: Classification vs. regression Neural Networks Neural networks learn by example and solve problems using historical and present data to forecast future values. Their architecture allows them to identify intricate relations lurking in the data in a way that replicates how our brain detects patterns. They contain many layers that accept data, compute predictions, and provide output as a single prediction. Decision Trees A decision tree is an analytics method that presents the results in a series of if/then choices to forecast specific options' potential risks and benefits. It can solve all classification problems and answer complex issues. As shown in Figure 3, decision trees resemble an upsidedown tree produced by algorithms identifying various ways of splitting data into branch-like segments that illustrate a future decision and help to identify the decision path. One branch in the tree might be users who abandoned the cart if it took more than three seconds to load. Below that one, another branch might indicate whether they identify as female. A "yes" answer would raise the risk as analytics show that females are more prone to impulse buys, and the delay creates a pause for pondering. Figure 3: Decision tree example Linear and Logistic Regression Regression is one of the most popular statistical methods. It is crucial when estimating numerical numbers, such as how many resources per service we will need to add during Black Friday. Many regression algorithms are designed to estimate the relationship among variables, finding key patterns in big and mixed datasets and how they relate. It ranges from simple linear regression models, which calculate a straight-line function that fits the data, to logistic regression, which calculates a curve (Figure 4). OVERVIEW OF LINEAR AND LOGISTIC REGRESSION Linear Regression Logistic Regression Used to define a value on a continuous range, such as the risk of user traffic peaks in the following months. It's a statistical method where the parameters are predicted based on older sets. It best suits binary classification: datasets where y = 0 or 1, where 1 represents the default class. Its name derives from its transformation function being a logistic function. It's expressed as y = a + bx, where x is an input set used to determine the output y. Coefficients a and b are used to quantify the relation between x and y, where a is the intercept and b is the slope of the line. It's expressed by the logistic function:where β0 is the intercept and β1 is the rate. It uses training data to calculate the coefficients, minimizing the error between the predicted and actual outcomes. The goal is to fit a line nearest to most points, reducing the distance or error between y and the line. It forms an S-shaped curve where a threshold is applied to transform the probability into a binary classification. Figure 4: Linear regression vs. logistic regression These are supervised learning methods, as the algorithm solves for a specific property. Unsupervised learning is used when you don't have a particular outcome in mind but want to identify possible patterns or trends. In this case, the model will analyze as many combinations of features as possible to find correlations from which humans can act. Figure 5: Supervised vs. unsupervised learning Shifting Left in Performance Engineering Using the previous algorithms to gauge consumer sentiment on products and applications makes performance engineering more consumer-centric. After all the information is collected, it must be stored and analyzed through appropriate tools and algorithms. This data can include error logs, test cases, test results, production incidents, application log files, project documentation, event logs, tracing, and more. We can then apply it to the data to get various insights to: Analyze defects in environments Estimate the impact on customer experience Identify issue patterns Create more accurate test scenarios, and much more This technique supports the shift-left approach in quality, allowing you to predict how long it will take to do performance testing, how many defects you are likely to identify, and how many defects might make it to production, achieving better coverage from performance tests and creating realistic user journeys. Issues such as usability, compatibility, performance, and security are prevented and corrected without impacting users. Here are some examples of information that will improve quality: Type of defect In what phase was the defect identified What the root cause of the defect is Whether the defect is reproducible Once you understand this, you can make changes and create tests to prevent similar issues sooner. Conclusion Software engineers have made hundreds and thousands of assumptions since the dawn of programming. But digital users are now more aware and have a lower tolerance for bugs and failures. Businesses are also competing to deliver a more engaging and flawless user experience through tailored services and complex software that is becoming more difficult to test. Today, everything needs to work seamlessly and support all popular browsers, mobile devices, and apps. A crash of even a few minutes can cause a loss of thousands or millions of dollars. To prevent issues, teams must incorporate observability solutions and user experience throughout the software lifecycle. Managing the quality and performance of complex systems requires more than simply executing test cases and running load tests. Trends help you tell if a situation is under control, getting better, or worsening — and how fast it improves or worsens. Machine learning techniques can help predict performance problems, allowing teams to course correct. To quote Benjamin Franklin, "An ounce of prevention is worth a pound of cure." This is an article from DZone's 2022 Performance and Site Reliability Trend Report.For more: Read the Report More

A Primer on Distributed Systems Observability

By Boris Zaikin CORE

This is an article from DZone's 2022 Performance and Site Reliability Trend Report.For more: Read the Report In the past few years, the complexity of systems architectures drastically increased, especially in distributed, microservices-based architectures. It is extremely hard and, in most cases, inefficient to debug and watch logs, particularly when we have hundreds or even thousands of microservices or modules. In this article, I will describe what observability and monitoring systems, the patterns of a good observability platform, and the observability subsystem may look like. Observability vs. Monitoring Before we jump directly to the point, let's describe what observability is, what components it includes, and how it differs from monitoring. Observability allows us to have a clear overview of what happens in the system without knowing the details or domain model. Moreover, observability lets us efficiently provide information about: The overall system, separate service failures, and outages The behavior of the general system and services The overall security and alerts We know what functions should cover the observability system. Below we can see what information should be gathered to properly design an observability and monitoring platform. Metrics – Data collection allows us to understand the application and infrastructure states — for example, latency and the usage of CPU, memory, and storage. Distributed traces – Allows us to investigate the event or issue flow from one service to another Logs – This is a message with a timestamp that contains information about application- or service-level errors, exceptions, and information. Alerting – When an outage occurs or something goes wrong with one or several services, alerts notify these problems via emails, SMS, chats, or calls to operators. This allows for quick action to fix the issue. Availability – Ensures that all services are up and running. The monitoring platform sends prob messages to some service or component (to the HTTP API endpoint) to check if it responds. If not, then the observability system generates an alert (see the bullet point for alerting). Also, some observability and monitoring platforms may include user experience monitoring, such as heat maps and user action recording. Observability and monitoring follow the same principles and patterns and rely primarily on toolsets, so in my opinion, the differentiation between the two is made for marketing purposes. There is no clear definition of how observability differs from monitoring; all definitions are different and high-level. Observability Patterns All complex systems based on microservices have recommendations and patterns. This allows us to build a reliable system without reinventing the wheel. Observability systems also have some essential patterns. The following sections discuss five of the most important patterns. Log Aggregation Pattern In distributed systems, logging can be difficult. Each microservice can produce a lot of logs, and it can be a nightmare to find and analyze errors or other log messages of each microservice. Therefore, the log aggregation pattern helps us here. It contains the central log aggregation service as a central log storage. Also, the service provides options to label, index, categorize, search, and analyze all logs. There are a few examples of log aggregation platforms like Grafana Loki, Splunk, Fluentd, and the ELK stack. Figure 1: Log aggregation pattern Health Check Pattern Imagine you have multiple services or microservices, and you need to know their current state. Of course, you can go to the logging aggregation service and check logs. But services may not produce logs in a starting state. Also, it may be the case where logging in is unavailable when the services fail. In all these instances, you need to implement health check patterns. You just need to create a health (or ping) endpoint in your service and point your log aggregation system to check and collect the checks of each service. You can also set up notifications or alerting when the service is unavailable — it saves a lot of time recognizing what service failed to start or went down. Figure 2: Health check pattern Distributed Tracing Pattern Imagine this scenario: you have multiple components, modules, and libraries in one or several microservices. You need to check the whole history of component execution or send the request to one microservice, and you need to check the execution history from one service component list to another. To do this, you need to have some distributed system that will collect and analyze all tracing data. Some open-source services allow you to do so, such as Jaeger, OpenTelemetry, and OpenCensus. Check out the Istio documentation for an example that demonstrates distributed tracing in action. Figure 3: Distributed tracing pattern Application Metrics Pattern Having distributed logging and tracing is essential; however, without application metrics, your observability system will not be complete. You may need to collect infra- and application-level metrics, such as: CPU Memory Disc use Services requests/response time Latency Collecting these metrics will not only help you understand what infrastructure size you need but will also help you save money on cloud providers. It also helps you to quickly mitigate outages caused by a lack of CPU or memory resources. Below is an example of the service that has a proxy agent. The proxy agent aggregates and sends telemetry data to the observability platform. Figure 4: Application metrics pattern Service Mesh for Observability A service mesh not only provides a central management control plane for microservices architecture but also provides a single observability subsystem. Instead of installing a separate tool for gathering metrics, distributed traces, and logs, we can just use one. For example, Azure provides an integrated service mesh add-on that can be set up in a minute. There is also an option to use Istio service mesh, which contains all features required for a proper observability subsystem. Moreover, it can gather metrics, logs, and traces for the control plane. For example, when we set up Grafana, Loki, or other tools, we also need to enable observability for them, as they may also fail while working or during the deployment process; therefore, we need to troubleshoot. Figure 5: Service mesh as observability Observability Architecture for Microservices As an example of the observability architecture, I'm going to use a smart heating system. Smart heating is an essential part of each home (or even smart home) that allows owners to: Manually manage heating in the apartment with an application. Automatically adjust heating depending on time and the temperature outside and inside. In addition, the system can do the following actions to help the owner: Turn on/off the heating when people are about to arrive at the apartment. Notify, alert, or just ask if something requires human attention or if something is wrong. Figure 6: Microservices architecture with an observability subsystem In Figure 6, you can see an architecture that is based on the microservices pattern, since it serves best and represents all system components. It contains main and observability subsystems. Each microservice is based on Azure Functions and deployed to the Azure Kubernetes Cluster. We deploy functions to Kubernetes using the KEDA framework. KEDA is an open-source, Kubernetes-based event autoscaling that allows us to automatically deploy and scale our microservices functions. Also, KEDA provides the tools to wrap functions to the Docker containers. We can also deploy microservices functions directly without KEDA and Kubernetes if we don't have a massive load and don't need the scaling options. The architecture contains the following components that represent the main subsystem: Azure operating as a microservice Azure Service Bus (or Azure IoT Hub) as a central messaging bus that microservices use to communicate Azure API Apps providing an API for mobile/desktop applications The essential part here is an observability subsystem. A variety of components and tools represent it. I've described all components in Table 1 below: COMPONENTS OF THE OBSERVABILITY SYSTEM Tool Description Prometheus Prometheus is an open-source framework to collect and store logs and telemetry as time series data. Also, it provides alerting logic. Prometheus proxy or sidecar integrates with each microservice to collect all logs, telemetry, and tracing data. Grafana Loki Grafana Loki is an open-source distributed log aggregation service. It's based on a labeling algorithm. It's not indexing the logs; rather, it's assigning labels to each log domain, subsystem, or category. Jaeger Jaeger is an open-source framework for distributed tracing in microservices-based systems. It also provides search and data visualization options. Some of the high-level use cases of Jaeger include: Performance and latency optimization Distributed transaction monitoring Service dependency analysis Distributed context propagation Root cause analysis Grafana (Azure Managed Grafana) Grafana is also an open-source data visualization and analytics system. It allows the collection of traces, logs, and other telemetry data from different sources. We are using Grafana as a primary UI "control plane" to build and visualize data dashboards that will come from Prometheus, Loki, and Grafana Loki sources. Let's summarize our observability architecture. Logging is covered by Prometheus and Grafana Loki, and distributed tracing is covered by Jaeger. All these components report to Grafana, which provides UI data dashboards, analytics, and alerts. We can also use OpenTelemetry (OTel) framework. OTel is an open-source framework that was created, developed, and supported by the Cloud Native Computing Foundation (CNCF). The idea is to create a standardized vendor-free observability language specification, API, and tool. It is intended to collect, transform, and export telemetry data. Our architecture is based on the Azure cloud, and we can enable OpenTelemetry for our infrastructure and application components. Below you can see how our architecture can change with OpenTelemetry. Figure 7: Smart heating with an observability subsystem and OpenTelemetry It is also worth mentioning that we do not necessarily need to add OTel, as it may add additional complexity to the system. In the figure above, you can see that we need to forward all logs from Prometheus to OTel. Also, we can use Jaeger as a backend service for OTel. Grafana Loki and Grafana will get data from OTel. Conclusion In this article, we demystified observability and monitoring terms, and we walked through examples of microservices architecture with observability subsystems that can be used not only with Azure but also with other cloud providers. Also, we defined the main difference between monitoring and observability, and we walked through essential monitoring and observability patterns and toolsets. Developers and architects should understand that an observability/monitoring platform is a tooling or a technical solution that allows teams to actively debug their system. This is an article from DZone's 2022 Performance and Site Reliability Trend Report.For more: Read the Report More

10 Essential Metrics for Effective QA Process Measuring

By Tania Zhydkova

Building an Open-Source Observability Toolchain

By Sudip Sengupta CORE

Making of Unreliable Systems

By Hossein Naderi

Accessibility Testing for Digital Products

The majority of people find using the internet on the web to be simple. But when we look at a different demographic group with difficulties, this is not the case. According to WHO, over 1 billion people live with some form of disability. Many of them face challenges in using digital products. Digital products must be functional, accessible, and helpful to this category of users as well; they must not make distinctions between users based on factors such as disabilities and impairments. To avoid these issues you must ensure accessibility in software development as well as accessibility in design. By following accessibility design principles and testing your software, you can eliminate barriers and ensure maximum accessibility for all. What Is Accessibility Testing? Accessibility testing is the practice of testing a product or application to ensure that it can be accessed and used by individuals with a wide range of abilities. Ensuring that your products are accessible improves user experience and caters to large groups of people. It is a growing field, as more and more digital products are produced. The latest annual Forrester Global State of Design Teams Survey found that 36% of respondents had a top-down commitment to accessibility. Accessibility testing is especially important in industries like healthcare, education, and government, where failure to adhere to accessibility guidelines can result in fines, lawsuits, and additional work. Web, app, and video accessibility cases rose by over a quarter in 2020 compared to the previous year, reaching over 3,500 cases. How To Get Started With Accessibility Testing Identify Common Types of Impairments There are many impairments and disabilities that impact a person’s ability to interact with your digital product. Start with these categories: hearing, color blindness, low vision, motor skill impairment, cognitive or learning disabilities, speech, non-visual, seizures, dexterity, and age-related impairments. Get Familiar With Relevant Standards and Regulations Accessibility standards outline what accessibility in digital products is. These standards cover a variety of aspects and types of accessibility testing, including web design, web content, and multimedia. These standards help to make digital products accessible to everyone: WCAG 2.0 (Web Content Accessibility Guidelines) is the most widely followed accessibility standard. It is made up of twelve guidelines arranged around four concepts: websites must be perceivable, operable, understandable, and robust. WAI-ARIA (Web Accessibility Initiative - Accessible Rich Internet Applications) is a technical specification that offers a framework for improving web content and application accessibility and interoperability. This guide is mostly intended for developers. The EU's Web Accessibility Directive improves access for individuals with impairments to public websites and mobile apps. The Directive mandates each website and mobile app to include an accessibility statement, a feedback system for users to report accessibility issues, and frequent monitoring of public sector websites and applications. The Americans with Disabilities Act (ADA) mandates some companies to offer reasonable accommodations for individuals with disabilities. Web content should be accessible to blind and deaf users, as well as those who require to navigate using speech, screen readers, or other assistive technology. Create User Journey Maps Once you identify common impairments you need to account for and guidelines you need to follow, you can create user journey maps. User journey mapping involves identifying all the possible ways that a user could interact with your product or application and identifying all of the steps or interactions that the user has to go through. This can help you identify gaps in accessibility. You can also include users with different impairments and disabilities in user research interviews. For example, you can interview people with motor skill impairments, color blindness, and cognitive or learning disabilities to better understand how your product can be more accessible for different users. Test Accessibility as Early as Possible Accessibility testing is frequently taken into account during the exploratory testing stage. But implementing accessibility testing in the initial test cycles would be much more beneficial. Early detection and remediation of accessibility problems are critical. Test Your Mock-Ups and Prototypes With Users With Disabilities Testing your mock-ups and prototypes with users with disabilities or impairments is another way to identify accessibility issues in your product. You can conduct remote interviews, user testing, or remote usability testing to help identify accessibility issues. You can also recruit testers from organizations that focus on accessibility, such as blind, deaf, and motor skill-impaired people, to conduct usability testing on your prototypes. Use Accessibility Testing Tools There are many tools that can help you follow accessibility guidelines and automate accessibility testing. Many of them are even designed with specific regulations in mind. W3C has created W3C Validator, a checker that helps you ensure that all of your code is correct and validated. It gives you instant results that are easy to read. Most web accessibility testing tools fall into the following categories: Color blindness simulators and color contrast analyzers — help you identify if your color combinations are usable for people with color blindness Readability analyzers — help you identify if your content is readable for people with various reading levels Keyboard accessibility tools — help you identify if your website is accessible for keyboard-only users Code analyzers — help you identify accessibility problems in your source code Over to You Accessibility testing is an important part of digital product development. You must ensure that your products are functional, accessible, and helpful for everyone, including people with disabilities and impairments. Including accessibility in your product design and development process can help you create products that cater to a larger audience and improve user experience.

By Tamara Badža

Application Performance Monitoring Tools — Why, When, What!

In this article, I reported my experience working on large applications and how monitoring came into my journey when I started producing software designed to solve business-critical problems. Solving customers' critical problems can generate great business opportunities, but you need to be ready for really high customer expectations in these situations. To serve these customers and seize these business opportunities, I quickly realized that it was needed to automate most of the daily activities that took up a lot of my time, negatively impacting productivity. We have new customers every month; applications become more and more complicated, bureaucracy increases as well, and emergencies that previously occurred once a month now force us to stay at work until late every day. I am unaware that my application was broken because a customer directly reported the error to me. These customers don’t report bugs or errors; they stop using an application looking for another team that is simply better organized. After more than ten years as a software engineer, I spent a lot of time selecting the best tools to improve my productivity. A lot of confusion has arisen in the world of monitoring, probably because so much data can be used in so many different ways. At first, the approach is challenging for developers to understand what is the best combination to solve their emergencies. In this article, I wrote about my experience trying to differentiate: When or in which situations monitoring can be effective. Why you should monitor some parts of your system and others not. What is the right tool for each specific monitoring problem. What Are Application Monitoring Tools? Application monitoring tools generally consist of two parts: The Agent The Analytics Platform The agent is a software package that developers install in their servers or applications (based on how the agent is designed). Its goal is to collect relevant information about application behavior and performance. This information is sent to the remote platform that analyzes that data and generates visual charts to help developers easily understand what’s happening in their applications. It’s able to send alerts to developers if something goes wrong. What They Are Not This is obviously a simplistic description that could cover a huge amount of tools out there. In fact, many tools look like application monitoring tools, but they have nothing to do with application monitoring. These similarities made it difficult for me to figure out the right tool to solve my productivity problems. Here is what I learned in my journey. Logs Management Tools The logs management tool is often the first kind of tool we tend to approach because since the beginning of the application development journey, watching application logs is one the most important activity every day to be informed about what’s happening inside the most important processes in our application. But when the application started to scale (it runs on multiple servers, requires a complex architecture, etc.) I realized that it was very difficult to extract relevant information from logs about application performance and monitor the impact of new releases over time. Like when the car was invented, people were initially looking for a faster horse because they were used to using the horse. Then they realized a different tool was needed to take it to the next level. Uptime Monitor Uptime monitoring tools can be described as a more sophisticated “ping.” The main purpose is simple: They ping your application endpoints from multiple regions to understand how well (or bad) it can be reached by users located in various geography. This information is useful to understand how the cloud infrastructure works to bring your application to the end-users (load balancer, CDN, network, etc.); it does not provide any information on what is happening inside your application. In my case, my application serves users all around the world, so external ping stats helped us to understand what regions suffer the highest latency by making decisions about in which regions we should place our servers. They monitor the external environment; you will never know if your database slows down. Server vs Application Monitoring This is the hardest difference to understand, and I have not found any interesting article that helped me clarify the separation of duties, rather than ads trying to sell me all kinds of tools. The Application runs on a Server, so they are obviously two strictly related system components. That’s why it might be confusing at first. But server and application monitoring accomplish two completely different needs. Server monitoring focuses on infrastructure, and it’s also basically provided for free by any decent cloud provider. Google GCP, AWS, and DigitalOcean provide you with the most important metrics by default, like CPU usage, Storage, Bandwith, and more, completely free with no extra cost other than running the VM itself. Understanding the time your VMs must scale up (or down) is an important necessity, but saying that the CPU is at 100% could mean everything and nothing: What part of your application do you need to refactor if your application consumes too many resources? How can you identify why a certain part of your app is slowing down, causing a negative experience for your users? How can you be aware if your application is firing exceptions, and why? As mentioned at the beginning of the article, Server monitoring works by installing an agent at the server level, so “outside” of your application. But it's virtually impossible to look at your application from the outside and know what’s happening inside your code. Application monitoring finally focuses you on “application.” These class of tools provides you with a software library, not a package to install in the OS. Developers install the integration library in their application like any other dependency without touching the server’s configuration. It automatically collects relevant information about your code performance, errors, and trends to alert you in case something goes wrong, like a sentinel. What Problem Does an Application Monitoring Tool Solve? Application monitoring tool provides metrics and alerts to identify bugs and bottlenecks in your application without waiting for the customers to report an issue. A well-designed application monitoring solutions provide developers with the information they need to connect application performance to business outcomes and identify and fix performance issues before they impact the end-users, enabling better technical support and the best continuity of service. It acts like a sentinel allowing you to visually explore how your code runs, doing 90% of the analysis work in complete autonomy. This is what Inspector is designed to do and how it is positioned in the monitoring market. Why Is Application Monitoring Important? It is important because happy customers are paying customers. Having an application is the easy part, relatively speaking; anyone can do it. The real work starts by building your rapport with the customer and making them number one. If you put the customer first, they’ll remain loyal fans of your application. On the other hand, one of the worst things for your business is error-prone, buggy software. Nothing will drive potentially paying customers away faster than waiting for the site to load up or finding it down altogether. So do whatever it takes to make them happy, and the revenue will follow. What Can You Monitor in an Application? You should be able to easily know how long your application takes to fulfill HTTP requests or complete background processes, like jobs, cron tasks, etc., to understand what are the most consuming processes in your system. Each execution cycle is typically called a “Transaction.” So during a transaction, the application can perform many different tasks like SQL queries, read/write files, call external systems, algorithms, etc. We call this list of tasks the “Timeline,” and you can explore it visually like in the image below: All of this information is automatically collected by the monitoring tool without any tricky configuration by developers. I really believe that clear and simple information is the most important thing to make better decisions. Learning why, when, and how to use monitoring tools was one of the most confusing parts of my developer journey, and I hope this experience can help you to have a more aware vision of your needs and what are the right tools you need to solve your problems and improve your productivity.

By Valerio Barbera

Configure Cucumber Setup in Eclipse and IntelliJ [Tutorial]

Selenium is a test automation framework extensively used by organizations looking to expedite the web-product testing process. As far as test automation frameworks in Java are concerned, TestNG is largely preferred by QA engineers for the development and enhancement of test suites. With the adoption of Agile methodology in the work environment, a diverse set of stakeholders like QAs, technical managers, and program managers — including non-technical team members — are willing to work collaboratively for the betterment of the product. The project lifecycle includes the business analysts and other stakeholders testing the application in a thorough manner. This is where the necessity to incorporate Behavior Driven Development (BDD) comes into the picture. Cucumber is a popular BDD tool that is widely used for test automation. In the first article of the Cucumber Selenium tutorial series, we will see how to set up Cucumber in Eclipse and IntelliJ for automated browser testing. If you are already familiar with Selenium with Java, you can also check our TestNG tutorial. What Is Cucumber Framework in Selenium? Cucumber is a Behavior Driven Development (BDD) framework used for automated browser testing. It enables diverse project stakeholders to come up with test scenarios that are written in plain English. This encourages participation from team members that do not come from a technical background. This simple text uses Gherkin language which can be easily understood by technical and non-technical people alike. Cucumber is written using the Ruby programming language but it supports other popular languages like Java, Python, and more. Advantages of Cucumber Framework Though there are a number of BDD tools like SpecFlow, Behave, and so on, Cucumber for Selenium automation is the most popular of them all. Here are the major reasons why QA engineers prefer Cucumber over other BDD frameworks: Technical and business teams can participate in test scenario development since the tests are written in plain and simple English. It supports multiple programming languages like Java, JavaScript, Ruby, Python, and so on. No prior coding experience is required to come up with test scenarios in Cucumber Selenium. Even a manual tester with zero exposure to coding can be a part of Cucumber framework creation. It helps in the creation of modular and reusable code. Now that we have covered the basics of the Cucumber framework, let’s look at how to setup Cucumber in Eclipse. In this IntelliJ Cucumber tutorial, we would also cover setting up Cucumber for IntelliJ and using Cucumber with Maven. Here are the pre-requisites for installing the Cucumber framework: Install Java Install Eclipse IDE for Java or Install IntelliJ Install Maven Install Selenium WebDriver Install JUnit Install Cucumber Once you are done with all the installations mentioned above, you can now proceed with Cucumber setup in Eclipse and IntelliJ (i.e., IDE of your choice). We will cover IntelliJ setup with Cucumber in the IntelliJ Cucumber tutorial section of this article. How to Configure Cucumber in Eclipse In this section, we will see how to add relevant JARs to a Java project so that you can setup Cucumber in Eclipse. In case you have already worked with the TestNG framework, follow the same procedure for adding required Jars to the project. In case you are just getting started with Selenium automation testing, you may follow the below steps for realizing Cucumber setup in Eclipse: Step 1 Open the Eclipse exe by double-clicking on the same. If you are opening it for the first time you will get an option to select the workspace where all the created projects will be saved. You can change your workspace directory anytime by using the Browse button. Since I already had a workspace, I got the following screen: Step 2 Click on the Launch button and you will see the Welcome window of Eclipse. You may close this window since it not required to setup Cucumber in Eclipse. Step 3 The next step is to create a Project. Navigate to File > New > Java Project for creating a new project. Step 4 Next, you will see a window to enter your project name. Give a name to your project and click on the Finish button. You will now be able to see your project in the Project Explorer window. Step 5 Now that the project is ready, it’s time to add the relevant Cucumber and Selenium JARs to your project. I have already downloaded the Selenium and Cucumber JARs since these JAR files are required for to setup Cucumber in Eclipse. Right-click on the Sample Project in the project explorer and navigate to Build Path > Configure Build Path. Step 6 Now click on Add External JARs button. Step 7 Browse to the location where you have downloaded the JAR files and select the downloaded Cucumber JARs and click Open. Step 8 Similarly, add all the Selenium JARs required for completing the Cucumber setup in Eclipse. Upon adding, you will see the JARs under the Libraries tab. Step 9 Finally, click on Apply and Close. With this, you have successfully imported the JARs required to setup Cucumber in Eclipse. The imported JARs will be visible under the "Referenced Libraries" tab of the project. You can take this certification as proof of expertise in the field of test automation with JavaScript to empower yourself and boost your career. How to Install Cucumber in IntelliJ Now that we have setup Cucumber in Eclipse, let’s look at how to configure Cucumber with IntelliJ (another popular IDE for Java development) in this section of IntelliJ Cucumber tutorial. Here are the steps to configure Cucumber in IntelliJ. Step 1 Launch the IntelliJ IDE and navigate to File > New > Project. Step 2 Select Java and click on Next until you arrive on the following screen: Step 3 Give a name to your project and click on Finish. Step 4 You will get a prompt to open the project in the same window or a new window. You can select "This Window." Step 5 You will see the project getting displayed in the project explorer. Step 6 Import the required JARs for Selenium and Cucumber, similar to how it was done with Cucumber setup in Eclipse. To do so, navigate to File > Project Structure > Modules. Step 7 As highlighted in the above screenshot select the Dependencies Tab. Add the required dependencies by clicking on the "+" sign towards the bottom and select the required JARs (or Directories). Step 8 Select the JARs through the "browse menu" and click on OK. Step 9 On similar lines, add the Selenium JARs. With this, the Cucumber Selenium dependencies are added to the project. Click on Apply and then press OK. Like Cucumber setup in Eclipse, the imported JARs will be available under the "External Libraries" section. Now that we have covered the aspects related to IntelliJ and Cucumber in IntelliJ Cucumber tutorial section, let’s move to the next section of this Cucumber Selenium tutorial. In the Maven tutorial, we saw how Maven eases the project configurations as the dependencies can be added directly in the project’s pom.xml. Maven will directly import the required jars once you specify them in the pom.xml. With this approach, you can do away with the manual addition of jar files to the test automation project. How to Create Cucumber Maven Project in Eclipse We will be using Cucumber setup in Eclipse (using Maven). The steps mentioned in the IntelliJ Cucumber tutorial section still remain the same. Hence, you can use either of the IDEs as per your convenience. This is the primary reason why we covered these two IDEs in this introductory blog of the Cucumber Selenium tutorial series. Step 1 Navigate to File > New > Project. Step 2 From the dialog, expand Maven, select Maven Project, and click on Next. Step 3 Select the checkbox to Create a Simple Project and click on the Next button. Step 4 Enter the details like the Group and Artifact ID, and click on the Finish button. You will see your Maven project under the project explorer. Pom.xml will contain package information for resolving the Cucumber and Selenium dependencies. Step 5 Click on the pom.xml tab at the extreme right. You will see an XML file that contains all the project details. Step 6 Fetch the maven dependencies from the Maven Repository and add the dependencies manually in pom.xml. Shown below is an example pom.xml where we added the required dependencies from the Maven repository: Save the project, and here you go. Your Maven project is all set to be used to run Cucumber tests using Selenium WebDriver. In the next article of the Cucumber Selenium tutorial series, we will demonstrate how to write Cucumber tests for automated browser testing. Conclusion Cucumber is one of the widely used BDD frameworks for performing automation testing. The major advantage of a BDD framework like Cucumber is that diverse set of team members can participate in creating impactful test scenarios. Along with Cucumber setup in Eclipse, we also covered setting up Cucumber in IntelliJ in the IntelliJ Cucumber tutorial section of this blog. In our first article of the Cucumber Selenium tutorial series, we took a deep dive into how to configure Cucumber and Selenium WebDriver with Eclipse. We also had a look at setting up Selenium WebDriver and Cucumber with IntelliJ IDEA IDE. In order to avoid manual addition of jar files to the project. This is where Selenium with Cucumber using Maven can be super helpful as the Cucumber Selenium dependency is added to the project pom.xml file. Now that we have covered the configuration aspects related to the Cucumber framework, we will look into the implementation aspects in subsequent articles. Happy testing!

By Harshit Paul

OTT Application Testing: What It Is and Why You Can’t Do Without It

Over two years ago, the COVID-19 pandemic completely changed the business landscape for dozens of industries. For example, the eCommerce industry saw a meteoric rise, while cinemas and theatres have not yet fully recovered from the multiple nationwide lockdowns. One of the industries whose rapid growth was also triggered by the coronavirus pandemic is the OTT or Over-the-Top industry. For millions of people stuck at home for months on end, with little to no social interaction, OTT services have become the go-to entertainment channel. Subsequently, the number of available streaming services also skyrocketed, and new players are regularly unveiled. At the same time, the quality of a streaming platform can be the one factor that makes the users fall in love with the service or eventually abandon it. In this article, we will look at what OTT testing is, why it’s needed, and what it usually includes. The Rise of OTT Platforms An average US user spends 485 minutes a day consuming media products, which is more time than most of us spend daily on eating, commuting, or hanging out with friends and family as busy adults. And most users will choose a subscription service based on two factors: the amount and quality of the content offered by a platform and the quality of the platform overall. At the same time, with an increasing number of streaming platforms, users are now facing a phenomenon called subscription fatigue. While the number of streaming platforms continues to grow, it can be harder and harder for users to maintain several subscriptions at once — both due to financial reasons and the lack of time to consume all the content available. This is why the quality of the service is now just as important as the amount and quality of the content on offer. Therefore, media streaming testing becomes one of the most effective ways to maintain the quality of the application at the required level and make sure the product can successfully compete with other streaming services. What Is OTT Testing? OTT testing, or software testing in media and entertainment, is the practice of performing software testing on an OTT media product. A typical approach to OTT testing doesn’t just include individual types of testing. All-encompassing quality assurance is required to check every single aspect of a streaming product. Ideally, OTT application testing should be performed by two subteams dealing with manual and automated QA respectively. This allows the team to achieve the wisest use of resources while maintaining a superior quality of the product. Types of Products and Platforms OTT Testing Deals With The OTT industry has dozens of both established and new players, but the absolute majority of them can be broken down into three categories. Here they are. Audio Apps These applications allow users to stream audio files using their smartphones, desktop computers, Smart TVs, or car stereo systems. The audio files may contain music, radio shows, podcasts, and more. The success of audio streaming apps depends on several factors: the amount and quality of available content, the absence of playback issues, the regular addition of new releases, the accessibility of the application on different platforms, sharing options, and more. Examples of audio OTT apps include Spotify, Deezer, YouTube Music, TuneIn, Pandora, and more. Video Apps Video OTT applications are probably the most popular type of streaming app out there. With a video OTT solution, users can stream movies, TV shows, concerts, sporting events, live video feeds, and other types of video content wherever they are. Like audio OTT applications, video streaming apps are available on different platforms, from mobile phones to Smart TVs. With an endless number of video streaming platforms, users are looking for high-quality, regularly updated content, a user-friendly interface, stable performance, and consistent operation on different platforms. Popular examples of video OTT platforms include Netflix, Hulu, Apple TV, HBO Max, and Amazon Prime. VoIP Apps VoIP OTT applications are the next generation of traditional VoIP solutions, allowing users to interact via audio or video calls, as well as text chats. A good OTT VoIP application will have a convenient interface, steady performance even when the connection is not ideal, and good security with data encryption mechanisms. Popular VoIP applications that utilize OTT technology include WhatsApp, Skype, Viber, Facebook Messenger, and Google Duo. The Importance of Testing OTT Apps As we’ve mentioned before, given the variety and quality of OTT platforms available today, users won’t think twice before cancelling their subscription after repeatedly facing playback issues or having concerns about their privacy. And while it’s natural for streaming services to rely on the amount of content available, especially when it’s original content that costs a lot to produce, great content alone isn’t enough to both attract new users and convince the existing ones to stay. So, timely and all-encompassing OTT application testing helps increase user retention by ensuring the top-notch quality of the application and is just as important as the content the platform has to offer. Key Types of OTT Testing A good media streaming testing strategy is a complex and comprehensive one. The central part of the strategy is not so much about the individual types of testing as it is about the aspects of the application that need to be tested. Still, it’s possible to single out a few most important types of testing that can be found in any media app testing plan. Here are the ones you cannot miss. UI/UX Testing The interface of a software solution is the first thing users see when they decide to give it a try and it’s something they will interact with day in and day out, provided that they decide to stick with the platform. This is why UI/UX testing should be one of the first types of testing performed on an OTT software platform. When two OTT services have a similar amount and quality of content, a good UX is often the one thing that will become the deciding factor for a user. Performance Testing When a user launches a streaming application, they expect a few key things: constant availability of both the platform and the content, quick response, instant switching to full-screen mode, and others. These are all the things that performance testing deals with. After thorough performance testing, you will know for sure how your application will perform under different circumstances and for different users. Load Testing Even though popular OTT platforms offer years worth of content that users can enjoy 24/7, streaming platforms regularly face peak load periods. For example, the first season of the highest-viewed show ever on Netflix, Squid Game, scored 1.65 billion hours watched by subscribers in just 28 days. Big sporting or cultural events also draw large viewing numbers. This is why it’s important to make sure the platform is equipped to withstand peak load with load testing. Compatibility Testing In this day and age, an OTT product owner cannot just make the product available for one or two platforms and call it a day. A streaming service, whether it’s an audio, video, or VoIP solution, is going to be used on dozens of different platforms, from smartphones and tablets to TVs and car entertainment systems, which means thousands of different devices/OS/browsers/network protocol combinations. This is why it’s important to take care of wide compatibility at the development stage and perform compatibility testing throughout. Network Testing The OTT technology heavily relies on network stability for an engaging user experience and smooth performance. Because all the content is stored outside of the user’s device, the stream will simply not launch or will constantly go into buffering mode in case of network problems. And although testing against every potential network problem is hardly possible, comprehensive network testing can definitely make the product equipped to face the most common connectivity issues. Accessibility Testing While developing and testing an OTT solution, it’s very important to keep in mind not only the fully-abled users who, without a doubt, constitute the majority of a typical streaming service audience, but also consider people with different disabilities, especially people who are hard of hearing or visually impaired. This means testing various aspects of the app that can help such users have a better experience using the product, from adjustable font sizes to voice-over functionality. Security Testing It may seem that OTT applications don’t require the same security approach as, for example, banking or fintech applications. However, OTT software products, whether they deal with audio, video, or communication, contain plenty of sensitive information. A security breach can have catastrophic consequences for the image and market position of the product, as evidenced by Yahoo, LinkedIn, and Facebook, to name a few. Timely security testing can protect the app owner from many privacy-associated issues and ensure a safe environment for the users. API Testing The use of APIs can significantly increase the number of options for making the functionality and user experience of an app richer. With the help of an API, you can introduce any new feature, from logging into the site using various social media services to adding dozens of payment options to the site. However, since APIs are typically created by third parties, it’s vital to test the way they are integrated into the main app with API testing. What Needs To Be Tested OTT services are not created equal. They offer different functionality, are aimed at different audiences, and have different marketing strategies. However, there are a few features and parameters that can be found in pretty much any OTT application. The presence of these features helps create a comprehensive OTT application testing strategy. These are the key things that need to be tested in a new or existing audio/video streaming application. Core Functionality This is the one aspect of an OTT solution that can make or break the success of the product. Solid functionality — whether it’s super innovative or simply meets the expectations of the users — will be the factor that not just lures in new customers, but also motivates the existing audience to continue using the product. This is why testing the core functionality of the product is the key element of any streaming testing strategy. Usability For a product designed for a wide audience, usability matters as much as the core functionality. No matter how many radio stations or original content your service offers, bad usability will drive users away faster than you can imagine. Most importantly, the product needs a user-friendly interface where every action is easy to perform. It’s also essential to check whether every element of the UI actually leads to the desired result. This is also one of the aspects of OTT application testing where manual QA is preferable to automated testing. Platform Performance Good performance is something most users don’t pay a lot of attention to, whereas bad performance is instantly noticeable and can significantly impact the product’s market position. There are many components to an OTT app’s performance, from how quickly the application reacts to users’ requests to how stable the playback is at any given moment. Moreover, it’s important to make sure that sudden spikes in user activity do not negatively affect performance. Connectivity Connectivity is the cornerstone of OTT technology: without a stable internet connection, a stunning interface or an impressive collection of content is not going to be worth much. And while the QA team working on a streaming application does not typically have any control over the network speed or quality on the client’s side, they need to ensure that the application does not create any obstacles that can prevent a steady internet connection. Storage OTT services don’t require users to download any media files to their devices, and that is one of their most appealing features. However, that means that the application is forced to rely even more on its own storage to give any number of users access to any piece of content they want. This is why storage requires special attention from the QA team. Whether you opt for your own storage solution or third-party cloud storage, this is one of the testing areas you cannot miss. Personalization Many industry experts believe that smart personalization is going to be the driving force of this generation’s race to the top of the OTT industry. In other words, a service that offers the most robust opportunities for content personalization and always gets it right when it comes to suggesting new content for users to check out is going to emerge as the winner in the increasingly competitive market. And only a QA team with plenty of experience with OTT products will be able to fully evaluate the personalization feature. Payment Functionality There are some OTT services that are monetized exclusively through ads, but they are a minority. Most platforms offer some form of subscription to give their users access to all content or an ad-free listening/watching experience. And having a paid subscription available on the site also means having a few payment methods on offer. Here, the testing team needs to focus on two important aspects: a flaw-free process of making a purchase and the uncompromised security of the payment gateway. Final Thoughts A comprehensive testing strategy will help you take care of the most important aspects of your product, ensuring positive audience reception, market longevity, and easy scalability. Stay tuned for our upcoming article on OTT platform testing, where we will cover the challenges commonly faced while testing OTT solutions, as well as streaming testing automation best practices and benefits.

By Anna Smith

What Is Black Box Testing?

Software testing is a fundamental part of the software development life cycle, whether for web or mobile application software. Not only does testing certify the quality of the software product, but it also provides the developer with an opportunity to enhance it. Almost every software application requires a single line of code or a series of complex routines. Therefore, the developer must do several tests to guarantee that the code runs appropriately and fulfills its intended purpose. Black box and white box software testing are the two forms of testing often performed by developers at this stage. In this article, we will focus on black box testing, its fundamental distinctions, and similarities between black box and white box testing. What Is Black Box Testing? Black box testing, often known as functional testing, is a technique that examines the functionality of the software without requiring knowledge of its internal code structure.It can be applied to all levels of software testing but is primarily used for higher acceptability and system-related levels. Black box testing is a process of testing a system and its behavior independently of its internal structure, design, and implementation. The tester gives input, and the output is viewed as part of this software testing method. This enables the identification of the system's response to expected and unexpected user activities, response time, usability difficulties, and reliability issues. Black box testing is a powerful method since it executes a system end-to-end. In the same way end-users "don't care" about how a system is designed or structured and expect a suitable answer to their requests, a tester can replicate user activity to determine whether the system delivers on its promises. Whether or not the program or application provides the advertised functionality can be ascertained by doing a black box test. In a black box test, all the individual components are examined, such as the user interface and experience, the web server or application server, the database, the dependencies, and the integrated systems. Other names for this type of testing include "opaque box," "closed box," "specification-based," and "eye-to-eye" testing. Types of Black Box Testing The following are types of black box testing: Functional Testing Functional testing primarily focuses on the critical features of the software, as well as the integration between key components and the overall system. This methodology involves smoke testing/sanity testing, integration testing, and system testing to test the software's unique functions and features. A typical example of this testing is verifying that only users with the correct credentials can log in, while those with incorrect credentials cannot. Non-Functional Testing Non-functional testing transcends the testing of features and functions. Instead of determining if the software can do an operation, it examines how it accomplishes that action. This type of testing examines the software's usability and comprehension, performance under peak loads, compatibility with relevant devices and browsers, and vulnerability to security threats. Regression Testing The functional parts of the program are subjected to regression testing to see if a new version demonstrates a regression or deterioration in its capabilities.This testing is undertaken to determine if a specific feature no longer functions in a recent version or if a previously efficient action now performs poorly. Black Box Testing Techniques The following are different black box testing techniques: 1. Equivalence Partitioning As an alternative to testing all possible inputs, testers can "partition" the available inputs into groups and then test just one "sample" input from each partition.It is sufficient for testers to check one birth date in the "under 18" group and one date in the "over 18" group if, for example, the system requests a user's birth date and delivers the same response for all users under the age of 18, and a different reaction for users beyond the age of 18. 2. Boundary Value Analysis The unique behavior of a system around a given boundary value is easily detectable by testers. It's possible, for instance, that a field will only allow numbers between 0 and 99.The boundary values (-100, -99, and -100) provide a convenient way for testers to verify proper input validation. 3. Simulation of a Decision Table There are a lot of systems that will provide you with results based on some inputs. After discovering such "rules," or sets of conditions, testers can determine the effects of each rule and create corresponding test cases. 4. Testing for State Change When transitioning from one state to another, specific systems elicit many responses. A classic example is a login system that permits users to authenticate but locks the account after a predetermined number of failed attempts. If testers determine a state transition mechanism, they can construct test cases that probe the system when it transitions between states.For instance, if a system locks the account after five unsuccessful login attempts, a test case can examine what occurs on the sixth login attempt. 5. Error Guessing This technique involves testing for frequent faults made by developers while constructing similar systems. For instance, testers can examine whether the developer handled null values in a field, text in a numeric field, or numbers in a text-only field, and sanitization of inputs—whether it is possible to submit user inputs containing executable code, which has security implications. An exceptional error guessing involves testing for known software vulnerabilities that can impact the tested system. Pros of Black Box Testing The following are advantages of black box testing: 1. The Rapid Development of Test Cases Testers are only concerned with the application's Graphical User Interface (GUI). Therefore, they do not inspect the source code for errors. 2. Testing Can Be Outsourced Testers are not required to understand the code; hence outsourcing black box software testing is possible. 3. End-user Experience Tests are conducted from the perspective of the end-users. 4. Critical evaluation Since the tester is unfamiliar with the code, they have no preconceived notions about the functionality of the code. Cons of Black Box Testing The following are disadvantages of black box testing: 1. Test Paths May Be Omitted or Repeated The testing procedure may be duplicated, or specific paths may be omitted entirely. When software designers have already executed the tests, they may be unnecessary. 2. Some Portions of the Application May Not Be Examined Because the tester lacks coding knowledge, certain application functions and features may not be checked. 3. Needs Precise and Explicit Test Specs Testers must be sure of what they must test to ensure the program meets the highest quality standards. Conclusion Black box testing is helpful in your arsenal, but it should not be the only one. It might contribute to establishing trust in the quality of your project. Nonetheless, if there are undocumented requirements, black box testing will not be used to prioritize bugs.

By Mario Olomu

Explaining Performance to Non-Technical Stakeholders

Whether you’re an e-commerce company, a SaaS provider, or a content publisher, understanding the performance of your website is important to everyone on the team—not just the developers. Performance is a huge part of the user experience and is directly tied to how well your website achieves its goals. But web performance is often measured in very technical terms, like Largest Contentful Paint, that cause most business folk’s eyes to glaze over. This language gap is a big part of the reason why many websites are so slow. Many only consider performance from their own perspective—“it’s fast for me”—and leave it at that. We simply lack the vocabulary to talk about the problem. To effectively talk about web performance with everyone on the team, you have to break through the jargon and talk about what really matters: the experience of the real users and how that impacts the business. Here are a few ways to do that: Push What’s Important Performance is rarely top of mind, and it’s unlikely anyone will go looking for it unless a user is complaining. Most users don’t complain—they just leave. To keep your users, you need to know about performance before it gets bad. You need to know when it’s important. At Request Metrics, it's our job to figure out what’s important and push the relevant information to you. We examine all your web performance data and push a summary to your inbox with what you need to know. And to make it sharable, we publish it as a PDF so that you can forward it to your client, your boss, or the executives in charge. BLUF: Bottom Line Up Front This old military acronym is about getting right to the most important information first—lives could depend on it! Your situation is (probably) not that dire, but the advice stands. If the most interesting performance anomalies are buried three clicks deep, it’s unlikely to be discovered. At Request Metrics, we expose the interesting things in plain language at the top of all of our reports and dashboards. Plain language descriptions of performance in Request Metrics. This takes the burden off the developer to interpret the data for their team. We can plainly state what’s happening, backed up with data. Describe Metrics as User Impact Web performance metrics are highly technical and confusing—even to developers. If you don’t understand what Cumulative Layout Shift is measuring, it is hard to care that the score isn’t good. Instead, start the conversation with the user impact of the metric, like “the website has a lot of frustrating shifting of content for 75% of our users” * * * Performance can be a powerful way to make your websites better and more effective—but only if you know how to have the conversation. Remember to Push what’s important to your stakeholders, put the bottom line up front, and describe metrics in terms of user impact. If you have the real user data and reports from Request Metrics, that will help.

By Todd Gardner

What Is Configuration Testing in Software Testing?

Configuration testing plays a critical part in the development life cycle by specifying the quality and portability of software. It helps to understand whether a program can be used on any operating system other than the one for which it was designed. Or whether there is any work required to make software run in the new environment. Let’s find out what is configuration testing with an example and how does it work? What Is Meant By Configuration Testing? Configuration testing is a type of testing that is performed to verify the best possible performance of a system and the least and most appropriate configuration that does not result in bugs and defects. Configuration testing ensures your app functions with as many different hardware elements as possible. For this purpose, it is tried out on different supported system configurations, by which we mean combinations of operating systems, browsers, drivers, etc. For example, Oracle Database and MySQL (databases), Chrome and Microsoft Edge (browsers), etc. The idea is to figure out whether the app is suitable for all of them or not. Every combination of software and hardware is incorporated into the testing process. The team detects the most suitable one among them. And now, let us consider what is meant by configuration testing from the perspective of its objectives. Objectives of Configuration Testing Even though the hardware is expected to be designed to a set of standards to function without delays and failures, there are cases when it does not work properly with specific hardware configurations. This happens because not all of the standards are kept. The primary purpose is to distinguish the optimal configuration of equipment that outputs the expected behavior attributes of a system and check its compatibility with the declared hardware and operating systems. It assists testers in system performance and availability validation when there are modifications in configurations. The basic objectives of configuration testing are: Partial app validation; Requirement failure identification (to check whether an app satisfies configurability requirements); Failure identification and minimization (to discover bugs that were missed out or not discovered productively during different testing phases); Performance verification regardless of system settings; Hardware modification evaluation; Identification of the most appropriate system configuration; App verification (it relates to how manageable app elements are during the entire development cycle); Software app analysis (by switching software and hardware resources). How to Perform Configuration Testing The process has a set of prerequisites or activities that should be kept to in the beginning. Here is how to do configuration testing correctly. A team creates a covering matrix with all possible variations of software and hardware setups. It is hard to check each and every variation effectively when there are so many potential ones available (e.g., 800 or more). Thus, it is crucial to underline the exact platforms that are going to be supported. Configurations are placed in order of importance. The most important ones are tested in detail first. Each of the previously prioritized combinations is checked by how it has been determined. The phase of planning configuration testing should never be skipped. A team decides on the hardware pieces, features, modes, and options needed for the software to work. Not all of the hardware features or modes of a device should be supported by the software. Thus, only the necessary ones should be picked out. What is the set of requirements for features and options for a configuration to include to run them? A set of unique features of the software that runs with the hardware combinations are determined. There is a limit to the number of probable scenarios that can be used by a tester. Thus, only those that matter are left. A team can create a table to compare potential combinations by putting together any related information in one place. What are the unique software features that support those hardware patterns? The test cases are created for every configuration. If a bug occurs, it is important to distinguish whether the reason is hardware or software. The results are reported to a team or a hardware manufacturer. Remember, preparation for the configuration test is the little bits and pieces to guide a tester through the process. Types of Testing There are two types of testing when it comes to system behavior analysis with multiple configurations. 1. Software Configuration Testing It comes after unit and integration testing and involves numerous operating systems, browsers, and platforms to compare and check whether an app is compatible with every potential piece of software or not. The testing environment is reset repeatedly, and the app is validated by running on different software. The procedure takes some time to complete as both processes of installation and uninstallation are laborious. To decrease time spent on software test scenarios, a team can use virtual machines. They mimic real-time configurations, saving costs and increasing productivity. 2. Hardware Configuration Testing This one comes after a new build is presented and assumes components connected to physical machines to check hardware combinations and the expected functioning of an app. The main task is to have various machines on which you can set up the software when a new build appears. Usually, a team automates the process for a specific set of hardware combinations because it requires substantial time and effort. However, this does not make the process simple instantly. It is challenging to complete 100% of all the tests. Thus, configurations are analyzed, and the most used hardware is selected and prioritized to conduct testing. The most likely configurations are selected among thousands of alternatives. There are two more types of configuration testing, divided based on testing level. client-level; server-level. A client-level technique has a client’s perspective and a similar concept to usability and functionality testing. A server-level technique is focused on the integration after a release and analyzes the communication between software and the external environment. Conclusion If configuration testing is not implemented, the risks of the configured elements not being attached to the software and the system collapsing are increased. Previously developed measurements for handling various configurations matter. They enable an app to interact with hardware and fulfill configurable requirements as expected, contributing to a bug-free application.

By Anna Smith

The Web Performance APIs Reference

Each of the following performance APIs is in different stages of the W3C’s specification maturity process. You can see each spec’s stage next to their title. Visit this article for a concise graphic of all the performance APIs’ maturity levels. The use of APIs to Boost Performance If you are actively wondering why you need to look at getting APIs into your system in the first place, you are not alone. Plenty of people have questioned what is so important about APIs and why it is that they need to focus on them, to begin with. They have a multitude of questions, but the answer all boils down to the fact that APIs provide real data that programmers can use to judge their own performance. The reality is, that most programmers are numbers-oriented people, and they like to see how their work contributes to the overall project that they are attempting to complete. APIs allow them to track those figures much more easily over time, and this is precisely what needs to be done to get projects done properly. Every API that they use contributes to great project success overall. NAVIGATION TIMING - W3C RECOMMENDATION The Navigation Timing API helps measure real user data (RUM) such as bandwidth, latency, or the overall page load time for the main page. Here is how developers check the page’s performance via JavaScript through the Performance Timing interface: var page = performance.timing, plt = page.loadEventStart - page.navigationStart, console.log(plt); // Page load time (PTL) output for specific browser/user in ms; Navigation timing covers the metrics of the entire page. To gather metrics about individual resources, you’ll use the Resource Timing API explained further down the list. HIGH RESOLUTION TIMING - W3C RECOMMENDATION The High-Resolution Timing API supports floating point timestamps and provides measurements to a microsecond level of detail so that it is not subject to system clock skew or adjustments. var perf = performance.now(); // console output 439985.4570000316 PAGE VISIBILITY - W3C RECOMMENDATION The visibility change event created by this spec is fired on the document whenever the page gains or loses focus. This event is very helpful in programmatically determining the current visibility state of the page. For example, the API can be applied if your user has several browser tabs open and you don’t want specific content to execute (e.g playing a video, or rotating images in a carousel). document.addEventListener(‘visibilitychange’, function(event) { if (document.hidden) { // Page currently hidden. } else { // Page currently visible. } }); RESOURCE TIMING - CANDIDATE RECOMMENDATION The Resource Timing API lets you dig deeper into understanding the behavior of each individual resource in a page. As an example, let’s pick the DZone logo, and retrieve the total loading time in ms: var img = window.performance.getEntriesByName(“http:// cdn.dzone.com/static/images/TOP_NAV_LOGO.png”)[0]; var total = parseInt(img.responseEnd - img.startTime); console.log(total); // output e.g. 281 (in ms) Beyond the main page’s performance (via the Navigation Timing API), you can also track real user experiences on a more granular basis (i.e. resource basis). PERFORMANCE TIMELINE - CANDIDATE RECOMMENDATION The Performance Timeline specification defines a unifying interface to retrieve the performance data collected via Navigation Timing, Resource Timing, and User Timing. // gets all entries in an array var perf = performance.getEntries(); for (var i = 0; i < perf.length; i++) { console.log(“Asset Type: “ + perf[i].name + “ Duration: “ + perf[i].duration + “\n”); } BATTERY STATUS - CANDIDATE RECOMMENDATION This API provides access to the battery status of your user’s battery-driven device. The specific metrics you can access are charging, chargingTime, dischargingTime and level. Events can also be fired based on these statuses: var battery = navigator.battery || navigator. webkitBattery || navigator.mozBattery || navigator. msBattery; if (battery) { console.log(“Battery charging? “ + battery.charging ? “Yes” : “No”); console.log(“Battery level: “ + battery.level * 100 + “ %”); console.log(“Battery charging time: “ + battery. chargingTime + “ seconds”); console.log(“Battery discharging time: “ + battery. dischargingTime + “ seconds”); }); USER TIMING - CANDIDATE RECOMMENDATION With the User Timing API, you can set markers to measure specific blocks or functions of your application. The calculated elapsed time can be an indicator of good or bad performance. performance.mark(“start”); loadSomething(); performance.mark(“end”); performance.measure(“measureIt”, “start”, “end”); var markers = performance.getEntriesByType(“mark”); var measurements = performance. getEntriesByName(“measureIt”); console.log(“Markers: “, markers); console.log(“Measurements: “, measurements); function loadSomething() { // some crazy cool stuff here :) console.log(1+1); } BEACON - WORKING DRAFT With the beacon API, you can send analytics or diagnostic code from the user agent to the server. By sending this asynchronously, you won’t block the rendering of the page. navigator.sendBeacon(“http://mywebserver.com”, ‘any information you want to sent’); ANIMATION TIMING - EDITOR’S DRAFT Instead of using setTimeOut or setInterval to create animations, this spec will introduce the requestAnimationFrame. This method grants the browser control over how many frames it can render; aiming to match the screen’s refresh rate (usually 60fps) will result in a jank-free experience. It can also throttle animations if the page loses visibility (e.g., the user switches tabs), dramatically decreasing power consumption and CPU usage. Check out Microsoft’s demo page comparing setTimeOut with requestAnimationFrame. RESOURCE HINTS - EDITOR’S DRAFT Predictive browsing is a great way to serve your users with exactly what they want to see or retrieve next. “Pre-browsing” refers to an attempt to predict the users’ activity with your page—i.e. is there anything we can load prior to the user requesting it? This spec will allow developers to give the User Agent hints on the download priority of a resource. The following pre-browsing attributes are meant to help you with pre-loading assets on your page. <link rel=”dns-prefetch” href=”//host_to_resolve.com”> <link rel=”subresource” href=”/javascript/mydata. json”> <linkrel=”prefetch” href=”/images/labrador.jpg”> <link rel=”prerender” href=”//example.org/page_2. html”> FRAME TIMING (NOT SUPPORTED YET) - EDITOR’S DRAFT This specification defines an interface to help web developers measure the frame performance of their applications by giving them access to measurements like frames per second and time to frame. – W3C NAVIGATION ERROR LOGGING (NOT SUPPORTED YET) - EDITOR’S DRAFT This specification defines an interface to store and retrieve error data related to the previous navigations of a document. – W3C BROWSER SUPPORT For a chart of the browser support for all of these standards, visit this blog post and scroll down to Browser support overview. Your best method for keeping up with upcoming standards is to subscribe to the Web Performance Working Group mailing list for any performance-related updates. DOWNLOAD YOUR FREE COPY TODAY

By Barbara Bermes

The Next Frontier for Observability: Data Ownership With OpenTelemetry

Observability is a mindset that lets you use data to answer questions about business processes. In short, collecting as much data as possible from the components of your business — including applications and key business metrics — then using an AI-powered tool to help consolidate and make sense of this huge volume of data gives you observability into your business. Having observability for your business and applications lets you make smarter decisions, faster. You save time troubleshooting and can proactively solve problems before they impact customers. Why Is Data Key? Data underlies the entire observability process. Observability is, after all, an evolution of monitoring, and to monitor something, you must collect data about it. As applications move to the cloud and as more modern development practices are adopted, the volume of data expands non-linearly. Adding one microservice could add dozens of cloud compute instances, storage block pools, load balancers, and other miscellaneous components. Data from every one of these services must be collected, retained, and analyzed in order to fully understand the system and what’s happening inside it. Without this data, or with incomplete data, the benefits that observability can offer will never be fully realized. The process of getting data out of your components and in to the observability system needs to be easy, repeatable, and flexible. In observability, we call the process of setting up an application and system to emit data “instrumentation.” This instrumentation process is work, no matter what vendor you choose, or how advanced they claim to be — there will always be some manual work required to inform your observability system when a new user transaction has begun, or to tag trace data with information about your users, or for much of the other advanced Observability functionality. Why Would I Care About Ownership of This Data? To try to make this process easy, many vendors suggest you deploy a proprietary agent that will just magic away all the complexity and “give you observability” after you install it. This is tempting, because it seems like less work to start with, and often in the demo it seems like you can get the same kind of results as you might with a different system. However, you’ll quickly determine that proprietary agents only give you the most basic data out-of-the-box, and that true observability still requires manual instrumentation work. Of course, now, when you do that manual work, you’re locking yourself into their ecosystem and environment. It’s a bad idea to be dependent on one vendor for anything — but especially not for something that’s critical to your business like an observability system. What if you want to leave that vendor for another vendor? What if you grow beyond their ability to scale and need to? If you’re stuck emitting telemetry data in a format that other vendors can’t consume, you’re stuck with that vendor and reliant on their roadmap and operational ability. Data ownership of your telemetry data gives you true independence. You can choose the technology that’s right for your business, no matter where it is in the observability journey, and you can also extend this data for whatever comes after observability as well. With an open data pipeline, you can also process data before sending it to your observability system — what if you don’t want to analyze certain data points, or if there are data residency requirements that apply for some customers, for example? With a proprietary agent, you might not have any control over what’s sent and analyzed. How Can You Own Your Data? The easy answer to this question is to embrace OpenTelemetry. OpenTelemetry is the future of observability, but also provides the most flexible solution for getting observability data from an application into an observability system. OpenTelemetry is an open standard for collection of metrics, traces, logs (and more to come) that can be ingested by almost every vendor (either through the native OpenTelemetry protocol or through an emitter for many other common formats used in observability.) Adopting OpenTelemetry means you control what data is ingested, what processing is done to it, and where it is emitted. This diagram from the OpenTelemetry project website illustrates the pipeline: (Above image from the OpenTelemetry Documentation, © 2022 The OpenTelemetry Authors, used under CC-BY-4.0 license) Why OpenTelemetry, Specifically? As you can see in the image above, the OpenTelemetry system is endlessly flexible. You can customize nearly everything about how data is collected, processed and emitted. You can create duplicate streams and send parts of the data to multiple vendors, or to a cold storage and a vendor, or practically anything else you can imagine. You can automatically redact sensitive data in the telemetry stream before it gets to the vendor. If you have a vendor that charges for ingest, you can sample on the head-end and save money (but we recommend you don’t sample!). OpenTelemetry is also extremely popular. It is the second-busiest CNCF project (second only to Kubernetes). Support for emitting OpenTelemetry is increasingly available in open-source products and projects, and we all know the best instrumentation is instrumentation that you don’t have to do at all. Nearly every major observability vendor contributes to OpenTelemetry, so you can truly feel confident that this technology is the future of the observability movement. What's Next? Now that you know a little about OpenTelemetry, how can you get started? The best way is to jump in and set up the OpenTelemetry collector. You can learn how to get started with OpenTelemetry at this link. Once you've installed the collector, spend some time learning about what functions it provides through receivers, processors, and exporters, and then instrument one of your applications to see how easy it is. You'll see quickly that retaining ownership of your telemetry data isn't difficult, and you'll be setting yourself up for success in the future from the first day. Evolving your reactive monitoring to proactive observability will be a lot easier once you've built a strong foundation with OpenTelemetry.

By Greg Leffler

5 Super Fast Ways To Improve Core Web Vitals

Does your website take more than 3 seconds to load? Is your website struggling with a high bounce rate? Then it requires immediate attention. Google considers Core Web Vital as one of the ranking signals. It is introduced to measure and evaluate the user experience of a website. The core web vitals consist of three metrics: Largest Contentful Paint (LCP) - This metric measures how fast the largest content (image, video, gif) loads to be displayed to the user. Cumulative Layout Shift (CLS) - This metric measures the movement in the layout that could cause users to lose focus and interest while consuming the content. First Input Delay (FID) - This metric calculates the time the browser takes to load the webpage to respond to a user's first interaction - clicking on a link or playing a video. Ensuring all these three metrics are satisfied and at the perfect level provides a great user experience and improves the search engine's rank. Here are the five ways to optimize your website’s loading speed and responsiveness to rank up high and deliver a great user experience: 1. Compress and Optimize the Image Size High-quality images consume more space. When the image size is big, your loading time will increase. If the loading time increases, the user experience will be affected. So, keeping the image size as small as possible is best. Compress the image size. If you have created your website using WordPress, you can use plugins like ShortPixel to compress the image size. If not, many online sites are available to compress image size. However, you might have a doubt - does compression affect the quality of the image? To some extent, yes, it will damage the quality, but only it will be visible while zooming in on the image. Moreover, use JPEG format for images and SVG format for logos and icons. It is even best if you can use WebP format. You can optimize the image using the Content Delivery Network (CDN) to improve the loading speed. A CDN is a network of server farms geographically distributed throughout the world. For example, if your main server is located in India, but the user is in Australia, in that case, your website information stored on CDN passes the data instantly pulling from a nearby server. 2. Optimize for Render Blocking Render blocking affects the First Input Delay metrics. When a browser loads your website, it first checks the CSS - because it is responsible for the overall look and layout - and renders it, followed by Java Script and renders it, then HTML and renders it, etc. This is the process. If the CSS and Javascript are big, it takes a lot of time to load and blocks the process by not allowing anything to run in the background. This would make the user wait a long time and get frustrated. To save from this, you can remove the critical design or stuff and include it in the codes for faster loading. You can also use plugins to check the build and heavy Java and CSS files to remove them. To find the scripts that are blocking the render, you can use the Google Pagespeed Insights tool. 3. Optimize Your Font Size Font size can also affect the User experience and loading time. When the font is heavy, like images, it takes a lot of time to load. If you use different font styles and sizes, the loading time will increase and damage the website layout. Here is an example of the layout damage - if a user is about to click on one of your interlinks, and because the fonts shift up if the user clicks on other links, it would affect the change in cumulative layout score and irritate the reader to continue proceeding with your website. So, the best practice to remove such confusion is to use 1 or 2 font styles throughout your website. That to use only the globally used font style like times new roman, Ariel, Robot, Open Sans, etc. Another way to obviate such hassle is to use preload fonts. Preload fonts are nothing but saving the optimized copy of your fonts stored in each cache so that it can be delivered fast. 4. Upgrade to a Fast Server and Hosting Provider Servers impact the performance of your website. If a server responds to a user request at a faster rate, then it is no problem. But, if the server takes too long to respond, it is time to upgrade your hosting plan or change your hosting provider. Initially, as a startup, a shared hosting platform might be enough. But, once your business has grown, you need to shift to separate hosting. If you find that even changing the hosting plan doesn't improve the site speed, it is time to change the hosting provider. There are several top hosting providers that offer high speed and high uptime. For instance, SiteGround offers a 99.99% uptime. Here are some additional tips to improve server response time: Check the server response time (collect data). Review any plugins that are increasing the loading speed. Because a certain plugin comes in with additional weight for the page and can impact negatively. Leave only the necessary ones. Use CDN for your site. 5. Set Proper Dimension One of the important metrics of the core web vitals is the Cumulative Layout shift. Imagine that you're scrolling through a website on your phone. You think that it is all set to engage with it. Now, you see a text which has a hyperlink that has grasped your interest, and you're about to click it. When you click it, all of a sudden, the text disappears, and there is an image in the place of the text. All the time, while you were reading the content on the website, the website was loading an image. In this scenario, don't you get irritated and frustrated? This is what the CLS measures. You must optimize the CLS to deliver a seamless user experience and improve ranking. To do that, you must give a proper dimension to the image, video, or graphic in the CSS. This is the reason for the instability of your website. Check the image's or other media's proper width and height, and then enter the right size in the CSS file. Conclusion Core Web Vitals are one of the top crucial ranking signals of Google. It measures how a user interacts with the site and enjoys it. Missing any metrics can disrupt the entire website’s ranking and performance. With the above given five tips, you can easily improve your core web vitals and reap benefits. Remember, there are plenty of small elements where you can work on and improve the vitals. But, these five are the most vital and cover most aspects.

By Joydeep Bhattacharya CORE

Performance

DZone's Best Performance Resources

Top Performance Experts

The Latest Performance Topics