Performance refers to how well an application conducts itself compared to an expected level of service. Today's environments are increasingly complex and typically involve loosely coupled architectures, making it difficult to pinpoint bottlenecks in your system. Whatever your performance troubles, this Zone has you covered with everything from root cause analysis, application monitoring, and log management to anomaly detection, observability, and performance testing.
The majority of people find using the internet on the web to be simple. But when we look at a different demographic group with difficulties, this is not the case. According to WHO, over 1 billion people live with some form of disability. Many of them face challenges in using digital products. Digital products must be functional, accessible, and helpful to this category of users as well; they must not make distinctions between users based on factors such as disabilities and impairments. To avoid these issues you must ensure accessibility in software development as well as accessibility in design. By following accessibility design principles and testing your software, you can eliminate barriers and ensure maximum accessibility for all. What Is Accessibility Testing? Accessibility testing is the practice of testing a product or application to ensure that it can be accessed and used by individuals with a wide range of abilities. Ensuring that your products are accessible improves user experience and caters to large groups of people. It is a growing field, as more and more digital products are produced. The latest annual Forrester Global State of Design Teams Survey found that 36% of respondents had a top-down commitment to accessibility. Accessibility testing is especially important in industries like healthcare, education, and government, where failure to adhere to accessibility guidelines can result in fines, lawsuits, and additional work. Web, app, and video accessibility cases rose by over a quarter in 2020 compared to the previous year, reaching over 3,500 cases. How To Get Started With Accessibility Testing Identify Common Types of Impairments There are many impairments and disabilities that impact a person’s ability to interact with your digital product. Start with these categories: hearing, color blindness, low vision, motor skill impairment, cognitive or learning disabilities, speech, non-visual, seizures, dexterity, and age-related impairments. Get Familiar With Relevant Standards and Regulations Accessibility standards outline what accessibility in digital products is. These standards cover a variety of aspects and types of accessibility testing, including web design, web content, and multimedia. These standards help to make digital products accessible to everyone: WCAG 2.0 (Web Content Accessibility Guidelines) is the most widely followed accessibility standard. It is made up of twelve guidelines arranged around four concepts: websites must be perceivable, operable, understandable, and robust. WAI-ARIA (Web Accessibility Initiative - Accessible Rich Internet Applications) is a technical specification that offers a framework for improving web content and application accessibility and interoperability. This guide is mostly intended for developers. The EU's Web Accessibility Directive improves access for individuals with impairments to public websites and mobile apps. The Directive mandates each website and mobile app to include an accessibility statement, a feedback system for users to report accessibility issues, and frequent monitoring of public sector websites and applications. The Americans with Disabilities Act (ADA) mandates some companies to offer reasonable accommodations for individuals with disabilities. Web content should be accessible to blind and deaf users, as well as those who require to navigate using speech, screen readers, or other assistive technology. Create User Journey Maps Once you identify common impairments you need to account for and guidelines you need to follow, you can create user journey maps. User journey mapping involves identifying all the possible ways that a user could interact with your product or application and identifying all of the steps or interactions that the user has to go through. This can help you identify gaps in accessibility. You can also include users with different impairments and disabilities in user research interviews. For example, you can interview people with motor skill impairments, color blindness, and cognitive or learning disabilities to better understand how your product can be more accessible for different users. Test Accessibility as Early as Possible Accessibility testing is frequently taken into account during the exploratory testing stage. But implementing accessibility testing in the initial test cycles would be much more beneficial. Early detection and remediation of accessibility problems are critical. Test Your Mock-Ups and Prototypes With Users With Disabilities Testing your mock-ups and prototypes with users with disabilities or impairments is another way to identify accessibility issues in your product. You can conduct remote interviews, user testing, or remote usability testing to help identify accessibility issues. You can also recruit testers from organizations that focus on accessibility, such as blind, deaf, and motor skill-impaired people, to conduct usability testing on your prototypes. Use Accessibility Testing Tools There are many tools that can help you follow accessibility guidelines and automate accessibility testing. Many of them are even designed with specific regulations in mind. W3C has created W3C Validator, a checker that helps you ensure that all of your code is correct and validated. It gives you instant results that are easy to read. Most web accessibility testing tools fall into the following categories: Color blindness simulators and color contrast analyzers — help you identify if your color combinations are usable for people with color blindness Readability analyzers — help you identify if your content is readable for people with various reading levels Keyboard accessibility tools — help you identify if your website is accessible for keyboard-only users Code analyzers — help you identify accessibility problems in your source code Over to You Accessibility testing is an important part of digital product development. You must ensure that your products are functional, accessible, and helpful for everyone, including people with disabilities and impairments. Including accessibility in your product design and development process can help you create products that cater to a larger audience and improve user experience.
In this article, I reported my experience working on large applications and how monitoring came into my journey when I started producing software designed to solve business-critical problems. Solving customers' critical problems can generate great business opportunities, but you need to be ready for really high customer expectations in these situations. To serve these customers and seize these business opportunities, I quickly realized that it was needed to automate most of the daily activities that took up a lot of my time, negatively impacting productivity. We have new customers every month; applications become more and more complicated, bureaucracy increases as well, and emergencies that previously occurred once a month now force us to stay at work until late every day. I am unaware that my application was broken because a customer directly reported the error to me. These customers don’t report bugs or errors; they stop using an application looking for another team that is simply better organized. After more than ten years as a software engineer, I spent a lot of time selecting the best tools to improve my productivity. A lot of confusion has arisen in the world of monitoring, probably because so much data can be used in so many different ways. At first, the approach is challenging for developers to understand what is the best combination to solve their emergencies. In this article, I wrote about my experience trying to differentiate: When or in which situations monitoring can be effective. Why you should monitor some parts of your system and others not. What is the right tool for each specific monitoring problem. What Are Application Monitoring Tools? Application monitoring tools generally consist of two parts: The Agent The Analytics Platform The agent is a software package that developers install in their servers or applications (based on how the agent is designed). Its goal is to collect relevant information about application behavior and performance. This information is sent to the remote platform that analyzes that data and generates visual charts to help developers easily understand what’s happening in their applications. It’s able to send alerts to developers if something goes wrong. What They Are Not This is obviously a simplistic description that could cover a huge amount of tools out there. In fact, many tools look like application monitoring tools, but they have nothing to do with application monitoring. These similarities made it difficult for me to figure out the right tool to solve my productivity problems. Here is what I learned in my journey. Logs Management Tools The logs management tool is often the first kind of tool we tend to approach because since the beginning of the application development journey, watching application logs is one the most important activity every day to be informed about what’s happening inside the most important processes in our application. But when the application started to scale (it runs on multiple servers, requires a complex architecture, etc.) I realized that it was very difficult to extract relevant information from logs about application performance and monitor the impact of new releases over time. Like when the car was invented, people were initially looking for a faster horse because they were used to using the horse. Then they realized a different tool was needed to take it to the next level. Uptime Monitor Uptime monitoring tools can be described as a more sophisticated “ping.” The main purpose is simple: They ping your application endpoints from multiple regions to understand how well (or bad) it can be reached by users located in various geography. This information is useful to understand how the cloud infrastructure works to bring your application to the end-users (load balancer, CDN, network, etc.); it does not provide any information on what is happening inside your application. In my case, my application serves users all around the world, so external ping stats helped us to understand what regions suffer the highest latency by making decisions about in which regions we should place our servers. They monitor the external environment; you will never know if your database slows down. Server vs Application Monitoring This is the hardest difference to understand, and I have not found any interesting article that helped me clarify the separation of duties, rather than ads trying to sell me all kinds of tools. The Application runs on a Server, so they are obviously two strictly related system components. That’s why it might be confusing at first. But server and application monitoring accomplish two completely different needs. Server monitoring focuses on infrastructure, and it’s also basically provided for free by any decent cloud provider. Google GCP, AWS, and DigitalOcean provide you with the most important metrics by default, like CPU usage, Storage, Bandwith, and more, completely free with no extra cost other than running the VM itself. Understanding the time your VMs must scale up (or down) is an important necessity, but saying that the CPU is at 100% could mean everything and nothing: What part of your application do you need to refactor if your application consumes too many resources? How can you identify why a certain part of your app is slowing down, causing a negative experience for your users? How can you be aware if your application is firing exceptions, and why? As mentioned at the beginning of the article, Server monitoring works by installing an agent at the server level, so “outside” of your application. But it's virtually impossible to look at your application from the outside and know what’s happening inside your code. Application monitoring finally focuses you on “application.” These class of tools provides you with a software library, not a package to install in the OS. Developers install the integration library in their application like any other dependency without touching the server’s configuration. It automatically collects relevant information about your code performance, errors, and trends to alert you in case something goes wrong, like a sentinel. What Problem Does an Application Monitoring Tool Solve? Application monitoring tool provides metrics and alerts to identify bugs and bottlenecks in your application without waiting for the customers to report an issue. A well-designed application monitoring solutions provide developers with the information they need to connect application performance to business outcomes and identify and fix performance issues before they impact the end-users, enabling better technical support and the best continuity of service. It acts like a sentinel allowing you to visually explore how your code runs, doing 90% of the analysis work in complete autonomy. This is what Inspector is designed to do and how it is positioned in the monitoring market. Why Is Application Monitoring Important? It is important because happy customers are paying customers. Having an application is the easy part, relatively speaking; anyone can do it. The real work starts by building your rapport with the customer and making them number one. If you put the customer first, they’ll remain loyal fans of your application. On the other hand, one of the worst things for your business is error-prone, buggy software. Nothing will drive potentially paying customers away faster than waiting for the site to load up or finding it down altogether. So do whatever it takes to make them happy, and the revenue will follow. What Can You Monitor in an Application? You should be able to easily know how long your application takes to fulfill HTTP requests or complete background processes, like jobs, cron tasks, etc., to understand what are the most consuming processes in your system. Each execution cycle is typically called a “Transaction.” So during a transaction, the application can perform many different tasks like SQL queries, read/write files, call external systems, algorithms, etc. We call this list of tasks the “Timeline,” and you can explore it visually like in the image below: All of this information is automatically collected by the monitoring tool without any tricky configuration by developers. I really believe that clear and simple information is the most important thing to make better decisions. Learning why, when, and how to use monitoring tools was one of the most confusing parts of my developer journey, and I hope this experience can help you to have a more aware vision of your needs and what are the right tools you need to solve your problems and improve your productivity.
Over two years ago, the COVID-19 pandemic completely changed the business landscape for dozens of industries. For example, the eCommerce industry saw a meteoric rise, while cinemas and theatres have not yet fully recovered from the multiple nationwide lockdowns. One of the industries whose rapid growth was also triggered by the coronavirus pandemic is the OTT or Over-the-Top industry. For millions of people stuck at home for months on end, with little to no social interaction, OTT services have become the go-to entertainment channel. Subsequently, the number of available streaming services also skyrocketed, and new players are regularly unveiled. At the same time, the quality of a streaming platform can be the one factor that makes the users fall in love with the service or eventually abandon it. In this article, we will look at what OTT testing is, why it’s needed, and what it usually includes. The Rise of OTT Platforms An average US user spends 485 minutes a day consuming media products, which is more time than most of us spend daily on eating, commuting, or hanging out with friends and family as busy adults. And most users will choose a subscription service based on two factors: the amount and quality of the content offered by a platform and the quality of the platform overall. At the same time, with an increasing number of streaming platforms, users are now facing a phenomenon called subscription fatigue. While the number of streaming platforms continues to grow, it can be harder and harder for users to maintain several subscriptions at once — both due to financial reasons and the lack of time to consume all the content available. This is why the quality of the service is now just as important as the amount and quality of the content on offer. Therefore, media streaming testing becomes one of the most effective ways to maintain the quality of the application at the required level and make sure the product can successfully compete with other streaming services. What Is OTT Testing? OTT testing, or software testing in media and entertainment, is the practice of performing software testing on an OTT media product. A typical approach to OTT testing doesn’t just include individual types of testing. All-encompassing quality assurance is required to check every single aspect of a streaming product. Ideally, OTT application testing should be performed by two subteams dealing with manual and automated QA respectively. This allows the team to achieve the wisest use of resources while maintaining a superior quality of the product. Types of Products and Platforms OTT Testing Deals With The OTT industry has dozens of both established and new players, but the absolute majority of them can be broken down into three categories. Here they are. Audio Apps These applications allow users to stream audio files using their smartphones, desktop computers, Smart TVs, or car stereo systems. The audio files may contain music, radio shows, podcasts, and more. The success of audio streaming apps depends on several factors: the amount and quality of available content, the absence of playback issues, the regular addition of new releases, the accessibility of the application on different platforms, sharing options, and more. Examples of audio OTT apps include Spotify, Deezer, YouTube Music, TuneIn, Pandora, and more. Video Apps Video OTT applications are probably the most popular type of streaming app out there. With a video OTT solution, users can stream movies, TV shows, concerts, sporting events, live video feeds, and other types of video content wherever they are. Like audio OTT applications, video streaming apps are available on different platforms, from mobile phones to Smart TVs. With an endless number of video streaming platforms, users are looking for high-quality, regularly updated content, a user-friendly interface, stable performance, and consistent operation on different platforms. Popular examples of video OTT platforms include Netflix, Hulu, Apple TV, HBO Max, and Amazon Prime. VoIP Apps VoIP OTT applications are the next generation of traditional VoIP solutions, allowing users to interact via audio or video calls, as well as text chats. A good OTT VoIP application will have a convenient interface, steady performance even when the connection is not ideal, and good security with data encryption mechanisms. Popular VoIP applications that utilize OTT technology include WhatsApp, Skype, Viber, Facebook Messenger, and Google Duo. The Importance of Testing OTT Apps As we’ve mentioned before, given the variety and quality of OTT platforms available today, users won’t think twice before cancelling their subscription after repeatedly facing playback issues or having concerns about their privacy. And while it’s natural for streaming services to rely on the amount of content available, especially when it’s original content that costs a lot to produce, great content alone isn’t enough to both attract new users and convince the existing ones to stay. So, timely and all-encompassing OTT application testing helps increase user retention by ensuring the top-notch quality of the application and is just as important as the content the platform has to offer. Key Types of OTT Testing A good media streaming testing strategy is a complex and comprehensive one. The central part of the strategy is not so much about the individual types of testing as it is about the aspects of the application that need to be tested. Still, it’s possible to single out a few most important types of testing that can be found in any media app testing plan. Here are the ones you cannot miss. UI/UX Testing The interface of a software solution is the first thing users see when they decide to give it a try and it’s something they will interact with day in and day out, provided that they decide to stick with the platform. This is why UI/UX testing should be one of the first types of testing performed on an OTT software platform. When two OTT services have a similar amount and quality of content, a good UX is often the one thing that will become the deciding factor for a user. Performance Testing When a user launches a streaming application, they expect a few key things: constant availability of both the platform and the content, quick response, instant switching to full-screen mode, and others. These are all the things that performance testing deals with. After thorough performance testing, you will know for sure how your application will perform under different circumstances and for different users. Load Testing Even though popular OTT platforms offer years worth of content that users can enjoy 24/7, streaming platforms regularly face peak load periods. For example, the first season of the highest-viewed show ever on Netflix, Squid Game, scored 1.65 billion hours watched by subscribers in just 28 days. Big sporting or cultural events also draw large viewing numbers. This is why it’s important to make sure the platform is equipped to withstand peak load with load testing. Compatibility Testing In this day and age, an OTT product owner cannot just make the product available for one or two platforms and call it a day. A streaming service, whether it’s an audio, video, or VoIP solution, is going to be used on dozens of different platforms, from smartphones and tablets to TVs and car entertainment systems, which means thousands of different devices/OS/browsers/network protocol combinations. This is why it’s important to take care of wide compatibility at the development stage and perform compatibility testing throughout. Network Testing The OTT technology heavily relies on network stability for an engaging user experience and smooth performance. Because all the content is stored outside of the user’s device, the stream will simply not launch or will constantly go into buffering mode in case of network problems. And although testing against every potential network problem is hardly possible, comprehensive network testing can definitely make the product equipped to face the most common connectivity issues. Accessibility Testing While developing and testing an OTT solution, it’s very important to keep in mind not only the fully-abled users who, without a doubt, constitute the majority of a typical streaming service audience, but also consider people with different disabilities, especially people who are hard of hearing or visually impaired. This means testing various aspects of the app that can help such users have a better experience using the product, from adjustable font sizes to voice-over functionality. Security Testing It may seem that OTT applications don’t require the same security approach as, for example, banking or fintech applications. However, OTT software products, whether they deal with audio, video, or communication, contain plenty of sensitive information. A security breach can have catastrophic consequences for the image and market position of the product, as evidenced by Yahoo, LinkedIn, and Facebook, to name a few. Timely security testing can protect the app owner from many privacy-associated issues and ensure a safe environment for the users. API Testing The use of APIs can significantly increase the number of options for making the functionality and user experience of an app richer. With the help of an API, you can introduce any new feature, from logging into the site using various social media services to adding dozens of payment options to the site. However, since APIs are typically created by third parties, it’s vital to test the way they are integrated into the main app with API testing. What Needs To Be Tested OTT services are not created equal. They offer different functionality, are aimed at different audiences, and have different marketing strategies. However, there are a few features and parameters that can be found in pretty much any OTT application. The presence of these features helps create a comprehensive OTT application testing strategy. These are the key things that need to be tested in a new or existing audio/video streaming application. Core Functionality This is the one aspect of an OTT solution that can make or break the success of the product. Solid functionality — whether it’s super innovative or simply meets the expectations of the users — will be the factor that not just lures in new customers, but also motivates the existing audience to continue using the product. This is why testing the core functionality of the product is the key element of any streaming testing strategy. Usability For a product designed for a wide audience, usability matters as much as the core functionality. No matter how many radio stations or original content your service offers, bad usability will drive users away faster than you can imagine. Most importantly, the product needs a user-friendly interface where every action is easy to perform. It’s also essential to check whether every element of the UI actually leads to the desired result. This is also one of the aspects of OTT application testing where manual QA is preferable to automated testing. Platform Performance Good performance is something most users don’t pay a lot of attention to, whereas bad performance is instantly noticeable and can significantly impact the product’s market position. There are many components to an OTT app’s performance, from how quickly the application reacts to users’ requests to how stable the playback is at any given moment. Moreover, it’s important to make sure that sudden spikes in user activity do not negatively affect performance. Connectivity Connectivity is the cornerstone of OTT technology: without a stable internet connection, a stunning interface or an impressive collection of content is not going to be worth much. And while the QA team working on a streaming application does not typically have any control over the network speed or quality on the client’s side, they need to ensure that the application does not create any obstacles that can prevent a steady internet connection. Storage OTT services don’t require users to download any media files to their devices, and that is one of their most appealing features. However, that means that the application is forced to rely even more on its own storage to give any number of users access to any piece of content they want. This is why storage requires special attention from the QA team. Whether you opt for your own storage solution or third-party cloud storage, this is one of the testing areas you cannot miss. Personalization Many industry experts believe that smart personalization is going to be the driving force of this generation’s race to the top of the OTT industry. In other words, a service that offers the most robust opportunities for content personalization and always gets it right when it comes to suggesting new content for users to check out is going to emerge as the winner in the increasingly competitive market. And only a QA team with plenty of experience with OTT products will be able to fully evaluate the personalization feature. Payment Functionality There are some OTT services that are monetized exclusively through ads, but they are a minority. Most platforms offer some form of subscription to give their users access to all content or an ad-free listening/watching experience. And having a paid subscription available on the site also means having a few payment methods on offer. Here, the testing team needs to focus on two important aspects: a flaw-free process of making a purchase and the uncompromised security of the payment gateway. Final Thoughts A comprehensive testing strategy will help you take care of the most important aspects of your product, ensuring positive audience reception, market longevity, and easy scalability. Stay tuned for our upcoming article on OTT platform testing, where we will cover the challenges commonly faced while testing OTT solutions, as well as streaming testing automation best practices and benefits.
Software testing is a fundamental part of the software development life cycle, whether for web or mobile application software. Not only does testing certify the quality of the software product, but it also provides the developer with an opportunity to enhance it. Almost every software application requires a single line of code or a series of complex routines. Therefore, the developer must do several tests to guarantee that the code runs appropriately and fulfills its intended purpose. Black box and white box software testing are the two forms of testing often performed by developers at this stage. In this article, we will focus on black box testing, its fundamental distinctions, and similarities between black box and white box testing. What Is Black Box Testing? Black box testing, often known as functional testing, is a technique that examines the functionality of the software without requiring knowledge of its internal code structure.It can be applied to all levels of software testing but is primarily used for higher acceptability and system-related levels. Black box testing is a process of testing a system and its behavior independently of its internal structure, design, and implementation. The tester gives input, and the output is viewed as part of this software testing method. This enables the identification of the system's response to expected and unexpected user activities, response time, usability difficulties, and reliability issues. Black box testing is a powerful method since it executes a system end-to-end. In the same way end-users "don't care" about how a system is designed or structured and expect a suitable answer to their requests, a tester can replicate user activity to determine whether the system delivers on its promises. Whether or not the program or application provides the advertised functionality can be ascertained by doing a black box test. In a black box test, all the individual components are examined, such as the user interface and experience, the web server or application server, the database, the dependencies, and the integrated systems. Other names for this type of testing include "opaque box," "closed box," "specification-based," and "eye-to-eye" testing. Types of Black Box Testing The following are types of black box testing: Functional Testing Functional testing primarily focuses on the critical features of the software, as well as the integration between key components and the overall system. This methodology involves smoke testing/sanity testing, integration testing, and system testing to test the software's unique functions and features. A typical example of this testing is verifying that only users with the correct credentials can log in, while those with incorrect credentials cannot. Non-Functional Testing Non-functional testing transcends the testing of features and functions. Instead of determining if the software can do an operation, it examines how it accomplishes that action. This type of testing examines the software's usability and comprehension, performance under peak loads, compatibility with relevant devices and browsers, and vulnerability to security threats. Regression Testing The functional parts of the program are subjected to regression testing to see if a new version demonstrates a regression or deterioration in its capabilities.This testing is undertaken to determine if a specific feature no longer functions in a recent version or if a previously efficient action now performs poorly. Black Box Testing Techniques The following are different black box testing techniques: 1. Equivalence Partitioning As an alternative to testing all possible inputs, testers can "partition" the available inputs into groups and then test just one "sample" input from each partition.It is sufficient for testers to check one birth date in the "under 18" group and one date in the "over 18" group if, for example, the system requests a user's birth date and delivers the same response for all users under the age of 18, and a different reaction for users beyond the age of 18. 2. Boundary Value Analysis The unique behavior of a system around a given boundary value is easily detectable by testers. It's possible, for instance, that a field will only allow numbers between 0 and 99.The boundary values (-100, -99, and -100) provide a convenient way for testers to verify proper input validation. 3. Simulation of a Decision Table There are a lot of systems that will provide you with results based on some inputs. After discovering such "rules," or sets of conditions, testers can determine the effects of each rule and create corresponding test cases. 4. Testing for State Change When transitioning from one state to another, specific systems elicit many responses. A classic example is a login system that permits users to authenticate but locks the account after a predetermined number of failed attempts. If testers determine a state transition mechanism, they can construct test cases that probe the system when it transitions between states.For instance, if a system locks the account after five unsuccessful login attempts, a test case can examine what occurs on the sixth login attempt. 5. Error Guessing This technique involves testing for frequent faults made by developers while constructing similar systems. For instance, testers can examine whether the developer handled null values in a field, text in a numeric field, or numbers in a text-only field, and sanitization of inputs—whether it is possible to submit user inputs containing executable code, which has security implications. An exceptional error guessing involves testing for known software vulnerabilities that can impact the tested system. Pros of Black Box Testing The following are advantages of black box testing: 1. The Rapid Development of Test Cases Testers are only concerned with the application's Graphical User Interface (GUI). Therefore, they do not inspect the source code for errors. 2. Testing Can Be Outsourced Testers are not required to understand the code; hence outsourcing black box software testing is possible. 3. End-user Experience Tests are conducted from the perspective of the end-users. 4. Critical evaluation Since the tester is unfamiliar with the code, they have no preconceived notions about the functionality of the code. Cons of Black Box Testing The following are disadvantages of black box testing: 1. Test Paths May Be Omitted or Repeated The testing procedure may be duplicated, or specific paths may be omitted entirely. When software designers have already executed the tests, they may be unnecessary. 2. Some Portions of the Application May Not Be Examined Because the tester lacks coding knowledge, certain application functions and features may not be checked. 3. Needs Precise and Explicit Test Specs Testers must be sure of what they must test to ensure the program meets the highest quality standards. Conclusion Black box testing is helpful in your arsenal, but it should not be the only one. It might contribute to establishing trust in the quality of your project. Nonetheless, if there are undocumented requirements, black box testing will not be used to prioritize bugs.
Configuration testing plays a critical part in the development life cycle by specifying the quality and portability of software. It helps to understand whether a program can be used on any operating system other than the one for which it was designed. Or whether there is any work required to make software run in the new environment. Let’s find out what is configuration testing with an example and how does it work? What Is Meant By Configuration Testing? Configuration testing is a type of testing that is performed to verify the best possible performance of a system and the least and most appropriate configuration that does not result in bugs and defects. Configuration testing ensures your app functions with as many different hardware elements as possible. For this purpose, it is tried out on different supported system configurations, by which we mean combinations of operating systems, browsers, drivers, etc. For example, Oracle Database and MySQL (databases), Chrome and Microsoft Edge (browsers), etc. The idea is to figure out whether the app is suitable for all of them or not. Every combination of software and hardware is incorporated into the testing process. The team detects the most suitable one among them. And now, let us consider what is meant by configuration testing from the perspective of its objectives. Objectives of Configuration Testing Even though the hardware is expected to be designed to a set of standards to function without delays and failures, there are cases when it does not work properly with specific hardware configurations. This happens because not all of the standards are kept. The primary purpose is to distinguish the optimal configuration of equipment that outputs the expected behavior attributes of a system and check its compatibility with the declared hardware and operating systems. It assists testers in system performance and availability validation when there are modifications in configurations. The basic objectives of configuration testing are: Partial app validation; Requirement failure identification (to check whether an app satisfies configurability requirements); Failure identification and minimization (to discover bugs that were missed out or not discovered productively during different testing phases); Performance verification regardless of system settings; Hardware modification evaluation; Identification of the most appropriate system configuration; App verification (it relates to how manageable app elements are during the entire development cycle); Software app analysis (by switching software and hardware resources). How to Perform Configuration Testing The process has a set of prerequisites or activities that should be kept to in the beginning. Here is how to do configuration testing correctly. A team creates a covering matrix with all possible variations of software and hardware setups. It is hard to check each and every variation effectively when there are so many potential ones available (e.g., 800 or more). Thus, it is crucial to underline the exact platforms that are going to be supported. Configurations are placed in order of importance. The most important ones are tested in detail first. Each of the previously prioritized combinations is checked by how it has been determined. The phase of planning configuration testing should never be skipped. A team decides on the hardware pieces, features, modes, and options needed for the software to work. Not all of the hardware features or modes of a device should be supported by the software. Thus, only the necessary ones should be picked out. What is the set of requirements for features and options for a configuration to include to run them? A set of unique features of the software that runs with the hardware combinations are determined. There is a limit to the number of probable scenarios that can be used by a tester. Thus, only those that matter are left. A team can create a table to compare potential combinations by putting together any related information in one place. What are the unique software features that support those hardware patterns? The test cases are created for every configuration. If a bug occurs, it is important to distinguish whether the reason is hardware or software. The results are reported to a team or a hardware manufacturer. Remember, preparation for the configuration test is the little bits and pieces to guide a tester through the process. Types of Testing There are two types of testing when it comes to system behavior analysis with multiple configurations. 1. Software Configuration Testing It comes after unit and integration testing and involves numerous operating systems, browsers, and platforms to compare and check whether an app is compatible with every potential piece of software or not. The testing environment is reset repeatedly, and the app is validated by running on different software. The procedure takes some time to complete as both processes of installation and uninstallation are laborious. To decrease time spent on software test scenarios, a team can use virtual machines. They mimic real-time configurations, saving costs and increasing productivity. 2. Hardware Configuration Testing This one comes after a new build is presented and assumes components connected to physical machines to check hardware combinations and the expected functioning of an app. The main task is to have various machines on which you can set up the software when a new build appears. Usually, a team automates the process for a specific set of hardware combinations because it requires substantial time and effort. However, this does not make the process simple instantly. It is challenging to complete 100% of all the tests. Thus, configurations are analyzed, and the most used hardware is selected and prioritized to conduct testing. The most likely configurations are selected among thousands of alternatives. There are two more types of configuration testing, divided based on testing level. client-level; server-level. A client-level technique has a client’s perspective and a similar concept to usability and functionality testing. A server-level technique is focused on the integration after a release and analyzes the communication between software and the external environment. Conclusion If configuration testing is not implemented, the risks of the configured elements not being attached to the software and the system collapsing are increased. Previously developed measurements for handling various configurations matter. They enable an app to interact with hardware and fulfill configurable requirements as expected, contributing to a bug-free application.
Observability is a mindset that lets you use data to answer questions about business processes. In short, collecting as much data as possible from the components of your business — including applications and key business metrics — then using an AI-powered tool to help consolidate and make sense of this huge volume of data gives you observability into your business. Having observability for your business and applications lets you make smarter decisions, faster. You save time troubleshooting and can proactively solve problems before they impact customers. Why Is Data Key? Data underlies the entire observability process. Observability is, after all, an evolution of monitoring, and to monitor something, you must collect data about it. As applications move to the cloud and as more modern development practices are adopted, the volume of data expands non-linearly. Adding one microservice could add dozens of cloud compute instances, storage block pools, load balancers, and other miscellaneous components. Data from every one of these services must be collected, retained, and analyzed in order to fully understand the system and what’s happening inside it. Without this data, or with incomplete data, the benefits that observability can offer will never be fully realized. The process of getting data out of your components and in to the observability system needs to be easy, repeatable, and flexible. In observability, we call the process of setting up an application and system to emit data “instrumentation.” This instrumentation process is work, no matter what vendor you choose, or how advanced they claim to be — there will always be some manual work required to inform your observability system when a new user transaction has begun, or to tag trace data with information about your users, or for much of the other advanced Observability functionality. Why Would I Care About Ownership of This Data? To try to make this process easy, many vendors suggest you deploy a proprietary agent that will just magic away all the complexity and “give you observability” after you install it. This is tempting, because it seems like less work to start with, and often in the demo it seems like you can get the same kind of results as you might with a different system. However, you’ll quickly determine that proprietary agents only give you the most basic data out-of-the-box, and that true observability still requires manual instrumentation work. Of course, now, when you do that manual work, you’re locking yourself into their ecosystem and environment. It’s a bad idea to be dependent on one vendor for anything — but especially not for something that’s critical to your business like an observability system. What if you want to leave that vendor for another vendor? What if you grow beyond their ability to scale and need to? If you’re stuck emitting telemetry data in a format that other vendors can’t consume, you’re stuck with that vendor and reliant on their roadmap and operational ability. Data ownership of your telemetry data gives you true independence. You can choose the technology that’s right for your business, no matter where it is in the observability journey, and you can also extend this data for whatever comes after observability as well. With an open data pipeline, you can also process data before sending it to your observability system — what if you don’t want to analyze certain data points, or if there are data residency requirements that apply for some customers, for example? With a proprietary agent, you might not have any control over what’s sent and analyzed. How Can You Own Your Data? The easy answer to this question is to embrace OpenTelemetry. OpenTelemetry is the future of observability, but also provides the most flexible solution for getting observability data from an application into an observability system. OpenTelemetry is an open standard for collection of metrics, traces, logs (and more to come) that can be ingested by almost every vendor (either through the native OpenTelemetry protocol or through an emitter for many other common formats used in observability.) Adopting OpenTelemetry means you control what data is ingested, what processing is done to it, and where it is emitted. This diagram from the OpenTelemetry project website illustrates the pipeline: (Above image from the OpenTelemetry Documentation, © 2022 The OpenTelemetry Authors, used under CC-BY-4.0 license) Why OpenTelemetry, Specifically? As you can see in the image above, the OpenTelemetry system is endlessly flexible. You can customize nearly everything about how data is collected, processed and emitted. You can create duplicate streams and send parts of the data to multiple vendors, or to a cold storage and a vendor, or practically anything else you can imagine. You can automatically redact sensitive data in the telemetry stream before it gets to the vendor. If you have a vendor that charges for ingest, you can sample on the head-end and save money (but we recommend you don’t sample!). OpenTelemetry is also extremely popular. It is the second-busiest CNCF project (second only to Kubernetes). Support for emitting OpenTelemetry is increasingly available in open-source products and projects, and we all know the best instrumentation is instrumentation that you don’t have to do at all. Nearly every major observability vendor contributes to OpenTelemetry, so you can truly feel confident that this technology is the future of the observability movement. What's Next? Now that you know a little about OpenTelemetry, how can you get started? The best way is to jump in and set up the OpenTelemetry collector. You can learn how to get started with OpenTelemetry at this link. Once you've installed the collector, spend some time learning about what functions it provides through receivers, processors, and exporters, and then instrument one of your applications to see how easy it is. You'll see quickly that retaining ownership of your telemetry data isn't difficult, and you'll be setting yourself up for success in the future from the first day. Evolving your reactive monitoring to proactive observability will be a lot easier once you've built a strong foundation with OpenTelemetry.
Observability Practitioner, Director,
Director of Open Source Development,
Eric D. Schabell
Director Technical Marketing & Evangelism,