Load Testing Video Streaming With JMeter: Learn How

Video streaming services are used in sports events, music concerts, news and more. Before such events it is important to run performance tests on your streaming services, to make sure they don't crash when users are watching. In this article, we are going to learn how to test video streaming services by using Apache JMeter™.

The video streaming protocols we will test are taken from this list, which contains the most popular streaming protocol: HLS, Progressive Download, RTMP, MPEG-DASH. (Icecast is used mainly for audio streaming, for example, in radio services, so we excluded it from our list).

Each protocol will be discussed in two parts: script realization and KPI Monitoring. So first, let's discuss each part and learn what is important to take into consideration about each one. After that, we will proceed to the protocol performance testing examples.

When creating your performance scripts, it's important to take the following into consideration:

• Different devices - different video resolutions. When you watch a video using your phone, you have less video resolution than when you watch it using a laptop. You need to pay attention to the most popular resolutions in your tests when creating your scripts.
• You can configure different bandwidth speeds in JMeter. Learn more from this blog post "How to Simulate Different Network Speeds in Your JMeter Load Test".
• For testing from several regions/states, you can use JMeter's distributed mode and several AWS instances. Or, you can use BlazeMeter, which is easier and friendlier to use.
• Concerning user behavior, it is worth remembering that users do not always start watching the video from the beginning. If they're watching VOD, they can pause the stream or choose another one. This affects the way the server is load tested.

When analyzing script results, these are the KPIs we recommend you monitor:

1. Percentage of HTTP request errors
2. Average loading time of playlist for HLS, DASH
3. Average loading time of chunks for HLS, DASH
4. Bitrate (throughput);
5. Average loading time of starting a stream
6. Lag time

We can use JMeter to get the first four metrics. The fifth and sixth metrics can be calculated using timers and frame load time for some of the protocols. However, I am concerned about the correctness of the results for lag time through JMeter because of the calculation complexity and the features of the player used, which can affect the playback of the video stream. Therefore, at the end, I will propose a solution for obtaining these last two KPIs.

Note: items 1-6 from the list can also be obtained using the HLS commercial plugin developed by Ubik Ingénierie for video streams that work through HLS, MPEG-DASH, and Smooth protocols.

That's it for now. Let's proceed to the implementation of scripts, and after that, we will return to the problem of metrics calculation in the end.

Load Testing HLS/MPEG-DASH

HLS cuts video content into short blocks - chunks, and then these short fragments are delivered via HTTP. Our blog already contains an article about working with HLS through the HTTP Request Sampler, and also about working with the HLS Plugin. The first article contains an example for VOD.

Today, most video services work with an incoming RTMP stream. Then, the online video is encoded to HLS format and delivered to users. As an example, we will use a service for online broadcasts - Smashcast.tv. When viewing the broadcast, we request a playlist (manifest), from which we get links to the chunks and load them (only the last three chunks are used). After ~2 seconds, we re-request a playlist (at the same address as we did the first time). If it contains new chunks, then we will load them.

1. We have implemented the JMeter script, which you can find here, via the HTTP Request Sampler as shown in the 1st article, because this allows us to calculate the part of metrics that we cannot get using the HLS plugin from the second article. We recommend using the HLS plugin for your VOD testing.

This JMeter script does the following:

• After loading the manifest, the entire playlist is saved to the ArrayList object and when the manifest is reloaded, old and new playlists are compared to avoid loading old chunks.
• The Parallel Controller is used to ensure that manifest requests are not connected with chunk requests.
• A Dummy Sampler was added because if the loop was executed many times and the condition did not pass, after some time the execution will be interrupted.

2. The script metrics:

• Average loading time of manifest - average time for GET manifest sampler
• Average loading time of chunks - average time for GET chunk load sampler
• Bitrate - throughput for GET chunk load
• Average loading time of starting stream - the sum of the average time of samplers from GET /api/media/live to GET first chunk
• Percent of HTTP request errors

MPEG-DASH works exactly the same, but the manifest is XML, so you will only need to replace regexp for chunks extraction. See this example of an MPEG-DASH live stream. For VOD, we can extract all the necessary data from the manifest using regular expressions, but for live stream, we need to calculate the number of the actual chunk by the following formula:

latestSegmentNumber = (now - availabilityStartTime - duration)/duration + startNumber;

The calculation of the chunk number can be seen in this script for MPEG-DASH live streaming.

Load Testing Progressive Download

The Progressive Download technology is quite simple: you have a variety of videos at different resolutions that are available for download as video files on an HTTP Server. When a request is made for a particular video file, the video file is sent across to the player at a single Bitrate. The peculiarity of this technology is that a user of the device has a copy of this video, so this method is also called pseudo streaming. It is not used for online broadcasts, but is actively used for streaming small videos.

In fact, this is a simple download of a video file from the server and we can use the HTTP Request sampler for that. As an example, we will take a demo from this site.

The script is fairly simple and consists of only two requests (of the video and video upload, no need to upload playlists).

If we start watching the video from the middle, the server cuts the original video and sends us the cut part. This behavior is controlled by sending bytes to the video request header.

As for metrics, metrics connected to playlists, chunks, and bitrate are not suitable for this case, and we cannot calculate the last two (loading time of starting stream and lag time) with JMeter. So just make sure the average download time of the video file is less than its duration.

Load Testing RTMP

Another one of the most commonly used protocols for video streaming is RTMP. However, unlike the previous methods, it does not work via HTTP (due to this, its latency is lower than the protocols above). Therefore, implementation is a little more complicated.

1. You can learn how to do it from this article, and use Youtube as a service for streaming through RTMP.

2. All tools and the Java Sampler that were discussed in the RTMP article are intended only for creating the load and do not provide any additional KPI information, except for the information about the interrupted connections.

So, we are in a situation in which we can easily create a load, but we have difficulties getting some KPIs: lag time for all protocols and average loading time of starting a stream for RTMP and Processing Download.

A possible solution is to launch a UI Test with the load test in parallel. As a result, we will be able to get information about the behavior of the UI under the load on the server and also if it's a live stream, it makes sense to pay attention to the video latency.

Selenium is suitable for writing the test. Also, you can configure the launch of UI tests and JMeter tests by using Taurus. Apart from statistics, it also makes sense to run several UI Tests.

• Lag time can be calculated by using a timer that will be started when the download wheel appears and stops after it disappears.
• Time of starting stream can be calculated from the first download wheel.
• In addition to calculating metrics, we can check whether there are any problems with displaying a picture, like freezing, bad or black frames.

BlazeMeter will soon have a free recorder that will enable you to record a scenario and get a JMeter and Selenium script for it. By using the recorder, you will be able to record the live streaming scenario, get the scripts and adjust them to calculate the KPIs.

Also, many video streaming players have an API for calculating similar metrics. For example, here is a demo with metrics calculation for JWPlayer, but the source code of the application has to be modified for the correct event processing. Players based on Adobe Flash player usually have API too — Flowplayer.

Browser engines provide some metrics too. For instance, to get access to DecodedFrameCount for an html5 player on this site, you need to execute following js code:

document.getElementById("html5_player").webkitDecodedFrameCount;

For the demonstration you can paste it to the developer console:

You can see the full list of the provided metrics here.

As mentioned, BlazeMeter will soon provide a free Chrome Recorder Extension for creating JMeter and Selenium tests. Sign up here to get updates.

Your video streaming performance scripts can be run in BlazeMeter, where you can scale them for 1000s or 1000000s of VUs, run the test from all over the world, share tests and results with managers and colleagues and get insightful results in real time and over time.

Make sure you are ready for big events that require live streaming, like soccer or football tournaments or music shows, by performance testing enough time in advance. If your website crashes, your users will move on to your competitors, who tested their website properly.

To learn more about performance testing video streaming, view this free webinar, Using JMeter for Performance Testing HLS Video Delivery.