Testing MQTT Messaging Brokers

The Internet of Things is the concept of “things” interacting with the environment, or with each other, over the Internet. This concept is now quickly advancing together with the development and expansion of all types of wireless networks, cloud computing, and machine-to-machine communications.

IoT embraces not only home systems, appliances, and consumer products, but industrial objects as well. For example, home automation has become very popular nowadays. The home is equipped with numerous sensors, which collect climate-related information and motion data for people and pets both inside and outside the house, as well as data from home appliances and various equipment.

The collected data is then sent to other devices, which use it for operating. For instance, motion data is sent to devices that are responsible for switching on the lights, or information about the temperature inside the house is sent to the air-conditioning system or to the heater. In industrial plants, critical data is monitored to avert emergencies. In medicine, this technology is used in hospitals to monitor patients’ conditions.

These devices interact with each other over interfaces and protocols, and they are managed by different services. The more that such advanced devices appear, the more traffic they generate, and more resources are required to process their messages. Due to this complexity, performance testing of the services that facilitate and control the communication between numerous intellectual devices has become a very important task.

Understanding the MQTT Protocol

One of the protocols that is widely used in IoT is MQTT — Message Queue Telemetry Transport. MQTT is a compact, open protocol for data exchange.

The primary features of the MQTT protocol are:

• Asynchronous: The sender is not blocked by waiting for the delivery acknowledgment, and it can continue to send messages.
• Messages are small in size. The MQTT message consists of three parts: fixed header, which is always present, variable header, and the message body.
• It can work on unstable networks.

MQTT is an application-level protocol. It works over TCP/IP, using 1883 port by default. If SSL is used, the TCP port is 8883. MQTT uses the publisher-subscriber pattern for communication of clients with each other. As a rule, this messaging pattern is used in the IoT concept.

There are two communication parties in MQTT: clients and servers. The MQTT server is the message broker that controls communication between clients. Clients, or “things”, can be publishers and subscribers. Publishers send a message to the message broker, specifying a topic. Before receiving a message on a topic, subscribers have to subscribe to it. So subscribers receive messages on topics they have subscribed to.

Following this pattern, clients don’t communicate with each other over this protocol for security reasons. For example, without the server, publishers would need to know info about addresses of subscribers. If someone gets this info, they will be able to perform a DDOS attack on the subscriber. Thus, the server is a filter that distributes messages among subscribers, depending on their subscription. Without the server, the subscriber would have to receive all messages sent to it, regardless whether they need it or not. As a result, the subscriber could get overloaded and disabled. Of course, the server can also receive a lot of messages, but unlike the clients, it is designed to handle large loads. That’s why it’s very important to load test message brokers, as the communication and operability of the entire system depends on the brokers’ operability.

MQTT supports three service levels qualities:

• QoS = 0: Means one delivery at most. The message is delivered according to the capabilities of the underlying network. No response is sent by the receiver and no retry is performed by the sender. The receiver gets the message either once or not at all.
• QoS = 1: Means one delivery at least. This quality of service ensures that the message arrives at the receiver at least once, but there’s a probability of duplicating messages on the receiver’s side. If the publisher has not received the acknowledgment of delivery from the message broker, it sends the message again. After the duplicated message is received by the broker, the latter sends it again to all subscribers.
• QoS  = 2: Means one delivery exactly. This is the highest quality of service. It is used when neither loss nor duplication of messages is acceptable.

The higher the quality of the service level, the more resources are required to process MQTT packet. All of the subscribers that are subscribed to a certain topic, but were disconnected for some reason, will receive all messages after they are on if the quality of the service is 1 or 2.

The MQTT protocol has quite a few advantages. First, the implementation of this protocol doesn’t require a significant number of resources and it can be run and supported on many embedded platforms. Second, this protocol doesn’t require high throughput of the communication channel and it can be used in channels with a limited throughput.

How to Run an MQTT JMeter Load Test

Understanding the MQTT Plugin

The strategy of performance testing the message broker is imitating work publishers and subscribers working under different configurations and quality of service levels. You can use the cloud MQTT service for demo testing purposes.

2. Install the MQTT Apache JMeter Plugin, which fully implements communications over MQTT protocol. Download the plugin JAR file from this repository and add it to the /lib/e folder of JMeter. Unfortunately, this plugin is not part of the Plugin Manager.

This plugin adds three samplers to JMeter, which provide communication over MQTT:

• MQTT connection sampler
• MQTT Publisher sampler
• MQTT Subscriber sampler

To add these samplers, go right-click on a Thread Group -> Sampler -> MQTT connection sampler/MQTT Pub sampler/MQTT Sub sampler. Let’s investigate the settings of these samplers.

The MQTT Connection Sampler

This sampler is used to imitate idle clients, waiting for a signal or command. These clients don’t conduct any operation.

The settings of this sampler are the following:

• Server name or IP, port number: the URL or IP and the port number of the message broker, correspondingly.
• Timeout: the waiting time for the connection to the server.
• The sampler supports connection over two protocols: TCP or SSL. If SSL is selected, the check box Dual SSL authentication appears. If it’s checked, the additional fields appear for entering keys and certificates.
• Username and password are filled in if the server supports authorization.
• ClientId: a unique connection id. Each active connection has to have a unique id. In order to simplify the generation of unique ids, there’s an add random suffix clientid check box. If it’s set, a random value is added to the contents of this field.
• Connection keep time: the duration of the connection.
• Keep alive: the period between ping packets sent from the client to the server.
• Connect attempt max, reconnect attempt max: the number of attempts to connect to the server after the first unsuccessful attempt or after the disconnection. For an unlimited number of attempts, set this field to -1.

The MQTT Pub Sampler

This sampler imitates the work of clients that publish a message to the message broker. A few fields of this sampler have the same function as fields by the same name on the connection sampler. The other fields are:

• QoS: the field for selection of the quality of service level. 0 - at most once, 1 - at least once, 2 - exactly once.
• Topic name: the topic title the message is published under.
• Add timestamp in payload check box: if set, the time of publication is added to the message.
• Message type: defines the type of the message. It can be string, hex string when the hex encoded string in this field is converted to the binary one, or random string with fixed length. The message body supports JMeter variables and can be parameterized via BeanShell or JS223 preprocessors.

The MQTT Sub Sampler

This sampler imitates the work of clients that subscribe and receive messages from the message broker.

A few fields of this sampler have the same functions as fields by the same name on the connection and pub samplers. Additional fields are:

• Payload includes timestamp check box. If this check box is on, then the timestamp is expected in the received message.
• Sample on: a pull-down list that selects the subscriber’s mode. Specified elapsed time entry of this list means that sampler will send a ping message to the message broker every n ms, until the client gets one or a few messages. A few messages will be united together. Another entry — received number of messages — means that sampler finishes its work only after it receives n messages. n is the value specified in the neighboring field.
• If the check box debug response is set, the received message will be displayed under the response tab of the View Results Tree listener. It’s recommended to use this option only during debugging — switch it off during the script run.

Implementing the JMeter Script 

Now that we have examined all MQTT related samplers, let’s discuss the creation of the JMeter script. This script will imitate the case of collecting medical-related metrics in a hospital.

For the sake of simplicity, let’s consider that each hospital patient has three sensors: heart rate, blood pressure, glucose concentration. Each sensor acts as a publisher and sends messages at a certain rate. In addition, there’s a unit that collects information about each patient. The collected information is very important for making decisions. Therefore, the reliable work of the message broker is very important, so it needs load testing.

The test plan of the demo script contains four thread groups. Each group represents a certain type of client: publishers or subscribers. Three of them are publisher-clients and act as sensors that publish info about the condition of a patient — Heart Rate Group, Pressure Group, and Glucose Group. The last one is a subscriber-client and acts as a unit that collects information — Listening Group.

In the screenshot above, you can see a configured sub sampler. You can find the server name and port number in your Cloud MQTT account. You can also set the authorization data for your account there.

The QoS level is set to 2, as for our case, all messages have to be delivered. The check boxes Payload includes timestamp and Debug response are set and sample on parameter are configured to receive a number of messages and the value is set to 1. The loop count parameter of the thread group settings, where the sub sampler is located, is set to Forever so that after one MQTT packet is received, a new iteration of the thread group starts.

Each thread group that contains an MQTT pub sampler has a runtime controller that is the parent element of the MQTT pub sampler. The runtime controller adjusts the period when messages are sent. Besides the Pub Sampler, each Runtime controller contains a Counter to demonstrate the ability to use variables in messages, and a Constant Throughput timer to regulate message rates.

In the screenshot above you can see a configured Pub sampler. You can find part of the Sub Sampler configuration data in your Cloud MQTT Account. The topics that the messages will be published under, are specified. The check box add timestamp in payload is set. The message type in the demo script is selected as string and the message body contains a counter variable.

The script also contains a View Result Tree listener to monitor the result of script execution. The result of the script execution is shown in the screenshot below.

You can see on the left part of the screenshot that each call to publish the message is followed by the subscriber's call to read it. Also, each sub sampler message contains the timestamp of publication, which allows the performance engineer to calculate the delay.

To scale up the script, a CSV file is added to the test plan after running the test. This file contains MQTT samplers settings that are required to regulate sensors and units count. Each row in the CSV file contains authorization data and a topic title for both module types. The row also contains a client id for identifying messages.

Don’t forget to set the Number of Threads in the thread groups settings. The number of threads corresponds to the row count in the CSV file. It’s necessary to set the check box Track counter independently for each user in each Counter element.

One can see that requests are still correctly processed by message broker. The final demo script can be downloaded from this repository.

That’s it! You now know how to load test the MQTT messaging broker, which is an inherent element of IoT.