MQTT Essentials

The Internet of Things (IoT) has recently gained massive traction. IoT challenges enterprises, small companies, and developers with new problems to solve. While HTTP is the de facto protocol for the human web, communication between machines at scale requires a paradigm shift — steering away from request/response and leading toward publish/subscribe. This is where the ultra-lightweight, massively scalable, open, and easy-to-implement MQTT protocol enters the picture.  

The purpose of this Refcard is to explain what MQTT is and to show how it works. 

MQTT is a binary client-server publish/subscribe messaging transport protocol, standardized by OASIS. Designed with a minimal protocol overhead, MQTT is a good choice for a variety of machine-to-machine (M2M) and IoT applications, especially where a small code footprint is required and/or network bandwidth is at a premium. MQTT utilizes many characteristics of the Transmission Control Protocol (TCP) transport, so the minimum requirement for using MQTT is a working TCP stack, which is now available for even the smallest microcontrollers. 

The most recent version of MQTT is 5, which has many improvements over the first public MQTT release, MQTT 3.1. 

Use Cases 

MQTT excels in scenarios where reliable message delivery is crucial for an application, but a reliable network connection is not necessarily available — e.g., mobile networks. Typical use cases of MQTT include: 

• Telemetry 
• Automotive 
• Smart home 
• Energy monitoring 
• Chat applications 
• Notification services 
• Healthcare applications 

This section describes the basic components of MQTT and how they work. We'll review publishing and subscribing to messages, using topics, subscribing with wildcards, using quality of service to guarantee message delivery, clearing sessions, and keeping connections alive. 

Publish/Subscribe 

MQTT implements the brokered publish/subscribe pattern, which decouples a client ("publisher," those sending a particular message) from other clients ("subscribers," those receiving the message). This means that the publisher and subscribers don't know about the existence of one another. The clients do not know each other, but they know the message broker, which filters all incoming messages and distributes them to the correct subscribers. 

To demonstrate this process, the figure below shows a temperature sensor publishing temperature telemetry to the “temperature” topic on an MQTT Broker, which then delivers the telemetry to a laptop and a mobile device — subscribers to the “temperature” topic. 

Figure 1: MQTT publish/subscribe 

This decoupling of sender and receiver can be differentiated in three dimensions: 

1. Space decoupling – Publisher and subscriber do not need to know each other (e.g., by IP address and port). 
2. Time decoupling – Publisher and subscriber do not need to be connected at the same time. 
3. Synchronization decoupling – Operations on both components are not halted during publishing or receiving messages. 

MQTT Message Types 

MQTT has 14 different message types. Typically, end users only need to employ the CONNECT, PUBLISH, SUBSCRIBE, and UNSUBSCRIBE message types. The others are used for internal mechanisms and message flows. 

Table 1: MQTT message types 

Topics  

A topic is a UTF-8 string that is used by the broker to filter messages for each connected client. A topic consists of one or more topic levels, each of which is separated by a forward slash (topic level separator). In comparison to a message queue, a topic is lightweight. There is no need for a client to create the desired topic before publishing or subscribing to it because a broker accepts each valid topic without any prior initialization. 

MQTT Topic Wildcards 

MQTT Topic Wildcards can be used for topic filters when subscribing to MQTT messages. These wildcards are useful if a client wants to receive messages for different topics with similar structure at once. Wildcards are not allowed in topic names when publishing messages. The wildcard characters are reserved and must not be used in the topic. These characters cannot be escaped. 

Table 2: MQTT Topic Wildcards 

Valid MQTT topic examples: 

• my/test/topic 
• my/+/topic 
• my/# 
• my/+/+ 
• +/# 
• # 

Quality-of-Service Levels 

Each MQTT publish is sent with one of three Quality-of-Service (QoS) levels that are associated with different guarantees with regard to the reliability of message delivery. Both client and broker provide additional persistence and redelivery mechanisms to increase reliability in case of network failures, restarts of the application, and other unforeseen circumstances. 

MQTT relies on TCP, which has reliability guarantees on its own. Historically, QoS levels were needed to overcome data loss on older and unreliable TCP networks. This can still be a valid concern for mobile networks today. 

• Level 0: At most once delivery – The sender tries to send the message with best effort and relies on the reliability of TCP. No retransmission takes place. 
• Level 1: At least once delivery – The receiver will get the message at least once. If the receiver does not acknowledge the message, or the acknowledgement gets lost on the way, it will be resent until the sender gets an acknowledgement. Duplicate messages can occur. 
• Level 2: Exactly once delivery – The protocol makes sure that the message will arrive exactly once at the receiver. This increases communication overhead but is the best option when neither loss nor duplication of messages are acceptable. 

Last Will and Testament 

A Last Will and Testament (LWT) message can be specified by an MQTT client when connecting to the MQTT broker. If that client does not disconnect gracefully, the broker sends out the LWT message on behalf of the client when connection loss is detected. 

Retained Messages 

Each sent MQTT message can be sent as a retained message. A retained message is a last known good value and persists at the MQTT broker for the specified topic. Every time a new client subscribes to that specific topic, it will instantly receive the last retained message on that topic. This is also the case for matching wildcards. 

Clean/Persistent Sessions 

When a client connects to an MQTT broker, it has the choice of requesting a persistent session. The broker is responsible for storing session information of the client if the client requested a persistent session. The session information of a client includes: 

• All subscriptions of the client 
• All QoS 1 and 2 messages that are not processed yet 
• All QoS 1 and 2 messages the client missed while being offline 

Persistent sessions are often used for MQTT clients on constrained devices and clients that must not miss any messages for certain topics — not even when they are disconnected. When a client reconnects, the broker will send all missed messages for a subscription with a QoS level of 1 or 2. Persistent sessions are most useful for clients that subscribe to topics; publishing-only clients don't profit from persistent sessions. 

Clean sessions are often used by publishing-only MQTT clients that are not interested in any state. 

Heartbeats 

An MQTT CONNECT message contains a keepAlive value in seconds, where the client can specify the maximum timeout between message exchanges. This allows the broker to detect a half-open connection and close the connection to the (already disconnected) client if the keepAlive value is exceeded by more than 150% of the value. 

So if a connection between broker and client is still established, the client sends a PINGREQ message to the broker within the keepAlive interval if no other message exchange occurs. The broker responds with a PINGRESP message. 

Every client specifies its keepAlive value when connecting, and the maximum value is 65535 seconds (18h 12m 15s). 

Beyond the MQTT protocol itself, an MQTT broker should be scalable, highly available, and secured. This section describes these features.  

Scaling MQTT 

In a brokered architecture, it's critical to avoid a single point of failure and to think about scaling out, since typically, only one broker node is used. In the context of MQTT, there are two popular strategies applicable: bridging and clustering. 

Bridging 

Some brokers implement an unofficial bridging protocol that makes it possible to chain brokers together. Bridging allows forwarding messages on specific topics to other MQTT brokers. Bridge connections between brokers can be uni- or bidirectional. Technically, a bridge connection to another broker is a connection where the broker behaves like an MQTT client and subscribes to specific topics. 

Pros: 

• Useful for forwarding messages on specific topics 
• Different broker products can be chained 
• Hierarchical broker architectures are possible 

Cons: 

• No shared state between brokers 
• Bridge protocol is not officially specified 

Brokers that implement bridging: HiveMQ, mosquitto, RSMB, and IBM Websphere MQ. 

Clustering 

Many enterprise MQTT brokers implement clustering, which supports high-availability configurations and allows for scaling out by adding more broker nodes. When a cluster node is no longer available, other cluster nodes can take over so that no data or messages are lost. Often, brokers implement elastic clustering, and nodes can be added or removed any time.  

Pros: 

• High availability and scalability 
• MQTT semantics across cluster nodes 

Cons: 

• No standard 
• Broker-specific 

Brokers that implement clustering: Apache ActiveMQ, HiveMQ, RabbitMQ. 

If broker implementation allows, clustering and bridging can be used together, enabling messages from one broker cluster to be forwarded to another isolated cluster. 

Securing MQTT 

Security is an important part of any communication. MQTT itself keeps everything as simple as possible and relies on other proven technologies for safeguards instead of reinventing the wheel. 

Username/Password Authentication 

An MQTT CONNECT message can contain a username and password. The broker can authenticate and authorize with this information if such a mechanism is implemented. Many open-source brokers rely on access control lists, while other enterprise brokers allow coupling with user databases and/or LDAP systems. 

Transport Security: TLS 

When using MQTT, add transport layer security (TLS) if possible. With TLS, the complete communication between client and broker is encrypted, and no attacker can read any message exchanged. If feasible, X509 client certificate authentication adds an additional layer of security to the clients: trust. Some MQTT brokers, like HiveMQ, allow the use of X509 certificates in the plugin system for further processing (e.g., authorization). 

Other Security Mechanisms 

Most enterprise MQTT brokers add additional security mechanisms — e.g., a plugin system where concrete logic can be hooked in. Additional security for MQTT communications can be gained when adding the following to clients/brokers: 

• Payload encryption – This is application-specific. Clients can encrypt the payload of their PUBLISH messages. The shared secret has to be provisioned to all communication participants beforehand. 
• Payload signing – If the MQTT broker of choice supports intercepting MQTT messages (e.g., with a plugin system), every received message payload can be intercepted and signed with a private key before distributing. The distributed messages can then be verified by the MQTT clients to make sure no one has modified the message. 
• Complex authentication protocols – For many enterprise MQTT brokers, additional authentication methods can be implemented (e.g., OAuth 2, Kerberos, OpenID Connect). 
• Authorization/topic permissions – Securing access to topics is often done with a permission concept. Some brokers offer restricting publish/subscribe permissions with a plugin system. This makes sure no one can subscribe to more information than needed, and that only specific clients can publish on specific topics. 

Implementing MQTT Brokers

A variety of high-quality MQTT brokers are available. The table below shows the most popular open-source and commercial broker implementations:

Table 3: MQTT message types 

A variety of MQTT client implementations are available for most of the popular operating systems and programming languages. The following tables give an overview of the most prominent client libraries and tools: 

Table 4

Table 5

This section provides several short tutorial examples for using MQTT.  

Example Using MQTT on the Command Line 

Trying MQTT on the Linux and Mac OS X command lines is easy. Install an MQTT Broker, such as FairCom EdgeMQ, Mosquitto, or HiveMQ, and start it. Installing FairCom EdgeMQ, Mosquitto, or HiveMQ is as simple as downloading, unzipping, and running the broker’s executable. To try MQTT without even installing a broker, the following hosted brokers are available for free: 

Table 6

After installing Mosquitto, two command-line utility programs are available for testing MQTT. They are mosquitto_sub and the mosquitto_pub.  

Open two terminal windows, one for publishing and one for subscribing.  

In the first terminal window, run mosquitto_sub with the following command line to subscribe to the topic named my/topic. It waits for messages published to that topic and displays the messages in the terminal window. 

# Subscribing to a MQTT topic with QoS 2 and debug output 

mosquitto_sub – h broker.mqttdashboard.com –t 'my/topic' –q 2 -d 

In the second terminal window, run mosquitto_pub with the following command line to publish a message to the topic named my/topic:   

# Publishing a MQTT message with QoS 2 

mosquitto_pub –h broker.mqttdashboard.com –t 'my/topic' –m 'my_message' –q 2 

After publishing the message, the terminal running mosquitto_sub should display the message. 

Example Using Paho Client Libraries in a Java Program

Eclipse Paho is an umbrella project that provides scalable open-source MQTT client implementations for various languages. The following examples use the Eclipse Paho Java library for the MQTT client. 

Obtaining the Library 

With Maven: pom.xml:

...... 

<repositories> 

<repository> 

<id>Eclipse Paho Repo</id> 

  <url>https://repo.eclipse.org/content/repositories/paho-releases/</url> 

</repository> 

</repositories> 

.... 

<dependencies> 

<dependency> 

      <groupId>org.eclipse.paho</groupId> 

      <artifactId>org.eclipse.paho.client.mqttv3</artifactId> 

      <version>1.0.2</version> 

</dependency> 

</dependencies>

With Gradle: build.gradle:

repositories { 

maven { url 'https://repo.eclipse.org/content/ 

repositories/paho-releases/' } } 

dependencies { 

compile( [group: 'org.eclipse.paho', name: 'org. 

eclipse.paho.client.mqttv3', version: '1.0.2'] ) } 

Publishing a Message

Publishing messages is straightforward. After connecting, publishing is a one-liner with the publish() method: 

MqttClient mqttClient = new MqttClient( 

"tcp://broker.mqttdashboard.com:1883", //1 

"refcard-client"); //2 

​

mqttClient.connect(); 

mqttClient.publish( 

"topic", //3 

  "message".getBytes(), //4 

  0, //5 

  false); //6 

mqttClient.disconnect(); 

1. The server URI 
2. The MQTT client ID 
3. The MQTT topic 
4. The payload as byte array 
5. The QoS Level 
6. Retained flag 

Subscribing to Topics 

In order to subscribe to topics, an MqttCallback must be implemented. This callback is triggered every time an event (like messageArrived) occurs, and it must be implemented before connecting to the broker.  

mqttClient.setCallback(new MqttCallback() { //1 

@Override 

public void connectionLost(Throwable throwable) { 

//Called when connection is lost. 

} 

@Override 

public void messageArrived(String topic, MqttMessage mqttMessage) throws Exception { 

System.out.println("Topic: " + topic); 

System.out.println(new String(mqttMessage. getPayload())); 

System.out.println("QoS: " + mqttMessage. getQos()); 

System.out.println("Retained: " + mqttMessage. isRetained()); 

} 

​

  @Override 

public void deliveryComplete(final IMqttDeliveryToken iMqttDeliveryToken) { 

//When message delivery was complete 

} 

}); 

​

mqttClient.connect(); 

mqttClient.subscribe("my/topic", 2); //2 

1. Implement the MqttCallback to process messages that match the subscription  
2. Subscribe to a topic with QoS 2 

Connecting With Additional Options 

For more sophisticated MQTT applications, there are additional options for establishing a connection to the broker available: 

MqttConnectOptions options = new MqttConnectOptions(); 

options.setCleanSession(true); //1 

options.setKeepAliveInterval(180); //2 

options.setMqttVersion(MqttConnectOptions.MQTT_ VERSION_3_1_1); //3 

options.setUserName("username"); //4 

options.setPassword("mypw".toCharArray()); //5 

options.setWill("will/topic", //6 

"message".getBytes(), //7 

1, //8 

                true);//9 

mqttClient.connect(options); 

If a clean or persistent session should be used 

1. Interval in seconds for heartbeats 
2. MQTT version (3.1 or 3.1.1) 
3. Username for authentication 
4. Password for authentication 
5. Topic for Last Will and Testament (LWT) 
6. LWT message 
7. LWT QoS 
8. LWT retained flag 

Example Using MQTT in a Web Browser or Node.js 

HTML5 WebSocket provides full-duplex communication over a TCP connection. Most modern web browsers implement this specification, even on mobile devices. MQTT can be used in conjunction with WebSocket to allow any web application to behave like a full-featured MQTT client. A library that uses WebSocket for MQTT, like the Paho JavaScript Client, is needed.  

Figure 2: MQTT over WebSocket 

Advantages of using MQTT in web applications: 

• Quality of Service semantics – With QoS 1 and 2, there's assurance that a message arrives on the client or broker at least once/exactly once, even if the internet connection dropped in the meantime. 
• Queuing – When using QoS 1 or 2 and a persistent session, the broker will queue all messages a client misses from its subscriptions when it is not connected. On reconnect, all messages are delivered instantly to that client. 
• Retained messages – Messages that are retained on the server are delivered instantly when a web application subscribes to one of these topics. 
• Last Will and Testament – If a client doesn't disconnect gracefully, it's possible to publish a message to a topic in order to notify all subscribers that the client went offline. 

Connecting With Paho JavaScript 

A website can be connected to an MQTT broker easily by using the Paho JavaScript library. Typically, the following code is executed as soon as the page is loaded: 

var client = new Messaging.Client(hostname, port, clientid); 

var options = { 

//connection attempt timeout in seconds 

timeout: 3, 

//Called if the connection has successfully been established 

onSuccess: function () { 

alert("Connected"); 

}, 

    //Gets Called if the connection could not be established 

onFailure: function (message) { 

    alert("Connection failed: " + message.errorMessage); 

} 

}; 

//Connect the MQTT client 

client.connect(options); 

Publishing With Paho JavaScript 

After a connection is established, the client object can be used to publish messages: 

var message = new Messaging.Message(payload); 

   message.destinationName = topic; 

   message.qos = qos; 

   client.send(message); 

Subscribing With Paho JavaScript 

In order to process messages, a callback is needed for handling each arriving message. After assigning the callback, subscribing to concrete topics is possible: 

/*Gets called whenever a message is received for a 

subscription*/ 

   client.onMessageArrived = function (message) { 

      //Do something with the push message you received 

   }; 

client.subscribe("testtopic", {qos: 2}); 

MQTT is a mature, proven technology with a variety of commercial MQTT brokers and established client libraries for all major programming languages. MQTT does for the Internet of Things (IoT) and operational technology (OT) what messaging has done for information technology (IT): It makes integration less expensive, faster to implement, and easier to maintain. Because of its simplicity, ease of use, and low cost, MQTT is becoming a viable alternative to other messaging technologies, such as AMQP, Kafka, and IBM MQ. 

For more information, visit mqtt.org Getting Started page.