The Internet of Things (IoT) has recently gained massive traction. IoT challenges enterprises, small companies, and developers with new problems to solve. While HTTP is the de-facto protocol for the human web, communication between machines at scale requires a paradigm shift— steering away from request/response and leading towards publish/subscribe. This is where the ultra-lightweight, massively scalable, and easy-to-implement protocol MQTT enters the picture.

MQTT is a binary client-server publish/subscribe messaging transport protocol, standardized by OASIS. It is lightweight, open, simple, and easy to implement. Designed with a minimal protocol overhead, this protocol is a good choice for a variety of Machine-to-Machine (M2M) and Internet of Things applications, especially where a small code footprint is required and/or network bandwidth is at a premium. MQTT utilizes many characteristics of the TCP transport, so the minimum requirement for using MQTT is a working TCP stack, which is now available for even the smallest microcontrollers.

The most recent version of MQTT is 3.1.1, which has many improvements over the first public MQTT release, MQTT 3.1.

Use Cases

MQTT excels in scenarios where reliable message delivery is crucial for an application but a reliable network connection is not necessarily available, e.g. mobile networks. Typical use cases of MQTT include:

• Telemetry
• Automotive
• Smart Home
• Energy Monitoring
• Chat Applications
• Notification Services
• Healthcare Applications

MQTT implements the brokered publish / subscribe pattern. The publish / subscribe pattern decouples a client (“publisher”), which is sending a particular message from other clients (“subscribers”), which are receiving the message. This means that the publisher and subscribers don’t know about the existence of one another. The clients do not know each other, but they know the message broker, which filters all incoming messages and distributes them to the correct subscribers.

This decoupling of sender and receiver can be differentiated in three dimensions:

• Space decoupling: Publisher and subscriber do not need to know each other (for example, by IP address and port)
• Time decoupling: Publisher and subscriber do not need to be connected at the same time
• Synchronization decoupling: Operations on both components are not halted during publishing or receiving messages

MQTT has 14 different message types. Typically, end users only need to employ the CONNECT, PUBLISH, SUBSCRIBE, and UNSUBSCRIBE message types. The other message types are used for internal mechanisms and message flows.

A topic is a UTF-8 string, which is used by the broker to filter messages for each connected client. A topic consists of one or more topic levels. Each topic level is separated by a forward slash (topic level separator).

In comparison to a message queue, a topic is very lightweight. There is no need for a client to create the desired topic before publishing or subscribing to it, because a broker accepts each valid topic without any prior initialization.

MQTT Topic Wildcards

MQTT Topic Wildcards can be used for topic filters when subscribing to MQTT messages. These wildcards are useful if a client wants to receive messages for different topics with similar structure at once.

Wildcards are not allowed in topic names when publishing messages. The wildcard characters are reserved and must not be used in the topic. These characters cannot be escaped.

Valid MQTT Topic Examples

• my/test/topic

• my/+/topic

• my/#

• my/+/+

• +/#

• #

Each MQTT publish is sent with one of three Quality of Service (QoS) levels. These levels are associated with different guarantees with regards to the reliability of the message delivery. Both client and broker provide additional persistence and redelivery mechanisms to increase reliability in case of network failures, restarts of the application, and other unforseen circumstances.

MQTT relies on TCP, which has reliability guarantees on its own. Historically QoS levels were needed to overcome data loss on older and unreliable TCP networks. This can still be a valid concern for mobile networks today.

A Last Will and Testament (LWT) message can be specified by an MQTT client when connecting to the MQTT broker. If that client does not disconnect gracefully, the broker sends out the LWT message on behalf of the client when connection loss is detected. See the section "Pub / Sub With Paho" for an example.

Each sent MQTT message can be sent as a retained message. A retained message is a last known good value and persists at the MQTT broker for the specified topic. Every time a new client subscribes to that specific topic, it will instantly receive the last retained message on that topic. This is also the case for matching wildcards.

When a client connects to an MQTT broker, it has the choice of requesting a persistent session. The broker is responsible for storing session information of the client if the client requested a persistent session. The session information of a client includes:

• All subscriptions of the client
• All QoS 1 / 2 messages which are not processed yet
• All QoS 1 / 2 messages the client missed while being offline

An MQTT CONNECT message contains a keepAlive value in seconds where the client can specify the maximum timeout between message exchanges. This allows the broker to detect a half-open connection and close the connection to the (already disconnected) client if the keepAlive value is exceeded by more than 150% of the value.So if a connection between broker and client is still established, the client sends a PINGREQ message to the broker within the keepAlive interval if no other message exchange occurred. The broker responds with a PINGRESP message.Every client specifies its keepAlive value when connecting and the maximum value is 65535 seconds (18h 12m 15s).

A variety of high-quality MQTT brokers are available. The table below shows the most popular open source and commercial broker implementations.

A variety of MQTT client implementations are available for most of the popular operating systems and programming languages. These lists give an overview of the most popular MQTT client libraries and MQTT client tools.

MQTT Client Libraries

MQTT Client Tools

Trying MQTT on the command line is very easy. Install either mosquitto or HiveMQ as the MQTT broker and start it. Download HiveMQ at http://www.hivemq.com/download and download the mosquitto client tools with the package manager of choice or via http://www.mosquitto.org.

To try MQTT without even installing a broker, the following hosted brokers are available for free:

Open two terminal windows, one for publishing and one for subscribing.

# Subscribing to a MQTT topic with QoS 2 and debug output
mosquitto_sub – h broker.mqttdashboard.com –t ‘my/topic’ –q 2 -d

# Publishing a MQTT message with QoS 2
mosquitto_pub –h broker.mqttdashboard.com –t ‘my/topic’ –m ‘my_message’ –q 2

Now you should receive the message the publisher sent with the subscribing client.

Eclipse Paho is an umbrella project which provides scalable open-source MQTT client implementations for various languages. The following examples use the Eclipse Paho Java library for the MQTT client.

Obtaining the Library

With Maven: pom.xml

......
<repositories>
<repository>
<id>Eclipse Paho Repo</id>
  <url>https://repo.eclipse.org/content/repositories/paho-releases/</url>
</repository>
</repositories>
....
<dependencies>
<dependency>
      <groupId>org.eclipse.paho</groupId>
      <artifactId>org.eclipse.paho.client.mqttv3</artifactId>
      <version>1.0.2</version>
</dependency>
</dependencies>

With Gradle: build.gradle

repositories {
maven { url ‘https://repo.eclipse.org/content/
repositories/paho-releases/’ } }
dependencies {
compile( [group: ‘org.eclipse.paho’, name: ‘org.
eclipse.paho.client.mqttv3’, version: ‘1.0.2’] ) }

Publish a Message

Publishing messages is straightforward. After connecting, publishing is a one-liner with the publish() method.

MqttClient mqttClient = new MqttClient(
“tcp://broker.mqttdashboard.com:1883”, //1
“refcard-client”); //2

mqttClient.connect();

mqttClient.publish(
“topic”, //3
  “message”.getBytes(), //4
  0, //5
  false); //6
mqttClient.disconnect();

1. The server URI

2. The MQTT client ID

3. The MQTT topic

4. The payload as byte array

5. The QoS Level

6. Retained Flag

Subscribe to Topics

In order to subscribe to topics, an MqttCallback must be implemented. This callback is triggered every time an event (like messageArrived) occurs. This callback must be implemented before connecting to the broker.

mqttClient.setCallback(new MqttCallback() { //1
@Override
public void connectionLost(Throwable throwable) {
//Called when connection is lost.
}

@Override
public void messageArrived(String topic, MqttMessage mqttMessage) throws Exception {
System.out.println("Topic: " + topic);
System.out.println(new String(mqttMessage. getPayload()));
System.out.println("QoS: " + mqttMessage. getQos());
System.out.println("Retained: " + mqttMessage. isRetained());
}

  @Override
public void deliveryComplete(final IMqttDeliveryToken iMqttDeliveryToken) {
//When message delivery was complete
}
});

mqttClient.connect();

mqttClient.subscribe("my/topic", 2); //2

1. Implement the MqttCallback in order to process messages which match the subscription

2. Subscribe to a topic with Quality of Service level 2

Connecting With Additional Options

For more sophisticated MQTT applications, there are additional options for establishing a connection to the broker available.

MqttConnectOptions options = new MqttConnectOptions();
options.setCleanSession(true); //1
options.setKeepAliveInterval(180); //2
options.setMqttVersion(MqttConnectOptions.MQTT_ VERSION_3_1_1); //3
options.setUserName("username"); //4
options.setPassword("mypw".toCharArray()); //5
options.setWill("will/topic", //6
"message".getBytes(), //7
1, //8
                true);//9

mqttClient.connect(options);

1. If a clean or persistent session should be used

2. Interval in seconds for heartbeats

3. MQTT version (3.1 or 3.1.1)

4. Username for authentication
5. Password for authentication
6. Topic for Las tWill and Testament
7. Last Will and Testament message
8. Last Will and Testament QoS
9. Last Will and Testament Retained Flag

Figure 2: MQTT Over Websockets

HTML5 websockets provide a full-duplex communication over a TCP connection. Most modern web browsers implement this specification, even on mobile devices. MQTT can be used in conjunction with websockets to allow any web application to behave like a full-featured MQTT client. A library that utilizes websockets for MQTT like the Paho Javascript Client is needed.

The advantages of using MQTT in web applications are:

• Quality of Service semantics: With QoS 1 and 2, there’s an assurance that a message arrives on the client or broker at least once/exactly once, even if the Internet connection dropped in the meantime.
• Queuing: When using QoS 1 or 2 and a persistent session, the broker will queue all messages a client misses from its subscriptions when it is not connected. On reconnect, all messages are delivered instantly to that client.
• Retained messages: Messages that are retained on the server are delivered instantly when a web application subscribes to one of these topics.
• Last Will and Testament: If a client doesn’t disconnect gracefully, it’s possible to publish a message to a topic in order to notify all subscribers that the client went offline.

Connecting With Paho JavaScript

A website can be connected to an MQTT broker easily by using the Paho Javascript library. Typically the following code is executed as soon as the page is loaded.

var client = new Messaging.Client(hostname, port, clientid);

var options = {
//connection attempt timeout in seconds
timeout: 3,

//Called if the connection has successfully been established
onSuccess: function () {
alert("Connected");
},

    //Gets Called if the connection could not be established
onFailure: function (message) {
    alert("Connection failed: " + message.errorMessage);
}
};

//Connect the MQTT client
client.connect(options);

Publishing With Paho JavaScript

After a connection is established, the client object can be used to publish messages.

Subscribing With Paho JavaScript

In order to process messages, a callback is needed for handling each arriving message. After assigning the callback, subscribing to concrete topics is possible.

In a brokered architecture it’s critical to avoid a single point of failure and to think about scaling out, since typically only one broker node is used. In the context of MQTT there are two different popular strategies applicable:

Bridging

Some brokers implement an unofficial bridging protocol which makes it possible to chain brokers together. Bridging allows forwarding messages on specific topics to other MQTT brokers. Bridge connections between brokers can be uni- or bidirectional. Technically, a bridge connection to another broker is a connection where the broker behaves like an MQTT client and subscribes to specific topics.

Pros:

• Great for forwarding messages on specific topics
• Different broker products can be chained
• Hierarchical broker architectures possible

Cons:

• No shared state between brokers
• Bridge protocol is not officially specified

Brokers which implement bridging: HiveMQ, mosquitto, RSMB, Websphere MQ / IBM MQ

Clustering

Many enterprise MQTT brokers implement clustering, which supports high availability configurations and also allows for scaling out by adding more broker nodes. When a cluster node is no longer available, other cluster nodes can take over so that no data or messages are lost. Often brokers implement elastic clustering, and nodes can be added or removed any time.

Pros:

• High availability and scalability
• MQTT semantics across cluster nodes

Cons:

• No standard

• Broker-specific

Brokers which implement clustering: Apache ActiveMQ, HiveMQ, RabbitMQ

If broker implementation allows, clustering and bridging can be used together, enabling messages from one broker cluster to be forwarded to another isolated cluster.

Security is a very important part of any communication. MQTT itself keeps everything as simple as possible and relies on other proven technologies for safeguards instead of reinventing the wheel.

Username / Password Authentication

An MQTT CONNECT message can contain a username and password. The broker can authenticate and authorize with this information if such a mechanism is implemented. Many open-source brokers rely on Access Control Lists while other enterprise brokers allow coupling with user databases and/or LDAP systems.

Transport Security: TLS

A best practice when using MQTT is to add transport layer security if possible. With TLS, the complete communication between client and broker is encrypted, and no attacker can read any message exchanged. If feasible, X509 client certificate authentication adds an additional layer of security to the clients: trust. Some MQTT brokers, like HiveMQ, allow the use of X509 certificates in the plugin system for further processing (e.g. authorization).

Other Security Mechanisms

Most enterprise MQTT brokers add additional security mechanisms, e.g. a plugin system where concrete logic can be hooked in. Additional security for MQTT communications can be gained when adding the following to clients / brokers:

• Payload encryption: This is application-specific. Clients can encrypt the payload of their PUBLISH messages. The shared secret has to be provisioned to all communication participants beforehand.
• Payload signing: If the MQTT broker of choice supports intercepting MQTT messages (e.g. with a plugin system), every received message payload can be intercepted and signed with a private key before distributing. The distributed messages can then be verified by the MQTT clients to make sure no one has modified the message.
• Complex authentication protocols: For many enterprise MQTT brokers, additional authentication methods can be implemented (e.g. OAuth 2, Kerberos, OpenID Connect, etc.).
• Authorization / Topic Permissions: Securing access to topics is often done with a permission concept. Some brokers offer restricting publish / subscribe permissions with a plugin system. This makes sure no one can subscribe to more information than needed, and that only specific clients can publish on specific topics.

MQTT 3.1.1 is the most recent MQTT release and was published in October 2014. While most popular MQTT brokers and MQTT client libraries support MQTT 3.1.1, some older implementations still use 3.1. While mainly backwards- compatible, the two versions have subtle differences.

The following features were added to MQTT 3.1.1:

• Session present flag: If a client connects with a persistent session (which means it doesn’t use a clean session), an additional flag was introduced in the CONNACK message to indicate that the broker already has prior session information of the client like subscriptions and queued messages.
• Error codes on failed subscriptions: Prior to MQTT 3.1.1, it was impossible for clients to find out if the MQTT broker didn’t approve a subscription, which could be the case when using fine-grained permissions for MQTT topics. The new spec changes that and adds a new error (0x80) in the MQTT SUBACK message, so clients can react on forbidden subscriptions.
• Anonymous MQTT clients: The MQTT client identifier can be set to zero byte length. The MQTT broker will assign a random client identifier to the client temporarily.
• Immediate publishes: MQTT clients now have the ability to send MQTT PUBLISH messages before waiting for a CONNACK response of the MQTT broker.
• No client identifier restrictions: MQTT 3.1 had a limit of 23 bytes per client identifier. With the removal of this artificial restriction, client IDs can now use up to 65535 bytes.