Managed MQTT Broker Comparison — Performance Test and Development

This is the third article in the "Managed MQTT Broker Comparison" series. Link to the previous articles in the series:

Managed MQTT Broker Comparison - Product Packages and Pricing

Managed MQTT Broker Comparison - Console / Dashboard Features

Performance Test

According to the comparison in previous articles, I select EMQX Cloud and Cloud MQTT. The specification is the Standard Plan of EMQX Cloud with 1000 connections and CloudMQTT Loud Leopard with 1000 connections. 

The bench software I choose – XMeter. XMeter is a fully managed MQTT load testing service based on Jmeter.  There are three testing scenarios that can be set easily.

EMQX Cloud Performance Test

The dashboard is very comprehensive for monitoring all aspects of the connection. You can easily get understand the connection situation in current.

Publish Mode

The cloud broker receives 1000 messages per second and sends zero because there are no subscriber connections.

Performance Metrics in XMeter, the details can be found here.

One-to-One Mode

The broker receives 500 messages per second and sends 500 messages per second on balance.

Performance Metrics in XMeter, the details can be found here.

Subscribe Mode

The broker receives one message per second from only one publisher and sends 1000 messages to 1000 subscribers per second.

Performance Metrics in XMeter, the details can be found here.

Test Scenarios on Metrics

From the metrics dashboard, the moment of client connections and client disconnections. Meanwhile, the message received, the message sent, and the message dropped correspondingly.

CloudMQTT Performance Test

There is no real-time indicator of connections in the CloudMQTT Dashboard. We can find the clients and messages in MQTT metrics in the long-term chart. 

Publish Mode

Performance Metrics in XMeter, the details can be found here.

One-to-One Mode

Performance Metrics in XMeter, the details can be found here.

Subscribe Mode

Performance Metrics in XMeter, the details can be found here.

Test Scenarios on Metrics

Comparison of the Performance

Both EMQX Cloud broker and Cloud MQTT broker have a good performance from the connection test. In one-to-one mode, Latency is quite observable in CloudMQTT. However, the response success rate is promised.

Integration Capability

“Integration“ means sending messages to other services or systems from MQTT Broker, which in most architecture designs, is a messaging middleware for collecting and delivering. Integration capability is essential to build a whole system with a messaging broker as middleware. 

EMQX Cloud

The integration capabilities are different between the Standard Plan and the Professional Plan.  In the Standard Plan, data can be forwarded to the developer's web service via the rules engine and bridged to other MQTT Brokers, while in the Professional Plan, the options are more diverse. In addition to the features supported in the Standard Plan, data can be forwarded to 30+ data services like message queues or databases in the Professional Plan.

HiveMQ Cloud

It’s introduced that HiveMQ Enterprise Extension for Kafka can be enabled in the Dedicated version. 

Cedalo —  ProEdition

 No data integration yet.

CloudMQTT

CloudMQTT provides two data integration capabilities. A bridge can forward messages to another broker or AWS Kenisis, which is available for the dedicated instance. 

API

EMQX Cloud

Developers can manage authentication, access control, clients, subscription, topics, messages, and metrics through the APIs. With all these interfaces, you can develop a comprehensive cloud MQTT management application that will be well integrated with other systems.

HiveMQ Cloud

Backup, trace, and client information are available through the API. It’s impossible to develop a whole mirror system because of the limited fields. 

Cedalo —  ProEdition

No API provided yet. 

CloudMQTT

It provides the API for authentication and instance management, lacking the capability for querying the clients' information.

Documentation and SDKs

EMQX Cloud

The structure of the documentation is clearly designed. The team detailed the key factors and essential concepts for developers. The content of the documentation is script translated from its Chinese pages with some terminology or syntax issues.

SDK Languages: Python,  Go, Java, Nodejs, C, C#. 

HiveMQ Cloud

The documentation of HiveMQ Cloud is mixed with other products instead of being presented alone. It’s a bit confusing whether the feature is or is not exclusive to the cloud because the documents are shared. 

SDK Languages: Java, Python, Javascript, Go, C, Dart.

Cedalo —  ProEdition

The documentation is thorough and includes an introduction to the basics of the MQTT, making it very friendly for developers who are new to MQTT.

SDK Languages: Not found. 

CloudMQTT

Several major features are introduced, but the content is not too comprehensive enough.

SDK Languages: Ruby, Python, Nodejs, Java, Go, .Net, PHP.

Community and Event

EMQX Cloud

EMQX owns 10.5 stars on Github, indicating a fairly active community. In addition to the open-source project, there are video presentations of key features and Webinar clips on the Youtube channel. 

HiveMQ Cloud

Developers can post questions or issues on the HiveMQ community site and get answers from the team in time. The Youtube channel is also rich in content, including MQTT essentials to help beginners quickly on board and regular Webinars to bring developers the latest news. 

Cedalo —  ProEdition

You can get help by submitting questions in the forum. 

CloudMQTT

No community or event-related content was found on the website.

Comparison of the Development Capabilities and Community

Congratulations to EMQX for winning the most medals in development capabilities and community.