Introduction to MQTT Protocol: The Cornerstone of IoT

This article is an excellent piece from the Kanxue forum.

Author from Kanxue forumID: emqttx

With the arrival of the 5G era, the grand vision of the Internet of Things (IoT) is becoming a reality. The number of connected IoT devices reached 7 billion in 2018[^1], and in the next two years, the number of smart water and electricity meters alone is expected to exceed 1 billion[^2].

The massive number of devices connected and managed brings significant challenges to network bandwidth, communication protocols, and platform service architecture.

For IoT protocols, it is essential to address several key issues in IoT device communication: the network environment is complex and unreliable, memory and flash storage capacities are small, and processor capabilities are limited.

The MQTT Protocol is an IoT communication protocol based on the publish/subscribe model. With its simple implementation, support for QoS, and small message size, it occupies a significant share of the IoT protocol landscape:

The Birth of the MQTT Protocol

MQTT was created by Andy Stanford-Clark of IBM and Arlen Nipper (then of Arcom Systems, later CTO of Eurotech).

According to Arlen Nipper in an IBM Podcast, MQTT was originally named MQ TT, with a space between MQ and TT. Its full name is: MQ Telemetry Transport, developed in the early 1990s while he was working on a pipeline SCADA system for Conoco Phillips, a real-time data transmission protocol.

The purpose was to allow sensors to communicate with IBM’s MQ Integrator via bandwidth-limited VSAT. Since Nipper came from a remote sensing and data acquisition background, he named it according to industry conventions.

Design Principles of the MQTT Protocol

According to Nipper’s introduction, MQTT must be simple and easy to implement, support QoS (due to the complex network environment), be lightweight and bandwidth-efficient (as bandwidth was expensive at that time), be data agnostic (not concerned with the payload data format), and have persistent session awareness (always knowing whether the device is online).

The following introduces several core features of MQTT (version 3.1.1) corresponding to these design principles.

Flexible Publish/Subscribe and Topic Design

The publish/subscribe model is a decoupling solution compared to the traditional client/server model. The publisher communicates with consumers through a broker, whose role is to correctly send the received messages to consumers based on certain filtering rules. The advantages of the publish/subscribe model over the client/server model include:

Publishers and consumers do not need to know about each other’s existence in advance, such as not needing to communicate each other’s IP address and port in advance.
Publishers and consumers do not need to operate simultaneously since the broker is always running.

In the MQTT protocol, the aforementioned filtering rules are called topics. For example: all messages published to the topic ‘news’ will be forwarded by the broker to subscribers who have subscribed to ‘news’:

In the above image, the subscriber has pre-subscribed to ‘news’, and then the publisher sends a message ‘some msg’ to the broker, specifying it to be published to the ‘news’ topic. The broker matches the topic and decides to forward this message to the subscriber.

The topics in MQTT have a hierarchical structure and support wildcards + and #:

+ is a wildcard that matches a single level. For example, ‘news/+’ can match ‘news/sports’, and ‘news/+/basketball’ can match ‘news/sports/basketball’.
# is a wildcard that matches one or more levels. For example, ‘news/#’ can match ‘news’, ‘news/sports’, ‘news/sports/basketball’, and ‘news/sports/basketball/x’, etc.

Topics in MQTT do not need to be created in advance; when a publisher sends a message to a topic or a subscriber subscribes to a topic, the broker will automatically create that topic.

Minimizing Bandwidth Consumption

The MQTT protocol minimizes the additional consumption of the protocol itself, with the message header requiring a minimum of only 2 bytes.

The message format in MQTT consists of three parts:

The main message types of MQTT include:

CONNECT / CONNACK
PUBLISH / PUBACK
SUBSCRIBE / SUBACK
UNSUBSCRIBE / UNSUBACK
PINGREQ / PINGRESP
DISCONNECT

Among them, the PINGREQ / PINGRESP and DISCONNECT messages do not require a variable header and have no payload, meaning their message size only consumes 2 bytes.

In the variable-length header of the CONNECT message, there is a field for Protocol Version. To save space, it only uses one byte. Therefore, the version number is not stored as the string “3.1.1” but is represented by the number 4 to indicate version 3.1.1.

Three Optional QoS Levels

To adapt to different network environments of devices, MQTT has designed three QoS levels: 0, 1, 2:

At most once (0)
At least once (1)
Exactly once (2)

QoS 0 is a “fire and forget” message sending mode: the sender (which may be the publisher or broker) sends a message and does not care whether it was received by the other party, nor does it set any retransmission mechanism.

QoS 1 includes a simple retransmission mechanism; the sender sends a message and waits for the receiver’s ACK. If no ACK is received, the message is retransmitted. This mode ensures that the message reaches at least once but does not guarantee that the message is not duplicated.

QoS 2 is designed with a slightly more complex retransmission and duplicate message discovery mechanism, ensuring that the message reaches the other party and strictly arrives exactly once.

Session Persistence

MQTT does not assume that devices or brokers use TCP’s keep-alive mechanism but instead designs a protocol-level keep-alive mechanism: in the CONNECT message, a Keepalive field can be set to specify the sending interval of the keep-alive heartbeat packets PINGREQ/PINGRESP.

When the broker does not receive a PINGREQ from the device for an extended period, it will consider the device to be offline.

Overall, keep-alive has two functions:

Detecting the death of the peer or network interruption
Maintaining the connection from being disconnected by network devices during long periods of inactivity

For devices that want to receive messages missed during offline periods after coming back online, MQTT designs persistent connections—by setting the CleanSession field to False in the CONNECT message, the broker will store for the terminal:

All subscriptions of the device
QoS 1 and QoS messages that have not been confirmed by the device
Messages missed while the device was offline

Online Status Awareness

MQTT designs a Last Will message, allowing the broker to help the device publish a last will message to a specified topic in case it detects an abnormal offline situation.

In fact, in some implementations of MQTT servers (such as EMQ X), when a device comes online or offline, the broker publishes device status updates through certain system topics, which better fits practical application scenarios.

Choosing an Open Source MQTT Server

So far, several popular open-source MQTT servers include:

1. Eclipse Mosquitto

An MQTT server implemented in C language. The Eclipse organization also includes a large number of MQTT client projects: https://www.eclipse.org/paho/#

2. EMQ X

An MQTT server developed in Erlang, with a powerful built-in rule engine that supports many other IoT protocols like MQTT-SN, CoAP, LwM2M, etc.

3. Mosca

An MQTT server developed in Node.JS, simple and easy to use.

4. VerneMQ

An MQTT server also developed in Erlang.

Considering aspects like support for MQTT 5.0, stability, scalability, and clustering capabilities, EMQ X performs the best:

Developed using Erlang OTP, with good fault tolerance (a language proven in the telecommunications field, which has achieved 99.9999999% availability in switch devices^5)
The official has a large number of extension plugins available. Many authentication plugins and data storage (backend) plugins are available for selection. It supports various relational databases, NoSQL databases, and common message queues like Kafka, RabbitMQ, Pulsar, etc.
Supports clustering and horizontal scaling of nodes.
A single node supports 2,000K concurrent connections.

Supports rule engines and encoding/decoding.

Quick Experience with the MQTT Protocol

Online MQTT Server

EMQ X MQTT IoT cloud service provides an online public MQTT 5.0 server, allowing you to quickly start learning, testing, or prototyping the MQTT protocol without any installation.

For detailed access information of this MQTT server, please visit the official EMQ webpage: Free Online MQTT Server.

Online MQTT Client

EMQ also provides an MQTT online client tool that supports browser access. This tool supports connecting to the MQTT server via ordinary or encrypted WebSocket ports and also supports caching connections for convenient next-time access.

[^1]: The number of connected devices that are in use worldwide now exceeds 17 billion, with the number of IoT devices at 7 billion… https://iot-analytics.com/state-of-the-iot-update-q1-q2-2018-number-of-iot-devices-now-7b/

[^2]: The estimated installed base of smart meters (electricity, gas, and water) is expected to surpass the 1 billion mark within the next 2 years. https://iot-analytics.com/smart-meter-market-2019-global-penetration-reached-14-percent/

– End –

Introduction to MQTT Protocol: The Cornerstone of IoT

Kanxue ID: emqttx

https://bbs.pediy.com/user-897093.htm

*This article is original by Kanxue forum emqttx, please indicate the source if reprinted from Kanxue community.

☟

Activity Zone

Leave a message below this article,

The top liked message can receive a Kanxue forum conversion invitation code (worth 1000 snow coins).

Using the invitation code can successfully convert temporary members to formal members!