Comprehensive Summary of MQTT Protocol

This post is from a handsome young graduate from Beijing in 2017 – Xiao Xiao Su.

1. Concept

MQTT is an IoT communication protocol based on the Publish/Subscribe model. With its simplicity, support for QoS, and small message size, it occupies a significant share of IoT protocols.

MQTT (Message Queuing Telemetry Transport) is a “lightweight” communication protocol based on the publish/subscribe model, built on the TCP/IP protocol, and released by IBM in 1999. The greatest advantage of MQTT is that it can provide real-time and reliable messaging services for connecting remote devices with minimal code and limited bandwidth. As a low-overhead, low-bandwidth instant messaging protocol, it has a wide range of applications in IoT, small devices, and mobile applications.

Summary:

As noted above, MQTT is commonly used in the IoT field, based on the subscription/publish model, characterized by low overhead and low bandwidth consumption.

2. Protocol Specifications & Features

Due to the unique environment of IoT, MQTT follows the following design principles:

1. Simplification, adding no unnecessary features;

2. Publish/Subscribe (Pub/Sub) model, facilitating message transfer between sensors, decoupling the Client/Server model, which means that there is no need to know each other’s existence (IP/port) in advance, and they do not need to run simultaneously;

3. Allows users to dynamically create topics (no need to pre-create topics), zero operational cost;

4. Minimize transmission volume to enhance transmission efficiency;

5. Consider factors such as low bandwidth, high latency, and unstable networks;

6. Support for continuous session maintenance and control (heartbeat);

7. Understand that client computing power may be very low;

8. Provide Quality of Service (QoS) management;

9. Do not impose restrictions on the type and format of transmitted data, maintaining flexibility (referring to application layer business data).

The MQTT protocol is designed for communication between remote sensors and control devices in low bandwidth and unreliable networks, featuring the following main characteristics:

1. An open messaging protocol, simple to implement.

2. Uses a publish/subscribe messaging model, providing one-to-many message publishing, decoupling application programs.

3. Message transmission that masks the content of the payload (application data carried by the protocol).

4. Based on TCP/IP network connections, providing ordered, lossless, and bidirectional connections. The mainstream MQTT is based on TCP connections for data pushing, but there is also a UDP-based version called MQTT-SN. These two versions have different advantages and disadvantages due to their different connection methods.

5. Quality of Service (QoS) support, ensuring reliable transmission;

6. 1-byte fixed header, 2-byte heartbeat message, minimizing transmission overhead and protocol exchanges, effectively reducing network traffic. This is why it is said that it is very suitable for communication between sensors and servers in the IoT field, where the computing power and bandwidth of embedded devices are relatively weak; using this protocol to transmit messages is most appropriate.

7. Online status awareness: Using Last Will and Testament features to notify relevant parties of abnormal client disconnections. Last Will: The will mechanism is used to notify other devices under the same topic that the device sending the will has disconnected. Testament: The will mechanism functions similarly to Last Will.

Summary:

All MQTT services must be designed and implemented according to this specification. We only need to understand these specifications without needing to memorize them.

3. QoS Message Quality Mechanism

QoS0

“At most once,” message publishing completely depends on the underlying TCP/IP network. Message loss or duplication may occur. This level can be used for cases such as environmental sensor data, where losing one reading is not significant because another will be sent soon. This method is mainly used for ordinary app push notifications; if your smart device is not connected to the network during message push, it won’t receive the message when it reconnects.

(TIPS: Send it out and forget it, regardless of whether the broker received it)

QoS1

“At least once,” ensuring the message arrives, but duplication may occur.

(TIPS: At least once means it is sent to the broker once. If it is not delivered to the broker, it will continue to send until it is delivered. If the first sending does not get a response from the broker and a second sending occurs, but the first message has already reached the broker, duplication will happen. The same principle applies when sending from the broker to the consumer client.)

QoS2

“Exactly once,” ensuring the message arrives only once. This level can be used in strict billing systems, where message duplication or loss could lead to incorrect results. This highest quality of message publishing service can also be used for instant messaging apps’ push notifications, ensuring users receive the message and only receive it once.

MQTT’s Quality of Service (QoS) guarantee is not end-to-end; it is between the client and the server. The level of QoS for the subscriber receiving the MQTT message ultimately depends on the QoS of the published message and the QoS of the topic subscription.

The QoS is defined between the producer and broker, and between the consumer and broker, with the final QoS of the message being the minimum of the two.

4. Publish/Subscribe, Topics, Sessions

MQTT communicates and exchanges data based on the Publish/Subscribe model, which is fundamentally different from the HTTP Request/Response model.

Subscribers will subscribe to a topic from the message server (Broker). Once successfully subscribed, the message server will forward messages under that topic to all subscribers.

(TIPS: Here, it is important to note that messages are sent to all consumers subscribed to the topic.)

Topics are distinguished by the ‘/’ separator. Topics containing wildcards ‘+’ or ‘#’ are called Topic Filters;

‘+’: Represents a wildcard for one level, e.g., a/+, matches a/x, a/y

‘#’: Represents a wildcard for multiple levels, e.g., a/#, matches a/x, a/b/c/d

Note: ‘+’ matches one level, ‘#’ matches multiple levels (must be at the end).

Session

Each client establishes a session with the server after connecting, with stateful interactions between the client and server. Sessions exist across networks and may span multiple continuous network connections between the client and server.

5. Methods in the MQTT Protocol

The MQTT protocol defines several methods (also known as actions) to represent operations on specific resources. This resource can represent pre-existing data or dynamically generated data, depending on the server’s implementation. Generally, resources refer to files or outputs on the server. The main methods include:

(1) CONNECT: Client connects to the server

(2) CONNACK: Connection acknowledgment

(3) PUBLISH: Publish message

(4) PUBACK: Publish acknowledgment

(5) PUBREC: Published message received

(6) PUBREL: Published message released

(7) PUBCOMP: Publish complete

(8) SUBSCRIBE: Subscription request

(9) SUBACK: Subscription acknowledgment

(10) UNSUBSCRIBE: Unsubscription

(11) UNSUBACK: Unsubscription acknowledgment

(12) PINGREQ: Client sends heartbeat

(13) PINGRESP: Server heartbeat response

(14) DISCONNECT: Disconnect

(15) AUTH: Authentication

6. MQTT Protocol Packet Structure

In the MQTT protocol, an MQTT packet consists of three parts: Fixed header, Variable header, and Payload. The structure of an MQTT packet is as follows:

(1) Fixed header. Present in all MQTT packets, indicating the packet type and grouping identifier, such as connection, publishing, subscription, heartbeat, etc. The fixed header is mandatory and must be included in all types of MQTT protocols.

(2) Variable header. Present in some MQTT packets, the packet type determines whether the variable header exists and its specific content. The variable header is not optional; it means that this part exists in some protocol types and does not exist in others.

(3) Payload. Present in some MQTT packets, indicating the specific content received by the client. Like the variable header, in some protocol types, there is message content, and in others, there is not.

The fixed header exists in all MQTT packets. The fixed header contains two parts: the first byte (Byte 1) and the remaining message length (starting from the second byte, with a length of 1-4 bytes), where the remaining length is the byte count of the remaining content in the current packet, including the variable header and data in the payload. The remaining length does not include the bytes used to encode the remaining length.

The remaining length uses a variable-length structure to encode, which uses a single byte to represent values from 0-127. Values greater than 127 are processed as follows. The low 7 bits of each byte are used to encode data, and the highest bit indicates whether there are subsequent bytes. Thus, each byte can encode 128 values plus an indicator bit. The remaining length can be represented using up to four bytes.

Packet Type

Position: The 7-4 bits (Bit[7-4]) of the first byte indicate a 4-bit unsigned value

The high 4 bits of the first byte determine the message type; 4 bits can determine 16 types, with 0000 and 1111 being reserved fields.

The MQTT message types are as follows:

Flags

Position: The 0-3 bits (Bit[3-0]) of the first byte. This means that the bits Bit[3-0] are used as message identifiers.

The low 4 bits (bit3~bit0) of the first byte are used to represent control fields for certain message types, and only a few message types have control bits, as shown in the following image:

(1): Where Bit[3] is the DUP field; if this value is 1, it indicates that this packet is a duplicate message; otherwise, it is the first published message.

(2): Bit[2-1] is the QoS field:

If Bit 1 and Bit 2 are both 0, it indicates QoS 0: at most once;

If Bit 1 is 1, it indicates QoS 1: at least once;

If Bit 2 is 1, it indicates QoS 2: exactly once;

If both Bit 1 and Bit 2 are set to 1, the client or server considers this to be an illegal message and will close the current connection.

Variable Header

The variable header means that the message header can change. Some message types contain a variable header, while others do not. The variable header is located between the fixed header and the message content, and its content varies based on the message type.

Protocol Name

The protocol name is a UTF-8 encoded string representing the name of the protocol MQTT. The offset and length of this string will not change in subsequent versions of the MQTT specification.

Servers supporting multiple protocols use the protocol name field to determine whether the data is an MQTT message. The protocol name must be the UTF-8 string “MQTT”. If the server does not wish to accept CONNECT but wants to indicate its MQTT server identity, it can send a CONNACK message with a reason code of 0x84 (unsupported protocol version) and must then close the network connection.

Protocol Version

The unsigned value indicates the client’s version level. The protocol level for version 3.1.1 is 4, and the protocol version field for MQTT v5.0 is 5 (0x05).

MQTT Session (Clean Session)

When an MQTT client initiates a CONNECT request to the server, it can set the session via the ‘Clean Session’ flag.

‘Clean Session’ set to 0 indicates the creation of a persistent session, where the session remains and offline messages are saved until the session times out and is deregistered when the client disconnects.

‘Clean Session’ set to 1 indicates the creation of a new temporary session, which is automatically destroyed when the client disconnects.

Will Flag/Will QoS/Will Retain

If the Will Flag is set to 1, it means that if the connection request is accepted, the server must store a Will Message and associate it with the network connection. Later, when the network connection is disconnected, the Will Message must be published unless the server receives a DISCONNECT packet that deletes the Will Message.

The Will Message will be published in certain situations, including but not limited to:

The server detects an I/O error or network failure.

The client fails to communicate within the Keep Alive time.

The client closes the network connection without sending a DISCONNECT packet.

The server closes the network connection due to protocol errors.

If the Will Flag is set to 1, the Will QoS and Will Retain fields in the connection identifier will be used by the server to indicate the QoS level used when publishing the Will Message.

The Will Retain bit indicates whether the Will Message should be retained after publishing.

If the Will Flag is set to 0, then Will Retain must be 0.

If the Will Flag is set to 1:

If Will Retain is set to 0, the server must publish the Will Message without retaining it.

If Will Retain is set to 1, the server must publish the Will Message and retain it.

User Name Flag

If the User Name Flag is set to 0, the username does not need to appear in the payload.

If the User Name Flag is set to 1, the username must appear in the payload.

Password Flag

If the Password Flag is set to 0, the password does not need to appear in the payload.

If the Password Flag is set to 1, the password must appear in the payload.

If the User Name Flag is set to 0, then the Password Flag must also be set to 0.

MQTT Connection Keep Alive Heartbeat

The purpose of the heartbeat:

PINGREQ packets sent from the client to the server can be used to:

1: Notify the server of the client’s alive status when no other control packets are sent from the client to the server.

2: Request a response from the server to confirm whether it is alive.

3: Confirm the validity of the network connection. PINGRESP packets are sent from the server to the client in response to PINGREQ packets. They indicate that the server is alive.

The MQTT client initiates a CONNECT request to the server, setting the keep-alive period through the KeepAlive parameter.

Keep Alive is a time interval measured in seconds. It is represented by 2 bytes and refers to the maximum time interval from the completion of sending one control packet to the start of sending the next. The client is responsible for ensuring that the interval between sending two control packets does not exceed the value of Keep Alive. If there are no other control packets to send, the client must send a PINGREQ packet.

The client can send a PINGREQ packet at any time without regard to the Keep Alive value, using PINGRESP to determine whether the network connection with the server is normal.

If the value of Keep Alive is non-zero and the server has not received a control packet from the client within one and a half Keep Alive cycles, the server must disconnect the network connection as a network failure.

If the client sends a PINGREQ and does not receive a PINGRESP packet within a reasonable time, it should close the network connection with the server.

If the value of Keep Alive is 0, the maintenance mechanism is turned off. This means that in this case, the server will not disconnect silent clients.

MQTT Last Will Message

When an MQTT client sends a CONNECT request to the server, it can set whether to send a Last Will Message (Will Message) flag, along with the Will Message topic (Topic) and content (Payload).

When the MQTT client goes offline abnormally (without sending a DISCONNECT message to the server before disconnecting), the MQTT message server will publish the Last Will Message.

A common variable header is the Packet Identifier (message ID).

A message ID consists of 2 bytes, with the high byte first and the low byte second. Protocol types that include Packet Identifier are:

PUBLISH (QoS > 0), PUBACK, PUBREC, PUBREL, PUBCOMP, SUBSCRIBE, SUBACK,

UNSUBSCRIBE, UNSUBACK

The message ID starts from 1 and increments. Once a message ID is used up, it can be reused. For PUBLISH (QoS 1), if the sender receives PUBACK, the message ID is used up.

For PUBLISH (QoS 2), if the receiver receives PUBCOMP, the message ID is used up.

For SUBSCRIBE and UNSUBSCRIBE, the completion marker for the message ID is when the sender receives the corresponding SUBACK and UNSUBACK.

Additionally, the message IDs of the client and server are independently allocated; the client and server can use the same message ID simultaneously.

Payload

Some message types contain a Payload, which refers to the message carrier.

The Payload of PUBLISH refers to the message content (the content published by the application). The Payload of CONNECT includes information such as Client Identifier, Will Topic, Will Message, Username, Password, etc.

Message types that include Payload are as follows:

Summary:We introduced the message format of the MQTT protocol, which includes Fixed Header, Variable Header, and Payload. Because the MQTT message format is very concise, it can transmit data efficiently.

Fixed Header includes the first byte, with the high 4 bits used to indicate the message type and the low 4 bits used for type control. Currently, only PUBLISH uses the type control field. Other control fields are reserved and must remain consistent with the protocol definition.

The Fixed Header also includes Remaining Length, which is the length of the remaining message, with a maximum length of 4 bytes. Theoretically, a single MQTT message can transmit up to 256MB of data. Remaining Length = length of Variable Header + Payload.

The Variable Header is the variable part of the header; some message types require a variable header, which varies based on the message type. For example, Packet Identifier is used in publishing, subscribing, and unsubscribing messages.

The Payload is the message content, appearing only in certain message types, and its content and format vary based on the message type.

To all the beauties of the harem!

Xiao Hei is calling all old students of Chuan Zhi to submit articles!!

To help everyone understand their strengths and weaknesses, plan reasonably, and respond effectively,

To remain undefeated in career battles!

For details, see below

1. Submission Types

Learning Sharing (learning methods, insights…)

Technical Articles (Java, Frontend, Python, Big Data, Artificial Intelligence…)

Interview Sharing (interview questions, stories, experiences…)

Workplace Insights (onboarding strategies, workplace anecdotes, gossip…)

No restrictions on style or theme; as long as you want to write, Xiao Hei will help you promote it!

2. Submission Requirements

1. Must be original and first published, plagiarism is rejected; a personal introduction of up to 30 words can be included at the end.

2. Around a thousand words; submissions can include images, but copyright issues must not be involved.

3. Submission Incentives

1. All authors who seriously submit will receive a small gift, regardless of whether their work is selected. If the submission is selected, an upgraded gift will be provided. Unselected articles will receive feedback for improvement, increasing the chance of publication.

2. Each qualified submission (regardless of whether published) can accumulate 1 point; published submissions can accumulate 5 points. Points can also be exchanged for additional prizes.

3. New reading, sharing, likes, and comments rankings will be established; each quarter, the top author on each list will receive a mystery gift prize!!!

4. How to submit

Submissions should be sent directly as an attachment to Xiao Hei’s email: [email protected]