Introduction to MQTT Protocol Concepts

Background

From this chapter onwards, unless otherwise specified, the version of MQTT referred to in this article is 3.1.1.

The MQTT protocol is one of the commonly used protocols in the Internet of Things, known as a “lightweight IoT messaging protocol”. MQTT and HTTP both belong to the seventh layer (application layer).

For friends who are not very familiar with network layering, please click: “Network OSI Seven Layer Model and Functions of Each Layer and TCP/IP”. Reference: “What is the Difference Between MQTT and Modbus in IoT Gateways”, “Introduction to MQTT” Document: “MQTT Protocol Chinese Version Materials”, “MQTT Protocol English Version Materials”.

History of MQTT

In the Internet of Things, open-source and open standards are fundamental elements. The development history of MQTT is roughly as follows:

• In 1999, IBM and its partners invented the MQTT protocol.

• In 2004, MQTT.org opened a forum for widespread participation.

• In 2011, IBM established the Eclipse open-source project Paho and contributed code. Eclipse Paho is the Java implementation of MQTT.

• In 2013, the OASIS MQTT Technical Committee was formed.

• In 2014, MQTT was officially recognized as the recommended standard for IoT transport protocols.

Concepts

MQTT is a machine-to-machine (M2M) / Internet of Things (IoT) connectivity protocol. It is designed as an extremely lightweight publish/subscribe messaging transport protocol. It is very useful for remote connections that require small code footprint and/or have very precious network bandwidth, specifically designed for constrained devices and low-bandwidth, high-latency, or unreliable networks. These principles also make this protocol an ideal choice for emerging “machine-to-machine” (M2M) or Internet of Things (IoT) connected devices, as well as for mobile applications with very high bandwidth and battery power requirements. For example, it has been used for sensors communicating with proxies via satellite links, dial-up connections with medical service providers, and a range of home automation and small device scenarios. It is also an ideal choice for mobile applications because it is small in size, low in power consumption, has minimal data packets, and can efficiently distribute information to one or more receivers.

Why not choose one of the many other network protocols?

Most developers are already familiar with HTTP web services. So why not connect IoT devices to web services? Devices can send their data in the form of HTTP requests and receive updates from the system in the form of HTTP responses. This request-response model has some serious limitations:

1. HTTP is a synchronous protocol. The client must wait for the server’s response. Web browsers have this requirement, but at the cost of sacrificing scalability. In the IoT field, a large number of devices and likely unreliable or high-latency networks make synchronous communication problematic. Asynchronous messaging protocols are more suitable for IoT applications. Sensors send readings, allowing the network to determine the best route and timing for delivering them to target devices and services.

2. HTTP is one-way. The client must initiate the connection. In IoT applications, devices or sensors are often the clients, meaning they cannot passively receive commands from the network.

HTTP is a 1-1 protocol. The client makes a request, and the server responds. Delivering messages to all devices on the network is not only difficult but also expensive, which is a common use case in IoT applications.

1. HTTP is a heavyweight protocol with many headers and rules. It is not suitable for constrained networks.

For the above reasons, most high-performance, scalable systems use asynchronous message buses for internal data exchange instead of web services. In fact, the most popular messaging protocol used in enterprise middleware systems is called AMQP (Advanced Message Queuing Protocol). However, in high-performance environments, computing power and network latency are usually not issues. AMQP is focused on achieving reliability and interoperability in enterprise applications. It has a large feature set but is not suitable for resource-constrained IoT applications.

In addition to AMQP, there are other popular messaging protocols. For example, XMPP (Extensible Messaging and Presence Protocol) is a peer-to-peer instant messaging (IM) protocol. It heavily relies on features that support IM use cases, such as presence status and media connections. Compared to MQTT, it requires much more resources on devices and networks.

So why is MQTT so lightweight and flexible? One key feature of the MQTT protocol is the publish and subscribe model. Like all messaging protocols, it separates the data publisher from the users.

Let’s first take a look at what a typical MQTT network topology looks like before introducing related concepts.

Roles

Implementing the MQTT protocol requires communication between client and server, during which there are three identities in the MQTT protocol: publisher (Publish), broker (Broker) (server), and subscriber (Subscribe). Among them, both the message publisher and subscriber are clients, while the message broker is the server, and the message publisher can also be a subscriber. When application messages are transmitted via MQTT, they have associated quality of service (QoS) and topics (Topic).

Complete Process

• 1) Start the server broker.

• 2) The subscriber subscribes to relevant topics from the server broker.

• 3) The publisher publishes topic information to the server broker.

• 4) The server broker pushes messages to all subscribers subscribed to that topic.

Client refers to programs or devices that use MQTT. Clients always connect to the server through the network. They can:

• 1) Publish application messages to other relevant clients.

• 2) Subscribe to request to receive relevant application messages.

• 3) Unsubscribe to remove the request for receiving application messages.

• 4) Disconnect from the server.

Server refers to a program or device that acts as an intermediary between the client sending messages and the client requesting subscriptions. The server can:

• 1) Accept network connections from clients.

• 2) Accept application messages published by clients.

• 3) Process clients’ subscription and unsubscription requests.

• 4) Forward application messages to qualified client subscriptions.

Terminology

Subscription refers to a subscription that includes a topic filter and a maximum quality of service (QoS) level. Subscriptions are associated with a single session. A session can contain more than one subscription. Each subscription in a session has a different topic filter.

Topic Name is a label attached to application messages that is known to the server and matches the subscription. The server sends a copy of the application message to each matching client subscription. The topic name is used to identify the channel of the published message information. Subscribers use it to determine the information they are interested in. It is a hierarchical structure, using a slash “/” as a separator (similar to RESTful style).

Topic Filter is an expression included in the subscription that represents one or more relevant topics. Topics can also be filtered using wildcards. Here, “+” can filter one level, while “#” can only appear at the end of the topic to indicate filtering any level. It is worth noting that MQTT allows wildcards for subscribing to topics, but does not allow wildcard broadcasting.

For example:

building-b/<span>floor</span>-<span>5</span>: represents the device on the 5th floor of Building B. +/<span>floor</span>-<span>5</span>: represents any building's 5th floor device. building-b/<span>#</span>: represents all devices in Building B.

Session refers to the state interaction between the client and server. Some sessions last as long as the network connection, while others can extend across multiple consecutive network connections between the client and server.

Control Packet refers to the information data packets sent over the network connection. The MQTT specification defines fourteen different types of control packets, one of which (PUBLISH packet) is used for transmitting application messages.

The messages transmitted by MQTT consist of two parts: topics (Topic) and payloads (payload):

• (1) Topic can be understood as the type of message. After subscribing (Subscribe), the subscriber will receive the message content (payload) of that topic;

• (2) Payload can be understood as the content of the message, referring to the specific content that the subscriber needs to use.

Network Structure

Design Specifications

Due to the unique environment of the Internet of Things, MQTT follows the following design principles:

• (1) Minimalism, avoiding unnecessary features;

• (2) Publish/Subscribe (Pub/Sub) model, facilitating message transmission between sensors, one-to-many message publishing;

• (3) Allow users to dynamically create topics with zero operational cost;

• (4) Minimize the transmission volume to improve transmission efficiency;

• (5) Consider factors such as low bandwidth, high latency, and unstable networks;

• (6) Support continuous session control;

• (7) Understand that client computing power may be very low;

• (8) Provide quality of service management;

• (9) Assume data ignorance, not enforcing the type and format of transmitted data, maintaining flexibility.

Features

• Open messaging protocol, simple to implement

• Publish-subscribe model, one-to-many message publishing

• Based on TCP/IP network connection, providing ordered, lossless, bidirectional connection.

• 1-byte fixed header, 2-byte heartbeat message, minimizing transmission overhead and protocol exchange, effectively reducing network traffic.

• Message QoS support, ensuring reliable transmission

Applications

The MQTT protocol is widely used in the fields of IoT, mobile internet, smart hardware, vehicle networking, and energy.

• IoT M2M communication, IoT big data collection

• Android message push, WEB message push

• Mobile instant messaging, such as Facebook Messenger

• Smart hardware, smart furniture, smart appliances

• Vehicle networking communication, electric vehicle station data collection

• Smart cities, remote medical care, remote education

• Electricity, oil, and energy industries

Subscriptions, Topics, and Sessions in MQTT Protocol

1. Subscription

Subscription includes a topic filter and maximum quality of service (QoS). Subscriptions are associated with a session. A session can contain multiple subscriptions. Each subscription in a session has a different topic filter.

2. Session

Each client establishes a session with the server upon connection, with state interaction between the client and server. Sessions exist across a network and may span multiple consecutive network connections between the client and server.

3. Topic Name

A label connected to an application message that matches the server’s subscription. The server sends messages to each client that matches the subscribed label.

4. Topic Filter

A wildcard filter for topic names used in subscription expressions, indicating multiple topics matched by the subscription.

5. Payload

The specific content received by the message subscriber.

Brief Introduction to MQTT-SN Protocol

This chapter is supplemented after discovering it while downloading the MQTT source code.

MQTT-SN (Sensor Networks) is the sensor version of the MQTT protocol, as the MQTT protocol based on TCP is still too heavy for some sensors that may only have a few dozen bytes of memory and cannot run the TCP protocol. MQTT-SN makes appropriate optimizations for memory-constrained microprocessors to allow it to run on such processors.

In other words, a conversion is needed between MQTT and MQTT-SN to communicate. This conversion operation is performed by the MQTT-SN gateway.

Origin of the Name MQTT-SN

The original name was MQTT-S, but it caused misunderstanding, so it was renamed to MQTT-SN:

As part of the job of applying the same or similar license terms to the MQTT-S specification as those on the MQTT specification, we are proposing a small name change. The new name would be MQTT-SN, standing for exactly the same long name, MQTT for Sensor Networks. Some people had assumed that the S in MQTT-S stood for secure, so we hope this change will avoid that confusion. – Ian Craggs

Purpose of MQTT-SN

MQTT for Sensor Networks is aimed at embedded devices on non-TCP/IP networks, such as Zigbee. MQTT-SN is a publish/subscribe messaging protocol for wireless sensor networks (WSN), with the aim of extending the MQTT protocol beyond the reach of TCP/IP infrastructure for Sensor and Actuator solutions.

It is a specific version of the MQTT protocol adapted for sensor devices (abbreviated as SA), generally running in embedded battery-powered electronic components, with transmission through wireless sensor networks (WSN) based on the IEEE 802.15.4 specification, also possessing enterprise-level features with data-centric publish/subscribe characteristics.

In summary, it is customized for low-power, battery-driven, resource-constrained devices that do not support TCP/IP protocol stack networks, such as the commonly used ZigBee (or XBee), which is agnostic to the underlying transport network it relies on, as long as the network supports bidirectional data transmission and gateways, it can support the upper-layer MQTT-SN protocol transmission. For example, simple datagram services, as long as they support a source endpoint sending data to a specific destination endpoint, are sufficient to support MQTT-SN protocol. Broadcast datagram transmission services are also essential for the automatic discovery process of gateways and terminals. To reduce broadcast storms, MQTT-SN defines broadcast path depth (broadcast range or broadcast radius).

Some Terms and Terminology

• Topic ID, a two-byte 16-bit natural number (Java language short type, range 0-65535), corresponding to the topic topic name.

• Gateway/Server, uniformly referred to as a gateway in MQTT-SN, mainly handling interactions with MQTT-SN clients, abbreviated as gateway.

• MQTT-SN terminal and client, uniformly referred to as the client, which is actually embedded sensor devices or electronic components, resource-constrained, running in wireless personal area networks.

• IEEE 802.15.4, the complete stack has a data upper limit of 128 bytes, generally choosing UDP (compared to the 20-byte TCP protocol, UDP message headers only require 8 bytes) for data transmission.

• Low-speed network/current network refers to LR-WPAN (low-rate wireless personal area network).

MQTT-SN vs MQTT

Although MQTT-SN is designed to be as close to MQTT as possible, the network scenarios for low-power, battery-driven, resource-constrained devices are characterized by low-speed bandwidth, high connection failure rates, and physical layer data packet upper limits of 128 bytes. The document outlines the following differences:

1. CONNECT messages are split into three messages (CONNECT, WILLTOPIC, WILLMSG), the latter two are used for clients to convey will topics and will messages, etc.

2. In PUBLISH messages, the topic (topic name) is replaced with a two-byte natural number (topic ID), which requires the client to obtain the corresponding topic ID through a registration process.

3. Predefined (predefined) topic ID and topic name, eliminating the intermediate registration process, requiring clients and gateways to specify them in their firmware in advance.

4. The protocol introduces an automatic discovery mechanism to help clients discover potential gateways. If multiple gateways exist, they can coordinate as master-slave backup or load balancing.

5. “Clean session” can be applied to subscription persistence, and it has also been extended to will features (will topic and will message).

6. Support for offline keep-alive mechanisms is added for sleeping devices, where the broker needs to cache messages when there are messages, sending them when the client is awakened.

MQTT-SN Architecture Diagram

In the MQTT-SN architecture diagram, there are three components:

1. MQTT-SN Client

2. MQTT-SN Gateway, which can exist independently or be integrated into the MQTT server. It needs to handle the conversion between MQTT-SN and MQTT protocols.

3. MQTT-SN Forwarder, responsible for forwarding current client data to gateways that cannot be accessed directly. When the gateway is not directly accessible to the client, the role of the forwarder becomes prominent. The forwarder encapsulates MQTT-SN messages and forwards them to the gateway, and unpacks messages from the gateway to send to the client. The gateway cannot tamper with the original data.

MQTT-SN Transmission Gateway

The MQTT-SN gateway transmission method is clear in the image below. A transparent gateway establishes a TCP connection channel to the MQTT server for each client, which can consume network resources, but the model is simple.

1. An aggregation gateway establishes only one TCP connection channel to the MQTT server, with all clients sharing one channel, which is economical.

The gateway needs to decide which messages need to be interacted with the remote MQTT Server, such as only selecting PUBLISH and SUBSCRIBE messages sent by clients.