Previously, we introduced AWS’s IoT solution in the article “AWS IoT Solution Analysis“. An important protocol used in the IoT solution is MQTT, which we will learn about here.
Introduction to MQTT Protocol
MQTT (Message Queuing Telemetry Transport) is an instant messaging protocol developed by IBM. This protocol supports all platforms and can connect almost all networked devices to external systems, serving as a communication protocol for sensors and actuators (for example, connecting homes to the internet through Twitter).
Although HTTP is the de facto standard for web pages, large-scale communication between machines (Machine-to-Machine, M2M) requires a different model: the traditional request/response model is no longer suitable, and it is replaced by the publish/subscribe model. This is where the lightweight and scalable MQTT (Message Queuing Telemetry Transport) can demonstrate its capabilities.
MQTT is a messaging protocol based on a binary message publish/subscribe programming model, originally proposed by IBM, and has now become an OASIS standard. Due to its simplicity, it is very suitable for IoT scenarios that require low power consumption and limited network bandwidth, such as:
· Remote sensing data
· Automotive applications
· Smart homes
· Smart cities
· Healthcare
Design Features of MQTT
Since the environment of the Internet of Things is very special, MQTT follows the following design principles:
1. Minimalist, avoiding unnecessary features.
2. Publish/subscribe (Pub/Sub) model that facilitates message transmission between sensors.
3. Allows users to dynamically create topics, with zero operational costs.
4. Minimizes transmission volume to improve transmission efficiency.
5. Considers factors such as low bandwidth, high latency, and unstable networks.
6. Supports continuous session control.
7. Understands that client computing capabilities may be very low.
8. Provides quality of service management.
9. Assumes data is unknown, not enforcing the type and format of transmitted data, maintaining flexibility.
Getting Started with MQTT Protocol
Using the MQTT protocol, devices can easily connect to IoT cloud services, manage devices, and process data, which can then be applied to various business scenarios, as shown in the figure below:
Publish/Subscribe Model
Unlike the synchronous request/response model, the publish/subscribe model decouples the relationship between the client that publishes messages (the publisher) and the client that subscribes to messages (the subscriber). This means that the publisher and subscriber do not need to establish a direct connection. For example, when you call a friend, you have to wait until they answer before you can start communicating, which is a typical synchronous request/response scenario; whereas sending an email to a friend’s mailing list is different; you send the email and go about your business, and your friends can check the email whenever they have time, which is a typical asynchronous publish/subscribe scenario.
Those familiar with programming are certainly familiar with this design pattern, as it offers the following benefits:
· Publishers and subscribers do not need to know each other, only need to recognize the same message broker.
· Publishers and subscribers do not need to interact; publishers do not have to wait for subscribers to confirm, avoiding locking.
· Publishers and subscribers do not need to be online at the same time; they can freely choose when to consume messages.
Topics
MQTT classifies messages through topics, which are essentially UTF-8 strings but can represent multiple hierarchical relationships using slashes. Topics do not need to be created; they can be used directly.
Topics can also be filtered using wildcards. The + wildcard can filter one level, while # can only appear at the end of a topic to filter any level.
For example:
· building-b/floor-5: Represents devices on the 5th floor of Building B.
· +/floor-5: Represents devices on the 5th floor of any building.
· building-b/#: Represents all devices in Building B.
Note that MQTT allows subscribing to topics with wildcards, but does not allow broadcasting with wildcards.
Quality of Service
To meet different scenarios, MQTT supports three different levels of Quality of Service (QoS) to provide message reliability:
· Level 0: At most once. The message sender will try to send the message, but will not retry in case of failure.
· Level 1: At least once. If the message receiver does not acknowledge receipt, the message sender will resend to ensure the message is received at least once, which may cause duplicate messages.
· Level 2: Exactly once. This guarantees that the message is neither lost nor duplicated, but may increase latency and reduce concurrency. Level 2 is most suitable when losing or duplicating messages is unacceptable.
Quality of service is an old topic. The no-loss and no-duplicate guarantees provided by Level 2 are often ideal, but the multiple round-trip confirmations do impact concurrency and latency. Level 1’s at-least-once guarantee is perfectly acceptable in scenarios like log processing, so systems like Kafka leverage this characteristic to reduce confirmations and significantly improve concurrency. Level 0 is suitable for data that is of little value, where you can take it or leave it.
Message Types
MQTT has 14 different message types:
1. CONNECT: Client connects to the MQTT broker.
2. CONNACK: Connection acknowledgment.
3. PUBLISH: New published message.
4. PUBACK: Acknowledgment of a new published message, a response to a PUBLISH message with QoS 1.
5. PUBREC: The first part of a QoS 2 message flow, indicating the message has been recorded.
6. PUBREL: The second part of a QoS 2 message flow, indicating the message has been released.
7. PUBCOMP: The third part of a QoS 2 message flow, indicating the message has been completed.
8. SUBSCRIBE: Client subscribes to a topic.
9. SUBACK: Acknowledgment of a SUBSCRIBE message.
10. UNSUBSCRIBE: Client unsubscribes from a topic.
11. UNSUBACK: Acknowledgment of an UNSUBSCRIBE message.
12. PINGREQ: Heartbeat.
13. PINGRESP: Acknowledgment of heartbeat.
14. DISCONNECT: Client gracefully notifies the MQTT broker before terminating the connection.
Similarities and Differences Between MQTT and Kafka
Although both evolved from traditional Pub/Sub messaging systems, they have evolved in different directions. Here are some prominent points:
1) Kafka is designed for data integration scenarios, unlike traditional Pub/Sub message buses, it provides massive message processing, high fault tolerance, and guarantees the order of data streams through a distributed architecture.
2) MQTT is optimized for IoT scenarios, providing multiple QoS options (exactly once, at least once, at most once) and features like hierarchical topics and last will.
In short, both are products of traditional messaging systems combined with different scenarios. However, they can be used together. For example, MQTT can be used to receive data uploaded by IoT devices, which can then be integrated with Kafka, and finally distributed to HDFS for archiving, data warehouses for OLAP analysis, and Elasticsearch for full-text search. This architecture is very suitable for large IoT projects, as it can handle massive data while also providing good scalability.
References
1) Getting Started with MQTT: http://dataguild.org/?p=6817