You have successfully built an excellent IoT system: device data transmission is stable and smooth, and the real-time dashboard is functioning well.
However, once asynchronous operations are involved, problems arise: critical firmware updates sent to offline smart locks fail; data processing nodes experience load imbalance, with some nodes idle while others are overwhelmed.
The real-time advantages of MQTT become inadequate when facing asynchronous operations, which is the fundamental reason many IoT projects become complex.
To bridge this gap, architects are forced to introduce independent, powerful message queues like RabbitMQ or Kafka. This solution not only increases the burden of deployment and maintenance but also compels backend teams to learn different protocols such as AMQP and adapt to entirely new client libraries.
What if these issues could be resolved without introducing any additional systems? What if your developers could achieve both real-time publish/subscribe and reliable message queue functionality using only the familiar MQTT protocol and existing client libraries?
Based on the EMQX message queue functionality, this unified and efficient message processing model has become a reality.
Now, let’s explore how this built-in feature addresses three common challenges in the IoT field.
Three Common Challenges, One Integrated Solution
The rule engine is an SQL-based data processing component built into EMQX, designed specifically for IoT scenarios, enabling efficient and low-cost real-time data flow and processing. Nowadays, most IoT systems look like this:
EMQX message queue fills the functional gap between real-time MQTT publish/subscribe and persistent message queues.
Here are some solutions to specific challenges:
Practical Application: Building a Highly Available Workflow
Scenario Description
Assume there is an AI-based image processing process where IoT cameras publish images to the topic <span><span>images/process</span></span>, and a group of worker node applications need to subscribe to this topic to retrieve images and perform OCR (Optical Character Recognition) tasks.
Challenges Faced
Using the standard MQTT publish/subscribe model, you will face two issues:
-
If there are no subscribers online, messages will be discarded directly.
-
Even if you use shared subscriptions to distribute the load, messages still lack persistence: if a worker node crashes after receiving a message but before processing it, that task will be permanently lost.
Solution:
We can map the topic to a persistent queue:
-
Message Publishing: The producer (camera) publishes messages to the topic
<span><span>images/process</span></span>as usual. -
Persistent Storage: EMQX receives the message and persists it to disk according to the queue rules configured for the
<span><span>images/+</span></span>topic filter. -
Queue Subscription: Worker node applications subscribe to the shared queue via the topic
<span><span>$q/images/+</span></span>. -
Smart Scheduling: By setting the distribution strategy to
<span><span>least_inflight</span></span><span>, EMQX intelligently assigns the next task to the node with the least pending tasks, effectively avoiding bottlenecks.</span>
Final Result:
We have achieved a highly resilient and persistently reliable workflow completed within EMQX: tasks are never lost, load is perfectly balanced, and the costs of external message brokers, deployment, and management are completely eliminated.
Practical Application: Using Last Value Queue for State Synchronization
Scenario Description
Assume you have a cloud application for controlling smart lights. Users want to change the color of the lights, but due to unstable Wi-Fi connections, the smart lights are temporarily offline. During this offline period, users change their minds multiple times, sending commands for blue, red, and green in succession.
Challenges Faced
If using a standard queue, the smart light will receive the backlog of three commands when it reconnects. It will sequentially change to blue, red, and green, resulting in a confusing effect. However, we only want it to achieve the final desired state: green.
Solution:
This is a perfect use case for the Last Value Queue.
You can configure the topic <span><span>lights/light-123/commands</span></span> as a last value queue and specify a key in the message. For example, using a user property named <span><span>command_type</span></span><span>, with a value of </span><code><span><span>set-color</span></span><span>.</span>
Now, when commands are published, the queue will only store the latest message for the <span><span>set-color</span></span><span> key. The messages for "blue" and "red" will be automatically discarded when the "green" message arrives.</span>
Final Result:
When the smart light reconnects and subscribes, it will only receive one message: the final “green” command. The device will immediately reflect the final expected state, rather than outdated history.
Difference Between Last Value Queue and MQTT Retained Messages
This is a common point of confusion. While both may seem similar, their purposes are entirely different.
We can use an analogy to understand their differences:
MQTT retained messages are like a notice posted at the entrance of a topic. It only retains the latest message for the entire topic, and any new subscribers connecting will receive a copy of this notice.
EMQX Last Value Queue is like a smart mailbox designed specifically for consumers. It can retain the latest instructions for multiple different keys (e.g., set color and update firmware) and can track the position (offset) that consumers have read. It is a truly state-aware communication channel.
Here is a detailed comparison of the two:
Unified Platform, Simplified Architecture
By integrating persistent queue functionality directly into the MQTT broker, EMQX effectively eliminates the architectural and operational complexities associated with deploying independent messaging systems.
More importantly, this implementation is entirely based on the MQTT protocol. Your devices and backend application teams can use the same set of MQTT clients and libraries they are already familiar with, while handling real-time topics and persistent queues. No new protocols are introduced, no additional complexity, just a unified and powerful messaging platform.
When practicing functionality, keep the following points in mind:
-
Select the appropriate scheduling strategy: For simple task distribution, polling mode can be used; however, in scenarios with uncertain processing times for worker nodes, it is recommended to adopt the least unacknowledged messages strategy.
-
Closely monitor queue status: Real-time monitoring of queue depth and consumer latency is essential for timely application capacity expansion.
Click to read the original text, and start building more resilient IoT applications.
Click “Read the original text” to learn more