Introduction: The First Groundbreaking Product to Introduce QUIC to MQTT
In the recently released version 5.0, EMQX has innovatively introduced QUIC support.
QUIC is the underlying transport protocol for the next-generation internet protocol HTTP/3. Compared to TCP/TLS protocols, it effectively reduces connection overhead and message latency, providing a flexible and efficient transport layer for modern mobile internet.
This article will provide a detailed analysis of MQTT over QUIC, showcasing the advantages and value of this leading technology implementation for IoT scenarios, helping everyone to more effectively leverage the QUIC support capabilities of EMQX 5.0 for more efficient and cost-effective IoT data transmission across various MQTT application scenarios.
QUIC is a general-purpose transport layer network protocol built on top of UDP, originally proposed by Google as a replacement for TCP+TLS, aimed at improving user experience.
Compared to existing TLS over TCP solutions, QUIC has many advantages:
-
Quickly establish low-latency connections (1 RTT or 0 RTT)
-
End-to-end encryption, with authentication through TLS 1.3 during the handshake
-
Avoid head-of-line blocking through multiplexing
-
Improved congestion control with pluggable congestion control strategies
-
Multipath support for smooth connection migration
-
-
Supports existing networks without requiring upgrades
Due to its efficient transmission and multiplexing capabilities, QUIC has become the underlying transport protocol for the next-generation internet protocol HTTP/3.
Introduction to HTTP/3 Protocol
In October 2018, the IETF’s HTTP and QUIC working groups jointly decided to refer to the HTTP mapping over QUIC as HTTP/3 to expedite its establishment as a global standard. On June 6, 2022, the IETF standardized HTTP/3 as RFC 9114.
The goal of HTTP/3 is to provide fast, reliable, and secure web connections on all types of devices by addressing the transport-related issues of HTTP/2. HTTP/3 uses semantics similar to the HTTP/2 version, including the same request methods, status codes, and message fields. The fundamental difference is that HTTP/2 uses TCP/TLS as its underlying protocol, while HTTP/3 uses the QUIC protocol.
According to W3Techs statistics, at least 40% of internet traffic is based on QUIC, with 25% of the top 10 million websites already supporting HTTP/3, including leading sites like Google, YouTube, and Facebook.
The Application Prospects of QUIC in MQTT Communication Scenarios
MQTT is an IoT communication protocol based on TCP, and its compact message structure enables stable transmission on severely constrained hardware devices and low-bandwidth, high-latency networks; features such as heartbeat mechanisms, last will messages, and QoS levels can address various IoT scenarios.
However, due to the limitations of the underlying TCP transport protocol, MQTT has inherent disadvantages in certain complex network environments:
-
Network switching leads to frequent connection interruptions
-
Difficulties in re-establishing connections after disconnection: the operating system releases resources slowly after disconnection, and the application layer cannot promptly perceive the disconnection status, resulting in significant overhead for Server/Client during reconnection
-
Data transmission in weak network environments is limited by congestion, packet loss detection, and retransmission mechanisms
For example, users in the vehicle network often face similar problems: vehicles may operate in mountainous areas, mines, tunnels, etc., where entering signal dead zones or passively switching base stations can lead to connection interruptions. Frequent connection interruptions and slow connection recovery speeds degrade user experience. In some applications requiring real-time and stable data transmission, such as L4 level autonomous driving, customers need to incur significant costs to mitigate this issue.
In such scenarios, the low connection overhead and multipath support characteristics of QUIC show their advantages. Through deeper exploration, we believe that MQTT Over QUIC can effectively address this dilemma — Based on QUIC’s 0 RTT/1 RTT reconnection/new connection capabilities, it can significantly enhance user experience in weak networks and unstable network paths.
The Implementation of MQTT over QUIC in EMQX 5.0
EMQX currently implements QUIC Stream as the transport layer, where the client initiates the connection and creates the Stream, and EMQX and the client interact over a bidirectional Stream.
Considering complex network environments, if the client fails to complete the QUIC handshake for any reason, it is recommended that the client automatically fall back to traditional TCP to avoid losing communication with the server.
Currently, EMQX 5.0 has implemented the following features:
-
Advanced Congestion Control: Effectively reduces data loss rates, maintaining stable data transmission even under network fluctuations during testing
-
Operation and Maintenance Friendly: Reduces overhead caused by large-scale reconnections (time overhead, client/server performance overhead) and minimizes unnecessary application layer state migrations that may lead to system overload (0 RTT)
-
More Flexible Architectural Innovations: For example, Direct Server Return (DSR) mode, where only ingress/request traffic goes through LB, while egress and response traffic bypass LB and return directly to the client, reducing LB bottlenecks
-
Reduced Handshake Latency (1 RTT)
-
Multipath Support, Smooth Connection Migration: QUIC can continue connection states on new quintuplets when switching from 4G to WiFi or due to NAT Rebinding changes, especially suitable for mobile devices that frequently change networks
-
Agile Development and Deployment: The protocol stack implementation is in userspace, allowing for rapid iteration
-
End-to-End Encryption: Unencrypted packet headers contain minimal information, reducing the impact of intermediate nodes in the transmission path, leading to better security and more controllable user experiences
Additionally, more capabilities are yet to be explored:
-
Independent Streams for Different Topics: Each topic can have independent Streams to eliminate the impact of long blocking from other topics, such as receiver long blocking or flow control, and can also implement priority topic features.
-
Streams with Different QoS: For example, in “flow control,” QoS 0 transmission should yield to higher QoS transmissions.
-
Segregating Control Messages into Different Streams: MQTT control messages can be sent unidirectionally or bidirectionally. For instance, the client can asynchronously send an UNSUBSCRIBE request through a “control stream” to request the server to stop sending data it is no longer interested in.
-
More Granular Send/Receive Side Collaborative Flow Control: Implementing flow control for each stream and overall connection to achieve more granular traffic control.
Testing Comparison of QUIC vs TCP/TLS
In a laboratory environment, we simulated tests of the performance of QUIC and TCP/TLS under different scenarios based on EMQX version 5.0.
Testing Environment
-
Testing Platform: EMQX 5.0 Single Node
-
Server Specifications: AWS EC2 M4.2xlarge (8 cores, 32GB)
-
Operating System: Ubuntu 20.04
-
Number of Clients: 5000
-
Loadgen Parallelism: 8
-
Latency Measurement: P95
Client Connection Latency
Testing the latency of handshake, connection establishment, and subscription completion under different network latencies. Compared to TLS, QUIC has certain advantages under higher network latencies.
0 RTT Reconnection Latency
Testing the latency required to re-establish a connection after disconnection.
Since QUIC can carry application layer data on the first packet in a 0 RTT scenario, the application layer responds faster compared to TCP’s round trip handshake.
QUIC protocol supports 0 RTT handshakes, allowing the server to send NST packets to the client after the initial handshake is completed. The client can use NST to quickly rebuild the connection, skipping many steps in the 1 RTT round trip.
The benefits of 0 RTT effectively reduce the handshake overhead between the client and server and improve performance (handshake latency), with EMQX defaulting to send NST packets to clients, effective for 2 hours.
0 RTT also supports early data, which can carry application layer data on the first packet, enabling quick recovery or restart of application layer services. However, since early data in 0 RTT cannot prevent replay attacks, QUIC recommends not to carry data that would change application state in 0 RTT.
* Note: EMQX does not support early data by default; this test is for comparative validation only.
The test results indicate that if the MQTT layer protocol is designed properly, QUIC performs better than pure TCP after the initial handshake.
Server Resource Usage During Connection/Reconnection
Testing the CPU and memory usage on the server during new connections and reconnections after disconnection, comparing the resource overhead during QUIC 1 RTT and 0 RTT handshakes with TLS. The test results indicate that QUIC’s CPU and memory usage are both better than TLS, but rebuilding the connection consumes more bandwidth than TLS.
* Note 1: Mainly for MQTT clearing sessions, the extra overhead of kicking out old connections
Note 2: Mainly due to the large number of QUIC initialization handshake packets caused by transmission path MTU verification
This test simulates the changes in business layer message transmission during large-scale client address migration.
Traditional TCP/TLS clients must perceive the disconnection at the application layer before reconnecting, which is a slow process accompanied by many unnecessary retransmissions. QUIC handles this more smoothly, maintaining the connection at the transport layer without requiring reconnection and making the application layer unaware (if needed, the application layer can subscribe to address changes).
In the case of client source IP address/port changes, QUIC message sending is unaffected. In contrast, TLS connections experience message sending interruptions after changes. Even if the client can reconnect to EMQX through the reconnection mechanism, there will be a time window where no operations can be performed.
This result indicates that QUIC is very suitable for environments where networks frequently need to switch.
Network Packet Loss Testing
Testing data transmission under weak network conditions. We conducted three tests: EMQX terminated TCP/TLS, QUIC, and nginx terminated TCP/TLS.
The test scenario: EMQX publishes QoS 1 messages at a rate of 20K/s, injecting network errors during this process: 20% out-of-order (the order of packets sent and received does not match), 10% packet loss, with additional network switching interference every 30 seconds during the QUIC test.
In this case, the data received by the QUIC server showed slight fluctuations but did not lose messages; whereas TLS experienced congestion and packet loss due to poor network conditions. This result indicates that QUIC can provide reliable transmission in weak network environments.
* Note: In the yellow circles, we removed network errors, and it can be seen that TLS has resumed normal sending and receiving, with consistent packet counts and no backlog, while QUIC merely transitioned from slight fluctuations to smoother performance.
Easier Use: MQTT over QUIC SDK
NanoSDK 0.6.0 (https://github.com/nanomq/NanoSDK/) is the first C language MQTT over QUIC SDK based on the MsQuic project.
NanoSDK adds QUIC support to the transport layer of NNG, enabling protocols like MQTT and nanomsg to transition from TCP to UDP, thereby providing a better IoT connection experience. Internally, it maps QUIC Streams to MQTT connections and implements 0 RTT fast handshake reconnection functionality.
* For message example code, please refer to the NanoSDK QUIC Demo: https://github.com/nanomq/NanoSDK/blob/main/demo/quic/client.c
We will also encapsulate and gradually release SDKs for languages like Python and Go based on NanoSDK, allowing more users to quickly experience the advantages of MQTT over QUIC.
Additionally, the related SDKs will support QUIC fallback, automatically switching to TCP/TLS 1.2 when QUIC is unavailable, ensuring that services can operate normally in various network environments.
NanoSDK communicates messages between EMQX and the client via QUIC
Combining QUIC features with IoT scenarios, we have planned numerous features for MQTT over QUIC, such as setting topic priority through distinguishing control channels, implementing unreliable real-time stream transmission to address high-frequency data transmission scenarios, and flexible topic and data channel (Stream) mapping to reduce interference between topics. These features will be gradually presented in future versions.
EMQ is also actively promoting the standardization of MQTT over QUIC. After becoming the only Chinese company with voting rights in the OASIS MQTT Technical Committee in 2018 and participating in the formulation of the 5.0 protocol standard, EMQ has now submitted related drafts for MQTT over QUIC.
We believe that in the near future, the underlying protocol for MQTT will support both TCP and QUIC, benefiting the entire IoT industry.
As we can see, QUIC is very suitable for weak network environments where traditional TCP/IP networks can ensure UDP MTU size or where networks frequently switch.
For IoT scenarios where devices are constantly on the move (such as vehicle networks, mobile collection, etc.), or scenarios that require frequent disconnections and are not suitable for long connections (such as devices needing to periodically sleep), QUIC has enormous potential and is a more suitable choice for the underlying protocol. This is also the reason why EMQX 5.0 introduced QUIC support.
MQTT over QUIC’s pioneering implementation in EMQX 5.0 puts EMQ at the forefront of the global IoT messaging server field once again. EMQ will continue to drive product iteration and upgrade through continuous technological innovation, aiming to provide infrastructure support and business innovation momentum for the IoT field through leading products.
Source: IoT Technology
