Wolfram Hempel is the co-founder of deepstreamIO, a German tech startup that provides high-performance, secure, and scalable real-time communication services for mobile clients and IoT devices. The text was translated by Wei Jia; please credit Gao Ke Yong Jian for reprints.
Want to send a request to the server and get a response? Just use HTTP! It’s very simple. But when you need to communicate through a persistent bidirectional connection, like with WebSockets, you have other options as well. This article will briefly explain MQTT, XMPP, STOMP, AMQP, WAMP, and other alternatives. One of the frequently cited XKCD comics:
Wait, There’s No Such Thing as a “Real-Time Protocol”!
Indeed, but here I will use the term (Realtime Protocol) as a catch-all for a set of protocols designed for message distribution, data synchronization, and request/response via persistent bidirectional connections.
Let’s categorize them based on their respective purposes:
Pure Messaging
Lower-level protocols (like TCP) are designed to transmit a message from a sender to a receiver. They do not concern themselves with how the message should be constructed, nor do they care about message requests, retrieval, storage, and ensuring security and reliability. Protocols like WebSockets, which sit on top of TCP, add some extra features, such as transmitting metadata using headers, splitting large messages across multiple packets, simple authentication, and routing/redirecting information. Essentially, they are still point-to-point methods of exchanging data. When it comes to building larger, more complex systems, you need a higher-level communication paradigm:
Publish-Subscribe
The publish-subscribe model is a widely used communication method in large-scale systems for many-to-many stateless messaging. “Subscribers” can subscribe to “message topics” (topics, channels, events, namespaces), while “publishers” can publish messages to those “topics,” and all subscribers will receive them through routing. This paradigm is very flexible, efficient, and scalable. It isolates the sender from the receiver, allowing subscribers to freely subscribe to or unsubscribe from topics. It’s similar to our daily subscription to newspapers. There are many protocols that support publish-subscribe, such as MQTT, STOMP, WAMP, and more. So how should we choose?
MQTT
MQTT (Message Queue Telemetry Transport) [2] is a binary protocol primarily used for communication between servers and low-power IoT devices. It sits on top of the TCP protocol and, in addition to providing the fundamental publish-subscribe functionality, offers several other features: different message delivery guarantees (delivery guarantee), “at least once” and “at most once.” It achieves message recovery after reconnection by storing the last confirmed received message.
It is very lightweight and is designed to work well in unstable network environments from both a design and implementation perspective.
When Should You Use It?
It is more suitable for IoT scenarios, supports development in almost all languages, and browsers can also send and receive MQTT messages via WebSocket.
Moreover, there are many options for MQTT Brokers, such as Mosquitto [3] or VerneMQ [4], as well as cloud-based MQTT platforms like HiveMQ [5] or CloudMQTT [6].
STOMP
Streaming Text Oriented Messaging Protocol (STOMP) [7] is a WebSocket communication standard. On top of the usual publish-subscribe semantics, it provides reliable message delivery through a begin/publish/commit sequence and acknowledgment mechanism.
Due to its simplicity and ease of implementation, almost all programming languages have client implementations of STOMP. However, it does not offer advantages in message size and processing speed.
When Would You Use It?
When combined with middleware like Apache Apollo [8] that supports multiple protocols, many interesting integrations can be achieved. For example, constantly updating IoT device data in browser charts.
Binary or Text-Based?
So far, we have discussed two protocols: one binary and the other text-based. Let’s quickly compare the differences: information exchange between computers is achieved by controlling the opening or closing of light or electricity in the cable (logical switches) or controlling the peaks or troughs of WiFi signals. In principle, this is a binary form. Therefore, at this level, all protocols are binary protocols. Once information is sent, the receiver has two options: it can either assemble the 0/1 stream into a byte sequence to retrieve (parse) the information; or it can perform additional steps to convert it into text and then parse this text. The former method is referred to as (binary-based). It saves some expensive operations and is the standard form for transmitting any non-text information, such as images, audio, video, or files. Of course, it can also be used to send text information. For instance, by adding a few bytes to each message to express metadata, such as the length or type of the message, allowing only the actual message data to be converted into text. However, since information exchange in many publish-subscribe architectures is text-based, many protocols choose to simply convert the entire information into text, reducing complexity and improving readability, with the trade-off of requiring additional computational tasks after message reception.
WAMP
Web Application Messaging Protocol (WAMP) [9] attempts to develop an open, text-based protocol standard that combines a publish-subscribe request/response programming model while possessing powerful routing and message delivery strategies. It is currently widely used for integrating crossbar.io [10] routers and autobahn’s high-speed caching clients [11].
Pusher / PubNub & Co
Those real-time communication platforms as a service (Realtime platform-as-a-service) products, such as Pusher or PubNub, usually use their own proprietary protocols. Pusher has publicly released detailed specifications of their JSON-based protocol [12] and encourages third parties or the community to help build client implementations in different languages. While PubNub is somewhat more closed, they currently support a range of other open protocols for interaction, such as MQTT.
Queues
In some scenarios, a simple publish-subscribe model is not enough. This is where queues come into play: they support various message and storage patterns.
By using get/acknowledge strategies, consumers receive some messages from the queue, acknowledge their “consumption,” process them, and then take the next batch of messages. This is a powerful and commonly used method.
Imagine you are building a product similar to Instagram that allows users to upload images and apply various filters. The process of applying filters is computation-intensive. Therefore, it cannot be performed directly during the upload, so the server receiving the image simply records the location in the file system and then adds the task of “which image needs which filter” to the task queue.
Another machine dedicated to processing filters can retrieve this task from the task queue, read the original image file, apply the filter, and confirm that the task has been consumed (completed). After that, to horizontally scale the image processing capability, you simply need to add more consumers (machines processing filters) to handle the task queue.
This model is very easy to scale and can incorporate complex routing strategies, task allocation strategies, etc.
AMQP
Advanced Message Queuing Protocol (AMQP) [13] is a leader among various messaging queue protocols. RabbitMQ [14] and HornetQ [15] are popular middleware implementations of this protocol.
When Would You Use It?
When a simple publish-subscribe model cannot meet your requirements. AMQP is very reliable and powerful. Of course, it and its implementations are not lightweight enough. If you need a lighter option, the next content may be better:
ZeroMQ
ZeroMQ [16] is both a protocol and a set of protocol implementation components. It provides a faster and more decentralized alternative than AMQP.
When Would You Use It?
When you need massive throughput and a message queue that supports your complex workflows without a single point of failure, ZeroMQ is a good choice, though you need to be prepared for a steep learning curve.
JMS
Java Messaging Service (JMS) [17] is both a protocol and the standard implementation of the Java messaging service specification, and it is also part of the Java Enterprise Edition (JEE) specification.
When Should You Use It?
When you are using the Java stack and have paid for the Java Enterprise Platform, JMS is the best choice. If not, consider the aforementioned options first.
Request/Response
Sometimes we just need to send a single request and wait for a response, which can be accomplished with HTTP requests. But since you have already established a persistent connection with the server, why not take advantage of it? This communication process, which uses a persistent connection for request/response, is usually referred to as Remote Procedure Calls (RPC) or Remote Method Invocation (RMI). AMQP or ZeroMQ can implement this pattern through a response queue, while JMS can directly support Java RMI.
DataSync
DataSync is the latest optional solution for implementing real-time communication. DataSync synchronizes data from the data store to the client. Changes made by the client to the data will be synchronized to all subscribers. DataSync hides the details of maintaining data state in real-time communication applications, reducing complexity and greatly speeding up development, but it is still a protocol standard that is not open.
Currently, DataSync is available on several PaaS platforms, such as deepstreamHub [18], Firebase [19], or Realm [20].
In Summary
By utilizing the Distributed Realtime Protocol (DRP) of deepstream [21], we are confident in finding a way to combine all the above concepts into a single protocol while maximizing efficiency in message size, scalability, and interoperability.
References
-
https://xkcd.com/
-
http://mqtt.org/
-
https://mosquitto.org/
-
https://vernemq.com/
-
http://www.hivemq.com/
-
https://www.cloudmqtt.com/
-
https://stomp.github.io/
-
https://activemq.apache.org/apollo/
-
http://wamp-proto.org/
-
http://crossbar.io/
-
http://crossbar.io/autobahn/
-
https://pusher.com/docs/pusher_protocol
-
https://www.amqp.org/
-
https://www.rabbitmq.com/
-
http://hornetq.jboss.org/
-
http://zeromq.org/
-
http://docs.oracle.com/javaee/6/tutorial/doc/bncdq.html
-
https://deepstreamhub.com/
-
https://firebase.google.com/
-
https://realm.io/
-
https://deepstream.io/
Recommended Reading
-
What Do I Care About When Designing Message Middleware?
-
How to Implement a Long-Connected Messaging System Supporting Millions of Users
-
Redis Author’s New Work: Disque Distributed Memory Queue (Part 1)
-
Identifying a JVM Design Flaw Leading to Gradually Increasing GC Times
This article was authored by Wolfram Hempel and translated by Wei Jia. Please credit the source for reprints. For technical original and architectural practice articles, feel free to submit through the public account menu “Contact Us.”
High Availability Architecture
Changing the Way the Internet is Built
Long press the QR code to follow the “High Availability Architecture” public account