How to Choose Serialization Protocols in MQTT Messaging: Text Serialization vs. Binary Serialization

Serialization of String Messages

In the previous article, we used the MQTTnet framework to implement an MQTT server, an MQTT publisher process, and an MQTT subscriber process. During the messaging process, we directly passed the string from the console. Since MQTT is an application layer protocol, it is based on TCP for data transmission. We know that TCP itself is a byte stream-based transmission protocol. Therefore, our string will ultimately be serialized into a byte array for data transmission. Let’s first look at the code for the publisher sending messages:

string msg;
while (true)
{
    Console.WriteLine("Please enter the message to send:");
    msg = Console.ReadLine();
    msg = "time:" + DateTime.Now.ToString("yyyy-MM-dd HH:MM:ss") + " msg:" + msg;
    var applicationMessage = new MqttApplicationMessageBuilder()
        .WithTopic("mytopic")
        .WithPayload(msg)
        .Build();
    var result = await mqttClient.PublishAsync(applicationMessage, CancellationToken.None);
    Console.WriteLine("Successfully published message to MQTT server.....");
}

As we can see, in the WithPayload function, we directly set a string. Now let’s take a look at how MQTT implements the source code of the WithPayload function. In the following code, we can clearly see that this function calls the Encoding.UTF8.GetBytes(payload) method to convert the string into a byte array before sending it. The Encoding.UTF8.GetBytes is the simplest serialization function provided by C#. This direct serialization of a string is actually text serialization. In actual development, such as game development, we usually use binary serialization protocols instead of text serialization protocols.

public MqttApplicationMessageBuilder WithPayload(string payload)
{
    if (string.IsNullOrEmpty(payload))
    {
        return WithPayload(default(byte[]));
    }

    var payloadBuffer = Encoding.UTF8.GetBytes(payload);
    return WithPayload(payloadBuffer);
}

Text Serialization vs. Binary Serialization

Text serialization converts objects into a human-readable text format. Common text serialization formats include JSON, XML, and YAML. The advantages and disadvantages of text serialization are as follows:

Advantages1. Readability: The serialized data is human-readable, making it easier to debug and log.2. Cross-platform: Most programming languages support parsing and generating common text formats like JSON and XML.3. Flexibility: Text formats can be easily modified and extended.Disadvantages1. Performance: Text serialization is usually slower than binary serialization due to the need for string parsing and generation.2. Space efficiency: Text formats typically occupy more storage space than binary formats.

Binary serialization converts objects into a compact binary format. Common binary serialization formats include Protocol Buffers, MessagePack, and Avro. The advantages and disadvantages of binary serialization are as follows:

Advantages1. Performance: Binary serialization is usually faster than text serialization because it does not require string parsing.2. Space efficiency: Binary formats typically occupy less storage space than text formats.Disadvantages1. Readability: The serialized data is in binary format, making it difficult for humans to read and debug.2. Cross-platform: While many binary formats are cross-platform, not all programming languages support all binary formats.

The final conclusion is: binary serialization is efficient, fast, compact, and has high transmission efficiency. In actual business development, we generally use binary serialization. Of course, in this article, we will use the MessagePack library to serialize objects into binary byte arrays.

MessagePack is an efficient binary serialization format. Unlike JSON serialization, MessagePack is faster and smaller. It was created by Japanese engineer Sadayuki Furuhashi and is maintained as an open-source project on GitHub. MessagePack is suitable for scenarios that require high performance and high space efficiency, such as network communication, storage, and transmission of large amounts of data. The GitHub open-source address is: https://github.com/MessagePack-CSharp/MessagePack-CSharp. This project has 6.1k stars and is an excellent binary serialization open-source project.

Next, we will demonstrate step by step how to use the MessagePack library.

Defining Message Format

Similar to our regular HTTP requests, we may encapsulate a common format for an HTTP message body, which contains three fields: code, data, and message. Here, we use the same model to carry the message content. This is a very simple C# class, and we will not go into further details; it is self-explanatory.

public class Message<T>
{
    public int Code { get; set; }
    public string Msg { get; set; }
    public T Data { get; set; }
}

Installing the MessagePack Package

To use the MessagePack framework, we must first install the NuGet package: MessagePack. The installation command is as follows:

dotnet add package MessagePack

For the object that carries the message body, we must add the MessagePackObject attribute to the class and add the Key attribute to each field in the class, marking the sequence of each field, generally starting from 0 and incrementing sequentially. The modified message class is as follows:

[MessagePackObject]
public class Message<T>
{
    [Key(0)]
    public int Code { get; set; }
    [Key(1)]
    public string Msg { get; set; }
    [Key(2)]
    public T Data { get; set; }
}

Message Serialization

MessagePack has a basic class called MessagePackSerializer, which provides two fundamental functions: Serialize and Deserialize, responsible for serializing an object into binary and deserializing binary back into an object, respectively. We use the builder pattern to construct the message object we will transmit.

public class MessageBuilder<T>
{
    private int _code;
    private string _msg;
    private T _data;
    public static MessageBuilder<T> Create<T>()
    {
        return new MessageBuilder<T>();
    }

    public MessageBuilder<T> WithCode(int code)
    {
        _code = code;
        return this;
    }

    public MessageBuilder<T> WithMsg(string msg)
    {
        _msg = msg;
        return this;
    }

    public MessageBuilder<T> WithData(T data)
    {
        _data = data;
        return this;
    }

    public byte[] Build()
    {
        var message = new Message<T>
        {
            Code = _code,
            Msg = _msg,
            Data = _data
        };
        byte[] bytes = MessagePackSerializer.Serialize(message);
        return bytes;
    }
}

Message Deserialization

Directly use the basic Deserialize function to deserialize the message.

public class MessageParser
{
    public static Message<T> Parse<T>(byte[] bytes)
    {
        var message = MessagePackSerializer.Deserialize<Message<T>>(bytes);
        return message;
    }
}

Modifying the MQTT Publisher Program

With the above message serialization and deserialization functions, we can directly call them. Replace the message serialization process in the MQTT publisher program; here we only show the core part of the code:

string data;
while (true)
{
    Console.Write("Please enter the message to send:");
    data = Console.ReadLine();
    data = "time:" + DateTime.Now.ToString("yyyy-MM-dd HH:MM:ss") + " msg:" + data;
    byte[] bytes = MessageBuilder<string>.Create()
        .WithCode(200)
        .WithMsg("msg")
        .WithData(data)
        .Build();

    var applicationMessage = new MqttApplicationMessageBuilder()
        .WithTopic("mytopic")
        .WithPayload(bytes)
        .Build();
    var result = await mqttClient.PublishAsync(applicationMessage, CancellationToken.None);
}

Modifying the MQTT Subscriber Program

For the subscriber program, we need to modify the callback function that receives the message.

var mqttClientOptions = new MqttClientOptionsBuilder().WithTcpServer("127.0.0.1").Build();
mqttClient.ApplicationMessageReceivedAsync += e =>
{
    Message<string> message = MessageParser.Parse<string>(e.ApplicationMessage.Payload.First.Span.ToArray());
    Console.WriteLine($"Received message: clientid:{e.ClientId}, topic:{e.ApplicationMessage.Topic}, message:{message.Data}");
    return Task.CompletedTask;
};

Running Results

Next, we will sequentially start the MQTT server process, the subscriber process, and the publisher process, and then observe the running results.

First, here are the results from the server process:

Next, the subscriber:

Finally, the publisher, where we published three messages, and the subscriber successfully received them.