Overview of BifroMQ Open Source MQTT Middleware Architecture

In today’s Internet of Things (IoT) field, the MQTT protocol has become an indispensable key element, providing core technical support for efficient and real-time data transmission. However, the projects offering MQTT capabilities in the market are mainly concentrated on traditional, existing “out-of-the-box” IoT device management platforms, leading to insufficient attention and exploration of lower-level MQTT protocol implementations. We firmly believe that the characteristics of the MQTT protocol have broader application prospects, extending beyond just the IoT field. Driven by this belief, after years of practice and technological accumulation, we launched the open-source project BifroMQ, focusing on the efficient processing of the MQTT protocol, especially its outstanding performance when handling large-scale loads.

Compared to traditional IoT device management platforms, BifroMQ is designed with a distinct philosophy. We boldly abandon the traditional paradigm of “all-in-one solutions” and focus on the demands of high-performance MQTT protocols. We believe that the characteristics of the MQTT protocol have broader application paradigms; therefore, BifroMQ is dedicated to fully leveraging the potential of the MQTT protocol, particularly its extraordinary performance in handling large-scale loads.

Unlike traditional MQTT implementations that integrate logic with downstream systems (e.g., Kafka), BifroMQ prefers to “interface” with downstream systems in a manner compliant with MQTT standards. For instance, through the Shared Subscription mechanism, we externalize integration logic, achieving more decoupled bridging and the ability to forward messages to heterogeneous systems while maintaining high message throughput. This design philosophy makes BifroMQ a highly customizable middleware, capable of easily adapting to various downstream systems and meeting the needs of different business scenarios.

BifroMQ’s mission is to construct an MQTT messaging system suitable for large-scale multi-tenancy, supporting the business characteristics of Serverless cloud services. This mission is driven by a profound understanding of the characteristics of the MQTT protocol and confidence in its broader applications. Unlike traditional enterprise architectures, Serverless cloud services emphasize on-demand resource allocation, elastic scalability, and highly flexible service construction. Therefore, BifroMQ, with its multi-tenant architecture and resource-sharing mechanisms, provides the characteristics of “shared resources, exclusive experiences” for Serverless-type businesses.

Although BifroMQ’s goal is to build multi-tenant Serverless cloud services, multi-tenancy + shared resources essentially abstract single-tenant exclusive resources. Consistent with our philosophy, through the free combination and configuration of core components, BifroMQ easily supports the common “Shared Nothing” enterprise-level cluster deployments. In fact, this is also the default deployment mode of the BifroMQ open-source version. This flexibility allows BifroMQ to adapt to various scales and demands of system architectures.

The architecture design of BifroMQ is based on “first principles,” analyzing the technologies required to achieve design goals from a technical perspective and how to organically combine these technologies to achieve overall optimization. Therefore, it is challenging to comprehensively understand its connotation solely from the project structure and code logic. Below, we will provide a high-level description of BifroMQ’s overall architecture from three main perspectives.

▎Modular Architecture Based on Decentralized Cluster Management

First, setting aside the implementation logic of the MQTT protocol itself, the various functional service modules of BifroMQ are built on a decentralized underlying cluster construction capability (base-cluster), as shown in the figure below:

Modular architecture of BifroMQ’s decentralized cluster

The cluster logic built by base-cluster is divided into two layers: Underlay Cluster and Overlay Cluster. The Underlay Cluster is referred to as the “Host” Cluster, where each Host in BifroMQ corresponds to a service process (Process) running on the operating system. The Host Cluster employs a Gossip-like protocol (SWIM Protocol) to implement Host member service (Membership) and provides an abstraction of Host addresses for the Overlay Cluster. The Overlay Cluster is referred to as the “Agent” Cluster, which provides Agent-level address abstraction (Agent Address) based on Host addresses. In BifroMQ, an Agent corresponds to a logical service (Service) that implements specific functions. These logical services (uniformly implemented through the base-rpc framework) include modules for both client and server roles, utilizing the capabilities of the Agent Cluster to achieve service registration and discovery.

The benefits brought by this architecture include:

▎Modular Load Isolation and Collaboration

Next, we describe BifroMQ’s structure from the perspective of protocol implementation. When architecting from the standpoint of building Serverless cloud services, it is necessary to consider the rationality of functional implementation from the perspective of load isolation and management, that is, the multi-tenancy of loads. We have split the workloads of the MQTT protocol so that each type of load can exist as an independent sub-cluster, and these sub-clusters collaborate to build complete MQTT capabilities. Below are the modules related to MQTT loads in BifroMQ:

The collaborative relationships between these modules are illustrated in the figure below:

Collaboration relationships of BifroMQ modules 1

Collaboration relationships of BifroMQ modules 2

It is worth mentioning that the bifromq-dist, bifromq-inbox, and bifromq-retain modules fully utilize the capabilities of base-kv to achieve distributed strong consistency persistence for critical loads. In the operational scenarios of Serverless cloud services, this aspect is particularly important for ensuring SLA.

▎Neutrality: Standard-Oriented and Scalable Integration

As mentioned earlier, BifroMQ’s positioning is to implement MQTT standard middleware, emphasizing neutrality and integrability. It does not operate on the same level as other “all-in-one IoT platform” projects that contain MQTT capabilities. Therefore, the standard for judging whether “BifroMQ possesses a certain feature or will support a certain feature in the future” is straightforward: any capability defined by the MQTT protocol falls within the scope of what the bifromq project needs to support, while any functionality beyond the MQTT protocol’s definition is preferred to exist as independent components or services and integrate with BifroMQ in a manner compliant with MQTT standards. This approach helps build larger-scale cloud services and promotes the maturity of related technologies. Of course, BifroMQ’s integrability is also reflected in its integration with various business systems as middleware, mainly through three mechanisms: Plugin, API, and Metrics:

Typical scene integration architecture

In summary, this is an overall introduction to the BifroMQ technical architecture. Please look forward to a series of in-depth articles we will publish next, which will explore each component and design principle of BifroMQ. At the same time, we welcome you to scan the code to join the BifroMQ developer group to provide feedback on the article content. If you have cooperation needs regarding BifroMQ, please scan the QR code below to fill out the demand questionnaire, and we will contact you as soon as possible.

Finally, to help everyone better understand the technical architecture of BifroMQ, we will hold an online live class on BifroMQ on September 21. We welcome everyone to pay attention to the information in the above BifroMQ developer group, lock in the live broadcast address in advance, and communicate and learn with peers!