Continuing from the previous article “Ten Discussions on High-Performance Network Programming in Linux | C++11 Implementation of 22 High-Concurrency Models”, many are interested in various Server implementations in C++. Therefore, I have compiled a list of high-performance open-source network frameworks I have encountered over the years, based on 9 <span>C++</span> network frameworks to implement an <span>Echo Server</span> example.
GitHub code address: https://github.com/linkxzhou/mylib/tree/master/c%2B%2B/high_performance_server
To facilitate testing, the current project uses <span>Bazel</span> for building, reducing some library dependencies. The build steps are as follows:
# ========== Install Bazel
# macOS
brew install bazel
# Ubuntu/Debian
sudo apt install apt-transport-https curl gnupg
curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor > bazel.gpg
sudo mv bazel.gpg /etc/apt/trusted.gpg.d/
echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
sudo apt update && sudo apt install bazel
# CentOS/RHEL
sudo dnf copr enable vbatts/bazel
sudo dnf install bazel
# ========== Install Dependencies
# macOS
brew install libuv libevent libev boost
brew install folly wangle proxygen # optional
# Ubuntu/Debian
sudo apt-get install libuv1-dev libevent-dev libev-dev libboost-all-dev
# CentOS/RHEL
sudo yum install libuv-devel libevent-devel libev-dev boost-devel
# ========== Build All Servers
./build_all_bazel.sh
# Or build directly with Bazel
bazel build //...
# Build Specific Server
bazel build //ace:ace_echo_server
bazel build //libevent:libevent_echo_server
bazel build //boost_asio:boost_asio_echo_server
bazel build //seastar:seastar_echo_server
bazel build //libev:libev_echo_server
bazel build //libuv:libuv_echo_server
bazel build //mongoose:mongoose_echo_server
bazel build //proxygen:proxygen_echo_server
bazel build //wangle:wangle_echo_server
Open Source Frameworks
| Framework | Type | C++ Standard | Features | Representative Open Source Projects |
|---|---|---|---|---|
| libevent | Event-Driven | C++11 | Cross-platform, lightweight, widely used | Memcached, Tor, Chromium, tmux |
| libev | Event-Driven | C++11 | High performance, Linux optimized | Node.js (early versions), PowerDNS, Varnish |
| libuv | Event-Driven | C++11 | Node.js core, cross-platform | Node.js, Julia, Luvit, pyuv |
| Boost.Asio | Asynchronous I/O | C++11 | Feature-rich, standardized | Beast (HTTP/WebSocket), cpp-netlib, Riak |
| ACE | Object-Oriented | C++17 | Enterprise-level, rich patterns | TAO (CORBA), OpenDDS, JAWS Web Server |
| Seastar | No Shared | C++17 | Extremely high performance, modern design | ScyllaDB, Redpanda, Seastar HTTP Server |
| Wangle | Asynchronous | C++17 | Facebook open-source, Pipeline architecture | Proxygen, McRouter, Facebook Services |
| Proxygen | HTTP Specific | C++17 | Facebook HTTP library | Facebook Web Services, Instagram API |
| Mongoose | Embedded | C++11 | Lightweight, easy to integrate | ESP32 projects, IoT devices, embedded web servers |
Performance Comparison
Framework Overview
1. libevent – Event-Driven Architecture
Features:
- Cross-Platform Compatibility: Supports multiple platforms such as Linux (epoll), macOS (kqueue), Windows (IOCP)
- Multiple I/O Multiplexing: Automatically selects the optimal I/O mechanism (epoll/kqueue/select)
- Buffered Event Handling: Built-in buffer management simplifies network programming
- HTTP Support: Built-in HTTP server and client functionality
- Timer Support: High-precision timers and timeout handling
- Thread Safety: Supports event handling in multi-threaded environments
- Memory Management: Efficient memory pool and buffer management
Underlying Architecture:
Core Components:
<span>event_base</span>: Core of the event loop, managing all events<span>evconnlistener</span>: Connection listener, handling new connections<span>bufferevent</span>: Buffered event handling, automatically managing read/write buffers- Callback Functions:
<span>echo_read_cb</span>,<span>echo_event_cb</span> - Platform Adaptation: Automatically selects the optimal I/O multiplexing mechanism
2. libev – High-Performance Event Loop
Features:
- Extreme Performance: Designed for high performance, minimizing system call overhead
- Linux Optimization: Deeply optimized for epoll performance, supporting Linux-specific features
- Lightweight Design: Simple code, minimal memory footprint
- Multiple Event Types: Supports I/O, timers, signals, subprocesses, and more
- Nesting Event Loops: Supports nested calls of event loops
- High-Precision Timers: Efficient timer implementation based on red-black trees
- Signal Handling: Safe asynchronous signal handling mechanism
Underlying Architecture:
Core Components:
<span>ev_loop</span>: High-performance event loop, supports multiple backends<span>ev_io</span>: I/O event monitor, monitoring file descriptors<span>ev_timer</span>: High-precision timer, implemented based on red-black trees<span>accept_cb</span>: Connection accept callback, handling new connections<span>client_cb</span>: Client data processing callback
3. libuv – Cross-Platform Asynchronous I/O
Features:
- Node.js Core: Core dependency of Node.js, validated in large-scale production
- Unified Cross-Platform: Unified API abstracts asynchronous I/O across different platforms
- Thread Pool: Built-in thread pool for handling file I/O and CPU-intensive tasks
- Asynchronous File Operations: Complete asynchronous file system API
- Process Management: Cross-platform process creation and management
- Network Abstraction: High-level network API supporting TCP, UDP, pipes
- Event Loop: Single-threaded event loop + multi-threaded work pool
Underlying Architecture:
Core Components:
<span>uv_loop_t</span>: Cross-platform event loop, unifying asynchronous mechanisms across different platforms<span>uv_tcp_t</span>: TCP handle, encapsulating network connections<span>uv_read_start</span>: Start asynchronous data reading<span>uv_write</span>: Asynchronous data writing, supporting batch writes- Thread Pool: Handles blocking operations, avoiding blocking the main thread
4. Boost.Asio – Asynchronous Network Programming
Features:
- C++ Standard Candidate: Modern design, potentially part of the C++ standard library
- Type Safety: Strong type system, compile-time error checking
- Coroutine Support: Supports C++20 coroutines, simplifying asynchronous programming
- Scalability: Supports custom I/O objects and protocols
- SSL/TLS: Built-in SSL/TLS support
- Timers: High-precision timers and deadlines
- Signal Handling: Asynchronous signal handling mechanism
Underlying Architecture:
Core Components:
<span>io_context</span>: I/O execution context, managing asynchronous operations<span>tcp::acceptor</span>: TCP acceptor, listening for new connections<span>session</span>: Session management class, encapsulating connection lifecycle- Asynchronous Operations:
<span>async_accept</span>,<span>async_read_some</span>,<span>async_write</span> - Coroutine Support: Integrated C++20 coroutines
5. ACE – Adaptive Communication Environment
Features:
- Enterprise-Level Framework: Mature framework validated by large enterprise systems
- Rich Design Patterns: Implements various network programming design patterns
- Highly Configurable: Supports compile-time and runtime configuration
- Cross-Platform: Supports 40+ operating systems and compilers
- Object-Oriented: Fully object-oriented design
- Componentized: Modular design, usable on demand
- Performance Optimization: Optimized for high-concurrency scenarios
Underlying Architecture:
Core Components:
<span>ACE_Reactor</span>: Core of the reactor pattern, supporting multiple implementations<span>ACE_Event_Handler</span>: Base class for event handlers<span>ACE_SOCK_Acceptor</span>: Socket acceptor<span>ACE_SOCK_Stream</span>: Socket stream, encapsulating network communication- Design Patterns: Reactor, Proactor, Acceptor-Connector
6. Seastar – No Shared Architecture
Features:
- No Shared Design: Each CPU core runs independently, avoiding lock contention
- User-Space Network Stack: Bypasses the kernel, directly operating on network hardware (DPDK)
- Coroutine Support: Coroutine model based on future/promise
- Memory Management: Custom memory allocator, reducing memory fragmentation
- CPU Affinity: Strict CPU core binding
- Zero-Copy: Minimizes data copy operations
- Modern C++: Heavily utilizes C++14/17 features
Underlying Architecture:
Core Components:
<span>app_template</span>: Application template, managing application lifecycle<span>server_socket</span>: Server socket, supporting multiple cores<span>connected_socket</span>: Connected socket, encapsulating network connections<span>future<></span>: Asynchronous operation result, supporting chained calls- No Shared Architecture: Independent memory, network queues, and schedulers for each core
7. Wangle – Pipeline Architecture
Features:
- Pipeline Design: Modular request processing pipeline
- Facebook Production: Widely used internally at Facebook
- Type Safety: Strongly typed Pipeline components
- Composability: Flexible processor combinations
- Protocol Agnostic: Supports multiple network protocols
- Load Balancing: Built-in load balancing and connection pooling
- SSL/TLS: Complete SSL/TLS support
Underlying Architecture:
Core Components:
<span>ServerBootstrap</span>: Server bootstrap, configuring the server<span>Pipeline</span>: Processing pipeline, chaining request processing<span>ByteToMessageDecoder</span>: Byte to message decoder<span>HandlerAdapter</span>: Handler adapter, connecting different types of handlers<span>IOBuf</span>: Efficient buffer management
8. Proxygen – HTTP Specific Library
Features:
- HTTP Specific: Optimized for HTTP/1.1 and HTTP/2
- Facebook Open Source: Foundation of Facebook’s internal HTTP services
- HTTP/2 Support: Complete implementation of HTTP/2, including server push
- Streaming Processing: Supports streaming uploads and downloads of large files
- Compression Support: Built-in gzip and deflate compression
- WebSocket: Complete WebSocket support
- Performance Monitoring: Built-in performance metrics and monitoring
Underlying Architecture:
Core Components:
<span>HTTPServer</span>: HTTP server, supporting HTTP/1.1 and HTTP/2<span>RequestHandler</span>: Request handler, managing the HTTP request lifecycle<span>RequestHandlerFactory</span>: Handler factory, creating request handlers<span>ResponseBuilder</span>: Response builder, constructing HTTP responses- HTTP/2 Features: Stream multiplexing, server push, header compression
9. Mongoose – Embedded Web Server
Features:
- Lightweight: Single-file implementation, easy to integrate
- Embedded Friendly: Suitable for resource-constrained environments
- Multi-Protocol Support: HTTP, WebSocket, MQTT, CoAP
- Cross-Platform: Supports embedded systems, desktops, and servers
- Zero Dependencies: Does not rely on external libraries
- Event-Driven: Based on event-driven asynchronous processing
- Built-in Features: File service, CGI, SSI support
Underlying Architecture:
Core Components:
<span>mg_mgr</span>: Connection manager, managing all network connections<span>mg_connection</span>: Connection object, encapsulating a single network connection<span>mg_http_listen</span>: HTTP listener, starting the HTTP service<span>mg_http_reply</span>: HTTP response, sending HTTP replies- Multi-Protocol: Unified interface for HTTP, WebSocket, MQTT, CoAP
References
- libevent Official Documentation
- libev Official Documentation
- libuv Official Documentation
- Boost.Asio Documentation
- ACE Official Website
- Seastar Official Documentation
- Wangle GitHub
- Proxygen GitHub
- Mongoose Official Documentation
- https://docs.seastar.io/master/tutorial.html
- https://zhuanlan.zhihu.com/p/30738569
- https://blog.csdn.net/Rong_Toa/article/details/108193340
- https://www.cnblogs.com/ahfuzhang/p/15824213.html “Memory Barriers: a Hardware View for Software Hackers”