Introduction

In today’s world of increasingly large software, with codebases ranging from tens of thousands to tens of millions of lines, quickly getting up to speed and understanding the business context is no easy task. By constructing relevant business scenarios and examining the internal state of the program, one can gain insights into the macro architecture of the software. The function call stack and key data structures are core components of the program’s state. Obtaining this information through a debugger helps focus on the main code paths, allowing one to maximize efficiency in a limited time.

Using GDB/trace tools to get started with open-source software is a common practice in the industry. Both GDB and trace tools have their strengths in debugging programs, with their core focus on function call stacks and key data structures, primarily revolving around Input/Output, streamlining the software’s process/thread model, and the logic of resource creation and maintenance.

This article consists of the following sections: ① Setting up the BRPC development environment; ② Using a synchronous program as an example to outline the internal architecture of BRPC and the processes of connection creation and destruction; ③ The core data structures of BRPC. This is the introductory article on BRPC, and subsequent articles will provide detailed introductions to timeout mechanisms, locks, thread scheduling, memory pools, and storage.

Environment Setup

• BRPC Compilation and Installation

  

• Using perf stat to view performance metrics

  

• Using perf to generate function call flowcharts

  

  

  Perf tool relies on the symbol table and C++ name mangling, which results in the generated function call graphs having limited reference value. Using –demangle did not help, so we will set perf aside for now and explore the BRPC framework primarily through GDB.

Introduction to BRPC

• Core Functions

  BRPC, as an RPC framework applied in business scenarios such as search, advertising, and recommendations, emphasizes high performance and protocol compatibility, providing core functions such as naming services and load balancing in service governance.

  High performance is a key advantage of the BRPC framework, which offers the following mechanisms for achieving high performance:

• Provides coroutine mechanisms to reduce overhead caused by context switching;

• Offers performance analysis capabilities for easier system performance analysis and troubleshooting;

• Provides traffic stress testing and replay tools for performance testing;

• Supports asynchronous programming to improve overall system throughput;

Detailed Explanation of Thread Model

The business logic of threads/processes is the “heart” of the software, being crucial for resource initialization, setting environment variables, and other related operations that prepare for thread/process logic. By focusing on the processing logic of threads/processes, we can connect the core data structures of the software and related business logic, forming the overall skeleton of the software.

BRPC is a multi-threaded asynchronous RPC framework. Using a simple echo_c++ (synchronous program) example, we will outline the overall architecture of BRPC. Both the client and server sides share the same thread model, and this article will use the client side as an example to introduce the core concepts of BRPC and related business processes.

• Thread Functions

  

Using the thread command to switch threads, and the bt command to view call stack information, the list of threads used by the echo_client client and their entry functions are as follows:

• echo_client: The working thread for user business logic, operates synchronously, constructs request content, and enters a blocking state after sending it through the API. Upon receiving a response, it displays the result;

• brpc_timer: Timer thread for handling scheduled tasks, with the entry function beingbtread::TimerThread:run()

• brpc_wkr: The core working function of the BRPC framework, responsible for handling core tasks during communication, with the entry function beingbtread::TaskControl::worker_thread()

• bvar_sample: Provides data for BRPC performance analysis, with the entry function beingbvar::detail::SamplerCollector::sampling_thread()

• Thread Interaction

  Function call stacks and variable information are key to problem localization. GDB can obtain function call stack information and variable content, but debugging multi-threaded/multi-process asynchronous programs poses significant challenges, requiring a business logic perspective to connect the entire business process.

  

By outlining the generation and consumption logic ofbthread_t tid, we can derive the logical relationships shown in the figure above. When the business logic code communicates with the server using the RPC framework, it calls thebthread_start_backgroud() interface to requestTaskMeta resources and saves the corresponding ID in_remote_rq; thebrpc_timer thread handles timeout requests, executing the same business logic when a task times out, generating the corresponding task ID.Thebrpc_wkr worker thread retrieves task IDs from the task queue, indexes the task ID toTaskMeta information, and executes related tasks.

echo_client/brpc_timer and brpc_wkr have a collaborative relationship, while thebrpc_wkr worker threads are in a competitive relationship, independently competing for tasks from the task queue and completing their respective tasks independently.

• Core Data Structures

  

The key data structure used in the task scheduling module isstruct TaskMeta, where the fn field records the method of the callback function, saving the callback method in this field. Using GDB to print the fn information, the main callback functions are as follows:

• brpc::Socket::ProcessEvent()

• brpc::GlobalUpdate(void*)

• brpc::Socket::KeepWrite()

• brpc::SocketMap::RunWatchConnections(void*)

• brpc::Controller::RunEndRPC(void*)

• brpc::PeriodicTaskThread(void*)

• brpc::EventDispatcher::RunThis()

Life Cycle of a Connection

The previous section introduced the thread model of BRPC, where threads interact throughtaskid and TaskMeta information, which is stored in the memory pool, allowing indexing of resource content through taskid, reducing memory copying and improving system performance.

The RPC client communicates with the server through sockets. On Linux platforms, BRPC uses the epoll asynchronous mechanism to handle business requests. From the information obtained in the previous section, the functions related to epoll and event handling mechanisms arebrpc::EventDispatcher::RunThis() and brpc::socket::ProcessEvent(), using these two functions as breakthrough points, we will use step-by-step tracing and set breakpoints to outline the complete lifecycle of a connection.

• Related Questions

• In the synchronous echo_client example, how does the client forward request data to brpc_wkr and then retrieve the response result from brpc_wkr?

• In the echo_client example, how is the connection to the backend service abstracted?

• A channel represents the connection information to the backend server, allowing a client to have multiple channels, thus enabling communication with multiple backend services.

• Data Reception Process

• Stack Information

• Data reception flow:

• Receive read events from epoll, triggering the socket callback process;

• Save the received data into the socket’s_read_buf;

• Using thecorrelation_id in the message to obtain thecontroller, fill in the response result in theResponse field of the controller, and release the lock;

• Core Data Structures

• InputMessenger: Used to handle client requests and server response results;

• Socket: An abstraction of network IO events, providing external interfaces for creating connections, reading and writing events, and is a core concept of BRPC.

• IOEventData: IO adaptation layer, enablingEventDispatcher to support various IO events such assocket/pipe/eventfd/timerfd;

• EventDispatcher: Used to dispatch IO events, serving as the cross-platform external interface of BRPC;

• IOEvent: Used to adapt different IO events, serving as the connection layer between the upper interface and specific IO events;

• Channel: The upper interface used by users to communicate with backend services, uniquely identified bysocketID, which corresponds one-to-one with the backend’s(serv_ip, serv_port).

• Data Sending Process

  

• In the synchronous client, the client uses the writev() system call to send data to the corresponding fd, wherebutil::IOBuf::pcut_into_file_descriptor() encapsulates the underlying related information;

• Creating Connections

  

• The echo_client uses the same socket to communicate with the backend service. If the connection has not been created during the first connection attempt, it will create the connection first.

• Connection Destruction

  

• When there are timeout tasks, a pending queue will be created, storing the queue inpending_q and triggering the destruction of the connection.