Chaos Engineering Practice: Fault Injection Testing for Distributed Architectures

1. What is Chaos Engineering

2. Chaosblade Chaos Engineering Experimental Tool

To better understand the implementation logic of Chaosblade, we obtained the source code of the open-source version of the tool and analyzed common commands, such as CPU faults, disk full, network blockage, and process hang-ups. Taking CPU faults as an example, we briefly analyze as follows.

(1) Command Description

Table 1 CPU Fault Command Description

(2) Implementation Principle

In the ./blade create cpu command, the CPU can be set to full load or a specific load percentage. The –cpu-count can specify the number of CPU cores to be fully loaded, and –cpu-list can specify the required CPU core numbers to be fully loaded. You can use forms like “1,3” to specify cores 1 and 3 to be fully loaded, or “0-3” to specify cores 0 to 3 to be fully loaded.

If you are not using the directly usable release version but the source code, then you need to compile the source code in front of Chaosblade, which will compile the files in the exec/bin/ directory into chaos_*** type executable files.

By examining the source code, we find that the logic for implementing CPU full load in chaos_burncpu is simply to keep the CPU running. The code snippet is as follows:

Shutting down the CPU full load process is also relatively simple, divided into two steps:

Step 1: Use the command ps -ef | grep … to find the pid of chaos_burncpu.

Step 2: Use the kill -9 pid command to end it.

If it is the directly executable release version, you can see these chaos_*** executable files under the bin/ directory. Then, when using the ./blade create cpu fullload command, it actually transforms internally and directly calls bin/chaos_burncpu, while the blade executable file is just a CLI entry implemented using Cobra. (Cobra is a library for creating powerful modern CLI applications and a program for generating application and command files.)

Other faults, such as disk faults, are implemented using the dd command to achieve file copying, simulating high-speed disk read and write, and enhancing I/O load by adjusting block size. The principle for insufficient disk capacity is to create files in the specified path using the dd command or fallocate command. Network delays, packet loss, packet duplication, local port occupation, and packet reordering are matched against the parameter classRule in a source file based on user input commands. The implementation principle is the tc command, which is used for traffic control in the Linux kernel, primarily by establishing a queue at the output port to control traffic acceptance.

By analyzing the implementation principles, we can understand the underlying logic of fault generation and better simulate the production of business abnormal scenarios.

3. Fault Injection Testing Practice

We use Nginx as the load balancer to simulate the deployment architecture of a distributed business system, using passive health checks to detect server status. If a backend instance fails, Nginx is responsible for gradually removing the failed nodes. When the system is stable, we use the Chaosblade tool for fault injection and collect transaction requests and processing situations when the faults occur, testing the overall system’s stability and robustness in abnormal scenarios. The system deployment architecture is as follows:

The pressure client is Xmeter, and Nginx is deployed on two 4C8G PC servers, primarily performing 7-layer reverse proxy and gateway functions to distribute transaction request traffic to backend server boards.

This test selects four common basic resource experimental scenarios from Chaosblade: CPU, disk, network, and processes. Referring to the Nginx testing environment deployment architecture, the fault node types are selected as backend servers and Nginx servers; for transaction requests, commonly used get and post requests are selected, totaling 48 test scenarios.

Table 2 Testing Scenarios

Before executing the fault scenario tests, a good observation index should be designed for comparison with the experimental results to determine whether the experimental results meet expectations. The observation index should be easy to obtain and clearly reflect the issues in the distributed architecture system under abnormal scenarios. The experimental subject is the stability of the distributed architecture system Nginx after using the Chaosblade tool for fault injection, so the observation index is selected as the transaction error rate, with an error rate set to less than 1% as the stable state of the system, which can be generated using the pressure testing tool Xmeter.

The testing steps are to set the client timeout to 30 seconds, with a concurrency of 10, and Nginx using health check mechanism recommended parameter configurations, continuously applying pressure for 10 minutes, with normal operation for 3 minutes before the fault occurs. Additionally, after completing the fault injection test, remember to destroy the testing environment.

Below, taking a 90% CPU load as an example, we explain the fault injection and fault destruction execution. The TOP command can be used to observe the CPU occupancy to confirm the fault operation.

During the experiment, we set an error rate of less than 1% as the stable state of the system and compared the experimental results with the stable state for analysis: for the distributed architecture constructed in this article, under the above abnormal scenarios, the process hang (fault node being the backend server) did not reach the system’s preset stable state in terms of error rate, while other situations met expectations. Future focus needs to be placed on the process hang situation, requiring further analysis to confirm whether setting the error rate of less than 1% as the stable state of the system is reasonable, why there is a difference between the backend server and Nginx server, and whether program optimization is necessary.

Table 3 Testing Results Summary

4. Summary and Outlook

Using the chaos engineering experimental tool Chaosblade for fault injection testing in the distributed architecture system Nginx can be roughly divided into ten steps:

(1) Determine the preliminary experimental requirements and subjects

Identify the distributed architecture system for abnormal scenario testing.

(2) Conduct feasibility assessments to determine the experimental scope

Conduct detailed assessments based on specific experimental requirements and subjects, usually including the following aspects: experimental indicators, experimental environment, experimental tools, fault injection scenarios, environmental recovery capabilities, etc., to determine a reasonable experimental scope.

(3) Design appropriate observation indicators

The design of observation indicators is one of the key factors for the success of the entire experiment. System observability has become a “mandatory function”; good system observability will provide strong data support for testing work, laying a solid foundation for subsequent experimental result interpretation, issue tracking, and final resolution.

(4) Design appropriate experimental scenarios and environments

Experimental scenarios: In actual production, fault scenarios vary widely, such as dependency faults, host faults, operating system faults, network faults, service layer faults, etc. When selecting experimental scenarios, it is important to base them on your experimental requirements and analyze where there may be weaknesses, thus designing fault scenarios for the experimental subjects accordingly to achieve precise fault generation.

Experimental environment: Choose an experimental environment applicable to the experimental subjects, such as development, testing, pre-production, or production environments. For non-production environments, simulate production traffic and architecture as closely as possible to verify the reliability of experimental scenarios and tools.

(5) Select appropriate experimental tools

Fault injection is a critical step in abnormal scenario testing. Currently, mainstream fault injection tools include Netflix’s open-source chaos engineering project – Chaos Monkey, PingCap’s open-source chaos engineering practice platform – Chaos Mesh, and Alibaba’s open-source chaos engineering experimental tool Chaosblade.

(6) Establish an experimental plan and execute the experimental process

The main purpose of the experimental plan is to determine relevant details, such as how the client simulates, how to set the server’s key parameters, whether there are requirements for the experimental duration, when the fault should occur, etc. After communicating with stakeholders and establishing the experimental plan, the experimental process can be executed according to the detailed steps.

(7) Collect experimental observation indicators

Collect the observation indicators designed in the early stage of the experiment. As mentioned earlier, observation indicators are key factors for determining the success of abnormal scenario testing; how to collect observation indicators in a timely and accurate manner should be determined during the early design phase, along with feasibility assessments.

(8) After the experiment is completed, clean up and restore the experimental environment

After completing the experiment, it is essential to clean up and restore the experimental environment; otherwise, the faults will persist and severely affect subsequent development and testing work. The Chaosblade experimental tool selected in this article requires manual execution of the fault destruction command unless the fault duration is set during the creation of the experiment. If other experimental tools are chosen, the fault destruction methods must also be strictly confirmed.

(9) Analyze experimental results

Compare the “experimental observation indicators” with the “stable state of the indicators”; the “stable state” refers to the state of normal system operation and can also be defined by certain indicators. Analyze whether the observation indicators collected under the abnormal scenarios of fault injection meet the requirements of the stable state; if they do, it indicates good system robustness, meaning that the system can still operate stably and without impact during faults; if not, the reasons and subsequent improvement measures need to be analyzed.

(10) Trace the root cause and resolve issues

For situations where the experimental observation indicators do not meet the stable state after fault injection, it is necessary to trace the root cause and resolve the issues. For cases that do not meet the stable state, further confirmation of the rationality of the stable state is required. If confirmed, then based on actual needs, targeted verification under other abnormal scenarios, fault nodes, request types, transaction types, etc. must be conducted to perform multi-angle comparative analysis to comprehensively analyze the reasons and identify the issues.

Currently, how to verify the high availability principles of distributed architecture systems and how to quickly and comprehensively construct abnormal scenarios in distributed architecture systems are challenges in testing work. In the future, we will leverage chaos engineering experimental tools like Chaosblade to build a unified fault rehearsal platform, quickly providing fault injection capabilities covering basic resources, Java applications, C++ applications, Docker containers, Kubernetes platforms, and more; and establish standardized fault rehearsal processes and fault recovery standards, assisting in generating rehearsal tasks based on business system architecture will be our key research direction.

Qin Hongyan: Member of the Platform Application Testing Team at BeiYan, mainly responsible for testing work related to lightweight cloud application development platforms, integrated gateway platforms, external service platforms, and more.

Shu Zhaoxin: Functional Group Leader of the Platform Application Testing Team at BeiYan, has been working on the front lines of testing, uniting and leading team members to conduct testing on various technical middle platforms such as Taihang, Light Cloud, AIR, and intelligent anti-fraud platforms, database platforms, etc. He deeply explores testing highlights to provide stable and user-friendly platforms for developers in the center and the entire bank, striving to create more space and value for everyone.