Exploration of Chaos Engineering Scenarios and Case Studies at ICBC

Written by / China Industrial and Commercial Bank Financial Technology Research Institute Cloud Computing Laboratory

ICBC adopts a “self-developed + open-source” approach to build a fault drill platform, providing capabilities such as fault orchestration, medium delivery, task scheduling, drill protection, drill visualization monitoring, environment recovery, automatic generation of drill reports, and one-click automated drills. This not only meets the daily chaos testing drill needs but also provides support for advanced capabilities such as red-blue offense and defense, emergency drills, and disaster recovery drills, continuously verifying and improving application high availability levels and operational management levels. Below, we will focus on elaborating on three aspects: the technical architecture of the chaos drill platform, fault injection capabilities, and experiences in building a high-availability expert database.

1. Introduction to the Technical Architecture of the Chaos Engineering Drill Platform

ICBC follows three main principles in planning the core capabilities of chaos engineering: first, to shield the differences between application architecture and underlying deployment architecture, achieving unified encapsulation for many underlying facilities, allowing users to focus only on the fault implementation content without needing to be concerned with underlying differences. Second, to provide core capabilities such as fault orchestration, task scheduling, and drill scenario configuration on top of fault injection tools, achieving enterprise-level platform support. Third, to support automatic matching of the high-availability expert database based on the characteristics of the application’s architecture, achieving one-click generation of multiple types of faults, thus lowering the usage threshold.

Based on the core points of capability encapsulation, supporting services, and drill strategies focused on the above three principles, ICBC divides the chaos drill platform into five levels, where infrastructure, underlying capabilities, and task scheduling achieve capability encapsulation, making it transparent to users; upper-level business and system integration achieve supporting services, such as integrating the chaos drill platform with continuous integration environments and performance testing platforms, forming automated drills and continuous protection capabilities based on high-availability expert database drill scenarios, comprehensively improving chaos drill efficiency. At the same time, the specialized services provided by the fault drill platform, such as chaos testing, red-blue offense and defense, and disaster recovery drills; platform management achieves core points of drill strategies, including process orchestration, fault injection, drill monitoring, security protection, and analysis measurement, providing users with convenience in drill implementation, observation, and measurement (as shown in Figure 1).

Figure 1  Schematic Diagram of ICBC’s Chaos Engineering Fault Drill Platform Architecture

2. Construction of Fault Injection Capabilities of the Chaos Engineering Drill Platform

ICBC’s fault drill platform shields the differences in application underlying deployment, making various basic deployment environments such as physical machines, virtual machines, and containers transparent, forming fault drill capabilities in three aspects: system, application, and container, covering more than 100 types of faults in three major categories. At the same time, based on the characteristics of ICBC’s cloud platform and platform capability support, rich fault injection capabilities are formed from top to bottom at the layers of SaaS, PaaS, and IaaS, relying on the platform’s visual configuration for flexible, efficient, and easy use to meet various drill scenario needs such as graceful startup, graceful shutdown, automatic scaling, container self-healing, flow limiting, circuit breaking, NOS cache circuit breaking, and database faults (as shown in Figure 2).

Figure 2  Schematic Diagram of ICBC’s Chaos Engineering Fault Capabilities and Expert Database

3. Experience Sharing on the High-Availability Expert Database of the Chaos Engineering Drill Platform

The high-availability expert database is a professional knowledge base composed of methods and strategies for validating application high availability levels, summarized by chaos engineering testers through production problem summaries, chaos testing practices, and industry scenario explorations. To reduce the difficulty of designing chaos drill scenarios and improve chaos drill efficiency, ICBC has focused on building a high-availability expert database for chaos drills in the financial field, which includes hundreds of drill scenario cases across six major categories, including application layer, database layer, platform layer, cache layer, message middleware layer, and routing layer, covering critical high-availability assurance capabilities such as application self-isolation, active-active in the same city, graceful startup and shutdown, flow limiting, degradation, and circuit breaking. Through the construction of the high-availability expert database in the financial field, the threshold for chaos testing and resource investment costs has been significantly lowered, and it lays a solid foundation for subsequent automated matching of test cases under the high-availability expert database based on the characteristics of the application’s architecture, achieving the high-level capability of one-click generation of fault drill tasks (as shown in Figure 2).

Quick payment, as a key business of ICBC, has very high requirements for system availability, stability, and fault tolerance. First, the overall transaction link of quick payment is complex, involving multiple business application systems. For different business scenarios, quick payment needs to call distributed services provided by different backend application systems. For personal financial services, the personal finance business system serves as the service provider, registering the services related to personal finance business with the registration center, while quick payment acts as the service consumer, defining personal finance services from the registration center. Second, the quick payment business scenario is designed as an active-active architecture, with business servers and registration centers deployed in both A and B parks. Regardless of which park or link experiences a fault, it will not affect the operation of the entire system. However, during actual operation, occasional infrastructure faults (network faults, storage faults, server downtime, etc.) often affect the success rate of quick payment transactions (as shown in Figure 3).

Figure 3  Schematic Diagram of Quick Payment Drill Example

The drill scenarios are based on critical nodes involved in the quick payment link and are designed to simulate faults at service provider/service consumer nodes and registration center cluster anomalies. Combined with the key nodes and potential fault issues of the infrastructure involved in quick payment, feasible fault drill scenarios are formulated and drilled one by one.

According to the full-link transaction process, faults such as network, hard disk, and process destruction are injected, observing changes in the transaction success rate, TPS, and transaction time of the quick payment link, while establishing system steady-state indicators for recovering transaction success rate and TPS to normal within one minute.

The drill scenarios are shown in Table 1, and the drill results are shown in Table 2.

Table 1  Drill Scenarios

Table 2  Drill Results

Based on the above drill results, two main issues were identified.

First, when all nodes of the “personal finance routing layer” in Park A fail, an avalanche effect occurs, ultimately leading to the overall unavailability of Park A. The failure of a single application node can trigger a park-level fault, indicating very low fault tolerance.

Second, during the recovery period of the registration center in Park A, a large number of service providers begin to re-register, and consumers start to re-subscribe. When providers only register a small number of services to the registration center, and consumers happen to pull the new service list and update it to the local cache, there will be a sharp reduction in available services, resulting in a large number of thread shortages and timeout exceptions during service calls.

Optimization measures: adopt a dual-park registration and subscription scheme. Service providers in Park A simultaneously register services to the registration centers of both Park A and Park B; consumers also subscribe to the registration centers of both Park A and Park B. The same applies to Park B. After re-validation, the above issues were resolved.

As of the end of 2023, ICBC has conducted stability testing based on chaos engineering for over 300 application systems, gradually establishing a high-availability expert database, fault automation capabilities, regular drill mechanisms, and red-blue offense and defense drill mechanisms. In 2023, a total of 9,971 chaos drill scenario tests were completed, uncovering 163 high-availability issues. Through the practice of chaos engineering, ICBC has significantly improved its system high-availability service capabilities and emergency response times for production issues.

In the future, ICBC will explore deeper into capabilities such as automatic steady-state judgment of systems and intelligent generation of drill cases based on large models and machine learning technologies, further enhancing the effectiveness of chaos engineering practices and ensuring the continuous improvement of high-level system stability assurance capabilities.

(This article was published in the “Financial Electrification” March 2024 issue)