He Bo: Establishing a System Stability Assurance Framework Centered on Chaos Engineering

He Bo, Director of the Financial Technology Committee of Zhongtai Securities Co., Ltd.

and General Manager of the Technology R&D Department

Background Analysis

With the rapid development of the financial industry, business demands are constantly increasing, and the speed of product iteration is accelerating, leading to larger system scales. The traditional monolithic architecture model can no longer meet the current business development needs of the financial industry, and distributed microservice architecture is increasingly being applied within it. Compared to traditional software architecture, the more complex deployment structure of microservices faces greater threats of failure, and the symptoms of microservice system failures are also more diverse. Additionally, due to the financial industry involving a large number of capital transactions, systems encompass multiple data centers, active-active configurations, disaster recovery, containers, virtual machines, and other complex infrastructures, making inter-system interactions particularly intricate, thereby increasing operational uncertainties.

Existing stability assurance measures focus on preventing the introduction of known system defects, lacking means to identify and repair failures that require specific external disturbances to trigger, only allowing for passive responses to failures when they occur, leading to uncontrollable progress and costs in failure response.

Introduction to Chaos Engineering

Chaos engineering is the discipline of experimenting on distributed systems, first proposed in August 2008 by Netflix. In 2012, Chaos Monkey was open-sourced as part of the Simian Army project, establishing the first open-source chaos engineering toolset, laying the foundation for the development of chaos engineering tools. In 2015, Netflix officially released the “Principles of Chaos Engineering,” which mainly introduced the purpose, significance, and methodology of chaos engineering experiments. In 2016, the chaos engineering commercial company Gremlin was established, marking the commercialization of chaos engineering. Since 2019, domestic enterprises have begun to introduce and practice chaos engineering.

The goal of chaos engineering is to enhance system fault tolerance and build confidence in the system’s ability to withstand unpredictable problems in production environments; it aims to identify failures before they cause outages, effectively “killing them in the cradle.” By actively injecting disturbances that may trigger failures into the system, it explores the system’s capacity to withstand disturbances, tests the system’s behavior under various pressures, identifies and repairs fault issues, and avoids severe consequences. Chaos engineering is a complex technical approach to improving the resilience of technical architecture, ensuring system availability through experimentation. This empirical validation method evidently builds more resilient systems while allowing us to gain a deeper understanding of various behavioral patterns of the system during operation, establishing confidence in operating highly available distributed systems while continuously creating more resilient systems.

Based on this, Zhongtai Securities has launched a pilot application of chaos engineering, selecting the internet finance business system, which has high business demand, as the pilot. By targeting system layer, application layer, and infrastructure layer, it injects business-level and architecture-level faults, successfully identifying issues such as excessive timeouts and node sub-health, significantly enhancing the company’s system stability and financial technology level.

Construction Method

Based on the promotion principle of “step-by-step implementation and hierarchical enhancement,” the company gradually promotes the implementation of chaos engineering in a manner of “pilot first, then promotion; focus first, then comprehensive.” The Zhongtai Securities internet finance business system consists of three major systems: retail business capability system, financial product system, and comprehensive financial service system, aimed at meeting the growing investment and financing needs of clients. The architecture of the internet finance system platform is divided into internet access layer, business service layer, basic service layer, and environment deployment layer. The business service layer includes specific business functions such as user center, market information center, news center, trading middle office, wealth management middle office, business processing, investment advisory middle office, and value-added tools. The system architecture adopts the SpringCloud microservice framework, combined with high-performance, low-cost distributed middleware such as caching, distributed storage, search engines, and message queues, as well as foundational services provided by data collection, real-time data computing, and log centers to construct the overall architecture of the business system.

As an important business system of the company, the internet finance system must meet high availability requirements, achieving high availability SLA guarantees and multi-region, multi-center high availability support. To further improve the system stability assurance framework, Zhongtai Securities decided to explore and practice the stability of the internet finance system based on chaos engineering principles.

Currently, we have developed a construction plan for conducting chaos engineering experiments in the internet finance business line, as detailed below.

1. Construction Plan for Chaos Engineering Simulation Environment

First, it is necessary to build a simulation environment for the internet finance system, with the simulation environment constructed 1:1 based on the production environment deployment architecture. The internet system architecture of Zhongtai Securities is shown in Figure 1.

Figure 1: Internet System Architecture of Zhongtai Securities

2. Deployment Plan for Chaos Engineering Monitoring System and Monitoring Indicators

The deployment of the monitoring system is based on a hybrid cloud deployment environment, monitoring business applications, basic services, and infrastructure (SaaS, PaaS, IaaS) from top to bottom. The main components of the monitoring system’s application status monitoring include: Exporter, Prometheus, Alertmanager, and Grafana. Additionally, to better monitor application and architecture states during the rehearsal process, a link monitoring and performance monitoring system has also been introduced. Various Exporter data collectors gather monitoring indicator data for basic server resources, database cluster monitoring indicators, message queue middleware cluster monitoring indicators, application service monitoring indicators, etc. Specific indicators include:

(1) Business Applications: Mainly including service availability monitoring (whether services and ports exist, whether they are in a dead state) and application performance monitoring (application processing capacity, such as transaction volume, success rate, failure rate, response rate, and time taken).

(2) Basic Service Layer: Includes performance indicators for various middleware, Docker containers, and cloud-native platforms.

(3) Infrastructure Layer: Monitors the performance of basic resources, CPU (CPU usage, per-core usage, CPU load), memory (application memory, total memory), disk IO (read/write speed, IOPS, average wait time, average service time), network IO (traffic, packet count, error packets, packet loss), connections (number of TCP connections in various states), etc.

3. Construction Plan for Chaos Engineering Platform

The chaos engineering platform is generally divided into five layers: upper business, platform modules, task scheduling, lower capabilities, and infrastructure. At the bottom is the infrastructure deployed by the company, including containers, virtual machines, physical machines, and other non-standard servers. Ultimately, the chaos engineering fault injection medium will be installed on these infrastructures to implement various fault injections.

The platform should mainly include functions such as environment management, application management, probe management, rehearsal planning, rehearsal observability, risk scenario library, and rehearsal reporting (as shown in Figure 2).

Figure 2: Rehearsal Indicator System

(1) Environment Management: Supports installation on virtual machines, physical machines, or cloud servers; supports cluster mode deployment, including virtual machine clusters and container clusters; supports cloud-native deployment, including cloud servers in cross-cloud and internal multi-cloud configurations; supports server types such as ARM/X86 server architecture; supports Linux operating systems.

(2) Application Management: Supports integration with Nacos and Zookeeper registration centers to achieve automatic application awareness; supports port matching, automatically matching probe-aware instances based on configured application ports; supports manual configuration of IP and port numbers.

(3) Probe Management: Installs probes on the tested clusters or machines to receive commands sent by the chaos engineering platform server to perform fault injections.

(4) Rehearsal Planning: The built-in rehearsal process includes fault injection → duration → injection recovery. The rehearsal plan can be organized through custom configurations of nodes, allowing for adding, editing, and deleting nodes. Supports serial and parallel scene orchestration; supports process reuse; each node supports retry and skip, and termination.

(5) Rehearsal Observability: Can be integrated with third-party monitoring tools to achieve observable rehearsal indicators.

(6) Risk Scenario Library: Supports custom configuration of scenario library categories, such as basic resource scenario library, traffic overload scenario library, etc.; supports setting specific expert scenarios, including basic information, configuration parameters, configuration processes, and observation indicators.

(7) Rehearsal Report: Supports custom configuration of rehearsal reports, including rehearsal time, rehearsal type, applications, instances, and other filters.

4. Standardization and Automation of Chaos Engineering Experiments

By abstracting the common steps of the project rehearsal process, the standardization of specific scenario rehearsal processes is achieved, which is conducive to accumulating standardized rehearsal scenarios and fault handling processes, empowering other businesses to quickly conduct experiments and reduce rehearsal costs. At the same time, through the automation of chaos engineering, more experimental sets can be covered.

Practice System and Experience

Through the implementation of this chaos engineering system, Zhongtai Securities has built a chaos engineering system capable of conducting rehearsals in a hybrid cloud environment (as shown in Figure 3). This has achieved refined management of reliability testing and established a continuous improvement mechanism based on monthly cycles, incorporating training, communication, and summary.

Figure 3: Rehearsal Platform Capability Construction

This practice involved simulating call delays, service unavailability, and machine resource saturation to observe whether the nodes or instances experiencing failures were automatically isolated or taken offline, whether traffic scheduling was correct, and whether the contingency plans were effective, while monitoring the overall QPS or RT of the system for impacts. Additionally, establishing an observability system is crucial for better monitoring the rehearsal process.

Through a layered monitoring system, rapid responses to failures during rehearsals can be achieved, and alerts can be notified, helping rehearsal participants quickly locate failures. Based on the improved rehearsal capabilities and observability system, the scope of failure nodes is gradually increased to verify the effectiveness of upstream service flow control, degradation, and circuit breaking; ultimately, the failure nodes are increased to service request timeouts, estimating the system’s fault tolerance threshold, and measuring the system’s fault tolerance capability; simulating upper-level resource loads to verify the effectiveness of the scheduling system; simulating unavailability of dependent distributed storage to verify the system’s fault tolerance capability; simulating unavailability of scheduling nodes to test whether scheduling tasks are automatically migrated to available nodes; simulating primary-backup node failures to test whether primary-backup switching is normal; verifying whether monitoring indicators are accurate, whether monitoring dimensions are complete, whether alert thresholds are reasonable, whether alerts are timely, and whether alert recipients are correct; and whether notification channels are available, thus enhancing the accuracy and timeliness of monitoring alerts; through fault assaults, randomly injecting faults into the system to assess the emergency response capabilities of relevant personnel, and whether the issue reporting and handling processes are reasonable, achieving the goal of training through combat, and exercising the ability to locate and solve problems.

In terms of team building, Zhongtai Securities prioritized forming a management team led by reliability testing leaders from various departments and maintenance personnel of the fault rehearsal platform, responsible for organizing tool promotion, test design, and test implementation, and established a dedicated implementation team composed of application architects, testers, developers, and operations personnel responsible for specific implementations. To ensure the better advancement of the practice system, a red-blue offensive and defensive system was introduced, and a GameDay culture was formed among the teams (as shown in Figure 4).

Figure 4: GameDay Culture

During the rehearsals, the red and blue teams confront each other, conducting practical network confrontations in a simulated network environment while ensuring normal business operations, promptly addressing real architectural vulnerabilities, testing and enhancing capabilities for monitoring security threats, emergency response, and security protection. Compared to traditional penetration testing, the key difference in red-blue confrontation is the introduction of dynamic responses from the defensive side, where both attacking and defending capabilities are gradually enhanced during the confrontation. The red team attempts to penetrate the internal network using various vulnerabilities, while the defensive side must possess comprehensive security protection capabilities to prevent the enemy from taking advantage of any opportunity. Through practical offensive and defensive rehearsals, the practical protection capabilities of important business systems can be effectively verified, identifying security vulnerabilities in the system, summarizing experiences and lessons learned, ultimately enhancing security protection capabilities.

In practical team operations, application architects are responsible for developing rehearsal plans for reliability testing based on the high availability architecture characteristics of application systems; testers are responsible for implementing specific tasks according to the rehearsal plans and conducting fault tolerance result analysis through monitoring indicators; developers and operations personnel are responsible for problem analysis and emergency response after failures occur. Through the division of labor between specialized management and specific implementation teams, the quality of reliability testing promotion has been effectively ensured. In terms of training, focus is placed on the characteristics and application methods of fault rehearsal tools; in terms of communication, regular reliability testing communication meetings are organized; in terms of summarization, excellent experiences are continuously distilled and solidified into a high availability expert repository, with targeted optimizations of tools or management processes to prevent recurrence of issues.

Practice Summary

Zhongtai Securities has conducted stability testing on internet finance services based on the chaos engineering framework, following a specific implementation route from small to large, from point to surface, first injecting faults into a specific request of the service, then into the entire service, and finally into the overall application architecture. Subsequently, network, disk, and process destruction fault injections were implemented along the entire chain process, continuously observing changes in the success rate and TPS of internet finance service access. In the future, chaos engineering testing will be normalized and integrated into the DevOps tool pipeline. While implementing chaos engineering in the internet finance business system, several technological innovations have been achieved, specifically in the following aspects.

First, a chaos engineering practice system for internet finance business has been established, empowering the stable development of internet finance business. Internet finance plays a very important role in the revenue of securities firms, and through the implementation of chaos engineering in the internet finance system, use case scenarios and tool systems for conducting rehearsals have been accumulated.

Second, a chaos engineering tool rehearsal system has been completed for hybrid cloud multi-scenario environments. Currently, production business systems typically involve multi-cloud environments, and through the implementation of this chaos engineering system, a rehearsal system capable of conducting rehearsals in hybrid cloud environments has been established.

Third, use case scenarios for chaos engineering rehearsals at the system, application, and business levels have been accumulated. Through the orchestration management of rehearsal indicators and business scenarios, the chaos engineering platform has accumulated a wealth of rehearsal cases, improving efficiency and saving costs for future chaos engineering implementations in new businesses.

Fourth, an assessment system for the stability of key systems in internet finance business has been improved. During the rehearsal practice process, it is necessary to assess the impact of corresponding rehearsal indicators, thus evaluating the stability of key business systems and identifying indicators that may affect system failures through chaos engineering rehearsals.

Fifth, the comprehensiveness of enterprise system stability assessments has been enhanced. Rich chaos engineering experimental scenarios support fault simulation capabilities at the IaaS, PaaS, and SaaS layers, facilitating multi-dimensional assessments of application system stability.

Meanwhile, relying on the practice of chaos engineering in the internet finance business system, the company’s R&D efficiency, system stability, and financial technology level have all been significantly improved. R&D efficiency has increased: through the rehearsal platform and simulation environment, rapid fault injection has reduced the time from 20 minutes to 1 minute.

System stability has improved: the establishment of an enterprise-level chaos engineering fault rehearsal platform has shortened problem discovery time, effectively reduced defect recurrence rates, and laid the foundation for enhancing production environment stability, achieving a reduction in production failure rates by 20% to 30% per year.

Financial technology levels have improved: through chaos engineering and simulation setups, the company’s chaos engineering implementation system has been supplemented, accumulating chaos engineering rehearsal scenarios and fault resolution solutions.

In the future, Zhongtai Securities will continue to research and develop stability models based on chaos engineering automated experiments and the principle of “minimizing explosion radius,” continuously enhancing the self-identification capability and testing effectiveness of reliability testing scenarios, integrating chaos engineering testing into the DevOps pipeline, conducting chaos engineering testing across multiple environments including development, testing, simulation, and production, normalizing chaos engineering testing, and safeguarding the stability of financial systems centered on distributed microservice architecture, continually enhancing the stability of cloud-native architecture systems.

(Column Editor: Zhang Lixia)

Recommended Reading

(Click the image to view exciting content)

He Bo: Establishing a System Stability Assurance Framework Centered on Chaos Engineering