【Abstract】 The banking industry widely uses databases, facing common challenges such as a variety of databases in non-core applications, complex SQL queries, and large volumes of historical data. These issues pose significant challenges for replacing domestic databases in non-core important systems. This article starts from the characteristics and transformation difficulties of these business systems, analyzes the implementation results of the multi-tenant deployment model of domestic databases in non-core important systems based on the general selection standards for domestic databases, combined with the actual usage of domestic databases.
【Author】Xiao Shao from Datang, a DBA at a large joint-stock bank data center, with over ten years of experience in database operation and maintenance for core business systems. He has participated in the launch of distributed banking core and credit card core system projects and is currently responsible for the construction of the distributed database operation and maintenance system for core systems. He is interested in domestic databases, distributed and containerization-related technologies.
1. Transformation of Domestic Databases in Non-Core Important Systems in the Banking Industry
With the wave of digitalization and the layered advancement of Xinchuang transformation, the transformation of domestic databases in the banking industry has also entered a deep water zone, fully unfolding in core and non-core business systems. However, traditional centralized databases still occupy a high proportion in the banking industry, and the transition from traditional centralized databases to domestic databases still faces a series of challenges such as application compatibility, system availability, performance and scalability, and migration costs.
1.1 Characteristics of Non-Core Important System Applications
Databases used in non-core applications in the banking industry are mainly traditional centralized databases such as Oracle and DB2. These commercial database products are mature and stable, with powerful functions that meet performance and high availability requirements. Some application systems also use open-source databases like MySQL or PostgreSQL, which provide more autonomy and flexibility in development specifications and SQL syntax. The characteristics of this type of application system and database usage are as follows:
-
Applications generally have a large number of complex queries and business analysis logic, using a large number of stored procedures and custom functions to meet customized requirements;
-
Some applications store a large amount of historical data, with single database capacities exceeding 10TB, and even single tables exceeding 1TB;
-
Some applications have non-standard table and index designs, non-standard SQL syntax, and issues such as redundant indexes and excessive indexes that do not comply with development specifications;
-
High availability architectures are implemented based on database technologies such as RAC, ADG, or storage replication, while some applications like MySQL and PostgreSQL still run as single instances;
-
Application systems and databases are deeply coupled with the existing development and operation maintenance system, including resource delivery, monitoring, change management, performance management, and emergency capabilities.
1.2 Difficulties in Transforming Non-Core Important Systems
Due to the variety of databases used in non-core transaction application systems and the complexity of business usage, there are several challenges to overcome during the actual upgrade and transformation process of domestic databases to meet minimal cost transformation and smooth migration:
-
Compatibility of existing stored procedures and complex SQL statements: Compared to traditional centralized databases, domestic databases have certain gaps in compatibility with stored procedures and SQL statements. Some applications may have hundreds of stored procedures, which require targeted development and optimization during the database migration process to meet application needs and reduce transformation work;
-
Performance differences in complex SQL statements: The optimizer of domestic databases generally has weaker performance in handling complex SQL statements, requiring application cooperation for optimization in cases of complex SQL statements such as multi-table joins;
-
Compatibility of various types of databases: Non-core business uses multiple databases, and the selection of domestic databases can either adopt one database compatible with various types such as MySQL, Oracle, PostgreSQL, or choose a domestic database to replace each type;
-
Adaptation to large data volume storage: Some applications may have tables with T-level data, and it is necessary to evaluate whether the performance of domestic databases can meet the storage of such large data volumes and the stability of operation and maintenance management;
-
Smooth migration of applications: It is necessary to assess whether the domestic replacement migration plan supports online migration, and the workload involved in modifying applications related to special character sets, stored procedures, and custom functions in traditional centralized databases;
-
Deployment of high availability architecture: It is necessary to verify whether the high availability architecture of domestic databases meets business continuity requirements, including stability and availability verification of the architecture, and guarantees of RPO and RTO;
-
Integration of operation and maintenance functions: The compatibility and integration of domestic databases with existing operation and maintenance processes and platforms, including resource delivery, monitoring indicators and emergency processes, version and change management, and upgrade maintenance.
In the construction of domestic replacement for core business systems, a large amount of manpower and resources are spent on application development optimization, high availability scenario verification, compatibility verification of domestic databases, and network operation testing. However, non-core business systems often cannot invest so much cost in domestic replacement upgrades, so it is necessary to consider domestic database products that minimize application transformation, are compatible with existing functions and SQL features, and have stable architecture and mature functions during the selection process.
2. Selection of Domestic Databases for Non-Core Important Systems
When selecting domestic databases for non-core important systems, it is essential to meet application performance and scalability requirements based on compatibility with existing application functions, scoring according to the selection standard template, and choosing products with mature and stable functions and architecture.
1) Domestic Database Selection Standards
During the selection process of domestic databases, scoring is conducted based on dimensions such as basic database functions (35%), database development functions (20%), high availability capabilities (10%), database operation and maintenance management (20%), database security functions (10%), and other database functions (5%). During the product POC testing or selection testing process, each detailed dimension is verified one by one, and the final evaluation results are summarized for reference in the final assessment of database products. The general selection standard template is shown in the table below, and the functional detail indicators should be evaluated according to each category:
Table 1 General Standards for Domestic Database Selection
2) Principles for Promoting the Use of Domestic Databases
Due to the characteristics of stored procedures, complex SQL, excessive data volume, and multiple database types in non-core business systems, the following principles should be followed during the promotion and use of domestic databases.
-
Prioritize application development transformation costs, ensuring that existing application database products can be compatible with most functions and requirements, including compatibility with Oracle and MySQL syntax. For new applications, functional development should be based on the standards and specifications of domestic databases;
-
Ensure application performance and system architecture scalability, prioritizing centralized deployment to guarantee the current performance, data storage, and high availability architecture of existing applications. When data volume or performance is limited by single-node constraints, use distributed databases for replacement;
-
Ensure sustainable development of architecture, combining industry technological evolution to ensure that the current architecture aligns with future development directions and is recognized by internal institutions in the financial industry.
Currently, most domestic distributed databases already support centralized deployment, combining both distributed and centralized modes. During the product selection and promotion process, it is necessary to meet both distributed and centralized deployment scenarios. The following table can be referenced during actual promotion and use, utilizing virtualization resource pools and storage virtualization to meet the computing performance and data storage requirements of centralized deployment:
Table 2 Deployment Principles for Domestic Databases in Non-Core Important Business
3. Multi-Tenant Architecture Design of Domestic Databases for Non-Core Important Systems
For non-core important systems in banks, the centralized deployment model has met most application needs. At the same time, utilizing the multi-tenant management capabilities of current domestic distributed databases, management is conducted according to tenant dimensions, fully utilizing the resources and operation management capabilities of the management cluster, reducing the deployment resources of application clusters. As shown in the figure below, application clusters or units correspond one-to-one with tenants, all managed under a unified database management cluster for resource isolation. The unified database cluster management under multi-tenancy has the following benefits:
Independence between tenants: including tenant resource management and isolation, user management, high availability deployment and configuration, backup recovery, change maintenance, and upgrades, etc.;
Unified operation and maintenance management interface: monitoring indicators, performance indicators, and other operation data are exposed through a unified standard interface, facilitating integration with existing operation and maintenance platforms;
Centralized database management cluster: multiple applications share a set of management clusters, which is more beneficial for operation and maintenance management compared to having one management cluster for each application, saving deployment resources. Otherwise, hundreds of applications would require hundreds of management clusters.
Figure 1 Multi-Tenant Deployment Model of Domestic Databases for Non-Core Important Systems
Most domestic distributed databases support multi-tenant management mode. Taking GoldenDB database as an example (as shown in Figure 1), multiple tenants are deployed under the same management cluster, with each tenant using Xinchuang virtual machines (domestic core + domestic OS), and the file system using storage resource pools for allocation, meeting the computing performance and storage space requirements of applications, avoiding the performance and storage capacity limitations of single PC servers.
Figure 2 Multi-Tenant Deployment Architecture Based on GoldenDB Database
Figure 2 shows a typical multi-tenant single-center deployment diagram based on GoldenDB database, where each application is independently deployed and managed according to tenants, and applications access the database through DB load balancing. In the management cluster deployment, the management nodes and global transaction nodes are shared among multiple tenants, while each tenant’s data nodes and computing nodes are virtualized hosts, and the file system is allocated through the virtualized Huawei all-flash OceanStor Dorado SAN storage resource pool, completing full-stack Xinchuang from hardware, operating system to database layer.
Non-core transaction systems have both OLTP and OLAP loads. For storage layers using local disk architecture, the following defects exist:
-
Coupling of computing and storage resources, making independent expansion difficult and causing resource waste: When expanding system capacity, data on nodes needs to be redistributed, which takes a long time and affects business. At the same time, storage and CPU resources are strongly bound, and when performance reaches a bottleneck, CPU utilization is often still low, leading to resource waste;
-
Long recovery time for disk failures: After a local disk failure in the database server, it can only rely on rebuilding the primary and standby database relationships, which takes a long time for data replication and affects the performance of the production database;
-
Databases deployed on local disks of servers have low disk utilization, lack of disk sub-health detection, and issues such as disk failures, slow disks, and wear lead to database failures, causing difficulties in diagnosis and other operation and maintenance problems.
Therefore, the storage layer of domestic databases adopts an all-flash SAN storage resource pool architecture to support OLTP and OLAP mixed loads.
4. Implementation Effects of Domestic Databases in Non-Core Important Systems in Banks
Using the multi-tenant management mode of distributed databases with centralized deployment architecture, a single database management cluster (limited by the resources of the management nodes) can support over 100 application systems and over 1000 database instance clusters. At the same time, each tenant supports independent deployment, relatively independent in high availability architecture and database management, and utilizes virtual machines and shared resource pools to support TPS<2000 and storage<10TB performance and storage capabilities.
Figure 3 Implementation Practice of Multi-Tenant High Availability Architecture Deployment of Domestic Databases in Non-Core Important Systems
1) High Availability
In the deployment architecture, each tenant is independently deployed (as shown in Figure 3), and management nodes are deployed according to a multi-center architecture. Each application is deployed in a single site, same city, or different locations based on its business continuity requirements. During disaster recovery drills, each tenant is also unaffected, achieving isolation and flexibility in deployment resources and architecture.
2) Horizontal and Vertical Scalability of Performance
Each application can scale horizontally and vertically based on performance and storage space growth, adjusting the number of computing node instances and the server configuration of computing nodes and data nodes to meet performance requirements; adjusting the file system mounted on virtual machine servers for storage space scaling. Based on virtualization of CPU, memory, and storage resources, flexible expansion of computing and storage performance is achieved, breaking through the performance capacity limitations of physical servers.
3) Unified Operation and Maintenance Management Platform
Multiple tenants are managed by a unified database cluster, including resource management, high availability management, user management, parameter management, monitoring alarms, performance indicators, backup recovery, and unified API interfaces for operation and maintenance management functions. In the actual production environment, database operation and maintenance management clusters are constructed according to different network areas, such as internet zones, office zones, core zones, and open network areas, with different applications deployed in each management area.
5. Summary of Experiences
This article mainly introduces some practical experiences in the selection and optimization transformation of domestic databases for non-core important systems in the banking industry. It first analyzes the current situation and characteristics of non-core important systems, dissecting the difficulties in database domestic transformation; then, based on the general model and standards for domestic database selection, it explores suitable deployment modes that match business and architectural characteristics; finally, the selection adopts a multi-tenant mode based on virtualization integrated with GoldenDB distributed database and all-flash SAN storage resource pool to support OLTP and OLAP mixed loads. Focusing on the multi-tenant deployment mode, it details the high availability deployment architecture, functional characteristics, and implementation effects. Compared to other domestic centralized or distributed databases and using local disk deployment, adopting a storage-computing separation architecture based on professional storage and using centralized deployment of multi-tenancy makes applications more independent and flexible in deployment and management, while reusing the resources of the management cluster, reducing the number of database clusters and improving resource utilization.
References:
1. White Paper on Database Transformation of Open Platforms in the Financial Industry
2. Report on Innovative Development of Databases in the Financial Industry 2023
3. Guide to Distributed Database Evaluation
Project Collaboration:
Gao Bin, System Engineer at a City Commercial Bank
Li Wentao, Storage Engineer at a City Commercial Bank
Gao Jian, Storage Engineer at Guiyang Bank
Wang Zhijun, System Engineer at a City Commercial Bank
Zhu Xiangdong, Senior Engineer at a Bank
Project Review:
Shen Junjie, System Architect at Guilin Bank
Click on the end of the article to read the original text and leave comments for discussion
If you find this article useful, pleaseshare, like or click“See” to let more peers see it
Recommended Materials/Articles:
-
Construction of a High-Performance and High-Availability Cloud-Native Platform Based on Domestic All-Flash NAS Storage by a Rural Commercial Bank
-
Design and Application Practice of a High-End All-Flash Xinchuang Storage Two-Location Three-Center Architecture for Important Transaction Systems in a Bank
-
Practice of Building Local Disaster Recovery Based on Domestic High-End All-Flash Storage by Inner Mongolia Rural Credit Union
-
Application and Practice of All-Flash Storage in Core Systems of Banks
-
Application of All-Flash Storage in Core High-Availability Systems of Banks: Deployment Practice and Batch Performance Improvement Testing
Welcome to follow the community “Storage” Technology Theme , which will continuously update quality materials and articles. Address:https://www.talkwithtrend.com/Channel/179
Download the twt community client APP
Long press to recognize the QR code to download
Or search for “twt” in the app store
Long press the QR code to follow the public account
*The content published by this public account only represents the author’s views and does not represent the community’s position; the cover image is authorized for use by copyright libraries