Author / Senior Information Technology Expert at China Postal Savings Bank Software Development Center, Pan Hua
China Postal Savings Bank Software Development Center, Wang Qifeng, Li Beichuan, Zhang Guozheng
In recent years, with the rapid development of technologies such as artificial intelligence, big data, and cloud computing, emerging technologies have been increasingly integrated with traditional scenarios, accelerating the digital transformation across various fields. Currently, the scale of enterprise information systems migrating to the cloud continues to expand, and system architectures are trending towards microservices and distributed systems. Due to varying maturity levels in the software development lifecycle, there may be discrepancies in system security levels and business capabilities. Business security data elements have become a critical issue that enterprises urgently need to address in terms of cybersecurity. At the same time, enterprises face diverse and comprehensive cybersecurity threats, such as Advanced Persistent Threats (APT), insider threats, supply chain contamination, and external real-time asset detection. It is crucial to ensure software security and reduce the risk of security incidents in the face of massive and heterogeneous systems.
Current State of Penetration Testing
Penetration testing simulates real attacks to comprehensively assess whether systems and networks have security vulnerabilities and to take timely remedial measures, significantly enhancing an organization’s cybersecurity defense capabilities. The commonly recognized types of testing are divided into white-box testing, gray-box testing, and black-box testing. Currently, business security penetration testing heavily relies on experienced penetration testers. There is a significant gap between the defects and vulnerabilities discovered by penetration testing tools in business logic and the ideal target requirements, necessitating tools that can understand business processes through human-like thinking. By utilizing machine learning and deep learning algorithms, it is essential to achieve automated business security penetration testing, which faces the following challenges.
1. Technically Challenging. Different information system development technology stacks and business processes exhibit significant differences. Furthermore, traditional vulnerability scanning products are based on feature matching for vulnerability verification. Business security penetration testing requires a hybrid approach, making it difficult to complete comprehensive business security penetration testing with a single tool or technology.
2. Difficulty in Ensuring Testing Accuracy. The diversity of vulnerabilities necessitates a combination of various detection methods. Traditional rule-matching methods lack autonomous learning capabilities, resulting in an inability to keep pace with evolving attack methods.
3. Low Testing Efficiency. Traditional business security penetration testing often relies on manual operations, leading to low testing efficiency. Testers need to manually execute test cases, resulting in lengthy testing cycles that cannot quickly complete security threat assessments.
4. High Resource Investment. To cover the scope of business security penetration testing as much as possible, a significant amount of human and time resources is required, thereby increasing the labor costs associated with business security penetration testing.
Although business security automated penetration testing faces some technical challenges, it is believed that these issues will gradually be resolved with continuous technological innovation and advancement. Enterprises need to strengthen technological research and development, enhance the professional quality and skill levels of testers, and promote the establishment of unified testing standards and specifications to address increasingly complex cybersecurity threats.
Design Concept
The goal is to provide enterprises with an efficient and accurate business security automated penetration testing framework (see Figure 1), employing targeted test case design to improve testing accuracy and effectiveness. The framework consists of a scheduling control center, a test case knowledge base, a vulnerability detection module, and a security evaluation module. The scheduling control center manages modules such as attack tracing, intelligent center, and attack-defense range, enabling intelligent scheduling orchestration and management of business security automated penetration testing. The test case knowledge base relies on advanced AI technologies, training models from a large amount of industry test case data, internal test case data, public information vulnerability sharing platforms, project requirement design documents, and feedback learning from test case data to generate a complete and accurate set of test cases using AI intelligence. The vulnerability detection module is responsible for managing testing tasks, assets, and vulnerability detection. The security evaluation module consists of test coverage evaluation, pass coverage evaluation, and comprehensive test evaluation, assessing the risk level of testing products and whether they meet the launch standards based on the test results from the scheduling control center and business weight, ensuring the business security of banking products.
Figure 1 Business Security Automated Penetration Testing Framework
Design of Business Security Automated Penetration Testing Framework
1. Scheduling Control Center Design. The scheduling control center is used to implement the invocation and management of various modules within the business security automated penetration testing framework, formulating detection plans and priorities, and scheduling the workflow of each module. It features a visual operation interface, supporting manual intervention and automated operation. It is responsible for managing attack tracing, intelligent center, and attack-defense range, achieving unified management of the business security automated penetration testing framework through the scheduling control center.
The attack tracing module is responsible for tracking and analyzing attack paths, utilizing time series technology to reconstruct attack scenarios, while continuously optimizing detection strategies based on chaos engineering.
The intelligent center integrates multimodal capabilities, importing business requirements, business design documents, and providing corresponding dialogue capabilities in the early stages of engineering design. It generates business scenario penetration test cases by calling different models for learning and generation, and through continuous learning and adjustment, it creates a complete business security test case knowledge base, including test paths, test methods, and more.
The attack-defense range utilizes a reconfigurable virtual target environment to verify the stability of tools. Continuous optimization of detection strategies can also be used for practical exercises to enhance the enterprise’s cybersecurity defense capabilities.
2. Test Case Knowledge Base Design. The data sources and composition of the business security penetration test case knowledge base interface with industry data, internal case data, information sharing vulnerability platform data, project business requirement documents, and continuously collected business case optimization data. Big data governance tools are used for data governance, forming a complete business security penetration test case knowledge base aimed at intelligently generating business security penetration test cases. AIGC is introduced based on deep learning training models, utilizing a large amount of business security penetration test case data to train and improve intelligent models, enabling them to intelligently generate business security penetration test cases based on functional requirements.
The test case knowledge base provides users with intelligent test case generation services, generating test cases based on user product requirements. Users can extract intelligently generated test cases according to product needs, forming new test case templates. Users can make minor adjustments to the new test case templates to create test cases that meet product requirements. The ultimate goal is to establish a business security penetration test case knowledge base tailored to the characteristics of the financial industry.
3. Vulnerability Detection Module Design. The vulnerability detection module is responsible for receiving penetration testing tasks issued by the scheduling control center, executing vulnerability detection based on penetration test cases. Penetration testing needs to be associated with the business-related asset situation for business security penetration testing, and the results of penetration testing are fed back to the scheduling control center.
Task management refers to executing tests on received business security penetration test cases based on tasks from the control center. The asset discovery module automatically discovers the network topology, application service port information, component information, and their dependencies to construct a complete asset view based on a combination of active probing and passive monitoring. The vulnerability detection integrates various vulnerability detection technologies, including static code analysis and dynamic monitoring at the web layer, and incorporates AI agents to provide corresponding detection methods based on different business scenarios of the business system; it also supports a plugin-based detection engine extension mechanism.
4. Business Security Penetration Testing Evaluation Design. The business security penetration testing evaluation assesses the security risk of vulnerabilities detected during penetration testing, providing a comprehensive evaluation based on business product-related elements, which is also an effective way to ensure business security. Before the comprehensive evaluation, the test case coverage and test case pass rate must meet preset standards. Ideally, the test coverage and pass rate should be close to or equal to 100%. However, due to many objective reasons in the production process, including business direction, product design ideas, and R&D issues, it is challenging to achieve comprehensive coverage in business security penetration testing. To address the objectively existing issues in project production, a method combining coverage and weight is designed for business security penetration testing evaluation (see Figure 2).
Figure 2 Business Security Automated Penetration Testing Evaluation Process
Step 1: Evaluate the test case coverage and case weight dimensions to obtain the business security test case coverage evaluation. When the percentage of generated test cases and case weights exceeds 95%, the business security test case is deemed to meet the standard. Step 2: Evaluate the test pass rate and test case weight pass rate dimensions to obtain the business security test case pass rate evaluation. When the percentage of test case pass rate and weight exceeds 90%, the business security test case pass rate is considered to meet the standard. Step 3: Under the premise of satisfying Steps 1 and 2, introduce evaluation grade classification and comprehensively evaluate the business security vulnerabilities summarized from the test results according to the enterprise’s “Security Management Specifications”. The comprehensive evaluation results ultimately determine whether the business can pass the test.
Multi-factor verification is employed to ensure the robustness of test cases and the security of test results. To facilitate understanding of the calculation of test case coverage and pass rates, a unified naming specification is established (see Table 1).
Table 1 Business Security Automated Penetration Testing Evaluation Model Comparison Table
(1) Test Case Coverage Evaluation Design. Users generate recommended test cases using the intelligent case knowledge base based on the functional points involved in the business requirements of the project to be tested. Users can select the actual required test cases to ensure that the test cases meet the standards for business security penetration testing. The formula for business security penetration test case coverage is the actual test cases divided by the recommended test cases, resulting in the case coverage rate. According to the above comparison table, UCC is the test case coverage rate, ATC is the actual test cases, and RTC is the knowledge base recommended test cases.
Using AI intelligence for big data analysis of business security penetration test cases and referencing banking security and testing specifications, when the case coverage and case weight are greater than or equal to 95%, the security level of the business security penetration test case is optimal, meeting the standards for business security penetration testing. Users fill in test case data in the template to keep the test case data complete, allowing it to enter the automated testing module.
(2) Test Case Pass Rate Evaluation Design. The test case pass rate is calculated based on the ratio of test cases to test case passes, using actual test case data, test pass case data, and case weight for evaluation. The test case coverage pass rate and test case weight pass rate are calculated based on the test results. The number of passing test cases TPUC divided by the actual number of test cases ATC gives the TPR, which is the test case pass rate.
Based on the number of passing test cases, the weight is calculated using the knowledge base weight, with W levels divided into high, medium, and low. The test case level weights W1, W2, W3, Wn, etc. are defined. The sum of the weight coefficients remains 1, facilitating the calculation of weight ratios and adhering to the principle that the higher the risk of the test case, the higher the weight. The weight W, actual test cases ATC, and actual passing test cases TPUC are used to calculate the test case weight pass rate TCWPR by dividing the W weight coefficient by the actual test cases ATC and then multiplying by the actual passing test cases TPUC.
Using AI intelligence big data analysis results, when the test case pass rate and case weight pass rate are greater than or equal to 90%, the business security penetration testing results meet the security standards.
(3) Comprehensive Evaluation Design for Vulnerability Levels. The risk levels of detected vulnerabilities are evaluated based on comprehensive evaluation rules, generating test reports in conjunction with the bank or enterprise’s “Security Management Specifications”, which also serve as the basis for whether to go live. The comprehensive evaluation rules combine external vulnerability risk warnings with internal security issues discovered, standardizing risk levels. A scoring system is established using base scores (external public vulnerability sharing platform warnings), environmental scores, system types, and business data security dimensions, categorizing vulnerabilities into four levels: critical, high, medium, and low. Base scores are derived from external vulnerability warnings including CVE, CNVD, CNNVD, etc., environmental scores come from network devices and operating environments, system types are based on the system type hosting the testing product, and business data security is based on data security levels.
Business type vulnerabilities are scored using a combination of base scores, environmental scores, system types, and business data security, calculated as: base score × base weight + environmental score × environmental weight + system type × system weight + business data security score × business data security weight. Risk levels are set based on the risk degree of each dimension, with scores ranging from 0 to 10. The vulnerability risk level scores are matched against the vulnerability risk level rules table (see Table 2) to determine the business vulnerability risk level. The number of vulnerability risk levels is then input into the go-live judgment rules table (see Table 3) to determine whether it can go live.
Table 2 Vulnerability Risk Level Rules Table
Table 3 Go-Live Judgment Rules Table
Based on the bank or enterprise’s “Security Management Specifications” and relevant industry security details, the go-live judgment rules table is designed. This table can be adjusted according to the actual situation of the enterprise to meet the security testing applications of various software business types. If the number of levels does not meet the go-live judgment rules, even if the test case coverage and weight exceed 95%, and the test case pass rate and weight exceed 90%, the test cannot pass, and vulnerabilities need to be fixed before retesting. After the comprehensive evaluation of vulnerability risk, the system generates a test report based on the test results.
Considering the broad applicability of the model, knowledge base weights, template coverage, template weight evaluation thresholds, and vulnerability risk judgment rules can be adjusted by enterprises according to their actual situations to ensure the safety and scientific nature of business security automated penetration testing case evaluations.
Application Scenarios of the Framework
The application scenarios of the business security automated penetration testing framework are extensive, as business security penetration testing is required throughout the enterprise information system development process. In different industry applications, only parameter adjustments are needed to comply with industry or enterprise security management specifications. Therefore, the validation of application scenarios is naturally set in the financial industry, which also needs to adhere to the software security management specifications of the financial industry.
It is stipulated that after software development is completed, a business security penetration test must be applied. The full process of application scenarios can be demonstrated through the intelligent test case knowledge base application, test case coverage calculation, test case pass rate calculation, comprehensive evaluation, and test report generation.
1. Information System Security Assessment Application Scenario. Continuous security assessments are conducted before the system engineering goes live, integrating the automated penetration testing framework into the DevSecOps environment to ensure that security testing is conducted with every code submission and deployment, promptly identifying and fixing security vulnerabilities and risk points in the information system. It can also provide safe improvement measures for developers; by designing and applying the automated penetration testing framework, the efficiency and coverage of security testing are significantly improved.
2. Attack-Defense Drill Application Scenario. Attack-defense drills simulate network attacks in a real environment, conducting comprehensive network security penetration testing on target business systems to identify network vulnerabilities and security risks within the business system, testing and enhancing the organization’s network security defense capabilities. By using the automated penetration testing framework, continuous and automated threat simulations can be achieved, improving the efficiency and effectiveness of drills and facilitating cross-departmental collaboration. Attack-defense drills can also test the intelligent security test case model, identifying defects in automatically generated test cases through attack-defense drills, continuously optimizing training data and permissions for automated test case generation calibration, while enhancing employees’ security awareness and emergency response capabilities, providing robust security guarantees for the business systems of enterprises and organizations.
3. Business Security Penetration Testing Application Scenario. The application scenarios for business penetration testing are quite broad, as business security penetration testing is required for project launches or iterations. Through penetration testing, potential internal and external security risks in business systems can be identified in a timely manner, preventing immeasurable economic and reputational losses to enterprises due to vulnerabilities.
The business security penetration testing model excels in test case writing, with the test case knowledge base assisting testers in writing test cases, improving the quality of test case writing, reducing the experience requirements for testers, and avoiding the risk of missed tests due to defects in test case design. The business security automated penetration testing case evaluation model is crucial for enterprises. It can comprehensively assess the security of systems and identify potential threats, guiding testing teams to conduct targeted and comprehensive testing, enhancing the overall security of systems; at the same time, it significantly improves employees’ security awareness, refines security policies, and reduces legal risks.
Conclusion
The business security automated penetration testing framework adopts a layered decoupled architecture design, enhancing scalability while supporting a plugin-based detection engine extension mechanism, allowing adjustments based on specific business needs and system characteristics to ensure the flexibility and practicality of the test case evaluation model. Through parameter and weight adjustments, it can be widely applied in software testing across different industries.
In the future, the business security automated penetration testing evaluation model will identify and detect potential security threats within business systems. Through in-depth analysis of test results, it can assess the scope of potential business risks and the paths of business vulnerabilities, further revealing system weaknesses, facilitating developers in addressing unknown security issues in business. It can identify issues in enterprises’ security policies and processes, enabling targeted improvements to security strategies, enhancing system defense capabilities, and continuously incorporating more emerging technologies, integrating proactive defense, intelligent response, and other new security capabilities to create an enterprise-level business security automated penetration testing platform.
(This article was published in the “Financial Electronics” October 2024 issue)
New Media Center
Director / Kuang Yuan
Editors / Yao Liangyu, Fu Tiantian, Zhang Jun, Tai Siqi
