|
“The Performance Evaluation Initiative for Typical Server Application Scenarios” has now been launched! — In order to standardize the construction of a full-stack standard system, promote the iterative upgrade of product technology, better showcase the advantages of the entire architecture and computing power, and support excellent product and solution brands in enhancing and expanding their market through official authoritative certification, the Green Computing Industry Alliance’s “Performance Evaluation Initiative for Typical Server Application Scenarios” (referred to as: Evaluation Initiative) has officially launched! This evaluation initiative includes big data, distributed storage, web applications, high-performance computing, centralized relational database management systems, ARM native cloud phones, and virtualization—seven typical server application scenarios. |
Today, we bring you a detailed interpretation of the “Performance Testing Methods for Server Application Scenarios: Virtualization”.
The drafting units of “Performance Testing Methods for Server Application Scenarios: Virtualization” include China Electronics Standardization Institute, Yunhong Information Technology Co., Ltd., Feiteng Information Technology Co., Ltd., Shenkangfu Technology Co., Ltd., Hunan Qilin Xin’an Technology Co., Ltd., Wuhan Yangtze River Computing Technology Co., Ltd., Huawei Technologies Co., Ltd., Arm Technology (China) Co., Ltd., and Anshengpei Semiconductor Technology Co., Ltd. We thank the above units for their strong support.
We welcome you to continue following the “Performance Testing Methods for Server Application Scenarios” series articles, and we will bring you a series of exhibitions of products/solutions evaluated through the “Performance Testing Methods for Server Application Scenarios: Virtualization”!
01
Background
The wave of digital transformation is sweeping across various industries, continuously driving the innovation of IT architecture and business models in enterprises with new technologies such as cloud computing, big data, and artificial intelligence. The new generation of data centers dynamically schedules and allocates infrastructure resources through virtualization and cloud computing technologies, solving various issues in traditional IT architecture, including low resource utilization and sharing rates, high procurement and operational costs, and long business launch cycles, helping enterprises move towards agile architecture and gain the value benefits brought by new technology stacks, thus further reducing costs and increasing efficiency, ensuring business quality and safety.
According to a report by CCW Research, in 2020, the scale of China’s server virtualization market was approximately 4.37 billion yuan (growth rate of about 16.8%), and the scale of the hyper-converged market (usually based on virtualization integrated hardware and software solutions) reached 10.59 billion yuan (growth rate of about 36.8%); the government, manufacturing, finance, telecommunications, medical, and education sectors are the top six industries with a high adoption rate of server virtualization, benefiting from the low cost, green low-carbon, elastic scheduling and scalability, and high reliability characteristics of virtualization technology.
The report reveals two market characteristics and changes:
1) Domestic virtualization software based on XEN or KVM has matured, possessing certain latecomer advantages in functionality, performance, and even price, thus becoming a growth driver in the main markets of government enterprises and medium to large enterprises, gradually encroaching upon and replacing the existing and incremental market of foreign VMware virtualization software;
2) With the advancement of information technology application innovation strategies, especially the ZTE-Huawei incident and the Sino-U.S. trade war deeply shaking the hearts of the Chinese people, opportunities for the rise of domestic chips represented by ARM architectures such as Kunpeng and Feiteng, as well as diversified routes such as Haiguang, Zhaoxin, Longxin, and Shenwei have emerged, leading to a new wave of development for virtualization as a key technology in cloud infrastructure, leveraging the efficiency, flexible scheduling, low construction, and management costs, and hardware-software decoupling advantages of virtualization, injecting vitality into the domestic independent software and hardware ecosystem and accelerating the process of information innovation.
From the computing power planning and server hardware selection process of clients in government and finance sectors, we have noted the important role of virtualization technology in simulating actual production business scenarios. The virtualization platform meets compatibility with various chip types and versions, as well as hardware conditions such as motherboard firmware, and provides support for various operating systems and application software on virtual machines; virtualization allows a physical server to run multiple isolated virtual machines simultaneously, each running different types of workloads that vary in resource demand according to the application. Compared to performance testing aimed at a single physical server and a single application, performance simulation testing aimed at virtual machines can more comprehensively reflect the performance capacity and level of the virtualization platform, simulating more closely to actual production business scenarios from multiple aspects such as multiple virtual machines in parallel, multiple virtualization ratios, and multi-core configuration calculations, thereby providing effective research data support for clients in technology selection, application migration, and computing power design and capacity planning.
02
Testing Methods
The overall idea of virtualization scenario testing is to combine the actual production points and configuration requirements of clients, focusing on four representative indicators of server performance: CPU, memory, storage, and network.
From these series of indicators, we design typical use cases aimed at physical servers, single virtual machines, and multiple virtual machines, using universal, industry-recognized testing tools, parameters, and methods. By simulating multiple times based on different computing power configurations and testing granularities to obtain average score results as baseline references, we can systematically evaluate the performance differences before and after virtualization, the performance capacity of the virtualization environment, and the horizontal comparison of different hardware under the same virtualization platform.
Additionally, for the virtualization platform itself, this “Method” also proposes “practical” angles such as long-term stability testing and execution efficiency of batch tasks, which have important practical significance for production-level applications.
Under the testing methods, we provide specific examples for reference. For example, in CPU computing capability testing, the performance of the CPU is crucial for CPU-intensive applications, including tasks such as compression and decompression, GCC compilation, operating system scheduling, as well as graphics processing and video encoding/decoding, thus the method design includes dimensions of comprehensive performance, integer performance, and floating-point performance.
Furthermore, since single-core performance testing mainly targets applications that only invoke single-core performance, and multi-core performance testing targets most applications that can invoke multi-core performance, the overall performance of the CPU will have a more pronounced impact for CPU-intensive applications; therefore, we further refine the above dimensions, using the universal computing performance benchmark tool CPUBench to distinguish single-core and multi-core application performance in both physical and virtual machine systems by adopting single-process and multi-process parallel execution methods.
It is worth noting that the testing of virtual machines is again divided into two categories: single virtual machine and multiple virtual machines, reflecting the performance of different resource deployment ratios used in different production environments (for example, a virtualization ratio of 1:6) and the performance capacity level of the virtualization platform.
Similarly, another representative player in computing capability—memory—also plays a crucial role. We mainly focus on the performance of memory bandwidth, which has high reference significance for the speed and stability of various applications during startup and operation.
Storage performance generally reflects application performance bottlenecks, and the performance of storage differs for different types of applications. For applications involving large file read and write, the influence of sequential read and write IOPS and bandwidth directly reflects the efficiency and speed of the application; random read and write IOPS and latency mainly affect applications such as structured databases and bulk small file operations, where random read and write performance plays a key role in the operational performance of such applications. Network performance primarily targets data transmission based on TCP or UDP under different modes, specifying different packet length parameters to observe network bandwidth quality reports in a request/response manner. When external network environment conditions are consistent, it can reflect the network card or virtualization network processing capability of the server.
-
Testing Configuration Description
The tested samples for this test used an ARM architecture server (chip HUAWEI Kunpeng 920 5251K) and an X86 architecture server (chip Hygon 7280), with the server virtualization software both being the Yunhong CNware series virtualization products. The virtual machine operating system selected is the Galaxy Kirin Operating System V10 version.
Caption: The CNware series virtualization products are independently developed by Yunhong and fully embrace the information innovation software and hardware ecosystem, leading support for one cloud and six cores (Kunpeng, Feiteng, Longxin, Haiguang, Zhaoxin, Shenwei), optimizing compatibility and synergy of various basic software and hardware in cloud computing, maximizing computing power performance, and providing neutral cloud computing solutions with financial-grade, hardware-software decoupling, and extreme performance.
Detailed configuration information is shown in the table below:
03
Testing Environment and Results
|
Testing Environment Introduction — This evaluation initiative is undertaken by the China Electronics Standardization Institute, and the alliance collaborates with the China Electronics Standardization Institute to create the open laboratory “GCC Open Lab”. Registered units can apply for the professional typical application scenario software and hardware compatibility testing environment provided by the China Electronics Standardization Institute or can build their own testing environment, with experts arranged by the institute for remote guidance and review to ensure the authority of the process and results. |
All testing data or functional implementation results in this report come from the joint laboratory of the Green Computing Industry Alliance, providing objective and fair testing data under the condition of ensuring similar testing environments as much as possible.
The goal of this test is to comprehensively evaluate the performance of servers based on typical chip architecture servers + Yunhong virtualization software from multiple dimensions. To highlight typicality and comparability as much as possible, this test defines the use of virtual machines with specifications of 4 cores, 8GB memory, and 100GB disk for testing, which is relatively common in client environments and thus has strong representativeness.
In the comprehensive performance testing item, the CNware virtual machine has smoothly demonstrated performance levels close to that of physical machines, thanks to the R&D tuning and performance breakthroughs of the CNware virtualization kernel. From the testing results of different architectures, it can be seen that in the comprehensive performance testing item, ARM single-core performance leads x86 single-core performance by about 6%, and multi-core performance leads by about 7%.
The memory performance testing results show that under single-thread conditions, ARM memory performance is comparable to x86; under multi-thread conditions, ARM outperforms x86 in all indicators, with Copy speed leading by about 42%, Scale speed leading by about 58%, Add speed leading by about 16%, and Triad speed leading by about 47%.
The storage performance testing items are divided into sequential read/write and random read/write, showing that the ARM architecture leads the x86 architecture by about 1.5 times in sequential read IOPS and bandwidth performance, leads x86 by about 56% in sequential write IOPS and bandwidth performance, leads x86 by about 14% in random read IOPS and bandwidth performance, and leads x86 by about 11% in random write IOPS and bandwidth performance.
The network performance testing results show that the ARM architecture’s bandwidth performance is comparable to x86 in TCP mode; in UDP mode and under packet length settings of 4B/32B/256B/1024B, the bandwidth performance improves between 20%-40%.
From the comparison of samples, the ARM architecture has more processor cores, achieving a higher virtualization ratio and performance capacity without contention; from the testing samples chosen, although ARM architecture’s single-core processing performance is not significantly different from traditional platforms, it is more prominent in multi-core processing and concurrent read/write access. This indicates that in virtualization scenarios, ARM servers can still leverage multi-core and high-concurrency performance advantages, enabling applications to achieve faster processing capabilities, higher bandwidth throughput, and lower latency based on software and hardware collaboration.
04
Summary and Outlook
The testing methods gather the results of extensive discussions among many excellent hardware and software vendors, reflecting the real requirements of technology selection testing and production applications; furthermore, from the testing results of this virtualization scenario, green computing servers based on ARM architecture demonstrate strong competitiveness compared to traditional architectures, exhibiting no significant shortcomings and even outperforming in certain dimensions. In terms of overall solutions, green computing servers combined with domestic virtualization software also exhibited excellent performance and high reliability in this evaluation, fully meeting the application performance demands of various industries.
We welcome you to continue following the “Performance Testing Methods for Server Application Scenarios” series articles, and we will bring you a series of exhibitions of products/solutions evaluated through the “Performance Testing Methods for Server Application Scenarios: Virtualization”!
In the future, the Green Computing Industry Alliance will continue to promote performance evaluations of typical computing business scenarios, driving the joint optimization of underlying server hardware and upper-layer application software to reflect the overall competitiveness of full-stack solutions. Meanwhile, we will collect more industry demands from alliance members, incorporating more typical server application scenarios (such as edge computing) and more green computing server configurations, making typical application scenario evaluations a core competitiveness of the Green Computing Industry Alliance.
Additionally, the alliance will conduct evaluations and research on green computing servers from more dimensions, such as artificial intelligence servers and green energy-saving (power management). The alliance will also study the system and mechanism to promote the implementation of alliance standards through evaluation certification, conducting preliminary research on the latest technology standards for green computing and upgrading testing environments to lay the foundation for subsequent compliance testing certification services.
The Green Computing Industry Alliance will continue to promote certification activities for green computing technology products, services, and management, enhancing the compliance, compatibility, and quality consistency of products and services, and promoting interoperability of products, thereby building trust throughout the entire industry chain. The alliance is committed to encouraging end-users and relevant parties to trust the certification results based on alliance standards, fully reflecting the value of standards and ensuring their effective implementation!
*Thanks to Yunhong Information Technology Co., Ltd. for providing material support for this article.
|
Welcome to register for participation in the “Evaluation Initiative”! Enhancing brand promotion, client recognition, and business opportunity acquisition —
|
|
Don’t hesitate… Participate in the “Evaluation Initiative”! How to participate? The “Performance Evaluation Initiative for Typical Server Application Scenarios” has received enthusiastic responses from alliance members since its launch. The first batch of products and solutions that passed the evaluation will be officially released at the “Green Computing Industry Alliance White Paper and Standard Results Release Conference” in July. In addition, the Green Computing Industry Alliance (GCC) will provide marketing support for the evaluated products through various forms such as “continuous online promotion, publishing evaluation rankings, and showcasing excellent cases”.
|
Click below “Read the Original“ to visit the alliance’s official website, download and view the standard text of “GCC 7005-2022 – Performance Testing Methods for Server Application Scenarios: Virtualization”.
Thank you for your long-term attention and support for the Green Computing Industry Alliance GCC! The alliance values your thoughts and feelings, and we welcome you to scan the QR code above to enter the alliance survey questionnaire, providing valuable opinions and suggestions for our work, and collaboratively building a new development for the green computing industry! (The deadline for this questionnaire is August 10, 2022, after which the alliance will select lucky participants for exquisite gifts!) Come and scan to participate~~
——— End of Text ———
About the Green Computing Industry Alliance Since its establishment in 2016, the Green Computing Industry Alliance aims to collaboratively build a green, open, autonomous, and shared ecological system, dedicated to promoting the development of the green computing industry, building a platform for industry communication and cooperation to enhance enterprises in PC, server, storage, operating systems, databases, and other fields, and promoting win-win cooperation in the computing field. It has now become a global alliance with the most complete partners in Arm-based infrastructure server chips, including Kunpeng, Feiteng, Ampere, and Marvell.
