Introduction
This article discusses the technology of storage switch networks, exploring solutions for replacing FC-SAN switch networking with NVMe over RoCE networking as a part of the trusted innovation industry. It fully illustrates how, under the premise of emphasizing autonomy and controllability, the technological content of selection and design can be improved, which is a goal every enterprise should pursue.
Author: Xu Gong
Storage Engineer, A Commercial Bank
1. Introduction
In recent years, driven by a series of policies such as the State Council’s “14th Five-Year Plan for Digital Economy Development”, the Central Bank’s “Financial Technology Development Plan (2022-2025)”, and the China Banking and Insurance Regulatory Commission’s “Guiding Opinions on the Digital Transformation of the Banking and Insurance Industries”, the digital transformation of the banking industry is entering a new phase of “accelerated running”, with greater emphasis on data management and innovation in information technology applications.
Among these, data is the most important production factor in the digital economy era and is also a key element supporting the digital transformation of financial institutions. It is crucial for banks to fully unleash the value potential of data elements during their digital transformation. To carry out data mining and analysis, and to empower business development with data, a stable and efficient data infrastructure must be established.
2. Current Storage Situation in Traditional Banking Environments
To cope with the large-scale application of OLAP business and relational databases in the banking industry, batch processing generally faces tight processing time windows. Using FC-SAN networks and centralized all-flash storage devices as the IT system infrastructure has become a common foundation to ensure high performance, high reliability, and low latency of business systems. The use of redundant fiber switches provides reliable stability and security, with servers and storage devices connected to mutually redundant SAN fiber switches via redundant fiber channels. FC-SAN provides a high-performance, highly reliable, and highly available solution.
3. Issues Faced by Traditional Storage Operations and Maintenance
Due to patent monopolies and other reasons, SAN fiber switches are 100% controlled by the United States, which poses a significant risk to the future of the financial industry in China regarding full autonomy and controllability. Fiber switches have become the biggest shortcoming and stumbling block in the construction of new infrastructure in China. To overcome the American technological barrier of fiber switches on the path to full autonomy and controllability, it is particularly necessary to explore a solution that can achieve better performance than using FC switches while ensuring comprehensive autonomy and controllability of IT technology facilities, including computing, storage, and networking.
4. Implementation of Storage Network Replacement in the Trusted Innovation Environment
In response to the new demands of digital transformation in the banking industry, starting from data infrastructure is undoubtedly key to solidifying the foundation. In recent years, the widespread application of GPUs and SSDs has rapidly improved the computing and storage performance of data centers, but network protocols have gradually become the performance bottleneck of data centers, urgently requiring the creation of a new “highway”.
Initially, NVMe, as a hard disk interface protocol, can fully leverage the performance of SSD media particles, releasing the enormous potential of storage systems. Today, NVMe SSDs have been widely adopted by cloud service providers with high performance requirements. According to predictions from Gartner, by 2022, NVMe SSDs will account for 52% of storage, surpassing half of the market and becoming truly mainstream. The comparison of IO paths between NVMe and SAS protocols is shown in Figure 1.
Figure 1 Comparison of IO Paths between NVMe and SAS Protocols
RDMA (Remote Direct Memory Access) is a technology that allows direct memory access from one computer to another without involving the operating systems of either side. It achieves high bandwidth, low latency, and low resource consumption. The dedicated InfiniBand network architecture for RDMA is closed and incompatible with existing networks, making it costly to use. However, with the emergence of RoCE (RDMA over Converged Ethernet) under the trend of network convergence, high-speed, ultra-low latency, and extremely low CPU usage RDMA can now be deployed on the most widely used Ethernet.
RoCE is a network protocol that carries RDMA over Ethernet, with performance comparable to IB networks, requiring Ethernet to support DCB features to ensure a lossless network. RoCEv2 is a network layer protocol that introduces IP to solve scalability issues, enabling routing capabilities across Layer 2 networks. The RDMA protocol stack is shown in Figure 2.
Figure 2 RDMA Protocol Stack
NVMe over RoCE is a type of NVMe over RDMA, as shown in Figure 3, utilizing RDMA’s high-performance remote access to achieve remote access to NVMe devices. NVMe over RDMA implements NVMe multi-IO queues directly on RDMA multi-queues, with RDMA’s SQ/CQ corresponding to NVMe SQ/CQ, achieving end-to-end multi-queue without the need to define a new message format, as NVMe protocol messages are directly used as RDMA data transmission.
Figure 3 NVMe over RoCE Queue Mapping
NVMe over RoCE networking is similar to iSCSI networking, both based on IP networks. NVMe over RoCE requires switches to support DCB lossless network configuration. All-flash storage systems need to support NVMe over RoCE interface cards, which typically support 25GE and 100GE rates.
Facing standard NVMe over RoCE networking (as shown in Figure 4)
Figure 4 NVMe over RoCE Networking
Compared to FC networking, which has shortcomings in automatic management and ease of use, most domestic manufacturers have enhanced network management plugins to enable hosts to automatically and quickly establish connections and quickly identify faults, better supporting the needs of core business applications. To maximize the performance of all-flash storage NVMe SSDs while expanding the application of NVMe in data centers, NVMe over Fabric has emerged. The choice of NVMe over RoCE (NoF+) by domestic IT infrastructure manufacturers not only ensures the innovation needs of information technology applications but also offers higher overall performance and cost-effectiveness. By coordinating servers, switches, and storage, and installing the SNSD (Storage Network Smart Discovery) plugin (Figure 5), NVMe-oF service information can be registered, enabling service change broadcasting. Servers can control access to NVMe-oF Targets by receiving service change broadcast notifications.
Figure 5 SNSD Enhanced Plugin
All-flash storage systems can support automatic discovery of storage devices; enhanced host drivers automatically discover storage devices in the network and establish connections without manual intervention; link failures automatically switch IO paths quickly; enhanced host drivers automatically disconnect storage based on received port change information and switch business to backup connections, as shown in Figure 6.
Figure 6 NVMe-oF Fast Failover
5. Conclusion
From the results of the transformation, the end-to-end NVMe environment after removing FC-SAN can achieve higher performance compared to FC. From the perspective of autonomy and controllability, the current NVMe over RoCE networking replacing FC-SAN switch networking can be regarded as the best solution for achieving comprehensive autonomy and controllability of IT infrastructure in China. The specific comparison is summarized in Table 1:
Table 1 Comparison between FC-SAN Switch Networking and NVMe over RoCE Enhanced Networking
Support the community by liking, sharing, or clicking “♡” on this article.
Related Recommendations:Kubernetes Container and Storage Integration Standards and PracticesWhat New Requirements Do Container Cloud Environments Pose for Storage Solutions?In the Context of Full Cloudification, Can Enterprise Storage Stand Alone?Architectural Design of Software-Defined Storage in Trusted Innovation CloudPerformance Analysis and Optimization of Storage Systems
Welcome to follow the community for the latest insights and opinions from industry experts related to “Storage”.
Address:https://www.talkwithtrend.com/Channel/179
Long press the QR code to follow the public account.
*The content published by this public account only represents the author’s views and does not represent the community’s position.