This article provides insights into the application and practice of Oracle databases in PowerVM virtualization environments, serving as a reference for Oracle architecture design. The content is compiled by community expert Chen Yongfa based on technical discussions within the community.
What factors and principles should be considered in the architectural planning and design of Oracle database clusters deployed in a virtualization environment?
The following issues need to be considered:
1. The heartbeat interaction data flow between different RAC cluster nodes will accumulate;
2. The resource consumption of host CPU and memory is greater than that of other partitions;
3. The distribution of different business type partitions on the same physical machine leads to significantly increased storage IOPS and throughput requirements, making it difficult for the same storage path to meet both demands simultaneously.
The design must meet the following requirements:
1. When deploying Oracle databases in a Power virtualization environment, it is recommended to fully consider the bandwidth requirements for production and heartbeat networks, using 10G networks whenever possible, ensuring redundancy to avoid insufficient network bandwidth or high availability issues;
2. Estimate the type and volume of storage I/O, and design more storage paths to meet the requirements based on different storage access types. Separate vscsi and npiv physical HBA cards, reserving more channels for npiv to increase redundancy, reduce congestion, and meet the different IOPS or throughput needs of each business partition;
3. Distribute and pair partitions with different business characteristics, such as pairing application partitions with database partitions, OLTP with OLAP, and critical systems with non-critical systems, to create staggered resource usage and balance the resource utilization of a single physical machine.
Does deploying a core database in a virtualization environment affect database performance?
There is definitely an impact, primarily in three areas: network, storage I/O, and CPU. The purpose of virtualization platforms is to improve resource utilization, which requires resource sharing, making it difficult to achieve exclusive resource allocation. Some virtualization platforms can directly connect to storage, while others cannot. Power virtualization can allocate storage directly to VIOC via NPIV, which can enhance storage I/O performance in a virtualization environment.
Databases are often the most challenging aspect of website architecture, with bottlenecks typically occurring here. Databases are highly sensitive to I/O, making them a focal point in architectural design, while I/O performance in virtualization environments has always been a bottleneck. Therefore, whether a virtualization environment can meet the requirements for running Oracle databases depends first on the product specifications. If the specifications meet the requirements, then conduct stress testing. If the stress test confirms that it can indeed meet the requirements, then it can be used confidently; otherwise, do not change the environment casually. For data warehouse types, it is recommended not to place them in a virtualization environment. Other transactional systems can be considered for virtualization based on business conditions and testing validation. From our current operational situation, except for data warehouses, everything else runs in the Power virtualization resource pool, with overall stable operation and no performance issues encountered.
Currently, following the design principles outlined in this article, our Oracle database environment has been migrated from traditional physical machines to the Power virtualization environment. Due to optimized basic network configuration, we have essentially eliminated node downtime caused by heartbeat network delays and congestion, significantly improving the stability of the RAC cluster compared to traditional physical machine methods. Storage I/O has been classified and dispersed as much as possible, and no I/O bottlenecks have been found. The key to the performance of a virtualization environment lies in whether monitoring is in place and whether management is diligent.
What considerations should be made for hardware selection in the application of Oracle databases in Power virtualization?
For hardware selection, it is advisable to choose high-end models with as much configuration as possible, as hardware redundancy and stability will be relatively better. When selecting hardware, there are a few basic principles to keep in mind. These principles are based on specific elements such as CPU performance, I/O, and average data throughput. However, the first consideration should be what operating system we are using and whether virtualization technology is needed.
For example, virtualization server environments respond well to high-performance and multi-core CPUs. Oracle products that require high-performance I/O are better suited for systems capable of running multi-threading, which need a 64-bit data path and high-speed interconnects. These high-speed interconnects include fiber channels for SAN (Storage Area Network), high-speed 10 GB Ethernet, and a high-performance panel for intelligent rack systems.
Of course, when selecting hardware, we should not only consider server performance. Practicality, integration, scalability, and management are also very important factors for the Oracle environment. These factors determine the final return on investment (ROI) and total cost of ownership (TCO), which will affect the overall budget of an enterprise. Products that are easy to upgrade have been proven over time to often have lower TCO and can increase ROI.
When choosing hardware for the Oracle environment, following the principles mentioned above will be helpful. These principles are not only about performance but also about cost considerations, utilization, expected growth, and the hardware lifecycle.
How to meet the different IOPS and throughput requirements of different business types simultaneously?
It is necessary to configure more storage paths based on different business types and storage service levels, distributing the data traffic of different business types to different storage paths. For example, separate vscsi and npiv methods, reserving more channels for npiv, can meet the needs of different business types while avoiding excessive traffic on a single path, ensuring the database’s access performance to storage and preventing path traffic congestion.
Design more storage paths to meet the requirements, separate the physical HBA cards of vscsi and npiv, reserve more channels for npiv, increase redundancy, reduce congestion, and meet the different IOPS or throughput needs of each business partition. Distribute and pair partitions with different business characteristics, such as pairing application partitions with database partitions, OLTP with OLAP, and critical systems with non-critical systems, to create staggered resource usage and balance storage I/O pressure.
Currently, most storage supports enabling QoS, provided that the storage itself has sufficient performance, and priority guarantees can also be made at the storage level.
How to design the heartbeat network for Oracle database clusters?
1. Hardware and Parameters
According to Oracle’s official recommendations, they first recommend using 10G Ethernet, at least using Gigabit Ethernet. If the load is very high, the private network can use Infiniband. However, this entirely depends on the specific business volume and load conditions of the customer’s production environment. This is just a reference; if conditions allow, configurations can be made according to the recommendations. The private network connection needs to use switches, as Oracle cluster installations do not support direct connections in private networks. The dual attack rate parameters of network cards and switches must be kept correct and consistent.
2. Network Card Binding
Various platforms have their own network card binding tools, providing load balancing and active-backup mode binding. First, to improve the high availability of public and private networks, network cards need to be bound. For Linux platforms, we need to control the specific strategy of network card binding in the configuration file “/etc/modprobe.d/dist.conf”:
mod=0, i.e., (balance-rr) Round-robin policy.
mod=1, i.e., (active-backup) Active-backup policy.
mod=2, i.e., (balance-xor) XOR policy.
mod=3, i.e., broadcast policy.
mod=4, i.e., IEEE 802.3ad Dynamic link aggregation.
mod=5, i.e., (balance-tlb) Adaptive transmit load balancing.
mod=6, i.e., (balance-alb) Adaptive load balancing.
For private network card binding, mode=3&6 will lead to ORA-600, while public network card binding mode=6 will lead to BUG9081436. For specific binding modes, in cases where the platform version is low and the network architecture is very complex, it is still recommended to use the active-backup mode, as it is more stable and less likely to cause inconsistent packet path issues. In load balancing mode, if network parameters are not set scientifically, it is easy to have packets sent from one physical network card but returned to another physical network card. When firewall rules are added to the network link, it can easily lead to packet loss issues.
For AIX platforms, change the parameter mode to NIB or Standard value. Standard determines which physical network card to use to send packets based on the target IP address, which is IP address-based load balancing and is less likely to cause the aforementioned packet loss issues.
3. SCAN IP
Oracle RAC introduced the SCAN (Single Client Access Name) feature after 11gR2. SCAN is a domain name that can resolve at least one IP and up to three SCAN IPs. Clients can access the database through this SCAN name, and the SCAN IP must be in the same subnet as the public IP and VIP. After enabling SCAN, a virtual service layer is added between the database and the client, which is the SCAN IP and SCAN IP Listener. Clients only need to configure the TNS information for SCAN IP, and through the SCAN IP Listener, connect to the backend cluster database. Thus, regardless of whether nodes are added or removed from the cluster database, it will not affect the client, and configuration does not need to be modified. There are some configuration considerations related to SCAN:
(1) The default gateway of the host must be in the same subnet as SCAN and VIP.
(2) It is recommended to resolve the SCAN name (11gR2 and higher) to at least three IP addresses in a round-robin manner through DNS, regardless of the cluster size.
(3) To avoid name resolution issues, if we set three SCAN addresses, then there should be no SAN records in the HOSTS file, as records in the HOSTS file are statically resolved, which contradicts DNS dynamic resolution.
4. Network Parameters
Kernel parameters related to the network on the operating system platform are very important, directly determining the stability and performance of data transmission in private and public networks. However, the relevant parameter settings vary for different operating systems.
1. Linux. For Linux platform kernel parameters, two are very important (net.core.rmem_default, net.core.rmem_max). The specific function explanations are as follows:
net.ipv4.conf.eth#.rp_filter: Packet reverse filtering technology.
net.ipv4.ip_local_port_range: Indicates the range of IPv4 ports that applications can use.
net.core.rmem_default: Indicates the default size of the socket receive buffer.
net.core.rmem_max: Indicates the maximum size of the socket receive buffer.
net.core.wmem_default: Indicates the default size of the socket send buffer.
net.core.wmem_max: Indicates the maximum size of the socket send buffer.
To achieve better network performance, we need to adjust the above two parameters from their default values to 2-3 times or even higher, and disable or set the reverse filtering function to disabled (0) or loose mode (2).
2. AIX. For AIX platform kernel parameters, the following settings are the best configurations extracted from Oracle’s official documentation:
tcp_recvspace = 65536;
tcp_sendspace = 65536;
udp_sendspace = ((db_block_size * db_multiblock_read_count) + 4096);
udp_recvspace = 655360;
rfc1323 = 1;
sb_max = 4194304;
ipqmaxlen = 512;
The first two parameters indicate the TCP window size, while the third and fourth parameters indicate the UDP window size. Enabling rfc1323 specifies window scaling and timestamp options as specified by RFC 1323 (TCP Extensions for High Performance). Window scaling allows TCP window sizes (tcp_recvspace and tcp_sendspace) to exceed 64KB (65536) and is typically used for large MTU networks. The default is 0 (off), and if you attempt to set tcp_sendspace and tcp_recvspace to greater than 64KB, you must first change this value to 1. ipqmaxlen specifies the number of packets to receive that can be listed in the IP protocol input queue. sb_max specifies the maximum buffer size allowed for a TCP and UDP socket.
Comparison of the advantages and disadvantages of fully virtualized database server deployment versus mixed deployment of virtual machines and physical machines?
1. This idea of combining physical and virtual machines is good. For critical systems, there is always a lot of concern when first moving to virtualization. This can be a gradual adaptation process, slowly transitioning to virtualization based on actual operational conditions;
2. During the virtualization migration process for Oracle database clusters, the adopted solution is “virtual-physical combination,” first migrating one physical node to a virtualized node, and after stable operation for 1 to 2 weeks, migrating the other physical node to a virtualized node;
3. The vast majority of Oracle databases are currently running in the Power virtualization resource pool. Compared to traditional physical machine environments, there are more improvements in standard uniformity, resource utilization, and maintenance workload. The management and operation of Oracle databases in virtualization environments are also gradually improving;
4. From our current operational situation, there have been no major issues in terms of reliability and performance. If running databases in a virtualization environment, it is recommended to reserve a certain margin of physical CPU and memory resource pools to prevent the system from needing to expand related resources online due to sudden performance demands.
Long press the QR code to follow the public account
