Introduction
Since the advent of server virtualization technologies like VMware, the efficiency, operational flexibility, and economic benefits of building enterprise data centers have greatly improved. Looking back a decade ago, when we wanted to deploy a new system, we first needed to apply for the purchase of a server. After it arrived, we had to physically move it to the data center, find a spot to install it in the rack, power it on, connect network cables, install the operating system, and by the time we could ping the new server’s IP, several months would often have passed. However, after the full promotion of virtualization in data centers, this process has become much easier: how many machines do we need? I just need to submit a request on the private “cloud” management platform, and after approval from the platform administrator, the required virtual machines are automatically deployed, with the entire process taking as little as a few hours.
While enjoying the many benefits of virtualization, we may not have realized that with the large-scale application of virtualization technology, the infrastructure, operation and maintenance, and organizational management of enterprise data centers have undergone significant changes, and the security risks faced are no longer the same as before.
Comparison of Traditional and Virtualized Environments
Let’s take the domain controller, which penetration testers love to challenge, as an example. In the past, the position and logical relationship of the domain controller in the network looked like this:
*Figure 1: Traditional deployment diagram, DC refers to Domain Controller, SA refers to Server Automation system.
The domain controller was installed on a dedicated physical server, and the domain administrator clearly knew its location in the data center, set complex operating system passwords, and carefully safeguarded them, promptly applying Microsoft patches, and strictly managing users and permissions, being sensitive to any signs of trouble on the server and their own terminals. At this time, the domain controller was like a solid fortress, making it very difficult to successfully penetrate from the network.
After using VMware virtualization, a typical deployment scenario looks like this:
*Figure 2: Virtualized deployment diagram, VM refers to Virtual Machine, VCenter refers to vSphere Center, vSAN refers to virtual storage, DC refers to Domain Controller.
All physical servers run the VMware ESXi system, and all virtual machine resources are centrally managed through VCenter. To provide a convenient resource application pathway for other personnel, a certain “cloud” management system may have been deployed, providing internal users with access through a Web Portal, and interacting with VMware’s interface through an automated deployment backend to allocate, change, and reclaim virtual machine resources. Whether it is the domain controller DC, VCenter, or the “cloud” management platform, they are all just ordinary VMs in the virtual resource pool.
Now let’s look at the domain controller DC: the domain administrator cannot ascertain which physical server its CPU and memory are running on, does not know where its disk is stored, does not know which network cable the TCP/IP packets will flow through, and is even unaware if anyone has tampered with it in the virtual world. Even if the administrator still guards the DC server as meticulously as before, they might be wondering if it is still as solid and inviolable as it used to be.
Layered Analysis of Risks in Virtualized Environments
Let’s consider the changes in the operational environment brought about by virtualization in enterprise data centers, as well as the various potential security risks that arise from this.
The application of virtualization has shifted the management of the “server” infrastructure from decentralized to centralized operations, from hardware servers to virtualized operating systems, to virtual machines, and various supporting systems. Now, the construction and maintenance work is concentrated in the virtualization team. In large enterprises, there are hundreds or thousands of physical servers and thousands of virtual machines to manage, which means this team needs a large number of personnel with different skills to complete such complex operational tasks. In the current domestic environment, it is almost impossible for them to all be professional security personnel or have the same level of security awareness, making certain mistakes in some processes almost inevitable.
1 First, to effectively manage and monitor a large number of physical servers, administrators must rely on the hardware management interfaces and out-of-band management networks provided by the servers. For example, HP’s iLO interface, Dell and Inspur’s IPMI interface can monitor the health status of server hardware, power and switch, operating system installation, remote console, and other functions through a Web or SSH interface.
Risk Point 1: Administrators may not have changed the default password for the out-of-band management interface, or have set weak passwords or commonly known passwords within the enterprise. The unified hardware management/monitoring platform (if any) may have vulnerabilities. The following image shows the weak password used for the HP iLO management interface that I discovered during an actual penetration test:
*Figure 3: HP server iLO management interface
And Inspur’s IPMI management interface:
*Figure 4: Inspur server IPMI management interface
2 Secondly, at the virtual host operating system level, administrators need to manage a large number of ESXi Servers, which may require outsourcing or on-site personnel to complete the system installation and initial setup before they can be put into use.
Risk Point 2: Some ESXi Servers may have used weak passwords and later forgotten to change them. All ESXi Servers may use the same password, which might be written in the operational manual.
Weak passwords are very easy to encounter. The SSH interface of ESXi can use VMware’s customized shell; the web interface can browse the virtual disks of the deployed clients; using the vSphere Client can perform all management operations. The following image shows an ESXi host for which I guessed the password during an actual penetration test, allowing me to browse the virtual machine’s disk files through the web interface:
*Figure 5: Browsing VMware data storage
Using the VMware SDK, virtual disks can be mapped to local access, avoiding the need to download large disk files:
*Figure 6: Using VMware SDK to remotely mount virtual disks
3 Furthermore, because dozens or hundreds of virtual clients are running on each virtual host, administrators are often hesitant to make any changes to the virtual host. Imagine something as simple as “restarting”; before taking action, the administrator may need to notify every virtual machine user in advance, shut down the virtual machines that can be stopped, temporarily migrate the virtual machines that cannot be stopped during the specified change window, restart, and then turn on the previously migrated virtual machines and notify users for business validation… Such caution is not without reason, as ensuring availability is the top priority for enterprise data centers. This also means that, except for performance issues or failures, patches or upgrades are generally not proactively installed on virtual hosts.
Risk Point 3: ESXi Servers may never have been patched, potentially containing security vulnerabilities.
For example, after a year or two, I could still find the OpenSSL Heartbleed vulnerability in the ESXi of a development testing environment:
*Figure 7: Heartbleed vulnerability obtaining 64K memory
Sometimes these vulnerabilities can leak internal SOAP interface (vpxa) session values, and with this session, many internal APIs can be called (these vpxa APIs may not even be known to the administrator, requiring you to refer to VMware’s technical documentation), for example, obtaining the storage list of all virtual clients:
*Figure 8: Using leaked session to obtain information in ESXi
4 Moving up to a higher level, we arrive at vCenter. What is the function of vCenter? Let’s take a look at VMware’s introduction:
Whether you have a dozen virtual machines or thousands, VMware vCenter Server is the simplest and most effective way to manage VMware vSphere. With VMware vCenter Server, you can manage all hosts and virtual machines in the data center from a single console, which aggregates performance monitoring functions for clusters, hosts, and virtual machines. VMware vCenter Server allows administrators to gain insights into the clusters, hosts, virtual machines, storage, guest operating systems, and other key components of the virtual infrastructure from one location.
In the language of penetration testers, this means: vCenter is the palace of the virtualization world, and the vCenter administrator is the king. Obtaining management permissions for vCenter allows one to rule over hundreds or thousands of virtual machines. Managing hundreds or thousands of virtual machines is certainly not a task for just one or two people. It may require dividing responsibilities among different individuals based on functional areas, and routine change operations may be delegated to on-site teams. This involves security issues related to accounts and permissions. Using simple passwords has become a high-probability event. Sometimes, for convenience, passwords may even be shared with you—once, I needed to reset a virtual machine’s root password and the on-site engineer gave me the vCenter admin account password to log in and connect to the console myself. Perhaps someone was testing an automated deployment program and wrote high-privilege account credentials in a script, and one day the server storing the script was compromised due to a weak password—this is just a possibility, but I have indeed encountered it in my environment.
Similarly, vCenter may also have vulnerabilities, and administrators are unlikely to be diligent about applying patches. In 2015, vCenter had a Java RMI remote code execution vulnerability CVE-2015-2342, which could easily compromise vCenter:
*Figures 9, 10, 11: Exploiting RMI vulnerability to spawn a System-level shell
Using mimikatz to read the vCenter admin password allows logging into vCenter:
*Figure 12: Logging into vCenter, gaining control of all virtual machines
This exploit may still be useful today.
On the main vCenter, the domain controller server may be among them, and you can now perform a hot clone operation, cloning an offline virtual machine, then using the vCenter console to log in, export the domain database, copy it to other virtual machines you control (for example, via shared virtual disks), and then delete the cloned machine. This process would leave the domain controller administrator completely unaware, and the domain controller itself would not generate any unusual system events.
*Figure 13: Hot cloning a target server
Another issue with vCenter is the virtual machine templates. New virtual machines are generally generated from limited templates, and their Administrator or root passwords are hardcoded. If there is no mechanism to modify the initial password before delivery to users, vCenter will continuously produce virtual machines with the same administrator password. Therefore, try scanning all virtual machines with the initial password to see what you find!
5 Let’s turn our attention further to the so-called “cloud” management platform. In reality, it may have other names; calling it “cloud” is just trendier. Its function is similar, providing a convenient channel for internal IT users to apply for, maintain, and destroy virtual machine resources through a web portal. This is a natural demand, and many third-party vendors have created such platforms. However, these platforms may also have various security issues. For example: how are the Web Portal accounts created and managed? How many users have administrative privileges? Are there any default passwords? Who manages the admin accounts on a daily basis? Does the Web Portal have common web vulnerabilities, such as SQL injection? Do the backend servers, including the database server, have weak passwords? Is the linkage with vCenter and vSphere done through vCenter accounts or API Keys? Are accounts or API Keys stored encrypted? And so on.
In practice, I once encountered a situation where both the Web Portal and backend servers were generated using the same Linux template, which had a uid of 0 for an admin user with a fixed password. The personnel of the “cloud” management platform only modified the root password and did not change the password for the admin account. Then, I discovered the database connection account in the Web platform, and found that the “cloud” management platform’s account password was stored using unsalted md5 hashes, with a large number of accounts synchronized from the OA system, all preset to the same password, and the platform administrator also used a simple password. Thus, this management platform was compromised. By logging in as the platform administrator, they could create their own application requests and approve them, allowing for unlimited deployment of virtual machines!
*Figure 14: A certain “cloud” platform—virtualization management platform
6. Supplement: Scanning and Discovery of VMware Products
As an internal penetration tester, how can one discover and identify VMware products (including vCenter, ESXi, etc.) in the enterprise environment? There are indeed some techniques. First, VMware products have specific service ports, such as 22, 80, 427, 443, 902, 9875, etc. Secondly, the service banner information or SSL certificate information may contain keywords like VMware or vSphere. This allows for quick discovery of VMware products in the network using scanners like zmap + banner grabbing. So, how do we determine the logical relationship between vCenter and the ESXi it manages? The key is the SLP protocol and the vpxa API. The SLP protocol can obtain the VMware hostname and ESXi version of the target IP address, for example:
~# /usr/bin/slptool 'unicastfindsrvs' 10.1.12.135 'service:VMwareInfrastructure'
service:VMwareInfrastructure://10.1.12.135,65535
~# /usr/bin/slptool 'unicastfindattrs' 10.1.12.135 'service:VMwareInfrastructure'
(product="VMware ESXi 6.0.0build-1921158"),(hardwareUuid="32393735-3733-4E43-4731-313954385050")And the vpxa API can query the vCenter address managed by ESXi:
The URL is: url_fmt = ‘https://%s/vpxa‘ %(ip)
The two SOAP requests are as follows:
apixml1='''<?xml version="1.0"encoding="UTF-8"?>
<soapenv:Envelopexmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd=" http://www.w3.org/2001/XMLSchema"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body><QueryVpxaStatusxmlns="urn:vpxa3"><_this
type="VpxapiVpxaService">vpxa</_this></QueryVpxaStatus></soapenv:Body></soapenv:Envelope>'''
apixml2='''<?xml version="1.0"encoding="UTF-8"?>
<soapenv:Envelopexmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body><GetVpxaInfoxmlns="urn:vpxa3"><_thistype="VpxapiVpxaService">vpxa</_this></GetVpxaInfo></soapenv:Body></soapenv:Envelope>'''The returned response message contains the vCenter address.
Next, you can use various historical vulnerabilities to test them.
Using the results of the above scans, a network diagram of all VMware relationships can be drawn, similar to the following effect:
*Figure 15: Overview of all VMware in the network
Let’s take a closer look:
*Figure 16: Logical relationships between VMware and ESXi, as well as vulnerability status
The ESXi in the diagram are all labeled with their versions; those without ESXi are vCenter. The vCenter points to the ESXi it manages. Either vCenter or ESXi may be isolated and unconnected. The red nodes indicate vulnerabilities. At that time, three vulnerabilities were tested: one was Heartbleed (SSL), another was the vCenter RMI vulnerability (CVE-2015-2342), and the last was an XXE vulnerability (caused by Apache Flex BlazeDS).
Conclusion
Virtualization has brought efficient operations and good economic benefits to enterprise data centers, but it has also introduced security threats and risks that are different from those in the past. This is something we may not have deeply considered in terms of specific response strategies, and we lack corresponding management measures and technical means. This puts the operational environment of enterprises at risk. How to identify, manage, and mitigate these risks is indeed a matter worth pondering for security practitioners. This article is summarized from the author’s practical work experience over the past few years, and I hope it can inspire some thoughts.
The related code for this article is available at: https://github.com/penoxcn/VMware, interested parties can refer to it.
* Author of this article: ipenox, this article belongs to the FreeBuf original reward program, reproduction without permission is prohibited
