Editor’s Note:
As one of the earliest participants in network virtualization, Li Li-Jun has long been dedicated to software architecture design and competitiveness in the field of network virtualization. In the blink of an eye, 11 years have passed, and he has grown from a “novice” to an expert in the field. The network virtualization product EVS (Elastic Virtual Switch) he participated in developing has progressed from prototype development to large-scale commercial use, covering scenarios such as cloud core and public cloud, making him one of the important “leaders” in Huawei’s network virtualization.
The story begins in March 2010 when I graduated with my master’s degree and left the ice city of Harbin, which was wrapped in a silver coat of snow, taking a green train south to settle in Hangzhou, a “paradise on earth” with scenery every ten steps, starting my journey of struggle.
At that time, Huawei had not yet built the Hangzhou Research Institute campus, and the office was in a shared office building in a technology park with other companies. The environment was not as good as it is now at the Hangzhou Research Institute, and there were black swans; I joined the application platform development department of Chinasoft, and the distributed parallel computing laboratory had not yet been established. When I arrived at the office area and saw every colleague busy and full of energy, I knew that this would be the beginning of my career.
The mindset and scene at that time are still fresh in my memory, and in a blink, it has been eleven years.
From “Fear of Blue” to “Blue Screen Master”
When Huawei launched the unified virtualization platform (UVP) in 2009, it was the first project I received after joining the company.
UVP can run multiple internally nested virtual machine operating systems in the operating system, providing an experience similar to that of real servers. It not only helps customers reduce procurement and maintenance costs but also greatly improves business deployment efficiency. When I joined the project team in May 2010, I participated in the code development of the virtual machine front-end driver alongside experienced employees. This front-end driver, which runs inside the virtual machine, is a virtual network card driver that can realize various functions of a real network card and is an important part of the entire UVP.
At first, as a novice, I really didn’t understand anything and could only learn. Those working in virtualization know that two essential introductory books are “Principles of System Virtualization” and “Linux Kernel Analysis.” I held these two books, chewing through them while writing code. At that time, I was young and ambitious, and I had to get to the bottom of any questions I didn’t understand.
To understand where the first line of code for the virtual operating system starts running, I spent more than half a month going through 400,000 lines of code in the open-source software Qemu. I had no time to read during the day due to work, so every night when it was quiet, I would read code on the computer. After the morning meeting the next day, I would actively share and communicate with everyone, deepening my understanding of virtualization principles through the process of reading code. Gradually, my work started to become more proficient.
After New Year’s Day 2011, the project encountered a very tricky problem. Due to the UVP platform’s inadequate response to certain exceptional scenarios, the Microsoft Windows operating system would occasionally crash when running in the virtual machine, often resulting in a blue screen. At that time, there were many forms and scenarios of the problem, some belonged to performance pressure testing scenarios, some were due to unstable physical networks, and the operating system had various distribution versions, with blue screen error codes varying widely. At that moment, stability in operation became the greatest project risk at the customer level.
To address this, we initiated a special project focused on blue screens in virtual machines, intending to systematically summarize and analyze the reasons behind these problems to eliminate this “roadblock.” At this time, the PL (project leader) approached me: “You are familiar with the underlying code, and you work diligently; you will be responsible for this special project while also honing your communication and closure skills with external issues.” I didn’t expect the PL to entrust such an important project to me, and a sense of trust welled up inside me. I suppressed my excitement and agreed, determined to do my best and live up to expectations.
Initially, everything went relatively smoothly, but as the work progressed, the special project entered deeper waters. Issues like damage to critical system information were particularly challenging, and we failed to close the loop within the planned timeframe; coupled with the team’s lack of understanding of the internal principles of the Microsoft operating system, progress in locating blue screen issues was significantly affected.
During that time, I faced numerous blue screens every day, leading to a psychological state of “blue fear,” making me somewhat anxious and restless, often wearing a worried expression late at night. The PL noticed this and came to chat with me, listening to my issues and saying, “The only fear in work is not knowing where the problem lies. Now that you know where the problem is, what is there to fear!” This statement suddenly enlightened me.
The next day, with the PL’s help, we connected with the Microsoft liaison. For the first time, I felt like an expert while communicating with Microsoft experts on equal footing, and they affectionately referred to me as “Engineer Li.” Through multiple discussions with Microsoft experts, we gained a clearer understanding of the meanings of blue screen error codes, the roles of the registry and key system files, and learned many analysis methods for problem logs and driver debugging in the Microsoft operating system, greatly enhancing the team’s combat capability.
Thus, I immersed myself in researching blue screen issues. I categorized the causes of blue screens and tackled them one by one, almost eliminating all causes of blue screens. However, unexpectedly, towards the end of the troubleshooting phase, a major issue arose— a virtual machine from a Shanghai operator suddenly blue-screened during operation, interrupting business traffic, and an error code that had never been encountered before appeared on the screen—0xc000021a.
After brainstorming with my colleagues to align our thoughts, we boldly hypothesized: it was either system overload or an abnormal end to a disk IO (input/output) process causing the blue screen. However, two days of information collection, suspicion investigation, and analysis verification yielded no breakthroughs. We were at a loss, and my colleagues were anxious like ants on a hot pan.
I immediately organized a discussion with Microsoft experts. Based on the previous two days of investigation and analysis, we repeatedly discussed and ultimately pinpointed the issue: damage to critical system files on the disk. My heart sank because pinpointing which specific file was damaged required a “clumsy method”—exporting the disk data and comparing the virtual machine’s data one by one. Since the disk files storing the Windows operating system were displayed in hexadecimal, comparing them was akin to “finding a needle in a haystack” among millions of data. The client was still waiting for us, and we could not hesitate; this was our only hope.
Without hesitation, everyone began dividing tasks to compare based on the string content of the data, guessing the location of the critical system files. Line by line, we compared, and soon our eyes were strained; when I looked up, hexadecimal numbers floated before my eyes. It felt like the scene in the novel “The Three-Body Problem” where the protagonist, an academician from the national academy, sees countdowns in his eyes.
I looked out the window to relax my eyes. Before I knew it, night had fallen, and the outside was pitch black, with the moon flickering in and out. Suddenly, a poem surfaced in my mind: “With the sun and moon in my heart, what fear is there of darkness?” Reflecting on the grandeur of ancient scholars, what do our small setbacks count for? I refocused and returned to the data comparison.
Wait? This string of characters seems… inconsistent? I rubbed my eyes, fearing I was mistaken. No, it’s it!
“Found it! Found it!” I couldn’t help but shout in the office, and the colleagues who were locating with me jumped up in excitement, gathering around. They crowded around my computer, not wanting to miss any steps of validation. We finally discovered that when the system software runs, if there is a sudden power outage, it will force a shutdown, and if some critical system file modification operations do not end correctly, it can lead to incomplete data, resulting in a blue screen upon restart. I immediately reconnected with the Microsoft experts to confirm, and they agreed with our pinpointing conclusion, praising our locating capabilities and technical level. At that moment, my colleagues showed excited and encouraged expressions.
After clarifying the cause, besides quickly helping the customer restore business, we also needed to consider how to completely resolve this issue to avoid future “needle-in-a-haystack” situations. We ultimately devised two measures: one was to write it into the emergency manual so that key system files could be quickly repaired in case of problems, and the other was to introduce a disk IO exception protection mechanism in the UVP platform under unexpected scenarios like power outages to ensure the integrity of disk data. This way, blue screen issues caused by such reasons would no longer occur, completely eliminating this roadblock.
Through our special team’s continuous efforts, not only were blue screen issues completely eradicated, but we also produced many tools for locating, code documentation, and Windows blue screen repair guides, ensuring the high-efficiency and high-quality delivery of the virtualization project. As a result, I was awarded the Technical Pioneer Award, and later whenever blue screen issues arose, everyone would turn to me for help, and I was happy to discuss and solve them, feeling like I had become the “Blue Screen Master.”
This battle not only greatly enhanced our team’s combat effectiveness but also gradually established my organizational skills and technical influence, further solidifying my confidence in career development.
Ten-Day Challenge, A Battle to Fame
In the following years, I continuously navigated through various projects in the virtualization field, from virtual network card drivers to UVP kernel maintenance and development, and finally to the delivery of the network virtualization product eVS (Elastic Virtual Switch). The work I handled became increasingly challenging, while my vision, knowledge, and technical reserves also expanded and accumulated continuously.
The functionality achieved by eVS is similar to that of physical switches and routers, providing data transmission for customer applications within virtual machines. For example, downloading short videos from Douyin or pushing hot data from Weibo is all achieved through the backend eVS of the virtual network card.
The main performance of data transmission lies in the transmission speed. At that time, the eVS was still using open-source software technology, and copying a large file was more than twice as slow compared to the local area network. For instance, encountering sudden announcements of marriage by celebrities or trending topics suddenly climbing to the top of searches would reveal the speed difference quite evidently. The goal set by the team was to elevate eVS performance to industry-leading levels, which was indeed a huge challenge for us at that time.
When the eVS performance optimization work started, the initial technical direction was to adopt a mainstream user-space high-performance packet processing framework, which could significantly shorten the processing overhead of the packet protocol stack path, reduce latency, and thereby increase transmission speed. At the beginning of 2014, due to external product pressure, we needed to complete performance verification quickly. The department’s TDT (Technical Development Team) manager called us into the conference room, introduced the situation, and immediately asked me, “I’ll give you 10 days to create the first eVS prototype. Are you confident?”
10 days? I quickly went through the core workload in my mind: developing a high-performance packet processing framework from scratch would take X man-days, developing physical and virtual network card driver modules would take X man-days, and determining the forwarding logic of eVS… Plus, with no experience, there would inevitably be some unknown issues along the way, so we had to leave some room for error.
10 days, can it be completed? I felt a bit uneasy inside. But in my heart, the biggest difficulty was not in the workload but in the technical implementation with zero experience. As long as this point was guaranteed, I was confident we could win this battle. With pressure comes motivation, what is there to fear! I decisively accepted this challenge, while also requesting support from Intel-related technical personnel on-site. Soon, the relevant developers, support personnel, and material resources were in place, and we formed a small task force of five, with me in charge.
Thus began a whirlwind 10-day battle!
With the ten-day challenge announced, we had to efficiently solve technical problems. First and foremost was the forwarding framework, as it is the foundation of the eVS architecture, closely coordinating with other components like a commander mobilizing and organizing the functions of each module. This mobilization function needs to be stable and reliable while hiding driver differences and supporting different types of network card drivers, making its design particularly critical. Fortunately, during the three years of work and study, I had read a lot of open-source code involved in various projects, such as Qemu/Linux, where there are many excellent design frameworks for decoupling, like how to abstract interfaces, how to allocate responsibilities of interfaces, or how to define the organization of key data structures at different layers… Drawing on these excellent experiences accelerated the detailed design of the framework.
To complete the task within 10 days, we needed to enhance efficiency and gather the entire team’s strength to focus on it. I provided each team member with detailed task breakdowns and plans. However, this was not enough; to motivate everyone, I set up a competition within the team—seeking the MVP. Like in the game Honor of Kings, each person would have a path, and the comprehensive “output damage” (code quality), “death count” (bugs), and “assist count” (helping other team members review code) would determine who was the MVP. Thus, my team and I entered a frenzied state of “growth” and “output.” To meet the ten-day challenge and not lose the competition, I wrote code until my hands were sore. The coding skills accumulated over three years were fully utilized, and at one point, I found myself in a state of “flow,” completely unaware of the passage of time.
Of course, since it was a team battle, individual performance was important, but more crucial was the collaboration. While achieving individual “growth,” we also had to insist on mutual assistance. Therefore, I organized pair programming sessions to review the codes developed by other colleagues, which also helped mutual understanding and ensured timely identification and closure of potential issues.
As one sub-function after another was verified and debugged, the deadline was drawing near. Finally, the time came on the 10th day after receiving the task. During the smoke test (the first use case test of basic functionality), the entire department watched us, witnessing the test results. The TDT manager had a half-smile on his face, which made me, who was originally confident, break into a sweat.
When the network ping between two virtual machines using eVS was successful, I knew the test had gone smoothly. The framework we designed in ten days achieved forwarding performance of X pps, meeting project expectations, and we successfully completed the task. We conquered the final “Crystal Tower” and achieved victory in the ten-day challenge. I heard cheers, and when I turned around, the team looked as if they had won a lottery of two hundred million; I held back my excitement inside, feeling that this was much more gratifying than winning ten consecutive victories in Honor of Kings!
The software architecture developed in the eVS prototype has been used to this day, widely utilized and continuously evolving in NFV and public cloud. I can say that after this battle, I gained fame. Subsequently, I began working as a system engineer (SE) and code committer, responsible for the crucial tasks of architecture safeguarding and competitiveness building for the eVS product.
Enduring Challenges, Remaining True to My Original Intent
At the beginning of 2015, we welcomed the first successful commercial product of eVS—cloud core NFV (Network Function Virtualization) network elements, which had particularly high performance requirements.
To ensure our product has a definite competitive advantage and remains at the forefront of the industry, we put significant effort into the packet loss rate, an important indicator reflecting product competitiveness, leaving no stone unturned even for the occasional low-probability packet loss issues. Sometimes, I would even think of solutions before bed and occasionally find inspiration in my dreams.
To let customers better understand our technical strength and how our products can genuinely help them solve problems, I also prepare thoroughly to discuss with clients alongside the frontline, meticulously addressing various stringent technical questions from clients, often spending an entire afternoon over tea.
Ultimately, the eVS product achieved industry leadership in various operator competition projects in the cloud core NFV telecom cloud scenario. Later, the eVS product further expanded into Huawei’s public and private cloud scenarios, becoming the main product in the field of virtualization.
Starting in 2019, with the large-scale commercial use of eVS in Huawei’s hybrid cloud, shipment volumes increased exponentially, and a more flexible and efficient self-developed architecture for management—”Gaea”—was born. This name originates from the ancient Greek mother of the earth, and we hope the new architecture will be a creative platform supporting various forwarding services.
This task is not just about designing and developing a module; it also requires building a completely new self-developed architecture that supports large-scale hybrid clouds, aiming to expand scale and reduce costs to support Huawei Cloud’s future business growth by a hundredfold. This is the biggest challenge I have faced since starting my career, as well as our team’s new mission. Today, the Gaea architecture has already been implemented and commercialized in the eVS 5.0 product, achieving revolutionary improvements in key technical indicators of configuration specifications, and the efficiency of data packet delays and session establishment has reached an industry-leading level, effectively supporting Huawei Cloud’s goal of large-scale commercial use of cloud server instance C6ne.
Looking back on the bits and pieces of these eleven years of work, I have been “leveling up” through various challenges, completing the transitions from developer to MDE, then to SE, and finally to architect. This has been the most meaningful and profound time in my career. The process of “architecture validation and analysis” has made me deeply feel that all technical research and innovation must start from value scenarios and needs, truly centering on the customer and making technology benefit humanity; high-quality code is the prerequisite for products to pursue industry leadership, safeguarding the code architecture is also a very proud thing; growth itself is a process of constantly stepping out of the comfort zone, the more adversity, the more one must swim against the current, and greater challenges motivate me to keep changing and improving myself.
“Complete what can be changed with optimism, have the courage to embrace what cannot be changed, and have the wisdom to distinguish the difference between the two.” I really like this sentence. In the future, I have two expectations for myself: one is to have good communication skills and mature thinking methods in doing things; the other is to become a full-stack technical expert with a sense of mission.
On the journey to seek wisdom, there will inevitably be numerous challenges ahead. I believe that as long as I remain true to my original intent, I will achieve my mission. The long wind will eventually break the waves, and the future is promising!
Source: Voice Community “Huawei People”
↓↓ Click “Read the Original” 【Join the Cloud Technology Community】
Related Articles:
A storage story from a Huawei veteran of twenty years
Impressive! A PhD sister shines in this black technology related to future data security
A PhD’s 22 years at Huawei (full of insights)
This Huawei “Doctor Strange” can handle it, challenging high-performance machine learning algorithm libraries, he is truly impressive!
A female PhD with an annual salary of 1.56 million joined Huawei! Netizens: A perfect blend of beauty and intelligence
After ten years at Peking University, he was mentally prepared for unemployment upon graduation, but…
Latest and most comprehensive 2020 Cloud Status Report “69-page PDF download”
RightScale’s 2019 Cloud Status Survey Report: 35% of cloud spending is wasted “attached 50-page PDF download”
For more articles, please follow
For a better reading experience, click here [Looking]👇