From Breakthrough to Implementation: The Journey of a Researcher–By | Yan Siyu–
With my left hand supporting the table, my knuckles tense, I create a stable arch for the cue; my right hand firmly grips the end of the cue, I hold my breath, gather strength, and push the cue. With a crisp strike, the colored balls roll towards the pocket along the predetermined trajectory. The ball goes in!
Since elementary school, I have loved the crisp sound of billiard balls colliding and watching the cue ball scatter the triangular formation into random patterns. Back then, I stood on tiptoe at the edge of the green table, trying to see the movement path of each shot, fascinated by this mechanical game in a small space.
However, the billiard hall—considered “somewhat chaotic” by my parents—affected the legitimacy of my participation in billiards and determined that my skill level remained at a novice stage. Yet, every time I took a shot, I realized that each successful pot required precise calculations and patient waiting, which unknowingly cultivated my focus and calmness.
In 2016, as an exchange student at Beijing University of Posts and Telecommunications, I interned in the Greater Bay Area and unexpectedly got the chance to communicate with Huawei experts. This was my first close encounter with Huawei and its research work. When I heard the expert say, “Research work is somewhat similar to playing billiards; you need to analyze the trajectory of the balls, break down problems, find breakthroughs, and then push forward step by step,” I couldn’t help but nod in agreement.
Thus began my story with Huawei. A year later, I joined the Data Communication Research Department and started working on data center network research.
Leading the Way: The Adventurer’s “Game”
In 2018, the demand for high-performance networks in data centers began to undergo tremendous changes. Clients in the finance and internet sectors started deploying high-performance applications such as distributed storage and machine learning, which raised the performance requirements for networks to new heights—throughput demands increased from 10G to 100G or even higher, latency needed to be reduced to microsecond levels, and sensitivity to packet loss became critical.
As client business gradually scaled up to thousands of nodes, issues of interconnection, optimization, and maintenance became greater challenges.
With a general direction for client needs established, the next step was to concretize these needs and transform them into our technical competitiveness. My team had accumulated considerable experience in traditional TCP (Transmission Control Protocol) network optimization, based on which we chose “RDMA (Remote Direct Memory Access) large-scale network deployment and optimization issues” as our entry point.
Initially, we designed an online optimization system based on expert experience, attempting to optimize network parameters through manual rules. However, during actual network testing, we found that the system was overly sensitive to transient changes in traffic, leading to increased fluctuations in network performance and even packet loss.
We began to reflect: transient adjustments are prone to disturbances; could we expand the time scale and turn it into steady-state adjustments? In other words, the optimization system might not achieve optimal results by considering only a few network states, but by taking more dimensions of network states into account, we might find the optimal solution.
Thus, we decided to introduce reinforcement learning AI algorithms, abstracting the entire DCN (Data Center Network) as a “observable, inputtable, and outputtable” closed-loop feedback control system. The system collects multi-dimensional network states in real-time as input, and the AI algorithm generates optimal optimization strategies; after executing the optimization strategy, the network immediately outputs new operational metrics and compares them with expected targets. The deviation generated from this comparison serves as feedback, driving the next round of optimization strategy iteration, thus achieving end-to-end automatic, continuous, and refined feedback adjustment.
To better advance the project and facilitate close collaboration with the development team, I, along with Zheng Xiaolong and others, was sent to Nanjing to form a joint project team with colleagues from the 2012 laboratory to discuss algorithm design and optimization.
However, internal disagreements arose regarding the selection of algorithms.
“The problem with centralized control algorithms is that they require collecting the state information of all devices, then calculating and issuing configuration instructions, which itself adds latency and is not conducive to the high-speed changing data center network traffic,” I explained while drawing the network topology on the whiteboard.
“But with distributed control algorithms, if each switch makes its own decisions, won’t there be a situation where they act independently? For example, the optimization actions of one switch could affect other devices, even causing oscillations,” a colleague who insisted on centralized control algorithms retorted.
“Distributed control algorithms are like an adaptive ecosystem, where each node can automatically adjust based on environmental changes without needing a central controller to command everything,” I tried to remain calm and patiently explain my viewpoint.
This debate continued for some time, and none of us could persuade each other, so we had to invite the head of the data communication research department and experts from the AI enabling department to act as “referees.” After listening to both sides’ proposals, they suggested a compromise: first verify the effectiveness of both schemes separately, and then decide based on actual performance.
Thus, we began a competition of algorithms. My colleagues and I designed multiple experimental scenarios, spending nearly a month simulating different network loads and traffic changes to test the performance of both algorithms. Ultimately, the “distributed control algorithm” scheme demonstrated advantages, quickly responding to traffic changes while maintaining good performance in large-scale networks, and was thus adopted, also incorporating some core ideas from the centralized control algorithm scheme to complement each other.
Everyone worked together towards a common goal of optimization, and we named the new scheme: AI ECN (Hyper-Converged Data Center High-Performance Network Traffic Optimization Technology).
Just when I thought things had come to a conclusion, however, issues with algorithm optimization followed one after another.
“How can we elevate the algorithm’s effectiveness to the next level?” was the topic I discussed most with Zheng Xiaolong. He excelled at analyzing problems from the perspective of mathematical models and could always clarify my thoughts with some simple logical deductions. He also had a habit of modeling and simulating on the whiteboard in the meeting room: “You can view the optimization process of each switch as an optimization problem, finding a balance between the speed of algorithm iteration and stability by dynamically adjusting the parameter update magnitude.”
“The core of distributed control is rapid response; if the execution efficiency of the algorithm on switch hardware is not high, the entire scheme loses its meaning.” Whenever I encountered difficulties combining device capabilities with algorithms, the then product SE (System Engineer) Wen Huafeng would always enlighten me with optimization suggestions I hadn’t considered from the perspective of hardware-software integration.
With their support, the algorithm optimization scheme in the lab finally yielded some results, but if the scheme remained in the lab, it would forever be just a “toy.” We sought out willing financial clients to conduct joint verification and optimization of the scheme based on scenarios closest to real customer business. Through collaboration with the OceanStor storage team, the 2012 network technology laboratory, and universities, the scheme not only targeted storage scenarios in the financial industry but also began to show effectiveness in high-performance networks for supercomputing centers.
Ultimately, AI ECN achieved the industry’s first breakthrough in applying AI algorithms to data center network devices, solving Ethernet packet loss and full-scenario generalization challenges, with significantly superior performance. As a fundamental characteristic of DCN hyper-convergence, it opened a new track for DCN Ethernet to replace dedicated networks like FC (Fibre Channel) and IB (InfiniBand). The paper on this technology was fortunate enough to become the first successful submission from the data communication DCN field to SIGCOMM (a top conference in the networking field), and it achieved a series of technical standardizations in organizations such as ODCC (Open Data Center Committee) and IEEE (Institute of Electrical and Electronics Engineers), leading new industry standards. I also received the “Pioneering Award” from the ICT Research and Algorithm Subcommittee in 2022 for this technology.
Breaking New Ground: Persisting in “Monetizing Technology”
Previously, the protocols used in storage networks on the market were basically monopolized by competitor C’s FC (network protocol name). In 2020, with changes in storage clients’ demands for network bandwidth and traffic, we saw an opportunity and decided to attempt using the more versatile and cost-effective Ethernet technology NoF (network protocol name) to directly “compete” with the relatively closed FC.
These are two completely different protocols, and I began to think from the client’s perspective: what would I be most concerned about when transitioning from FC to Ethernet? The answer quickly emerged in my mind: operations and maintenance.
Subsequently, I intensively visited banks and leading internet clients, and their concerns matched my expectations: FC has mature and stable performance visualization tools, but Ethernet is almost blank in this regard, making it difficult to meet their needs.
Therefore, I decided to tailor a measurement visualization solution for NoF and the storage network. The difference in the solution is that our tool can achieve more comprehensive results. With a thought, I immediately took action, utilizing my understanding of storage systems to construct a network and business mapping model, and led my teammates to quickly produce a demonstration prototype.
I took the prototype to Shanghai for in-depth discussions with the operations and maintenance team of a bank client.
“The demo we designed can monitor network traffic in real-time, identify business flow objects based on the network, and sense changes in business performance, allowing for early detection of anomalies and rapid problem localization to ensure application performance,” I explained while demonstrating.
“This is simply the ‘all-seeing eye’ of our operations and maintenance work!” The client’s excitement was palpable, and they even began to envision scenarios for deploying this technology in the future.
However, there was still a long way to go from demo to actual productization. Productization needs to consider the constraints of actual devices, and there was a significant bottleneck in the current switch devices. Imagine, the switch’s forwarding capacity is as high as 12.8T, while the measurement channel of the co-processor is only 100G, a difference of hundreds of times, like trying to boil dumplings in a teapot with a spout too small to let them out.
“The limitations of this device are just ridiculous,” I often murmured to myself while rubbing my sore eyes during that time.
My teammate Liu Ning, sitting at the adjacent workstation, would immediately stop what he was doing whenever he heard my voice, come over, and carefully examine the data on the screen, calmly offering a new solution: “Perhaps we can change the way the table is recorded in the microcode…”
After several attempts, it still didn’t work. Our daily work seemed to be on a “loop playback” mode. Repeated failures, repeated battles… I often felt powerless and had considered giving up. Where was the way forward?
Fortunately, through repeated discussions with surrounding teams, after four months, we finally found a breakthrough in the algorithm, breaking free from the constraints of switching devices: through feature compression algorithms, we ensured measurement accuracy while significantly reducing channel pressure and table resource usage, allowing the scheme to be implemented based on existing products.
In fact, in addition to algorithm breakthroughs, through communication with solution colleagues and clients, I gradually realized that the initial scheme might have an issue of over-design. The scheme’s functions were relatively broad, and I wanted NoF networks to carry too much, such as measuring various interaction delays and the time consumed at different stages of business, which inadvertently increased the difficulty of data processing and resource consumption exponentially. In reality, we only needed to select the most representative measurement indicators to reflect the real-time performance of NoF business. Therefore, I also recognized from another perspective: in R&D, we always think about making functions “all-encompassing,” but overlook that what clients truly need may just be core functionalities. We can change our thinking—not optimizing algorithms, but optimizing requirements. Sometimes, doing subtraction is more meaningful than doing addition.
This technology ultimately became a value-added feature of hyper-converged Ethernet products and was recognized as a potential high-value patent in data communication. More gratifyingly, this technology helped us achieve expansion sales of the NoF+ network for bank clients, and clients even hoped to migrate more traditional networks, such as NAS (Network Attached Storage), to the NoF+ network.
Watching Over the Spring of the “AI Intelligent Computing Network”
With the rise of AI, data center traffic has undergone tremendous changes. Unlike traditional DCN traffic, AI traffic congestion at a single point can amplify across the entire network, even damaging multiple innocent AI training tasks, like a domino effect that spirals out of control.
Once, while walking in the park after lunch with a colleague, they mentioned that during a technical sharing session with a certain internet company, they heard the vendor discuss the congestion diffusion problem encountered during the deployment of multiple AI tasks, which severely affected business performance.
Isn’t this the problem we studied years ago? The speaker was unaware, but I was intrigued. I stopped in my tracks, a glimmer of excitement in my eyes: “The remote precise flow control technology we had previously embedded might just solve this pain point?”
Five or six years ago, during a technical discussion with my team, I proposed the concept of remote precise flow control. We aimed to monitor traffic in real-time at the source, predict congestion risks, and proactively adjust traffic distribution to avoid congestion. However, at that time, network bandwidth resources were relatively abundant, and other technologies could barely cope with congestion issues, and the impact of congestion was relatively minor, making remote precise flow control seem somewhat “unnecessary.” Ultimately, this technology was only patented and temporarily shelved. But I always believed that future data centers would face more complex traffic problems. I felt like a “watcher,” waiting for the right moment for the technology to be implemented.
What is remembered will surely resonate. My colleague nodded in agreement with my suggestion: “Congestion issues in the AI era are unavoidable; this technology might become a key solution.”
This serendipitous opportunity led us to decide to re-simulate and optimize the remote precise flow control technology, giving it new life.
However, the process of implementing the technology was not easy; there were too many problems to solve at every step.
I still remember that the chip would introduce congestion, and the perception latency issue “stuck” us in our progress. My colleague Song Hexiang and I were at a loss. “How about we fly to Nanjing? There are many colleagues there who understand chips and switch products; we can brainstorm with them.” We hit it off and decided to go.
That night, in a conference room at the N4 building of the Nanjing Research Institute, we repeatedly simulated the chip congestion control process with product SE Liu Quan and platform development colleagues on the electronic screen. Each time we seemed on the verge of breaking through the darkness, we lost our way in the fog, unable to escape the predicament.
As the clock approached midnight, looking at the tired and somewhat dejected faces, I could clearly feel that everyone’s initial enthusiasm was gradually being worn down, and we were prepared to continue discussions and postpone our return date.
Just as we were about to adjourn, someone suddenly proposed a bold idea: “Could we predict congestion based on the trend of cache occupancy size in the switch?”
This was akin to a navigation system displaying whether a lane is congested, with a gradient process from green to yellow to red and finally deep red, assisting drivers in predicting congestion trends. In other words, based on cache occupancy size, we could predict the level of network congestion; the larger the cache occupancy, the more congested the network. We had an epiphany, immediately revitalizing our spirits, and began discussing this idea. Ultimately, by establishing a mathematical model relating congestion feedback time to cache occupancy growth, we found a fallback solution that could predict congestion based on changes in cache over different time windows, avoiding the accuracy control issues introduced by perception latency.
With this idea, after multiple optimizations, our algorithm was finally able to accurately predict congestion and quickly suppress traffic from the source, preventing exacerbation of congestion. Ultimately, this technology was successfully implemented in switch versions and chips.
More fortunately, this technology received great recognition in joint innovation with client S in Singapore and played a significant role in the overseas project’s first integration with NVIDIA GPUs. With the rapid development of AI intelligent computing business, I believe this technology can continue to play a role and open up larger market space.
Introducing the solution at the overseas S client intelligent computing joint innovation sharing session (the author is in the upper right)
In many nights when I couldn’t come up with a solution, I often recalled the moments at the billiard table: focusing on seemingly impossible obstacles, calmly analyzing, finding breakthroughs, and then winning with a single shot. Just like the path of technology research, it has never been smooth sailing; the meaning of life is not in pursuing perfection but in continuously breaking through oneself and finding one’s rhythm in passion.
Source: “Huawei People”
Please indicate the author and source when reprinting