Huawei R&D: The First Milestone – A Series of Unfortunate Events

Huawei R&D: The First Milestone - A Series of Unfortunate Events

In 1993, Huawei’s C&C08 project hardware team leader Chen Huirong was accustomed to waking up at two o’clock in the afternoon, and for several months, everyone returned to their homes almost at dawn. Whether in the hardware group or the software group, there was a group of night owls; the night was their best working time, as no one interrupted their thoughts, and no one competed for the analyzers and oscilloscopes. By May 1993, the prototype of the C&C08 digital program-controlled switch had been released, but its performance was very unstable. During basic testing, it behaved like a spoiled child, often leaving people puzzled. “There must be some reason hidden deep within!” Zhang Hongren, Chen Huirong’s partner, connected a device called a logic analyzer to the switch bus. The logic analyzer acted like a camera taking continuous pictures, quickly recording the information traveling along the bus over time for analysis, making it one of the most effective tools for hardware engineers. When the analyzer’s plug was connected to the bus, the entire system seemed to operate normally. Chen Huirong bet that he had found the solution; he soldered a few wires onto the bus, but Zhang Hongren firmly believed this would not work.

Huawei R&D: The First Milestone - A Series of Unfortunate Events

By dawn, the machine with a few wires seemed to be running normally, and Chen Huirong won the bet. For the next few days, everyone’s morning tea was paid for by Zhang Hongren. On the other side of the R&D department, the hardware design work for the multi-door machine was also in full swing. At that time, the design of the optical transmission analog circuit and code pattern circuit had just been completed, and no one was sure whether the entire high-speed 32Mb/s system could technically be realized. The hardware head Yu Houlin and several engineers were fiddling with programmable devices and oscilloscopes, trying various methods to eliminate interference on the high-speed clock and signal lines. The initial results were very frustrating: the entire C&C08 switch motherboard acted like a giant antenna, introducing various sources of interference, making it impossible for high-speed signals to be effectively regenerated. Yu Houlin pondered for weeks without coming up with an effective solution. In the control structure of the C&C08 switch, Li Yinan’s initial multi-door machine plan considered adopting a bus similar to those of Shanghai Bell and Fujitsu of Japan. At that time, the fastest publicly available bus standard was Intel’s Multibus II, so Li Yinan decided to implement the multi-door machine using Multibus. Since the initial plan used Intel’s bus products, Huawei’s R&D department ordered nearly $200,000 (equivalent to 2 million RMB at that time) worth of development boards and tools for the first time. To save time, Huawei allowed the R&D department to place the order without restrictions. However, after further consultation and research over the next few months, everyone believed that using this product was inappropriate: Huawei simply did not have the technical capability to implement such a fast bus. The $200,000 worth of development boards and tools were ordered in vain!

Huawei R&D: The First Milestone - A Series of Unfortunate Events

In mid-1993, Huawei’s financial situation was very tight, and many urgently needed components could not be purchased immediately due to a lack of funds. It was impossible to spend so much money on a batch of development boards and tools that were no longer very useful. Every day at work, the young Li Yinan felt nervous when the phone rang; the product had not yet succeeded, but it had already put the company in debt, and he felt a heavy burden on his shoulders. Later, with Zheng Baoyong’s efforts, they only compensated the supplier 200,000 RMB, saving the company a lot of losses. All levels of management at Huawei were well aware of the immense pressure the project team was under, and everyone encouraged the project team in various situations, without any blame or punishment for Li Yinan and the engineers who ordered the wrong development boards and tools. This spirit of taking risks has always been a part of Huawei’s development history. Even if some projects were not very successful, Huawei would continue to provide support in all aspects, never judging heroes by immediate success or failure. Huawei provided engineers with a great development platform, aiming to create practical products. During the development of the C&C08 switch, even though Huawei was so poor that it could not pay salaries, it spent a lot of money on product development. They had the latest development tools, including logic analyzers, digital oscilloscopes, and analog call generators, worth hundreds of thousands of RMB. What impressed the engineers the most was the development of circuit boards. When Liu Ping was working at Shanghai Jiao Tong University, he developed circuit boards and, to save money, would first build a test board using a breadboard after completing the circuit diagram, and only after testing would he order the actual board. But at Huawei, to speed up the process, once the circuit diagram was designed, it was immediately sent to Hong Kong for expedited production at double the price, and samples would be returned in a week. After debugging and modification, they would immediately order a second board. It was this rapid concentration of all efforts on a single product development that led to the success of the C&C08. According to the plan, the C&C08 2000-port module should have been installed on-site a month earlier, but at that time, the system of the C&C08 2000-port machine was still unstable, and the on-site installation was delayed repeatedly. The basic tests of the C&C08 multi-door machine system had not passed, and all the engineers in the development department were worried. All project managers in the development department assured Ren Zhengfei that they were confident in completing the task, and Ren Zhengfei still confidently introduced the latest technology of the digital machine in front of customers, but from the tired look on his face, it was possible to observe that he had also doubted whether this group of young people in the development department could succeed.

Huawei R&D: The First Milestone - A Series of Unfortunate Events

Entering Pizhou City: Rapid Breakthrough in the Urban Market

The first trial station for the C&C08 multi-door machine was selected in Pizhou, Jiangsu. This was Huawei’s first entry into the urban market, installed in a county-level telecom room. In the early 1990s, China’s telecom network was very underdeveloped, and only powerful individuals could install telephones at home. To develop the telecom network, the state provided a preferential policy to telecom departments, allowing them to charge users applying for telephone installation a fee of 5000 RMB. In this way, the telecom bureau took the users’ money and purchased switches from manufacturers at a price of one or two thousand RMB per line, while the cost of the switches was only three to four hundred RMB per line. Therefore, there was a large profit margin for both telecom operators and equipment manufacturers. This was the fertile soil for Huawei’s rapid development.

There is a significant gap between urban and rural areas in China; urban residents are wealthy, but the urban switch market has long been occupied by foreign models with “seven countries and eight systems.” Unless a new model has technical advantages over foreign models, no telecom bureau would be willing to give it a try. For the equipment department of the telecom bureau, buying foreign models is a safe choice, while purchasing domestically produced models with unstable technology could lead to significant issues, as a single switch bureau in the city covers a wide area, and any problems could have a large impact.

Huawei R&D: The First Milestone - A Series of Unfortunate Events

Huawei’s strategy in the switch market was to “surround the city from the countryside.” The rural market in China is vast, with high demands for after-sales service networks, an area where foreign manufacturers are weak, and the technical requirements are relatively lower than in cities. Therefore, Huawei started from the rural market based on its own technology and market conditions, quickly establishing a foothold in the rural market with the C&C08 2000-port switch. However, Huawei was not satisfied with just the rural market and soon used the advanced technology of the C&C08 multi-door machine to attack the urban market. The Pizhou telecom bureau had previously purchased a batch of Shanghai Bell’s S1240 switches, but due to the high demand for telephone installations, the capacity quickly became insufficient. When it came time to expand, they naturally wanted to buy more Shanghai Bell switches, but at that time, the orders for Shanghai Bell switches were already scheduled for the following year. They could not wait any longer, and with the efforts of Huawei’s Nanjing office, they finally decided to give Huawei’s switches a try. At that time, Xu Xubo, the director of Huawei’s Nanjing office, and Yang Jun, the salesperson responsible for the Pizhou project, seized this once-in-a-lifetime opportunity, allowing the newly born C&C08 multi-door machine to have its first trial. Thus, Huawei’s C&C08 multi-door machine was installed in the same machine room as Shanghai Bell’s switches. When the C&C08 multi-door machine was installed in the machine room, Huawei’s engineers felt ashamed compared to Shanghai Bell’s 1240 machines. Huawei’s switch looked so “rustic”: the cabinet was short and small, the color was dull gray, the rack rails were soft, and the circuit boards could not be easily inserted or removed. Just in terms of appearance, it lost to Shanghai Bell, and in terms of functionality, it was even more inferior.

Huawei R&D: The First Milestone - A Series of Unfortunate Events

However, being able to be placed in the same machine room as Shanghai Bell’s foreign machines was already an impressive achievement. At that time, the C&C08 multi-door machine could ensure stable phone calls, and everyone was grateful for that, as many new services could not be realized. After the machine began debugging, a major problem arose: it could not connect to the upper bureau in Xuzhou, and long-distance calls could not be made. Initially, everyone suspected that the relay board was faulty, and a new relay board was sent from Huawei, but it did not solve the problem. Then they suspected that the relay line was faulty, and someone was sent to Xuzhou to buy a new relay line, but that did not work either; Huawei sent people to bring a new relay line to test, but still no success. Huawei sent wave after wave of hardware developers, but the problem remained unresolved. More than a week passed with no progress, and everyone was a bit desperate. One day, Li Yinan, the person in charge of the C&C08 multi-door machine, said heavily to Liu Ping, “I may not be able to continue; you take over from here.” But fortune favors the prepared mind; by chance, hardware head Yu Houlin discovered that the grounding of the switch was not done properly. After fixing the ground wire, the problem was easily resolved; it was a minor issue. The bigger problem lay in the software. Due to the urgency of the telecom bureau to release the lines, as they had already collected money from users, the switch was connected to the lines before detailed testing was completed. However, the switch still had many issues, and everyone had to sleep during the day and debug at night when not many people were making calls to resolve the remaining problems. The biggest issue was that the time slots for phone calls were occupied and not released. In a program-controlled switch, one call occupies one time slot. When a user dials a number, a time slot is requested, and when they hang up, the time slot is released. Due to an error in program processing, sometimes the time slot would not be released. This problem did not appear during simulation debugging but emerged in the real environment, and it was difficult to track and resolve.

Huawei R&D: The First Milestone - A Series of Unfortunate Events

After tracking the issue for more than a week, everyone was ready to give up. Considering that if this problem accumulated, all the switch time slot resources would be used up, the entire switch would be paralyzed. Therefore, everyone adopted a temporary solution: they set the software to restart every night at 2 a.m. to clear and release all time slot resources, which they called “midnight crowing.” If a user happened to be on a call at that time, they would be out of luck, as the call would suddenly drop. In the C&C08 switch, “midnight crowing” lasted for more than half a year and was only resolved after multiple version upgrades. The engineers arrived in Pizhou in August 1994, originally hoping to return to Shenzhen for the National Day holiday. Unexpectedly, they did not finish until mid-October. During the final acceptance, Ren Zhengfei and Zheng Baoyong both came. In the evening at the engineers’ residence, Ren Zhengfei chatted with the engineers, and when the conversation became lively, he passionately said, “In ten years, to stand alongside AT&T and Alcatel, Huawei will occupy one-third of the world!” At that time, everyone burst into laughter, thinking, “The boss really knows how to boast.” It should be noted that at that time, everyone had to look up high to see Shanghai Bell, the leader in Chinese telecom equipment; and AT&T was the world-class leader in telecom equipment, and no matter how high they looked, they could not see its shadow. Ten years later, Huawei had entered the top five global suppliers of fixed telecom network equipment.

Today, Huawei’s development has proven that Ren Zhengfei was not boasting back then; those with ambition can achieve their goals! Meanwhile, AT&T, which spun off Lucent in 1995, merged with Alcatel ten years later due to competitive pressure from Huawei. History has witnessed this astonishing turnaround! A year later, when Liu Ping returned to Pizhou, the first cabinet of the C&C08 multi-door machine had already been placed at the entrance of the machine room as a shoe cabinet. Huawei replaced it with a new cabinet, which resembled Shanghai Bell’s switches.

Free reading of《Huawei R&D》, with each issue’s continuation uploaded, welcome to subscribe!

Huawei R&D: The First Milestone - A Series of Unfortunate Events

Leave a Comment