Injecting Thousands of Faults to Safeguard City Data Security

“A large number of alarms appeared in the government external network area of a certain city’s cloud platform”

“Confirming the affected cloud services”

“Quickly set up the war room”

“Check if the customer’s business is affected”

…….

Accompanied by a piercing alarm sound

The previously quiet Huawei Cloud Operations and Maintenance Monitoring Center

Suddenly became busy, and it was already early morning

For Huawei’s government cloud operations and maintenance personnel, this is just a routine part of their job, requiring 24/7 online presence to respond to various emergencies that may occur. However, the subsequent events were not so simple; a surprise attack in the cloud was quietly happening.

This was a surprise drill conducted on the Huawei government cloud. Through this surprise drill, the overall rapid recovery capability of the government cloud SRE team can be honed, verifying the high availability architecture of the entire product and achieving automatic recovery based on the cloud-native capabilities of the product.

Today, the development of digital cities is in full swing, with more and more national and livelihood-related businesses gradually “migrating to the cloud”. Huawei Cloud supports the government services and applications of over 150 cities across the country, making it crucial to ensure the secure and stable operation of these city government cloud platforms and services.

If a system failure occurs, it not only faces various data loss risks but may also lead to issues such as traffic system collapse, hindering travel, and paralysis of government service systems, forcing business operations to stop, affecting all aspects of people’s lives.

Huawei Cloud first reduces the probability of faults through high-quality product development, rigorous operation and maintenance processes, and systems. While continuously challenging zero faults, it employs certain technical means to control the intervals, impact scope, and fault recovery time, transforming the “uncertainty” brought by cloud computing into “certainty” through operation and maintenance.

Creating a “Resident + Operation and Maintenance Center” characteristic operation and maintenance model, with on-site operation and maintenance teams stationed locally with dedicated technical service managers providing personalized services; the remote operation and maintenance center builds a unified knowledge base, efficiently reusing original factory operation and maintenance experts and R&D support; guarding the safety of digital cities 24/7 while providing over 300 specialized services covering the entire lifecycle of business in the cloud.

Cloud operations and maintenance can demonstrate value in practice, supported by processes, tools, and operational mechanisms. Huawei’s government cloud SRE continuously discovers problems and accumulates experience based on several practical drills, already possessing strong evolutionary and iterative capabilities. To date, a complete set of integrated operation and maintenance services tailored for government cloud customers has been established to ensure the continuous, stable, and reliable operation of cloud applications.