Source: Hardware-Software Integration

Original Author: Chaobowx

Editor’s Note

The concept of computing power networks is gradually gaining traction. The vision of computing power networks is to make computing power ubiquitous and easily accessible. This vision is indeed very appealing.

In this article, we introduce two concepts: complex systems and complex computing. A complex system refers to a macro system formed by the integration of multiple systems; complex computing is the computational paradigm of complex systems.

1 Starting from Macroscopic Computing Power

What is performance? What is computing power? These two concepts are unified; performance is a microscopic concept while computing power is a macroscopic concept.

The relationship between performance and computing power can be simplified into the following formula: Total Computing Power = Chip Performance x Number of Chips x Utilization Rate.

These three parameters correspond to the three levels of computing power optimization: microscopic, mesoscopic, and macroscopic:

Microscopic level, which refers to the performance of a single chip, is primarily improved through advancements in technology, Chiplet packaging, and innovations in architecture and micro-architecture.
Mesoscopic level, the chip must support large-scale deployment. Here’s a counterexample: due to the numerous and rapidly changing AI algorithms, AI chips face difficulties in deployment, making large-scale mass production challenging. Chips that cannot be mass-produced have little significance for improving macroscopic computing power.
Finally, the macroscopic level, is the utilization rate of computing power. Even with many chips, if they are isolated, some systems may lack sufficient performance while most systems waste significant computing power, thus failing to fully utilize these computing resources. Statistics show that in cloud computing, the utilization rate of computing power is usually around 6%. If we can increase the utilization rate of computing resources to over 90%, it would create tremendous value. To enhance utilization, extensive work must be done at both the chip level and the macroscopic level.

The most direct and significant impact on macroscopic computing power is the utilization rate. It is necessary to connect all computing resources distributed across the cloud network edge into a vast resource pool for unified scheduling.

2 From Virtualization to Resource Pooling

According to virtualization levels, virtualization is divided into computer virtualization, operating system virtualization, and function virtualization. The common value of these three types of virtualization is:

Virtualization segments and combines resources based on certain time or space granularity;
Virtualization masks architectural/interface differences, providing consistent hardware/software for upper-layer software;
Virtualization offers a platform for upper-layer software systems to run with various combinations of lower-layer resources;
Upper-layer software systems are decoupled from lower-layer hardware/software systems, allowing the upper-layer software system to create/destroy, run/suspend, copy, and migrate;
Multi-system isolation/coexistence: while sharing resources, data isolation, performance isolation, fault isolation, and security isolation are maintained;
Enhancing system flexibility, resource utilization, hardware load balancing, and software high availability.

For example, assume there are 100 servers, and one physical server virtualizes 10 VMs, resulting in 1000 logical (or virtual) VMs belonging to 50 different sizes of private clusters (via VPC).

The dynamic coexistence of multiple clusters and systems is reflected in:

Hardware clusters: a collection of hardware devices for system scheduling, which can range from several to thousands, even millions of units;
Software multi-systems: multiple different specifications of software systems coexist on a single hardware through virtualization mechanisms;
Software multi-system clusters: a group of software systems form a software cluster, with multiple software clusters mixed and cross-deployed on a group of hardware clusters;
Dynamics: from a macroscopic perspective, the configurations of these hardware clusters and software clusters are frequently changing.

Many acceleration chips focus on specific domains: they only consider local aspects without taking into account the global picture.

Data center hardware is pre-configured, and when purchased, it is uncertain what software will run; thus, prioritizing sufficiently general and comprehensive hardware is essential.

Moreover, from the operational management perspective of cloud computing companies, it is ideal to minimize the variety of hardware models, ideally having uniform hardware specifications with only one model of hardware, while achieving differentiation of the “software running platform” through virtualization mechanisms.

From virtualization to resource pooling:

Virtualization is the foundation of pooling: virtualization focuses on individual hardware, while pooling focuses on the macroscopic whole;
Virtualization: segments resources into suitable granularity, then schedules resources through the creation and migration of virtualization instances;
The microscopic mechanism of resource pooling is virtualization, which achieves unified management, usage, and recycling of virtualized resources through the cloud operating system stack, even across cloud network edges;
Microscopic virtualization achieves high availability of the software running platform, while macroscopic resource pooling achieves high utilization of hardware resources;
Poolable (explicitly visible) underlying hardware resources include CPU, memory, GPU/DSA accelerators, storage, etc.

3 Macroscopic Characteristics of Complex Systems

First, let’s understand what macroscopic characteristics the systems targeted by complex computing possess?

First, it is unclear what the system is supposed to do. Traditionally, when designing chips and systems, we usually aim to understand the scenarios and then design our chips and systems based on the needs of those scenarios. The current challenge is that the needs of the scenarios are completely uncertain; not only do chip companies not understand, but customers themselves are also “unaware.” In the future, we need to aim at “nothing to aim at.”
Second, due to the uncertainty of what the system needs, the system must be all-encompassing, capable of performing anything.
Third, the system must be sufficiently professional and efficient in performing any task. We often say, “Professionals do professional things.” This implies that professionals can only perform tasks in their own field, while generalists may not be efficient in every area. Therefore, complex computing systems require both general and specialized capabilities (capable of doing anything efficiently).
Fourth, the system must be able to perform thousands of different tasks across various fields and scenarios simultaneously.
Fifth, the computing power and resources provided by the system must be ubiquitous and readily available. They should be accessible at the most needed places and times, appearing in the most suitable forms and ways; additionally, they should create more value for users and provide a better experience.
Lastly, a key point is that the system must continuously evolve to adapt to the rapid changes in user demands.

Of course, this does not imply that the capability of a single chip can support such a powerful system. Rather, it requires the collaborative or even integrated capabilities of thousands or even millions of individual chips to jointly support the stronger capabilities of a macro system.

4 Definition of Complex Computing

The definition of complex computing: ① based on a set of hardware clusters, ② running multiple system clusters, ③ dynamic, ④ cross-mixed computing. To elaborate:

A single hardware supports computations of multiple different specifications of systems;
A single hardware cluster supports computations of multiple system clusters, and system clusters are cross-mixed;
A scale of tens of thousands or even millions of computing devices, with completely dynamic and very frequent changes in hardware and software configurations;
Hardware needs to have sufficient consistency (as few models and specifications as possible), achieving differentiation of the software platform based on consistent hardware;
To meet the needs of all scenarios, a sufficiently general and comprehensive computing platform and system is required.

5 Scenarios of Complex Computing

5.1 From Cloud Computing to Cloud Network Edge

Friends in the cloud computing industry, upon seeing the concepts of complex systems and complex computing, will surely say, isn’t this just cloud computing? Indeed, complex computing is derived from the foundational characteristics of cloud computing.

These foundational characteristics of cloud computing also exist in scenarios such as edge computing, software-defined networking, and super terminal computing.

We attempt to summarize these characteristics, distilling them into the concept of complex computing, using this concept:

From the individual perspective, it guides the functional definition of the underlying chips and the design of the system architecture;
From the macroscopic perspective, it guides the overall coordination of macroscopic computing power resources and other related resources, providing capability support for global resource pooling and orchestration, further enhancing the utilization of macroscopic computing power.

5.2 Cloud Computing Scenario

Cloud computing primarily consists of a layered service system composed of IaaS, PaaS, and SaaS. Various XaaS services in cloud computing essentially represent the process of the system stack being gradually taken over by cloud operators. Users only need to focus on their core applications/functions.

5.3 Edge Computing Scenario

CDN (Content Delivery Network) is a service that uses servers closest to users to deliver music, images, videos, applications, and other files to users faster and more reliably, providing high-performance, scalability, and low-cost network content delivery services.

Edge computing and CDN share many similarities, both modifying DNS to provide a caching mechanism that is transparent to clients.

The essential difference between CDN and edge computing is:

CDN operates in read-only mode, whether the server pushes static or dynamic content;
Edge computing also needs to support multi-tenant and multi-system operations, with its system stack having certain similarities (reusability) with the cloud.

5.4 Super Terminal Scenario

On September 21, 2022, during the NVIDIA GTC 2022 Fall Conference, CEO Jensen Huang announced the self-driving chip set to be released in 2024. Due to its powerful performance of 2000 TFLOPS, NVIDIA has renamed it Thor, replacing the previous 1000 TOPS Altan.

The Thor SoC can achieve multi-domain computing, allowing it to partition tasks for autonomous driving and in-vehicle entertainment. Typically, these various types of functions are controlled by dozens of control units distributed throughout the vehicle. Manufacturers can leverage Thor to integrate all functions into one, rather than relying on these distributed ECUs/DCUs.

The biggest difference between super terminals and traditional terminals is: support for virtualization, support for multi-system operations, and support for microservices. Traditional APs like mobile phones, tablets, and personal computers run a single system: deploying an OS and running various applications, with software existing in conjunction with hardware. In contrast, super terminals such as autonomous vehicles require virtualization to partition hardware into different specifications, enabling multiple systems to run in different forms, with isolation among systems in aspects like environment, application, data, performance, fault, and security.

Autonomous vehicles typically need to support five main functional domains, including: power domain, body domain, autonomous driving domain, chassis domain, and infotainment domain, with each domain occupying one or more VMs.

5.5 In the Future, More Scenarios Will Require Complex Computing

Chip technology is becoming increasingly advanced, supporting larger system scales; upper-layer software applications are emerging rapidly, with existing applications continuously evolving. Systems are transitioning from single systems to complex systems that are mixed or even integrated.

As systems become more complex, the hardware supporting system computation is also becoming more complex; the more complex systems exist, the more scenarios will require complex computing coverage.

6 Challenges of Complex Computing

The underlying computing resources mainly include CPU, memory, network, and storage I/O, as well as accelerators like GPU and DSA. The core challenge of complex computing lies in how to gather a variety of resources with inconsistent architectures/interfaces into a pool.

Individual hardware must support excellent scalability. Individual hardware includes various heterogeneous processing resources that can form small resource pools; additionally, it must support tens of thousands of individual resources to connect and form larger resource pools.

Individual hardware needs to support system connectivity and integration, categorized into four stages based on the level of integration:

Stage one: Islands. All devices work independently;
Stage two: Interconnected. Devices are connected, allowing communication between them;
Stage three: Collaborative. C/S architecture is a typical collaboration; with collaboration, we have cloud network edges.
Stage four: Integrated. Collaboration is usually static; over time, the initial task division may not adapt to system development; integration represents dynamic and more adaptability; collaboration signifies multiple systems working together, while integration represents multiple systems merging into a larger system.

From the perspective of a macroscopic system, cloud servers, edge servers, terminal devices, and network devices are all consistent hardware. Through software orchestration, we select the most optimal resources to form the most suitable logical platform for software operation.

Computing power chips are like droplets, while computing power networks are like oceans. We must consider how to design this droplet better to integrate into this ocean, making the ocean more vast and magnificent.

(End of article)

Reprinted content only represents the author’s views

It does not represent the position of the Institute of Semiconductors, Chinese Academy of Sciences

Editor: Lu Dingci