Previously, we wrote an article about the NVL72 interconnection scheme, and some users in the community asked for an introduction to the meanings of these different cables in data centers. Let’s review where optical and copper interconnections are used in GB200.
The Abandoned NVL72 Optical Interconnection Scheme
There has been much discussion about the architecture and interconnection of GB200 in the community, and interested readers can join the discussion there.
Optical Cables
Active Optical Cable (AOC)
Definition of AOC:Active Optical Cable (AOC) is a cable technology that accepts the same electrical input as traditional copper cables but uses optical fibers between connectors. AOC employs electrical-to-optical conversion at the cable ends to enhance speed and transmission distance performance while maintaining compatibility with standard electrical interfaces.
We can see that the AOC component actually consists of four functional parts:
“High-density QSFP+ connector”: This is an SFF-8436 electronic connector that can be inserted into routers or switches.
“4-channel full-duplex active optical transceiver”: This optical transceiver is embedded within the housing, making it not directly visible. This transceiver component is responsible for optical-electrical (O-E) and electrical-optical (E-O) conversion.
MPO optical connector (black part): This connector is permanently connected to the housing and optical fiber. This permanent connection protects the optical interface from user contact and environmental contaminants.
Ribbon optical cable (the image shows a yellow jacket single-mode fiber, but multi-mode fiber versions are also available).
Of course, AOC also comes in many varieties:
10G SFP AOC, 25G SFP28 AOC, 40G QSFP+ AOC, 100G QSFP28 AOC, 200G QSFP56 AOC, and other different rates.
Here, let’s briefly explain SFP and QSFP: SFP stands for Small Form-Factor Pluggable, which is a small pluggable module, while QSFP is a quad-channel small form-factor pluggable module. For example, 40G QSFP+ consists of 4 x 10G channels, and 100G QSFP28 consists of 4 x 25G channels. Whether SFP or QSFP, they are just abbreviations for pluggable modules, and their interfaces can be MPO/MTP optical fiber connectors (parallel optical modules) or copper cables (such as QSFP electrical modules). Later versions include QSFP+ (40G), QSFP28 (100G), QSFP56 (200G/400G), etc.
To briefly explain the difference between QSFP28 and QSFP56: QSFP28 typically supports 4×25Gbps channels, with a total bandwidth of 100Gbps, and can reach a single-channel rate of up to 50Gbps when using PAM4 modulation, achieving a total bandwidth of 200Gbps. QSFP56 supports 4×50Gbps channels, enabling a data transmission rate of 200Gbps, with some supporting up to 400Gbps (4×100Gbps).
The emergence of AOC aims to replace copper cable technology in applications such as data centers and high-performance computing.
The initial driving force came from InfiniBand technology: as the data rate of this technology increased and the scale of data center clusters expanded, copper cable technology gradually reached its limits. For example, the 20 Gbps InfiniBand DDR technology limited traditional copper cable transmission distances to 8-10 meters, posing challenges for the physical layout of large clusters. Additionally, copper cables are bulky, difficult to manage, and susceptible to electromagnetic interference (EMI), which affects performance and reliability, especially in large-scale high-performance clusters.
Therefore, Intel and Luxtera invented AOC components to fill this gap. Intel’s design uses vertical cavity surface-emitting lasers (VCSELs) and a series of discrete components, while Luxtera’s “Blazar” series AOC products utilize CMOS photonics technology, integrating most transceiver functions onto a silicon chip, with only the laser and photodetector being discrete components.
AOC components have many advantages: if the network device is designed properly, they can be compatible with passive or active copper cables through universal electrical ports, allowing users to choose technology and cost according to their needs and easily reconfigure; by permanently connecting optical fibers with optical devices, they eliminate the need for expensive optical connectors with strict manufacturing tolerances (especially for parallel optical links). Their main advantages also include: longer transmission distances, higher bandwidth; secure and reliable transmission; minimal electromagnetic interference/radio frequency interference; low bit error rate (up to 10⁻¹⁵); and smaller size and lighter weight compared to copper cables.
Active optical cable components are designed to support multiple protocols, most of which are compatible with SFP+ Ethernet and InfiniBand electrical interfaces. Taking a typical 40 Gb/s QSFP+AOC as an example, the supported content is as follows:
- Multi-rate: 1.0 Gb/s – 10.3125 Gb/s (per channel)
- 4-channel full-duplex active optical transceiver
- InfiniBand SDR (2.5Gb/s), DDR (5 Gb/s), QDR (10 Gb/s)
- Ethernet 10G, 40G
- Fibre Channel 8G, 10G
- SAS, SATA 3G, 6G
- Fibre Channel Storage Area Network (SAN) 10G, 20G, 40G
- Myrinet 40G
Disadvantages of AOC
1. AOC integrates active components such as optical transceivers, lasers, and photodetectors, and the production process involves precise optical alignment and circuit integration, resulting in significantly higher manufacturing costs compared to traditional passive copper cables or passive optical cables.
2. Limited flexibility, as the optical fiber in AOC is permanently fixed to the optical modules at both ends, making it impossible to flexibly replace or adjust the length like independent optical modules + passive cables. However, regarding length adjustment, I believe it is acceptable; in almost all application scenarios, engineers will reserve enough cable, so we often see these cables coiled rather than being too short.
3. AOC has more components internally, leading to higher power consumption.
Cables
DAC
Direct-Attached Cable (DAC) is a copper cable with connectors on both ends. The connectors typically feature a locking mechanism or other devices to prevent accidental disconnection. They are called “direct-attached” cables because they are designed to connect directly to devices for interconnection. They are also referred to as “twinax” cables because they contain two pairs of twisted pairs, commonly known as “twinax” wire.
DAC cables are divided into two categories: passive DAC and active DAC. Passive DAC cables, also known as PCC, refer to cables that do not have signal conditioning functions to transmit data, making them cheaper than active DAC cables. We will discuss active DAC cables below.Generally, when we refer to DAC, we mean passive cables.
Depending on data rates and connector types, DAC cables typically have the following configurations: 10G SFP+ DAC cables, 25G SFP28 DAC cables, 40G QSFP+ DAC cables, 56G QSFP+ DAC cables, 100G QSFP28 DAC cables, 200G QSFP56 DAC cables, 400G QSFP-DD DAC cables, 40G DAC Breakout cables, 56G DAC Breakout cables, 100G DAC Breakout cables, 200G DAC Breakout cables, 400G DAC Breakout cables, etc.
Advantages of Passive Copper Cables
1. Low cost, as passive copper cables do not contain any active electronic components (such as signal amplifiers, processors, etc.), consisting only of copper wires and connectors, making their manufacturing costs significantly lower than those of active copper cables (Active DAC) and active optical cables (AOC), making them a highly cost-effective choice for short-distance connections.
2. Low power consumption, as passive copper cables do not require electronic components for signal processing or amplification, they consume almost no power, effectively reducing the energy consumption and heat dissipation pressure of devices, especially suitable for power-sensitive data center environments.
3. Low latency, as the passive design avoids signal delays from active components, providing better real-time data transmission, suitable for latency-sensitive short-distance interconnection scenarios (such as connections between servers and switches within a rack).
4. Simple structure and high reliability, with no complex electronic components, reducing failure points, longer lifespan, and stable performance against electromagnetic interference (EMI) over short distances, resulting in low maintenance costs.
Disadvantages of Passive Copper Cables
1. Limited transmission distance, as copper cables exhibit significant signal attenuation, and the passive design cannot amplify or compensate for the signal, thus transmission distances are typically limited to within 7 meters (some high-spec products can reach 10 meters), beyond which signal quality significantly deteriorates, failing to meet long-distance transmission needs.
2. Cables are relatively thick and less flexible; to ensure signal integrity, passive copper cables (especially high-bandwidth models like 100G QSFP28 DAC) have thicker copper wire diameters, making the cables overall stiffer, resulting in poor installation flexibility in narrow spaces, making them difficult to bend or organize.
This is very intuitive in server connections, as DAC is very thick, making installation in such confined spaces very inflexible.
3. Bandwidth limitations, although passive copper cables support various bandwidths (such as 10G, 25G, 100G, etc.), as bandwidth increases, their effective transmission distance further shortens, and high-frequency signals are more susceptible to loss and interference in copper cables, making it difficult to meet ultra-high-speed (such as 400G and above) long-distance transmission needs.
4. Limited anti-interference capability, as compared to optical cables (such as AOC), copper cables are more sensitive to electromagnetic interference (EMI) and radio frequency interference (RFI), and may experience unstable signals in strong electromagnetic environments (such as near power devices or motors).
ACC
If you just look at the appearance, they actually look quite similar.
But internally, they are different; ACC includes a Redriver:
We have previously discussed the differences between retimers and redrivers:
Discussing Retimer and Redriver in Data Centers
The main features of ACC are as follows:
In terms of signal processing, the connector contains active components such as signal amplifiers and equalizers, which can reduce signal attenuation and enhance signal integrity during long-distance transmission.
In terms of transmission distance, passive DAC typically reaches a maximum of 5-7 meters, while ACC can extend to about 10-15 meters, of course, the transmission distance is related to the transmission rate, making it suitable for medium-distance connections in data centers.
In terms of data transmission speed, it is similar to passive DAC, supporting speeds from 10 Gbps to 100 Gbps or even higher, depending on the cable type (such as SFP+, QSFP+, QSFP28, etc.).
In terms of latency, ACC maintains low latency similar to DAC, making it suitable for applications that require both distance and speed.
In terms of power consumption, although it is higher due to integrated electronic components compared to passive DAC, it is generally lower than optical cables (AOC), making it a more energy-efficient choice for medium-distance connections.
In terms of cost and flexibility, it is similar to DAC, more cost-effective than fiber solutions, especially in short to medium-distance scenarios, and more flexible and easier to manage than fiber.
AEC
AEC is where the redriver in ACC is replaced with a retimer; for specific differences, please refer to the previously mentioned article.
Compared to ACC, AEC not only includes re-driving but can also integrate retiming (CDR), DDM diagnostics, and custom signal optimization algorithms, making it suitable for a wider range of scenarios (such as 400G and other high-speed, long-distance requirements). The maximum transmission distance for 10G/25G ACC active copper cables can reach 15 meters, while for 40G/50G/100G/200G ACC active copper cables, the maximum transmission distance can reach 7 meters. Of course, due to the added functionalities in AEC, the cost will also be higher.
Let’s visually compare the use of DAC and AEC cables in a server rack:
Which Cables are Used in GB200?
For the interconnection scheme of GB200, you can also refer to the article we mentioned at the beginning.
Firstly, in the NVL72 scale-up interconnection, according to analysis by Semianalysis, NVIDIA did not choose optical connections but used 5184 copper cables, which is a much cheaper, lower power consumption, and more reliable choice.
Each GPU has a unidirectional bandwidth of 900GB/s. Each differential pair (DP) can transmit 200Gb/s of data in one direction, so each GPU requires 72 differential pairs for bidirectional transmission. Since each NVL72 rack contains 72 GPUs, a total of 5184 differential pairs are needed. Each NVLink cable contains one differential pair, so 5184 cables are required.
Moreover, according to Semianalysis, the interconnections used are all ACC.
In the Backend Networking of GB200, the optical connections are used for GPU-GPU inter-rack communication. Optical connections are used for long-distance connections (such as from compute racks to service racks) because copper cables have distance limitations. SA mentioned that if using a Top of Rack design, DAC/ACC copper cables can save approximately $32k per rack, but most deployments require optical due to power limitations.
For Frontend Networking, optical can also be used, but most customers (such as Amazon/Google/Microsoft) use custom NICs, requiring only 200G bandwidth per tray, and can save approximately $3.5k per system by using copper cables.
END
Currently, NVIDIA’s ConnectX cards are in high demand, and we have domestic alternatives available. Please add the WeChat below and note “Network Card”:
Knowledge Planet
We update industry information, foreign investment bank data, and selected analysis reports daily in the Knowledge Planet, and we will also organize key information into a cloud drive for continuous updates:
