In the preceding article, Dolphin scrutinized the competitive landscape of data center networking, particularly between two GPU titans catering to third-party markets: NVIDIA and AMD.
AMD exerted considerable effort utilizing Broadcom’s switches to enhance scalability from 8 to 72 cards, culminating in the development of Helios.
However, Helios merely managed to match the performance of NVIDIA’s GB300 at the time of shipment, falling short as NVIDIA’s offering evolved into Vera Rubin.
Although Helios equaled the GPU interconnection bandwidth, it faltered in cost efficiency and trailed in engineering innovation.
AMD’s advancements reach merely the threshold of adequacy, provoking a sense of futility akin to “utilizing a bygone dynasty’s sword against the officials of the current regime.”
Dolphin posits that the crux of the issue resides in AMD’s deficiency in comprehensive self-development. What might it entail to escalate self-development to a distinct zenith?
This discourse will pivot towards a preeminent figure in the realm of self-developed and internally utilized 1P ASICs — Google, a behemoth in cloud service provision (CSP). An overview of its networking architecture discloses:
I. An Expansive Scale-Up Domain, nearing 10,000 cards. For context, Vera Rubin and Helios possess only 72 cards;
II. A Singularly Innovative Topology, employing direct connections rather than the dual-mode switching paradigm of workstation areas (compute trays) paired with scheduling rooms (switch trays), constituting an exclusive Optical Circuit Switching (OCS) solution across the entire network;
III. A Comparatively Economical Networking Cost, undeniably a consequence of the former two facets.
Subsequently, Dolphin shall delve deeper into these three defining traits to examine Google’s architecture:
1. What mechanisms facilitate a Scale-Up domain housing nearly 10,000 cards?
2. How did the direct connection topology evolve from version 7 to version 8?
3. How competitive is Google’s solution in light of its transition to a TPU-as-a-Service model?
4. What are the pivotal objectives within the industrial ecosystem?
The ensuing analysis thoroughly elucidates these inquiries.
I. Mechanisms for Realizing a Scale-Up Domain with Nearly 10,000 Cards
We will utilize Ironwood v7, extensively deployed, as a benchmark.
1. Interconnection Protocol: ICI underpins half of the architecture
1) Intra-Pod Scale-Up: Both optical and electrical transmission leverage ICI
Google has meticulously crafted a proprietary inter-chip interconnection protocol known as ICI (Inter-chip Interconnection), benchmarking against NVLink and UALink, which facilitates communication between TPUs.
In its v8 iteration, the cumulative bidirectional bandwidth for a single TPU reaches an impressive 2,400GB/s, doubling from the 1,200GB/s recorded in v7 (as compared to NVIDIA’s 3.6TB/s).
Contrastingly, the frameworks of Vera Rubin and Helios embrace a method wherein “one protocol coincides with one transmission medium, employed within and outside the cabinet,” which may misleadingly appear as a symbiotic relationship.
However, as previously elucidated, the demarcation between Scale-Up and Scale-Out hinges on communication semantics rather than physical configuration.
Google exemplifies this understanding: within a Pod containing a specific number of TPUs, regardless of whether the inter-TPU connections traverse outside the cabinet or utilize optical or electrical transmission, the ICI protocol prevails; only when the scale surpasses this threshold does the system revert to the conventional Data Center Network (DCN) protocol based on Ethernet.
2) Cross-Pod Scale-Out: Jupiter DCN as a Complementary Framework
Prior to version 8, Google refrained from establishing a separate network for Scale-Out. The rationale lies in the substantial ICI domain, wherein tensor parallelism and expert parallelism—both highly sensitive to bandwidth—are predominantly contained within ICI, thus mitigating the demand for cross-Pod communication, which entails comparatively relaxed bandwidth constraints, pegged at a mere 100Gbps for each card.
Consequently, cross-Pod traffic directly utilizes the self-developed general-purpose Jupiter DCN, the universal Ethernet network employed across all Google data centers, simultaneously facilitating traffic for search, advertising, and storage.
This dual role implies that Jupiter DCN accommodates both front-end and back-end communications: the front end integrates storage, CPUs, and external networks, characterized by sporadic traffic with tolerance for latency; conversely, the back end interlinks accelerators, necessitating synchronous coordination.
2. Networking Solution: 3D “Building Blocks” plus Optical Circuit Switching (OCS)
Networking fundamentally concerns the topology — the intricate interconnections between nodes in a network. The networking models delineated in previous Dolphin articles primarily feature the 2D fat-tree networks embraced by numerous data centers.
Google’s topology, however, transcends this paradigm, adopting a 3D structural approach. Its networking hierarchy progresses from the smallest unit to the most expansive: chip → tray → Cube (or Rack) → Pod/Superpod.
In comparison to NVIDIA’s architecture, while the nomenclature of chip and tray layers bears a semblance, the assemblage of trays reveals strikingly different characteristics.
1) Intra-Tray Scale-Up: Physical Configuration 1×4, Topological Logic 2×2
A tray comprises 4 TPUs (as depicted below). Communication among TPUs adheres to the ICI protocol, with each TPU housing 6 ICI ports, enabling an aggregate bidirectional bandwidth of 1,200GB/s. Interaction between TPU and CPU is facilitated through PCIe (DAC).
To appreciate the significance of the 6 ports, one must contextualize them within the larger framework.
Google structures its TPUs into a three-dimensional grid, with 6 ports interfacing with adjacent nodes: above, below, left, right, front, and back (i.e., ±x, ±y, ±z), forming a 3D toroidal arrangement (illustrated below).
Returning to the tray, it is worth noting that the 4 TPUs are sequentially aligned, yet not interconnected in pairs.
Four of the 6 ports are designated for connections to neighboring trays, leaving only 2 ports for intra-board connectivity, thereby permitting each TPU to link with merely 2 chips on the same tray.
The linkage of the 4 ports external to the tray defines the cross-tray topology explored in the subsequent section.
2) Tray-to-Tray Scale-Up: Direct Connection Topology Ensures Connectivity
The most notable distinction from NVIDIA and AMD’s architectures resides in the tray interconnectivity: Google opts not to incorporate switching chips; instead, TPUs are directly interconnected within a 3D framework.
16 trays amalgamate to form a Cube (equivalent to a rack) comprising 64 TPUs, where the chips are arranged into a 4×4×4 3D topology. Each chip is envisaged as a six-sided cuboid, with cabling extending in six directions to secure direct connections with neighboring chips.
Upon examination of the topology in the accompanying figure, you will observe that certain TPUs utilize copper cables, while others employ optical connections. What accounts for this differentiation?
The choice between copper and optical transmission is contingent upon the TPU’s location within the Cube. Within a given Cube constituted by 64 chips, the innermost 8 TPUs achieve complete connectivity with their 6 proximate TPUs through copper (DAC+PCB).
When considering this 64-chip cube as an individual entity, it becomes apparent that some sides of the TPUs positioned at the edges lack nearby adjacent TPUs, necessitating optical connections to TPUs situated on the opposing side of the cube (as illustrated by the green line).
The TPUs on each outer perimeter interconnect sequentially, with signals transformed into optical formats via an optical module, relayed to the OCS switch.
We have meticulously cataloged the pertinent connection devices and parameters of TPUs situated at various positions (center, surface, edge, corner) within the Cube, detailed below.
Optical transmission at the Scale-Up layer represents Google’s innovative design. The entirety of the Scale-Up procedures for Vera Rubin NVL72 and Helios predominantly relies on copper: GPUs interface with NVSwitch/Tomahawk using backplane copper connectors, with optical modules making their appearance solely on the NICs and switches associated with Scale-Out; the GPUs themselves do not engage directly with optical signals.
Google’s optical modules are seamlessly integrated into the OSFP cage on the front panel of the TPU tray. The optical pathway follows this route: TPU → optical module → optical fiber → OCS → optical fiber → optical module → TPU, eschewing any intermediate electrical switching apparatus.
3) Cube-Cube Scale-Up: The Role of OCS
As previously indicated, the optical cable interconnections along the Cube’s external surface converge within the OCS switch.
In essence, within Google’s larger networking framework, the OCS is pivotal in amalgamating multiple Cubes into a more extensive domain: Pod (e.g., 4 Cubes, 256 TPUs) scaling up to Superpod (with an upper limit of 144 Cubes, equating to 9,216 TPUs). This functionality encapsulates Google’s most salient network hardware:
a. OCS switches function exclusively to redirect optical signals. The signal connections on the Cube’s outer surface link to the OCS through an optical module; the OCS determines whether each link’s terminal is looped back to form a 64-chip torus or integrated into a larger toroidal configuration. The OCS is limited to guiding light from one port to another.
b. Mechanism of Optical Signal Switching. The OCS comprises dual sets of 2D MEMS micromirror arrays. By adjusting each mirror along two axes, light beams from any input port are “guided” to any output port (illustrated in the accompanying figure).
The crucial differentiation from Electronic Packet Switch (EPS) solutions employed by AMD and NVIDIA is that EPS makes real-time forwarding determinations for each data packet, while OCS creates static optical paths between ports.
For instance, should port 1 require to transition from port 2 to port 4, reconfiguration of the mirrors becomes imperative, as OCS lacks intrinsic forwarding capabilities; in analytics, traditional EPS ports are fully interconnected, negating the necessity for such adjustments.
A fitting analogy resembles railway switches: numerous tracks can exist, yet only one can be in use concurrently, necessitating a switch alteration (i.e., adjusting the mirror orientation) to shift tracks.
Hence, one could also assert that Google’s routing pathways are pre-configured via software, as the prospect of undertaking several seconds for reconfiguration amidst low-latency workloads with synchronous parallelism is virtually untenable.
c. Absence of O (Optical)-E (Electrical)-O (Optical) conversions in the OCS switch. It is pertinent to note that in conventional EPS, the signal journey for cabinet-external communication follows this pathway: the chip emits an electrical signal, subsequently converted to light (E→O) by a local optical module and transported to the switch via optical fiber.
Upon reaching the switch, which recognizes only electrical signals, the inbound signals are transformed back into electricity (O→E), the switching chip discerns the forwarding route, and the outbound signals revert to light (E→O); upon traversing the optical fiber to the opposing cabinet, the optical module reconverts the signals into electricity (O→E) and inwardly channels them to the chip.
The OCS safeguards against two conversions between optics and electronics within the switch. Notably, it constitutes the sole “switching” device within this architecture, while being exempt from the power demands and expenses associated with switching ASICs. The schematic representation is provided below:
d. Flexible Topology Scaling. In practical implementation, Google’s expansion may not adhere strictly to the conventional Cube (64 TPUs) configuration.
Google possesses the capacity to configure TPUs across various scaling denominations ranging from 4 to 2048, a flexibility bestowed by the OCS switches’ pre-configuration.
4) Superpod-Superpod Scale-Out: OCS’s Aspirational Role as a Replacement for the Spine Layer.
Source link: Eu.36kr.com.
