The PCIe protocol is getting more complex as it matures, and with 6.0 still in the early days of adoption, interoperability is paramount for advanced AI data centers looking to deployment.

Broadcom Inc. recently introduced its end-to-end PCIe 6.0 portfolio, including its Interop Development Platform (IDP), which simplifies interoperability and system design with its advanced telemetry and diagnostics capabilities. The company has also collaborated with Micron Technology and Teledyne LeCroy to test its high-port switch and retimer for interoperability.

In a briefing with EE Times, Sreeni Bagalkote, product line manager for Broadcom’s data center solutions group, said PCIe 6.0 adoption is being driven largely by AI workloads, and it is the most difficult iteration yet to implement due to the adoption of Pulse Amplitude Modulation 4–level (PAM4) signaling. “You can’t use the standard packets anymore,” Bagalkote said.

Rather, PCie 6.0 uses fixed length packets, which also means standard error correction no longer works, Bagalkote added. “You have something called forward error correction that’s borrowed from [the] Ethernet world. It’s hugely complicated.”

Further adding to the complexity is the introduction of fabrics, he said, and Broadcom’s latest offerings, including retimers, enablement platforms and IDP, reflect the significant changes in PCIe 6.0.

Bagalkote said Broadcom was recently able to demonstrate with Micron the ability to run full speed traffic with 6.0 on the company’s SSDs, and that some customers have already received Broadcom’s boards and IDP to evaluate PCIe 6.0 devices.

Having been around more than 25 years, PCIe in the most successful interconnect, Bagalkote noted, having become foundational for other protocols like NVMe and the CXL. He said the evolution it has undergone as it moves into 7.0 has led to more complexity and, given its role in modern servers in data center, PCIe is the most important connectivity standard.

Bagalkote said many of its hyperscale customers are using PCIe as a hub for any telemetry diagnostics because it is a foundational layer, which is why Broadcom has analyzers built into its switches. This capability has become increasing important in the AI era with many GPUs and accelerators requiring their own NIC along with connectivity to storage, he said. “You can visualize how complicated this becomes compared to non-AI traditional compute servers.”

AI servers are expected to drive mainstream adoption of PCIe 6.0 in 2026, Bagalkote said, followed by mainstream compute.

Astera Labs, meanwhile, is preparing for PCIe 6.0 adoption with the recent expansion of its Cloud-Scale Interop Lab powered by its Scorpio Smart Fabric switches.

In a briefing with EE Times, Thad Omura, Astera’s chief business officer, said the expansion comes on the heels of the company introducing its PCIe6 “smart fabric” switches in October. Astera’s lab enables rigorous interoperability and performance testing of the Scorpio switches with a wide range of PCIe 6.x exercisers, analyzers, GPUs, CPUs, NICs, SSDs and switches across multiple PCIe generations and topologies. This allows customers to design with confidence, minimize interoperation risk and reduce overall development time and costs.

Omura said PCIe interoperability boils down to two levels: the communications level, the equivalent of “hearing,” and the more important protocol level, which is understanding. Astera’s lab is outfitted to conduct linkup tests with a broad range of devices, he said. “You want to make sure the two devices can hear each other.”

Astera’s connectivity system management and optimization software ensure the millions of links in the data center and the protocol are transferring along the whole network optimally, Omura said. “You want to be able to monitor it real time.”

If one link goes down it can affect thousands of GPUs working together, he said. “The whole workload gets impacted.” This speaks to increased complexity of PCIe. “It is becoming much more challenging to interoperate with PCIe 6 devices.”

Like Broadcom, Astera sees state-of-the-art AI platforms driving the need for PCIe 6, Omura added. “There are many active designs in process today. The GPUs are getting more and more powerful every single generation and can process data at a faster pace.”

These GPUs must be supported with sufficient bandwidth, he said, otherwise the most expensive component in the system is getting underutilized. “This is why all of AI platform designers are all trying to keep the GPUs as busy as possible with the maximum uptime, maximum utilization, and maximum data performance.”

PCI-SIG announced it was committed to releasing PCIe 7.0 in 2025 during the PCI–SIG Developers Conference, with the aim of doubling the data rate to 128 GT/s and up to 512GB/s bi–directionally via x16 configuration. It is now in the review phase and should be released later this year.

By: DocMemory
Copyright © 2023 CST, Inc. All Rights Reserved