When the PCI Special Interest Group (PCI-SIG) first announced PCIe 4.0 a few years back, the group made it clear that they were not just going to make up for lost time after PCIe 3.0, but that they were going to accelerate their development schedule to beat their old cadence. Since then the group has launched the final versions of the 4.0 and 5.0 specifications, and now with 5.0 only weeks old, the group is announcing today that they are already hard at work on the next version of the PCIe specification, PCIe 6.0. True to PCIe development iteration, the forthcoming standard will once again double the bandwidth of a PCIe slot – a x16 slot will now be able to hit a staggering 128GB/sec – with the group expecting to finalize the standard in 2021.

As with the PCIe iterations before it, the impetus for PCIe 6.0 is simple: hardware vendors are always in need of more bandwidth, and the PCI-SIG is looking to stay ahead of the curve by providing timely increases in bandwidth. Furthermore in the last few years their efforts have taken on an increased level of importance as well, as other major interconnect standards are building off of PCIe. CCIX, Intel’s CXL, and other interfaces have all extended PCIe, and will in turn benefit from PCIe improvements. So PCIe speed boosts serve as the core of building ever-faster (and more interconnected) systems.

PCIe 6.0, in turn, is easily the most important/most disruptive update to the PCIe standard since PCIe 3.0 almost a decade ago. To be sure, PCIe 6.0 remains backwards compatible with the 5 versions that have preceded it, and PCIe slots aren’t going anywhere. But with PCIe 4.0 & 5.0 already resulting in very tight signal requirements that have resulted in ever shorter trace length limits, simply doubling the transfer rate yet again isn’t necessarily the best way to go. Instead, the PCI-SIG is going to upend the signaling technology entirely, moving from the Non-Return-to-Zero (NRZ) tech used since the beginning, and to Pulse-Amplitude Modulation 4 (PAM4).

At a very high level, what PAM4 does versus NRZ is to take a page from the MLC NAND playbook, and double the number of electrical states a single cell (or in this case, transmission) will hold. Rather than traditional 0/1 high/low signaling, PAM4 uses 4 signal levels, so that a signal can encode for four possible two-bit patterns: 00/01/10/11. This allows PAM4 to carry twice as much data as NRZ without having to double the transmission bandwidth, which for PCIe 6.0 would have resulted in a frequency around 30GHz(!).


NRZ vs. PAM4 (Base Diagram Courtesy Intel)

PAM4 itself is not a new technology, but up until now it’s been the domain of ultra-high-end networking standards like 200G Ethernet, where the amount of space available for more physical channels is even more limited. As a result, the industry already has a few years of experience working with the signaling standard, and with their own bandwidth needs continuing to grow, the PCI-SIG has decided to bring it inside the chassis by basing the next generation of PCIe upon it.

The tradeoff for using PAM4 is of course cost. Even with its greater bandwidth per Hz, PAM4 currently costs more to implement at pretty much every level, from the PHY to the physical layer. Which is why it hasn’t taken the world by storm, and why NRZ continues to be used elsewhere. The sheer mass deployment scale of PCIe will of course help a lot here – economies of scale still count for a lot – but it will be interesting to see where things stand in a few years once PCIe 6.0 is in the middle of ramping up.

Meanwhile, not unlike the MLC NAND in my earlier analogy, because of the additional signal states a PAM4 signal itself is more fragile than a NRZ signal. And this means that along with PAM4, for the first time in PCIe’s history the standard is also getting Forward Error Correction (FEC). Living up to its name, Forward Error Correction is a means of correcting signal errors in a link by supplying a constant stream of error correction data, and it’s already commonly used in situations where data integrity is critical and there’s no time for a retransmission (such as DisplayPort 1.4 w/DSC). While FEC hasn’t been necessary for PCIe until now, PAM4’s fragility is going to change that. The inclusion of FEC shouldn’t make a noticeable difference to end-users, but for the PCI-SIG it’s another design requirement to contend with. In particular, the group needs to make sure that their FEC implementation is low-latency while still being appropriately robust, as PCIe users won’t want a significant increase in PCIe’s latency.

The upshot of the switch to PAM4 then is that by increasing the amount of data transmitted without increasing the frequency, the signal loss requirements won’t go up. PCIe 6.0 will have the same 36dB loss as PCIe 5.0, meaning that while trace lengths aren’t officially defined by the standard, a PCIe 6.0 link should be able to reach just as far as a PCIe 5.0 link. Which, coming from PCIe 5.0, is no doubt a relief to vendors and engineers alike.

Even with these changes, however, as previously mentioned PCIe 6.0 is fully backwards compatible with earlier standards, and this will go for both hosts and peripherals. This means that to a certain extent, hardware designers are essentially going to be implementing PCIe twice: once for NRZ, and again for PAM4. This will be handled at the PHY level, and while it’s not a true doubling of logic (what is NRZ but PAM4 with half as many signal levels?), it does mean that backwards compatibility is a bit more work this time around. Though discussing the matter in today’s press conference, it doesn’t sound like the PCI-SIG is terribly concerned about the challenges there, as PHY designers have proven quite capable (e.g. Ethernet).

PCI Express Bandwidth
(Full Duplex)
Slot Width PCIe 1.0
(2003)
PCIe 2.0
(2007)
PCIe 3.0
(2010)
PCIe 4.0
(2017)
PCIe 5.0
(2019)
PCIe 6.0
(2021)
x1 0.25GB/sec 0.5GB/sec ~1GB/sec ~2GB/sec ~4GB/sec ~8GB/sec
x2 0.5GB/sec 1GB/sec ~2GB/sec ~4GB/sec ~8GB/sec ~16GB/sec
x4 1GB/sec 2GB/sec ~4GB/sec ~8GB/sec ~16GB/sec ~32GB/sec
x8 2GB/sec 4GB/sec ~8GB/sec ~16GB/sec ~32GB/sec ~64GB/sec
x16 4GB/sec 8GB/sec ~16GB/sec ~32GB/sec ~64GB/sec ~128GB/sec

Putting all of this in practical terms then, PCIe 6.0 will be able to reach anywhere between ~8GB/sec for a x1 slot up to ~128GB/sec for a x16 slot (e.g. accelerator/video card). For comparison’s sake, 8GB/sec is as much bandwidth as a PCIe 2.0 x16 slot, so over the last decade and a half, the number of lanes required to deliver that kind of bandwidth has been cut to 1/16th the original amount.

Overall, the PCI-SIG has set a rather aggressive schedule for this standard: the group has already been working on it, and would like to finalize the standard in 2021, two years from now. This would mean that the PCI-SIG will have improved PCIe’s bandwidth by eight-fold in a five-year period, going from PCIe 3.0 and its 8 GT/sec rate in 2016 to 4.0 and 16 GT/sec in 2017, 5.0 and 32 GT/sec in 2019, and finally 6.0 and 64 GT/sec in 2021. Which would be roughly half the time it has taken to get a similar increase going from PCIe 1.0 to 4.0.

As for end users and general availability of PCIe 6.0 products, while the PCI-SIG officially defers to the hardware vendors here, the launch cycles of PCIe 4.0 and 5.0 have been very similar, so PCIe 6.0 will likely follow in those same footsteps. 4.0, which was finalized in 2017, is just now showing up in mass market hardware in 2019, and meanwhile Intel has already committed to PCIe 5.0-capable CPUs in 2021. So we may see PCIe 6.0 hardware as soon as 2023, assuming development stays on track and hardware vendors move just as quickly to implement it as they have on earlier standards. Though for client/consumer use, it bears pointing out that with the rapid development pace for PCIe – and the higher costs that PAM4 will incur – just because the PCI-SIG develops 6.0 it doesn't mean it will show up in client decides any time soon; economics and bandwidth needs will drive that decision.

Speaking of which, as part of today’s press conference the group also gave a quick update on PCIe compliance testing and hardware rollouts. PCIe 4.0 compliance testing will finally kick off in August of this year, which should further accelerate 4.0 adoption and hardware support. Meanwhile PCIe 5.0 compliance testing is still under development, and like 4.0, once 5.0 compliance testing becomes available it should open the flood gates to much faster adoption there as well.

Source: PCI-SIG

POST A COMMENT

120 Comments

View All Comments

  • mode_13h - Wednesday, June 19, 2019 - link

    Why do believe 6/7 years is the right number, if you want to avoid hardware failures? Except for some PSUs, nothing has a warranty that long.

    If downtime is really an issue, then you should replace stuff before it goes out of warranty. Otherwise, just suck it up and replace stuff when it breaks.

    BTW, speaking of power, I recommend using a high-quality PSU and UPS (i.e. with AVR). In my experience, you'll get more life out of your components, that way. Also, don't overclock, and either buy over-spec'd or ECC memory.
    Reply
  • DanNeely - Wednesday, June 19, 2019 - link

    experience with prior systems having elevated failure rates as they approached a decade old, combined with the increasing difficulty of getting parts. The 2 year window fits new product cycles where (pre 10nm faceplant) Intel was issuing significant upgrades every 2 years; a pattern that AMD appears to be copying with Zen. Demoted/retired before 8 years means I start looking around 6, with timing being tied to major product refreshes along with time/budget constraints. Reply
  • rocky12345 - Wednesday, June 19, 2019 - link

    I'm one of those that uses the hardware as long as I can before upgrading. Once I start to notice slow downs in my system I either change out things like hard drive & upgrade to something like a SSD drive. I always try to max out the memory in my case 32GB was my boards max so that is what it got. I over clock everything that can be over clocked to the max. This has allowed me to keep on using my system form 2012 up until today without any speed issues. The only reason I am going to upgrade my system this fall is because of the next gen consoles and the push in the market for more cores are more cool. I feel that my 4/8 CPU will become a huge bottleneck once the next gen consoles come out with their 8/16 CPU and the game industry really starts to support CPU's with those core counts and my fake 8 thread CPU will just slowly chug along in the games at that point. There is only one game right now that seems to make my CPU feel like it could use more cores and doe snot run 100% all of the time. I got a lot fo time in on this CPU but it is time to say good bye to it and demote it to some lesser tasks and not be on my desktop as my gaming station now I just have to figure out which new CPU to get in the fall. Reply
  • Mikewind Dale - Tuesday, June 18, 2019 - link

    I wonder if this will make it possible to implement an inexpensive Optane page file that rivals DRAM without having to use Optane DIMMs? I've got a friend who occasionally needs about 512 GB of RAM, and that's expensive. But PCIe 6.0 x4 will have 32 GB/s, which is almost as much as dual channel DDR4. So I could imagine installing an Optane M.2 x4 SSD and designating it as the location for the page or swap file. This might be easier and cheaper than buying a server board that supports Optane DIMMs or hundreds of GB of RAM. Reply
  • mode_13h - Wednesday, June 19, 2019 - link

    I'm not sure Intel will keep making Optane SSDs. I think their long-term goal is to use Optane DIMMs as leverage to help sell Intel CPUs.

    Perhaps Micron will continue offering 3D XPoint in SSDs, but probably only for the enterprise market. And those probably won't be much cheaper than RAM.
    Reply
  • willis936 - Tuesday, June 18, 2019 - link

    I've been to a few of the 802.3 meetings where PAM4's development has happened and I've drank with some of the people representing big players. I've also tested high speed ethernet and seen PCIe PHY testing. I will be shocked if there is PCIe 6.0 harware ready in the next ten years. PAM4 is not a trivial problem. It isn't "throw twice as many EQ taps and deal with half the SNR and call it good" scenario. Fortunately IEEE is all public but they'll need to actually understand how these lofty models are made and why before they can hope to implement their own PAM4 over a long, nasty channel. Reply
  • Yojimbo - Tuesday, June 18, 2019 - link

    Maybe they are better at it when they aren't drinking... Reply
  • willis936 - Tuesday, June 18, 2019 - link

    Drinking is the only way humans can get through a week of standards development. If you think it's easier than I'm making it sound then I invite you try to even get up to speed.

    http://www.ieee802.org/3/ck/public/18_05/index.htm...
    Reply
  • mode_13h - Wednesday, June 19, 2019 - link

    Are you considering the orders of magnitude shorter distances involved in PCIe than Ethernet over copper? PCIe can also be constrained to shielded PCB layers, whereas Ethernet over copper has to contend with more EMI.

    I'd like to believe that the PCIe SIG wouldn't publish such plans without anyone ever conducting lab tests to validate their feasibility.
    Reply
  • willis936 - Wednesday, June 19, 2019 - link

    Calls of interest and feasibility studies are industry standard. Following ethernet demonstrates good feasibility on paper.

    Ethernet is a lot more than twisted pair and RJ-45. SFP is good for up to 25 GBaud over a few meters. Going to 50 GBaud and still going a few meters on expensive channels is what the IEEE 802.3ck WG is currently trying to achieve. Most of where these very high baud rates see use is in optical, chip to chip, and chip to module applications. Those are all also covered by ethernet.
    Reply

Log in

Don't have an account? Sign up now