What is possibly NVLink?

nVidia today announced NVLink claiming 5x-12x bandwidth of PCIe. And should be used together with IBM POWER systems.

What's possibly NVLink? what's it made of?

Previous tech on bus link increased bandwidth by growing bus width. 4-bit, 8-bit, 16-bit, 32-bit. e.g. like PCI bus or parallel bus. The problem of growing parallel bus width is that things got complicated when ilnk is wider and longer. It is harder and harder to keep all signals in sync. So the on-board PCI bus stopped at 64-bit. Parallel SCSI for inter-enclosure stopped at 16-bit. They both changed the direction toward high-speed serial links. So now we have PCIe and SAS.

But the high-speed serial link has its shortcomings too. Serial links though eliminated the need to sync across all signals, has high demand for serdes design, high-freq signal handling, etc. It is advancing, but not as easy as adding more signal links. The latest PCIe gen3 grew from 5GHz to 8GHz, less than doubled clock rate. The other bahdwidth improvement came from more efficient 128b/130b LDPC.

As the most bandwidth demanding component, GPU has lead the way of link and memory bandwidth. so we have various GDDR, ultra-side memory bus, and the primary user of PCIe has always been GPU before the AGP, graphics-centric bus, was done.

So what can NVLink be?

According to the slide nVidia provides, GPU and CPU still have PCIe link between them. So it is possible NVLink is used as a wider side channel for more efficient data movement. Much like what SLI and Crossfire did in the past. Only this time CPU joins the party. This makes it look more like the internal connect found in recent AMD APUs, but for inter-chip.

But where does the 5x-12x bandwidth come from?

It is possible that nVidia and IBM came out tech similar to QPI or HT for chip-level communications. Given that serial link clock is unlikely to get huge boost, It is more possible that NVLink gets the 5x ratio partly from higher clock/transaction. Say 1.5x for chip/board-optimized usage. The other mostly come wider link. PCIe is currently limited to x16 configuration. For specific usage optimization it is possible nVidia choose to widen the link, provided the link distance won't go too far. x32 link is a safe bet, and x64 is a bit ambitious.

Another inter-chip connect example is the newly disclosed OPIO from Intel, used on Haswell parts. 16-bit x 4 clusters/direction x 6.4GT/s gives 51.2GB/s per direction or "102.4GB/s bidirectional".OPIO is focused on on-package low-power interconnect. NVLink is unlikely to achieve the same level of power consumption.

The bad thing is nVidia is always light on technical details. Sometimes even distort or hide truths. The good is that IBM is more open in this regard. We will see.