InfiniBand

From Free net encyclopedia

InfiniBand is a high-speed serial computer bus, intended for both internal and external connections.

It is specified and maintained by the InfiniBand Trade Association (IBTA).

Description

Like Fibre Channel, PCI Express, and many other modern interconnects, InfiniBand uses a bidirectional serial bus so that it can avoid the signal skew problems of parallel busses when communicating over relatively long (room- or building-scale) distances. Although it is a serial connection, it is very fast, with 2.5 gigabits per second (Gbit/s) links in each direction per connection. InfiniBand also supports double and quad data rates for 5 Gbit/s or 10 Gbit/s respectively. Links use 8B/10B encoding so every 10 bits sent carry 8 bits of data, such that the actual data rate is 4/5ths the raw rate. Thus the single, double and quad data rates carry 2, 4 or 8 Gbit/s respectively.

Links can be aggregated in units of 4 or 12, called 4X or 12X. A quad-rate 12X link therefore carries 120 Gbit/s raw, or 96 Gbit/s of user data. Most systems today use a 4X single data rate connection, though the first double-data-rate products are already entering the market. Larger systems with 12x links are typically used for various cluster and supercomputer interconnects and for inter-switch connections.

**Infiniband data rates, raw / data**
	SDR	DDR	QDR
1X	2.5 / 2 Gbit/s	5 / 4 Gbit/s	10 / 8 Gbit/s
4X	10 / 8 Gbit/s	20 / 16 Gbit/s	40 / 32 Gbit/s
12X	30 / 24 Gbit/s	60 / 48 Gbit/s	120 / 96 Gbit/s

InfiniBand uses a switched fabric topology so several devices can share the network at the same time (as opposed to a bus topology). Data is transmitted in packets of up to 4 kB that are taken together to form a message. A message can be a direct memory access read or write operation from/to a remote node (RDMA), a channel send or receive, a transaction-based operation (that can be reversed), or a multicast transmission.

Like the channel model used in most mainframes, all transmissions begin or end with a channel adapter. Each processor contains a host channel adapter (HCA) and each peripheral has a target channel adapter (TCA). These adapters can also exchange information for security or quality of service.

[edit]

History

Infiniband is the result of merging two competing designs, Future I/O, developed by Compaq, IBM, and Hewlett-Packard, with Next Generation I/O (ngio), developed by Intel, Microsoft, and Sun Microsystems. From the Compaq side, the roots were derived from Tandem's ServerNet. For a short time before the group came up with a new name, InfiniBand was called System I/O.

The addressing structure of Infiniband borrows heavily from IPv6.

InfiniBand was originally envisioned as a comprehensive "System Area Network" that would connect CPUs and provide all high speed I/O for "back-office" applications. In this role it would potentially replace just about every datacenter I/O standard including PCI, Fibre Channel, and various networks like Ethernet. Instead, all of the CPUs and peripherals would be connected into a single pan-datacenter switched InfiniBand fabric. This vision offered a number of advantages in addition to greater speed, not the least of which is that I/O workload would be largely lifted from computer and storage. In theory, this should make the construction of clusters much easier, and potentially less expensive, because more devices could be shared and they could be easily moved around as workloads shifted. Proponents of a less comprehensive vision saw InfiniBand as a pervasive, low latency, high bandwidth, low overhead interconnect for commercial datacenters, albeit one that might perhaps only connect servers and storage to each other, while leaving more local connections to other protocols and standards such as PCI.

So far, however, InfiniBand has seen more limited use. It is used today mostly for performance focused computer cluster applications, and there are some efforts to adapt InfiniBand as a "standard" interconnect between low-cost machines for either commercial or technical applications. A number of the Top 500 supercomputers have used InfiniBand including the low-cost System X built by Virginia Tech.

While not quite the ubiquitous network fabric replacement envisioned, InfiniBand is having some success against other high performance computing switch fabric vendors, notably Quadrics (QsNet) and Myricom (Myrinet). In the supercomputing segment, Infiniband's closest competition remains the low cost and relative ubiquity of gigabit ethernet. Ethernet also enjoys a large knowledgebase and installbase among networking and computing professionals. As gigabit Ethernet evolves toward 10-gigabit Ethernet, InfiniBand will face stiffer competition. Infiniband will retain a higher maximum throughput (on QDR hardware) overall, but at the 10 Gbit/s and below level, the primary advantage of Infiniband becomes its fabric architecture (rather than its speed). Further competing with Infiniband are the various TCP and Ethernet augmentations, such as TCP Offload Engines, and RDMA Ethernet.

Recently, SGI has released storage products with Infiniband "target adapters". This product essentially competes with architectures such as fibre channel, iSCSI, and other traditional Storage Area Networks. Such target adapter-based discs would become a part of the fabric of a given network, in a fashion similar to DEC VMS clustering. The advantage to this configuration would be lower latency and higher availability to nodes on the network (because of the fabric nature of the network).

Another novel InfinBand-based product is the upcoming HyperTunnel, a competing architecture for AMD Horus, which will tunnel HyperTransport over InfiniBand for use in cluster-based MPI applications and future NUMA hardware. Additionally, the Cray XD1 uses Mellanox on-motherboard InfiniBand switches to create a fabric for HyperTransport between compute nodes.

[edit]