InfiniBand

InfiniBand is a point-to-point high-speed switch fabric interconnect architecture that features built-in quality of service, fault tolerance and scalability. The InfiniBand Architecture (IBA) Specification defines the interconnect (fabric) technology for interconnecting processor nodes and I/O nodes to form a system area network that is independent of the host operating system and processor platform.

It is specified and maintained by the InfiniBand Trade Association (IBTA).

Founded in 1999, the InfiniBand Trade Association (IBTA) is comprised of leading enterprise IT vendors including Agilent, Dell, Hewlett-Packard, IBM, SilverStorm, Intel, Mellanox, Network Appliance, Oracle, Sun, Topspin and Voltaire. The organization completed its first specification in October 2000. In the past 12 months all major system vendors have announced InfiniBand products and hundreds of products have completed interoperability testing and are commercially available.

Description

Like Fibre Channel, PCI Express, and many other modern interconnects, InfiniBand uses a bidirectional serial bus so that it can avoid the signal skew problems of parallel busses when communicating over relatively long (room- or building-scale) distances. Although it is a serial connection, it is very fast, with 2.5 gigabits per second (Gbit/s) links in each direction per connection. InfiniBand also supports double and quad data rates for 5 Gbit/s or 10 Gbit/s respectively. Links use 8B/10B encoding so every 10 bits sent carry 8 bits of data, such that the actual data rate is 4/5ths the raw rate. Thus the single, double and quad data rates carry 2, 4 or 8 Gbit/s respectively.

Links can be aggregated in units of 4 or 12, called 4X or 12X. A quad-rate 12X link therefore carries 120 Gbit/s raw, or 96 Gbit/s of user data. Most systems today use a 4X single data rate connection, though the first double-data-rate products are already entering the market. Larger systems with 12x links are typically used for various cluster and supercomputer interconnects and for inter-switch connections.

**InfiniBand data rates, raw / data**
	SDR	DDR	QDR
1X	2.5 / 2 Gbit/s	5 / 4 Gbit/s	10 / 8 Gbit/s
4X	10 / 8 Gbit/s	20 / 16 Gbit/s	40 / 32 Gbit/s
12X	30 / 24 Gbit/s	60 / 48 Gbit/s	120 / 96 Gbit/s

InfiniBand Switch Latency is 140 ns. Real data latencies in a system are about 3 µs (depending on software/firmware).

InfiniBand uses a switched fabric topology so several devices can share the network at the same time (as opposed to a bus topology). Data is transmitted in packets of up to 4 kB that are taken together to form a message. A message can be a direct memory access read or write operation from/to a remote node (RDMA), a channel send or receive, a transaction-based operation (that can be reversed), or a multicast transmission.

Like the channel model used in most mainframes, all transmissions begin or end with a channel adapter. Each processor contains a host channel adapter (HCA) and each peripheral has a target channel adapter (TCA). These adapters can also exchange information for security or quality of service.

History

InfiniBand is the result of merging two competing designs, Future I/O, developed by Compaq, IBM, and Hewlett-Packard, with Next Generation I/O (ngio), developed by Intel, Microsoft, and Sun Microsystems. From the Compaq side, the roots were derived from Tandem's ServerNet. For a short time before the group came up with a new name, InfiniBand was called System I/O.

InfiniBand implementations are prominent in server clusters where high-bandwidth and low latency are key requirements. In addition to server clusters, InfiniBand unifies the compute, communications and storage fabric in the data center. Several InfiniBand blade server designs have been announced by major server vendors, accelerating the proliferation of dense computing. InfiniBand draws on existing technologies to create a flexible, scalable, reliable I/O architecture that interoperates with any server technology on the market. With industry-wide adoption, InfiniBand continues to transforms the entire computing market.

In addition to servers, InfiniBand enables both block and file based storage systems with a high performance interface that directly connects to the server cluster. This unification of servers and storage ultimately delivers higher performance with lower overall total cost of ownership by utilizing a single network for both clustering and storage connectivity.

InfiniBand is also being used for embedded computing, an area in which proprietary components are being replaced by higher-performance, standardized, off-the-shelf equivalents. InfiniBand benefits embedded applications economically and technically, through its inherent resiliency, scalability and highly efficient communications.

Recently, SGI has released storage products with InfiniBand "target adapters". This product essentially competes with architectures such as fibre channel, iSCSI, and other traditional Storage Area Networks. Such target adapter-based discs would become a part of the fabric of a given network, in a fashion similar to DEC VMS clustering. The advantage to this configuration would be lower latency and higher availability to nodes on the network (because of the fabric nature of the network).

Another novel InfiniBand-based product is the upcoming HyperTunnel, a competing architecture for AMD Horus, which will tunnel HyperTransport over InfiniBand for use in cluster-based MPI applications and future NUMA hardware. Additionally, the Cray XD1 uses Mellanox on-motherboard InfiniBand switches to create a fabric for HyperTransport between compute nodes.

External links

Serial buses | Computer buses

Gourt Home::Gourt :: InfiniBand

Description

History

External links