KNOW MORE ABOUT VIRTUAL INTERFACES AND REMOTE DIRECT MEMORY ACCESS (RDMA)

KNOW MORE ABOUT VIRTUAL INTERFACES AND REMOTE DIRECT MEMORY ACCESS (RDMA)

With the Virtual Interface Architecture we move away from the communication between physical devices and finally approach the applications. VIA facilitates fast and efficient data exchange between applications that run on different computers. This requires an underlying network with a low latency and a low error rate. This means that the VIA can only be used over short distances, perhaps within a data centre or within a building. Originally, VIA was launched in 1997 by Compaq, Intel and Microsoft. Today it is a fixed component of InfiniBand. Furthermore, protocol mappings exist for Fibre Channel and Ethernet.

Today communication between applications is still relatively complicated. Incoming data is accepted by the network card, processed in the kernel of the operating system and finally delivered to the application. As part of this process, data is copied repeatedly from one buffer to the next. Furthermore, several process changes are necessary in the operating system. All in all this costs CPU power and places a load upon the system bus. As a result the communication throughput is reduced and its latency increased. The idea of VIA is to reduce this complexity by making the application and the network card exchange data directly with one another, bypassing the operating system. To this end, two applications initially set up a connection, the so-called Virtual Interface (VI): a common memory area is defined on both computers by means of which application and local network card exchange data (Figure 3.46). To send data the application fills the common memory area in the first computer with data. After the buffer has been filled with all data, the application announces by means of the send queue of the Virtual Interface and the so-called doorbell of the VI hardware that there is data to send. The VI hardware reads the data directly from the common memory area and transmits it to the VI hardware on the second computer. This does not inform the application until all data is available

in the common memory area. The operating system on the second computer is therefore bypassed, too. The Virtual Interface (VI) is the mechanism that makes it possible for the application (VI consumer) and the network card (VI NIC) to communicate directly with each other via common memory areas, bypassing the operating system. At a given point in time a Virtual Interface is connected with a maximum of one other Virtual Interface. Virtual Interfaces therefore only ever allow point-to-point communication with precisely one remote Virtual Interface. A VI provider consists of the underlying physical hardware (VI Network Interface Controller, VI NIC) and a device driver (kernel agent). Examples of VI NICs are VI-capable Fibre Channel host bus adapters, VI-capable Ethernet Network cards and InfiniBand host channel adapters. The VI NIC realizes the Virtual Interfaces and completion queues and transmits the data to other VI NICs. The kernel agent is the device driver of a VI NIC that is responsible for the management of Virtual Interfaces. Its duties include the generation and removal of Virtual Interfaces, the opening and closing of VI connections to remote Virtual Interfaces, memory management and error handling. In contrast to communication over the Virtual Interface and the completion queue, communication with the kernel agent is associated with the normal overhead such as process switching. This extra cost can ,however, be disregarded because after the Virtual Interface has been set up by the kernel agent all data is exchanged over the Virtual Interface. Applications and operating systems that communicate with each other via Virtual Interfaces are called VI consumers. Applications generally use a Virtual Interface via an intermediate layer such as sockets or MPI. Access to a Virtual Interface is provided by the user agent. The user agent first of all contacts the kernel agent in order to generate a Virtual Interface and to connect this to a Virtual Interface on a remote device (server, storage device). The actual data transfer then takes place via the Virtual Interface as described above, bypassing the operating system, meaning that the transfer is quick and

the load on the CPU is lessened. 108 I/O TECHNIQUES The Virtual Interface consists of a so-called work queue pair and the doorbells. The work queue pair consists of a send queue and a receive queue. The VI consumer can charge the VI NIC with the sending or receiving of data via the work queues, with the data itself being stored in a common memory area. The requests are processed asynchronously. VI consumer and VI NIC let each other know when new requests are pending or when the processing of requests has been concluded by means of a doorbell. A VI consumer can bundle the receipt of messages about completed requests from several Virtual Interfaces in a common completion queue. Work queues, doorbell and completion queue can be

used bypassing the operating system. Building upon Virtual Interfaces, the Virtual Interface Architecture defines two different communication models. It supports the old familiar model of sending and receiving messages, with the messages in this case being asynchronous sent and asynchronously received. An alternative communication model is the so-called Remote Direct Memory Access (RDMA). Using RDMA, distributed applications can read and write memory

areas of processes running on a different computer. As a result of VI, access to the remote memory takes place with low latency and a low CPU load.