Free tips on Performance bottlenecks in file servers Acceleration of network file systems

Free tips on Performance bottlenecks in file servers Acceleration of network file systems and Current NAS servers and NAS gateways, as well as classical file servers, provide their storage capacity via conventional network file systems such as NFS and CIFS or Internet protocols such as FTP and HTTP. Although these may be suitable for classical file sharing, such protocols are not powerful enough for I/O-intensive applications such as databases or video processing. Nowadays, therefore, I/O-intensive databases draw their storage from disk subsystems rather than file servers. Let us assume for a moment that a user wishes to read a file on an NFS client, which is stored on a NAS server with internal SCSI disks. The NAS server's operating system first of all loads the file into the main memory from the hard disk via the SCSI bus, the PCI bus and the system bus, only to forward it from there to the network card via the system bus and the PCI bus. The data is thus shovelled through the system bus and the PCI bus on the file server twice (Figure 4.7). If the load on a file server is high enough, its buses can thus become a performance bottleneck. When using classical network file systems the data to be transported is additionally copied from the private storage area of the application into the buffer cache of the kernel on the transmitting computer before this copies the data via the PCI bus into the packet buffer of the network card. Every single copying operation increases the latency of the communication, the load on the CPU due to costly process changes between application processes and kernel processes, and the load on the system bus between CPU and main memory. The file is then transferred from the network card to the NFS client via IP and Gigabit Ethernet. At the current state of technology most Ethernet cards can only handle a small part of the TCP/IP protocol independently, which means that the CPU itself has to handle the rest of the protocol. The communication from the Ethernet card to the CPU is initiated by means of interrupts. Taken together, this can cost a great deal of CPU time (cf. Section 3.5.2, 'TCP/IP and Ethernet as an I/O technology').

Acceleration of network file systems

If we look at the I/O path from the application to the hard disks connected to a NAS server (Figure 4.15 on page 130), there are two places to start from to accelerate file sharing: (1) the underlying communication protocol (TCP/IP); and (2) the network file system (NFS, CIFS) itself. TCP/IP was originally developed to achieve reliable data exchange via unreliable trans- port routes. The TCP/IP protocol stack is correspondingly complex and CPU-intensive. This can be improved first of all by so-called TCP/IP offload engines (TOEs), which in contrast to conventional network cards process a large part of the TCP/IP protocol stack on their own processor and thus significantly reduce the load on the server CPU (Section 3.5.2). It would be even better to get rid of TCP/IP all together. This is where communication techniques such as Virtual Interfaces (VI) and Remote Direct Memory Access (RDMA) come into play (Chapter 3.7). Today there are various approaches for accelerating network file systems with VI and RDMA. The Socket Direct Protocol (SDP) represents an approach which combines the benefits of TOEs and RDMA-enabled transport (Section 3.7). Hence, protocols based on TCP/IP such as NFS and CIFS can – without modification–benefit via SDP from RDMA-enabled transport. Other approaches map existing network file systems directly onto RDMA. For example, a subgroup of the Storage Networking Industry Association (SNIA) is working on the protocol mapping of NFS on RDMA. Likewise, it would also be feasible for Microsoft to develop a CIFS implementation that uses RDMA instead of TCP/IP as the communication protocol. The advantage of this approach is that the network file systems NFS or CIFS that have matured over the years merely have a new communication mechanism put underneath them. This makes it possible to shorten the development and testing cycle so that the qualitative requirements of production environments can be fulfilled comparatively quickly. A greater step is represented by newly developed network file systems, which from the start require a reliable network connection, such as the Direct Access File System (DAFS, Section 4.2.5), the family of the so-called shared disk file systems (Section 4.3) and the virtualization on the file-level (Section 5.5). 4.2.5 Case study: the Direct Access File System (DAFS) The Direct Access File System (DAFS) is a newly developed network file system that is tailored to the use of RDMA. It is based upon NFS version 4, requires VI and can fully utilize its new possibilities. DAFS makes it possible for several DAFS servers together to provide the storage space for a large file system (Figure 4.8). It remains hidden from the application server – as the DAFS client – which of these DAFS servers the actual data is located in (Figure 4.9).

The communication between DAFS client and DAFS server generally takes place by means of RDMA. The use of RDMA means that access to data that lies upon a DAFS server is nearly as quick as access to local data. In addition, typical file system operations such as the address conversion of files to SCSI block addresses, which naturally also require the CPU, are offloaded from the application server to the DAFS server. An important function of file sharing in general is the synchronization of simultaneous accesses to file entries – i.e. metadata such as file names, access rights, etc. – and file contents, in order to protect the consistency of the data and metadata. DAFS makes it possible to cache the locks at the client side so that a subsequent access to the same data requires no interaction with the file server. If a node requires the lock entry of a different node, then this transmits the entry without time-out. DAFS uses lease-based

locking in order to avoid the permanent blocking of a file due to the failure of a client. Furthermore, it possesses recovery mechanisms in case the connection between DAFS client and DAFS server is briefly interrupted or a different server from the cluster has to step in. Similarly, DAFS takes over the authentication of client and server and furthermore can also authenticate individual users in relation to a client-server session. Two approaches prevail in the discussion about the client implementation. It can either be implemented as a shared library (Unix) or DLL (Windows) in the user space or as a kernel module (Figure 4.10). In the user space variant – known as uDAFS – the DAFS library instructs the kernel to set up an exclusive end-to-end connection with the

DAFS server for each system call (or for each API call under Windows) by means of a VI provider layer (VIPL), which is also realized as a library in user space. The VI- capable NIC guarantees the necessary protection against accesses or faults caused by other processes. The user space implementation can utilize the full potential of DAFS to increase the I/O performance because it completely circumvents the kernel. If offers the application explicit control over the access of the NIC to its private storage area. Although control communication takes place between the VI provider layer in the user space and he VI-NIC driver in the kernel, the CPU cost that this entails can be disregarded due to the low data quantities. The disadvantage of the methods is the lack of compatibility with previous applications, which without exception require an upgrade in order to use DAFS. Such a cost is only justified for applications for which a high I/O performance is critical.In the second, kernel-based variant the implementation is in the form of a loadable file system module (fDAFS) underneath the Virtual File System (VFS layer) for Unix or as an Installable File System (IFS) for Windows. Each application can address the file system driver as normal by means of the standard system calls and VFS or API calls and the I/O manager; DAFS then directs a query to the DAFS server. The I/O performance is slowed due to the fact that all file system accesses run via the VFS or the I/O manager because this requires additional process changes between user and kernel processes. On the other hand, there is compatibility with all applications. Some proponents of DAFS claim to have taken measurements in prototypes showing that data access over DAFS is quicker than data access on local hard disks. In our opinion this comparison is dubious. The DAFS server also has to store data to hard disks. We find it barely conceivable that disk access can be quicker on a DAFS server than on a conventional file server. Nevertheless, DAFS-capable NAS servers could potentially support I/O-intensive applications such as databases, batch processes or multi-media applications. Integration with RDMA makes it irrelevant whether the file accesses take place via a network. The separation between databases and DAFS servers even has the advantage that the address conversion of files to SCSI block addresses is offloaded from the database server to the DAFS server, thus reducing the load on the database server's CPU. However, file servers and database servers will profit equally from InfiniBand, VI and RDMA. There is therefore the danger that a DAFS server will only be able to operate very few databases from the point of view of I/O, meaning that numerous DAFS servers may have to be installed. A corresponding number of DAFS-capable NAS servers could be installed comparatively quickly. However, the subsequent administrative effort could be considerably greater. The development of DAFS was primarily advanced by the company Network Appliance, an important NAS manufacturer. The standardization of the DAFS protocol for communication between server and client and the DAFS API for the use of the file system by applications was performed by the DAFS Collaborative . Since the release of Version 1.0 in September 2001 (protocol) and in November 2001(API) the standardization efforts have slowed down. Initially the DAFS standard was submitted to the IETF in September 2001, but it did not get enough support across the storage industry. Instead of this we can observe more activity for an alternative approach,124 FILE SYSTEMS AND NAS the enablement of NFS to run RDMA-native and the adding of DAFS-like local sharing semantics. DAFS is an interesting approach to the use of NAS servers as storage for I/O-intensive applications. Due to a lack of standardization and widespread support across the industry, current DAFS offerings (2003) should be considered as a temporary solution for specific environments until alternatives like NFS over RDMA and CIFS over RDMA emerge. Furthermore, iSCSI is of interest to those who see DAFS as a way of avoiding an investment in Fibre Channel, the more so because iSCSI – just like NFS and CIFS – can benefit

from TCP/IP Offload Engines, the Socket Direct Protocol (SDP) and a direct mapping of iSCSI on RDMA (iSER) (Section 3.7). Shared-disk file systems (Section 4.3) also offer a solution for high-speed file sharing and in addition to that, storage virtualization (Chapter 5) provides high-speed file sharing while it addresses the increasingly expense management of storage and storage networks as well.