Know more about TCP/IP and Ethernet as an I/O technology LAN,WAN,MAN
From a technical point of view, Fibre Channel has some advantages in relation to IP storage: the Fibre Channel protocol stack is integrated and thus very efficient. In comparison to Fibre Channel, TCP/IP has a significantly higher protocol overhead. Furthermore, Fibre Channel has for some years been successfully used in production environments. By contrast, IP storage has not yet stood the test of time: the iSCSI standard was only passed by the IETF at the beginning of 2003. The ratification of FCIP is expected to take place in 2004. As a result, there are currently very few production environments in which one of the new IP storage protocols is used. In what follows we will describe the reasons why we nevertheless believe that IP storage will establish itself as an important technique for storage networks in a few years time. To this end, we will first explain the advantages and disadvantages of IP storage and then show in Section 3.5.3 a migration path from Fibre Channel to IP storage. Proponents of IP storage cite the following advantages in relation to Fibre Channel:
Ø common network for LAN, MAN, WAN, SAN, voice and probably video; • standardization and maturity of technology since TCP/IP and Ethernet have been in use for decades;
Ø more personnel are available with TCP/IP knowledge than with knowledge of Fibre Channel;
Ø TCP/IP have no distance limits;
Ø cheaper hardware, since competition is greater in the field of TCP/IP than Fibre Channel due to the higher market volume;
Ø availability of administration tools for TCP/IP networks. In the following we will discuss how these supposed advantages of IP storage are not as clear-cut as they might appear. However, we let us first also mention the supposed disadvantages of IP storage:
Ø lack of standardization of IP storage;
Ø lack of interoperability of IP storage;
Ø high CPU use for SAN data traffic via TCP/IP;
Ø greater TCP/IP overhead, since the protocol is not designed for mass data;
Ø high latency of TCP/IP/Ethernet switches;
Ø low exploitation of the bandwidth of Ethernet (20–30%) due to the typical collisions for Ethernet. In what follows we will also investigate the listed is advantages, some of which contradict the advantages of IP storage that are often put forward. It is correct that when using IP storage LAN, MAN, WAN and SAN can be operated via common physical IP networks (Figure 3.41). However, it should be borne in mind
that in many environments the LAN-MAN-WAN network is already working at its limit. This means that when using IP storage, just as when using Fibre Channel SAN, additional network capacity must be installed. It is questionable whether it is organizationally possible for the IP network for LAN to be managed by the same people who manage the IP network for IP storage: with LAN, access to data is restricted by the applications so that the LAN administrator cannot simply access confidential data. In IP storage, on the other hand, the administrator can access significantly more data. Nevertheless, there is increasingly a trend towards handling all data traffic over IP and Ethernet. Conventional data networks use almost traditional TCP/IP and its application protocols such as HTTP, FTP, NFS, CIFS or SMTP. In pilot projects Gigabit Ethernet is already being used for the networking of schools, authorities and households in Metropolitan Area Networks (MANs). It is therefore easily possible that Gigabit Ethernet will at some point supersede DSL for connecting companies, authorities and households the broadband Internet (the Internet of the future). In addition, telephoning over IP (Voice of IP, VoIP) has been in use in new office buildings for some time. If locations a long distance apart frequently have to be in telephone contact, telephoning over the Internet can save an immense amount of money.The standardization of all data traffic – from telephony through LAN to storage networks – to IP networks would have certain advantages. If only IP networks were sedin office buildings, the available bandwidth could be provided to different types of data traffic as required. In an extreme case, the capacity could be used for different purposes depending upon the time of day, for example, for telephone calls during the day and for network back-up during the night.
In addition, many companies rent dedicated IP connections for the office data traffic, which are idle during the night. FCIP allows the network capacity to be used to copy
data without renting additional lines. When writing tapes over a Gigabit Ethernet line of ten kilometres in length by FCIP, a throughput of 30 to 40 MByte/s has been measured. A higher throughput was limited by the tape drive used. Considerable cost savings are thus possible with FCIP because the WAN connections that are already available and occasionally not used can also be used. Furthermore, the standardization of all communications to TCP/IP/Ethernet ensures
further cost savings because the market volume of TCP/IP/Ethernet components is significantly greater than that of any other network technology segment. For example, the development and testing cost for new components is distributed over a much larger number of units. This gives rise to greater competition and ultimately to lower prices for Ethernet components than for Fibre Channel components. However, high-end LAN switches and high-end LAN routers also come at a price, so we will have to wait and see how great the price advantage is. The availability of personnel with knowledge of the necessary network technology is a point in favour of IP and Gigabit Ethernet. IP and Ethernet have been in use for LANs for many years. Knowledge regarding these technologies is therefore widespread. Fibre Channel, on the other hand, is a young technology that is mastered by few people in comparison to IP and Ethernet. There is nothing magical about learning to use Fibre Channel technology. However, it costs money and time for the training of staff, whichis usually not necessary for IP and Ethernet. However, training is also necessary for IP SANs, for example for iSCSI and iSCSI SAN concepts.It is correct to say that there are currently (2003) very few tools on the market that canhelp in the management of a heterogeneous Fibre Channel SAN. There are hardly any tools that show the topology of a Fibre Channel SAN, in which the network components and end devices of different manufacturers are used. The administration tools for TCP/IP networksare much broader here. However, here too expansions are necessary. For example, for storage administration we need to know which servers use which storage devices and how great the load is on the storage devices in question as a result of read and write access. Although it is currently possible to find this out for individual servers, there are no tools that help to determine the storage resource consumption of all servers in a heterogeneous environment (Chapter 8).
In connection with IP storage, the vision is sometimes put forward that servers will store their data on storage systems that export virtual hard disks on the Internet – TCP/IP makes this possible. However, we have to keep in mind the fact that the Internet today has a high latency and the transmission rates achieved sometimes fluctuate sharply. This means that storage servers on the Internet are completely unsuitable for time-critical I/O accesses such as database transactions. Even if the performance of the Internet infrastructure increases, the transmission of signals over long distances costs time. For this reason, a database server in London will never access virtual hard disks in New York. This scenario is therefore only of interest for services that tolerate a higher network latency, such as the copying, back-up, replication or asynchronous mirroring of data. Like Fibre Channel FCP, IP storage has to serialize the SCSI protocol and map it onto IP, TCP/IP or UDP/IP. Precisely this standardization is still in progress: different approaches to IP storage are currently being standardized for iSCSI, iFCP, mFCP and FCIP. These standards must first be implemented by various manufacturers and tested for interoperability. Anyone who wishes to use storage networks today (2003) is therefore forced to use Fibre Channel or put up with the proprietary IP storage solutions of individual manufacturers. The assertion that IP storage will have no interoperability problems because the under- lying TCP/IP technology has been in use for decades is nonsense. The protocols based
upon TCP/IP such as iSCSI or iFCP have to work together in a cross-manufacturer manner just like Fibre Channel SAN. In addition, there is generally room for interpretation in the implementation of a standard. Experiences with Fibre Channel show that, despite standardization, comprehensive interoperability testing is indispensable (Section 3.4.6). Interoperability problems should therefore be expected in the first supposedly standard- compliant products from different manufacturers. It is correct that TCP/IP data traffic is very CPU-intensive. Figure 3.42 compares the CPU load of TCP/IP and Fibre Channel data traffic. The reason for the low CPU load of Fibre Channel is that a large part of the Fibre Channel protocol stack is realized on the Fibre Channel host bus adapter. By contrast, in current network cards a large part of the TCP/IP protocol stack is processed on the server CPU. The communication between the Ethernet network card and the CPU takes place via interrupts. This costs additional computing power, because every interrupt triggers an expensive process change in the operating system. However, more and more manufacturers are now offering so-called TCP/IP offload engines (TOEs). These are network cards that handle most of the TCP/IPprotocol stack and thus greatly free up the CPU. Now even the first prototypes for iSCSI HBAs are available, which in addition to TCP/IP also realize the iSCSI protocol in hardware. Measurements have shown that the CPU load can be significantly reduced. The Fibre Channel protocol stack is a integrated whole. As a result, cut-through routing is comparatively simple to realize for Fibre Channel switches. By contrast, TCP/IP and Ethernet were developed independently and not harmonized to one another. In the TCP/IP/Ethernet protocol stack the IP layer is responsible for the routing. So-called level 3 routers permit the use of cut-through routing by analysing the IP data traffic and then realizing the cut-through routing a layer below on the Ethernet layer. It is therefore highly
probable that the latency of an Ethernet/IP switch will always be poorer than the latency of a Fibre Channel switch. How relevant this is to the performance of IP storage is currently unknown: Figure 3.23 shows that in today's Fibre Channel SANs the latencyof the switches is insignificant in comparison to the latency of the end devices. The economic advantages of IP storage discussed above would presumably be negated if IP storage required different IP/Ethernet switches than the switches for LAN/MAN/WAN data traffic. Proponents of Fibre Channel sometimes assert that TCP/IP and Ethernet is inefficient where there are several simultaneous transmitters because in this situation the collisions that occur in Ethernet lead to the medium only being able to be utilized at 20–30%. This
statement is simply incorrect. Today's Ethernet switches are full duplex just like Fibre Channel switches. Full duplex means that several devices can exchange data in pairs using the full bandwidth, without interfering with each other. To summarize the discussion above, IP storage will at least fulfil the performance
requirements of many average applications. The question of to what degree IP storage is also suitable for central applications with extremely high performance requirements has yet to be answered. Since no practical experience is available regarding this question, only theoretical considerations are possible. An I/O technique for high performance applications must guarantee a high throughput at a low CPU load and a low latency (delay) of data transmission. As discussed, even now the CPU load is under control with iSCSI HBAs. More bandwidth will very soon also be available between servers and storage devices than can be processed with the current level of technology. If the manufacturers stick to their announcements and launch the first 10-Gigabit Ethernet components for storage networks onto the market in 2004, it will be possible by means of trunking to bring together several 10-Gigabit Ethernet connections into one virtual connection that provides an even greater bandwidth. It is more difficult to make predictions regarding the effect of the latency of IP storage,
which will probably be higher, on the performance of applications. We will have to wait for relevant experiences in production environments. The current reference installations are not yet sufficient to make a comprehensive judgement that is proven by practical experience. Furthermore, in the more distant future we can hope for improvements if techniques such as Remote Direct Access Memory (RDMA), Virtual Interfaces (VI) and InfiniBand are drawn into storage networks, and protocols such as iSCSI, iSER and SDP are based directly upon these new techniques (Sections 3.6, 3.7 and 3.8). There is still a long way to go before IP storage can be viewed as a serious alternative to Fibre Channel SAN for the realization of storage networks. There are some technical
tasks to be dealt with that do not require a supreme engineering achievement, but simply have to be done. Many reputable manufacturers have announced the appearance of IP storage products, so it is now just a matter of time before IP storage becomes a serious lternative to Fibre Cannel SAN. We believe that IP storage will gain a large share of the market for storage networks in a few years due to the economic advantages over Fibre102 I/O TECHNIQUES
Channel discussed above. In our opinion IP storage has the potential to marginalize Fibre Channel in the long term, like the technologies of ATM and FDDI that once had similar ambitions. It would not be the first time that Ethernet has prevailed
