Learn more on Basics Interoperability of Fibre Channel SAN

Learn more on Basics Interoperability of Fibre Channel SAN

Fibre Channel SANs are currently being successfully used in production environments. Nevertheless, interoperability is an issue with Fibre Channel SAN, as in all new cross manufacturer technologies. When discussing the interoperability of Fibre Channel SAN we must differentiate between the interoperability of the underlying Fibre Channel network layer, the interoperability of the Fibre Channel application protocols, such as FCP (SCSI over Fibre Channel) and the interoperability of the applications running on the Fibre Channel SAN. The interoperability of Fibre Channel SAN stands and falls by the interoperability of

FCP. FCP is the protocol mapping of the FC-4 layer, which maps the SCSI protocol on a Fibre Channel network (Section 3.3.8). The FCP is a complex piece of software that can only be implemented in the form of a device driver. The implementation of hardware-like device drivers alone is a task that attracts errors as if by magic. The developers of FCP device drivers must therefore test extensively and thoroughly.

Two general conditions make it more difficult to test the FCP device driver. The server initiates the data transfer by means of the SCSI protocol; the storage device only responds to the requests of the server. However, the idea of storage networks is to consolidate storage devices, i.e. for many servers to share a few large storage devices. Therefore, with storage networks a single storage device must be able to serve several parallel requests from different servers simultaneously. For example, it is typical for a server to be exchanging data with a storage device just when another server is scanning the Fibre Channel SAN for available storage devices. This situation requires end devices to be able to multitask. When testing multitasking systems the race conditions of the tasks to be performed come to bear: just a few milliseconds delay can lead to a completely different test result. The second difficulty encountered during testing is due to the large number of components that come together in a Fibre Channel SAN. Even when a single server is connected to a single storage device via a single witch, there are numerous possibilities that cannot all be tested. If, for example, a Windows server is selected, there is still the choicebetween NT, 2000 and 2003, each with different service packs. Several manufacturers offer several different models of the Fibre Channel host bus adapter card in the server. If we take into account the various firmware versions for the Fibre Channel host bus adapter cards we find that we already have more than 50 combinations before we even

select a switch. Companies want to use their storage network to connect servers and storage devicesfrom various manufacturers, some of which are already present. The manufacturers of Fibre Channel components (servers, switches and storage devices) must therefore perform interoperability tests in order to guarantee that these components work with devices from third-party manufacturers. Right at the top of the priority list are those combinations that are required by most customers, because this is where the expected profit is the highest. The result of the interoperability test is a so-called support matrix. It specifies, for example, which storage device supports which server model with which operating system versions and Fibre Channel cards. Manufacturers of servers and storage devices often limit the Fibre Channel switches that can be used. Therefore, before building a Fibre Channel SAN you should carefully check whether the manufacturers in question state that they support the planned configuration. If the desiredconfiguration is not listed, you can negotiate with the manufacturer regarding the payment of a surcharge to secure manufacturer support. Although non-supported configurations canwork very well, if problems occur, you are left without support in critical situations. If in any doubt you should therefore look for alternatives right at the planning stage. All this seems bsolutely terrifying at first glance. However, manufacturers now support a number of different configurations. If the manufacturers' support matrices are taken into consideration, robust Fibre Channel SANs can now be operated. The operation of up-to- date operating systems such as Windows NT/2000, AIX, Solaris, HP-UX and Linux is particularly unproblematic. Fibre Channel SANs are based upon Fibre Channel networks. The incompatibility of the fabric and arbitrated loop topologies and the networking of fabrics and arbitrated loops has already been discussed in Section 3.4.3. Within the fabric, the incompatibility of the Fibre Channel switches from different manufacturers should also be mentioned. At the end of 2003 we still recommend that when installing a Fibre Channel SAN only the switches

and directors of a single manufacturer are used. Even though routing between switches and directors of different manufacturers may work as expected, and basic functions of the fabric topology such as aliasing, name server and zoning work well across different vendors in so-called 'compatibility modes'. But bear in mind that there is still only a very small installed base of mixed switch vendor configurations. A standard has been passed that addresses the interoperability of these basic functions, meaning that it is now just a matter of time before these basic functions work across every manufacturers' products.However, for new functions such as SAN security, inter-switch-link trunking or B-Ports,teething troubles with interoperability must once again be expected. In general, applications can be subdivided into higher applications that model and support the business processes and system-based applications such as file systems, databases and back-up systems. The system-based applications are of particular interest from the point of view of storage networks and storage management. The compatibility of network file systems such as NFS and CIFS is now taken for granted and hardly ever queried. As storage networks penetrate into the field of file systems, cross-manufacturer standards are becoming ever more important in this area too. A first offering is Network Data Management Protocol (NDMP, Section 7.9.4) for the back-up of NAS servers. Further down the road we expect also a customer demand for cross-vendor standards in the emerging field of storage virtualization (Chapter 5). The subject of interoperability will preoccupy manufacturers and customers in the field of storage networks for a long time to come. Virtual Interface Architecture (VIA), Infini- Band and Remote Direct Memory Access (RDMA) represent emerging new technologies that must also work in a cross-manufacturer manner. The same applies for Internet SCSI (iSCSI) and its variants like iSCSI Extensions over RDMA (iSER). iSCSI transmits the

SCSI protocol via TCP/IP and, for example, Ethernet. Just like FCP, iSCSI has to serialize he SCSI protocol bit-by-bit and map it onto a complex network topology. Interoperability will therefore also play an important role in iSCSI.

FREE TUTORS ON THE FIBRE CHANNEL PROTOCOL STACK FLOW CONTROL (Service classes)

Free Tutors on THE FIBRE CHANNEL PROTOCOL STACK

FC-1: 8b/10b encoding, ordered sets and link control protocolFC-1 defines how data is encoded before it is transmitted via a Fibre Channel cable(8b/10b encoding). FC-1 also describes certain transmission words (ordered sets) that are required for the administration of a Fibre Channel connection (link control protocol). 8b/10b encoding In all digital transmission techniques, transmitter and receiver must synchronize their clock-pulse rates. In parallel buses the bus rate is transmitted via an additional data line. By contrast, in the serial transmission used in Fibre Channel only one data line is available through which the data is transmitted. This means that the receiver must regenerate the transmission rate from the data stream.

The receiver can only synchronize the rate at the points where there is a signal change in the medium. In simple binary encoding (Figure 3.11) this is only the case if the signal changes from '0' to '1' or from '1' to '0'. In Manchester encoding there is a signal change for every bit transmitted. Manchester encoding therefore creates two physical signals for each bit transmitted. It therefore requires a transfer rate that is twice as high as that for binary encoding. Therefore, Fibre Channel – like many other transmission techniques – uses binary encoding, because at a given rate of signal changes more bits can be transmitted than is the case for Manchester encoding. The problem with this approach is that the signal steps that arrive at the receiver are not always the same length (jitter). This means that the signal at the receiver is sometimes a little longer and sometimes a little shorter (Figure 3.12). In the escalator analogy this means that the escalator bucks. Jitter can lead to the receiver losing synchronization with the received signal. If, for example, the transmitter sends a sequence of ten zeros, the receiver cannot decide whether it is a sequence of nine, ten or eleven zeros. If we nevertheless wish to use binary encoding, then we have to ensure that the data stream generates a signal change frequently enough that jitter cannot strike. The so-called 8b/10b encoding represents a good compromise. 8b/10b encoding converts an eight-bit

byte to be transmitted into a ten-bit character, which is sent via the medium instead of the eight-bit byte. For Fibre Channel this means, for example, that a useful transfer rate of 100 MByte/s requires a raw transmission rate of 1 Gbit/s instead of 800 Mbit/s. Incidentally, 8b/10b encoding is also used for the Enterprise System Connection Architecture (ESCON), Serial Storage Architecture (SSA), Gigabit Ethernet and InfiniBand. Finally, it should be noted that 1 Gigabyte Fibre Channel uses the 64b/66b encoding variant for a certain cable type (single lane with serial transmission). Expanding the eight-bit data bytes to ten-bit transmission character gives rise to the following advantages: • In 8b/10b encoding, of all available ten-bit characters, only those that generate a bit

sequence that contains a maximum of five zeros one after the other or five ones one after the other for any desired combination of the ten-bit character are selected. There- fore, a signal change takes place at the latest after five signal steps, so that the clock synchronization of the receiver is guaranteed.

• A bit sequence generated using 8b/10b encoding has a uniform distribution of zeros and ones. This has the advantage that only small direct currents flow in the hardware that processes the 8b/10b encoded bit sequence. This makes the realization of Fibre Channel hardware components simpler and cheaper.

• Further ten-bit characters are available that do not represent eight-bit data bytes. These additional characters can be used for the administration of a Fibre Channel link.

Ordered sets

Fibre Channel aggregates four ten-bit transmission characters to form a 40-bit transmission word. The Fibre Channel standard differentiates between two types of transmission word: data words and ordered sets. Data words represent a sequence of four eight-bit data bytes. Data words may only stand between a Start-of-Frame delimiter (SOF delimiter) and an End-of-Frame delimiter (EOF delimiter).Ordered sets may only stand between an EOF delimiter and a SOF delimiter, with SOF sand EOFs themselves being ordered sets. All ordered sets have in common that they begin with a certain transmission character, the so-called K28.5 character. The K28.5 character includes a special bit sequence that does not occur elsewhere in the data stream. The input channel of a Fibre Channel port can therefore use the K28.5 character to divide the continuous incoming bit stream into 40 bit transmission words when initializing a Fibre Channel link or after the loss of synchronization on a link. Link control protocol With the aid of ordered sets, FC-1 defines various link level protocols for the initialization and Administration of a link. The initialization of a link is the prerequisite for data exchange by means of frames. Examples of link level protocols are the initialization and arbitration of an arbitrated loop.

3.3.4 FC-2: data transfer

FC-2 is the most comprehensive layer in the Fibre Channel protocol stack. It determines how larger data units (for example, a file) are transmitted via the Fibre Channel network. It regulates the flow control that ensures that the transmitter only sends the data at a speed that the receiver can process it. And it defines various service classes that are tailored to the requirements of various applications. Exchange, sequence and frame FC-2 introduces a three-layer hierarchy for the transmission of data (Figure 3.13). At the top layer a so-called exchange defines a logical communication connection between two end devices. For example, each process that reads and writes data could be assigned its own exchange. End devices (servers and storage devices) can simultaneously maintain several exchange relationships, even between the same ports. Different exchanges help the FC-2 layer to deliver the incoming data quickly and efficiently to the orrect receiver in the higher protocol layer (FC-3). A sequence is a larger data unit that is transferred from a transmitter to a receiver. Only one sequence can be transferred after another within an exchange. FC-2 guarantees that sequences are delivered to the receiver in the same order they were sent from the transmitter; hence the name 'sequence'. Furthermore, sequences are only delivered to the next protocol layer up when all frames of the sequence have arrived at the receiver (Figure 3.13). A sequence could represent the writing of a file or an individual database transaction. A Fibre Channel network transmits control frames and data frames. Control frames contain no useful data, they signal events such as the successful delivery of a data frame. Data frames transmit up to 2112 bytes of useful data. Larger sequences therefore have to be broken down into several frames. Although it is theoretically possible to agree upon

different maximum frame sizes, this is hardly ever done in practice. A Fibre Channel frame consists of a header, useful data (payload) and a CRC checksum

(Figure 3.14). In addition, the frame is bracketed by a Start-of-Frame delimiter (SOF) and an End-of-Frame delimiter (EOF). Finally, six filling words must be transmitted by means of a link between two frames. In contrast to Ethernet and TCP/IP, Fibre Channel is an integrated whole: the layers of the Fibre Channel protocol stack are so well harmonizedwith one another that the ratio of payload to protocol overhead is very efficient at up to 98%. The CRC checking procedure is designed to recognize all transmission errors if the underlying medium does not exceed the specified error rate of 10−12 Error correction takes place at sequence level: if a frame of a sequence is wrongly transmitted, the entire sequence is retransmitted. At gigabit speed it is more efficient to resend a complete sequence than to extend the Fibre Channel hardware so that individual lost frames can be resent and inserted in the correct position. The underlying protocol

layer must maintain the specified maximum error rate of 10−12so that this procedures efficient.

Flow control

Flow control ensures that the transmitter only sends data at a speed that the receiver can receive it. Fibre Channel uses the so-called credit model for this. Each credit represents the capacity of the receiver to receive a Fibre Channel frame. If the receiver awards the transmitter a credit of '4', the transmitter may only send the receiver four frames. The transmitter may not send further frames until the receiver has acknowledged the receipt of at least some of the transmitted frames. FC-2 defines two different mechanisms for flow control: end-to-end flow control and link flow control (Figure 3.15). In end-to-end flow control two end devices negotiate the end-to-end credit before the data exchange. The end-to-end flow control is realized on the host bus adapter cards of the end devices. By contrast, link flow control takes place at each physical connection. This is achieved by two communicating ports negotiating the buffer-to-buffer credit. This means that the link flow control also takes place at the Fibre Channel switches.

Service classes

The Fibre Channel standard defines six different service classes for data exchange between end devices. Three of these defined classes (Class 1, Class 2 and Class 3) are realized in products available on the market, with hardly any products providing the connection- oriented Class 1. Almost all new Fibre Channel products (host bus adapters, switches, storage devices) support the service classes Class 2 and Class 3, which realize a packet- oriented service (datagram service). In addition, Class F serves for the data exchange between the switches within a fabric. Class 1 defines a connection-oriented communication connection between two node ports: a Class 1 connection is opened before the transmission of frames. This specifies a route through the Fibre Channel network. Thereafter, all frames take the same route through the Fibre Channel network so that frames are delivered in the sequence in which they were transmitted. A Class 1 connection guarantees the availability of the full bandwidth. A port thus cannot send any other frames while a Class 1 connection is open.

Class 2 and Class 3, on the other hand, are packet-oriented services (datagram services): no dedicated connection is built up, instead the frames are individually routed through the Fibre Channel network. A port can thus maintain several connections at the same time. Several Class 2 and Class 3 connections can thus share the bandwidth. Class 2 uses end-to-end flow control and link flow control. In Class 2 the receiver acknowledges each received frame (acknowledgement, Figure 3.16). This acknowledge-ment is used both for end-to-end flow control and for the recognition of lost frames. A missing acknowledgement leads to the immediate recognition of transmission errors byFC-2, which are then immediately signalled to the higher protocol layers. The higherprotocol layers can thus initiate error correction measures straight away (Figure 3.18).Users of a Class 2 connection can demand the delivery of the frames in the correct order.Class 3 achieves less than Class 2: frames are not acknowledged (Figure 3.17). Thismeans that only link flow control takes place, not end-to-end flow control. In addition, the higher protocol layers must notice for themselves whether a frame has been lost. The loss of a frame is indicated to higher protocol layers by the fact that an expected sequence is not delivered because it has not yet been completely received by the FC-2 layer. A switch may dispose of Class 2 and Class 3 frames if its buffer is full. Due to greater time-outvalues in the higher protocol layers it can take much longer to recognize the loss of aframe than is the case in Class 2 (Figure 3.19).We have already stated that in practice only Class 2 and Class 3 are important. In practice the service classes are hardly ever explicitly configured, meaning that in current Fibre Channel SAN implementations the end devices themselves negotiate whether theycommunicate by Class 2 or Class 3. From a theoretical point of view the two service classes differ in that Class 3 sacrifices some of the communication reliability of Class 2in favour of a less complex protocol. Class 3 is currently the most frequently used service class. This may be because the current Fibre Channel SANs are still very small, so that

frames are very seldom lost or overtake each other. The linking of current Fibre Channel SAN islands to a large SAN could lead to Class 2 playing a greater role in future due toits faster error recognition.