Basics Hardware components for Fibre Channel SAN

Basics Hardware components for Fibre Channel SAN

Within the scope of this book we can only introduce the most important product groups. It is not worth trying to give an overview of specific products or a detailed description of individual products due to the short product cycles. This section mentions once again some product groups that have been discussed previously and introduces some product groups that have not yet been discussed. It is self-evident that servers and storage devices are connected to a Fibre Channel network. In the server this can be achieved by fiting the host bus adapter cards (HBAs) of different manufacturers, with each manufacturer offering different HBAs with differing performance features. In storage devices the same HBAs are normally used. However, the manufacturers of storage devices restrict the selection of HBAs. Of course, cables and connectors are required for cabling. In Section 3.3.2 we discussed different copper and fiber-optic cables and their properties. Various connector types are currently on offer for all cable types. It may sound banal, but in practice the installation of a Fibre Channel SAN is sometimes delayed because the connectors on the cable do not fit the connectors on the end devices, hubs and switches and a suitable adapter is not

to hand. A further, initially improbably, but important device is the so-called Fibre Channel-to- SCSI bridge. As the name suggests, a Fibre Channel-to-SCSI bridge creates a connection between Fibre Channel and SCSI (Figure 3.30). These bridges have two important fields of application. First, old storage devices often cannot be converted from SCSI to Fibre Channel. If the old devices are still functional they can continue to be used in the Fibre Channel SAN by the deployment of a Fibre Channel-to-SCSI bridge. Second, new tape libraries in particular often initially only support SCSI; the conversion to Fibre Channel is often not planned until later.With a Fibre Channel-to-SCSI bridge the newest tape libraries can be operated directly in a Fibre Channel SAN and Fibre Channel connections retrofitted as soon as they become available. Unfortunately, the manufacturers have not agreed upon consistent name for this type of device. In addition to Fibre Channel-to-SCSI bridge, terms such as SAN router or storage gateway are also common. The switch is the control centre of the fabric topology. It provides routing and aliasing, name server and zoning functions. Fibre Channel switches support both cut-through routing and the buffering of frames. In new switches a number of ports between eight and about 250 and a data transfer rate of 200 MByte/s should currently (2003) be viewed as standard. In Fibre Channel SANs that have already been installed, however, a large base of switches exists that still work at 100 MByte/s.

 

 

Resilient, enterprise-class switches are commonly referred to as 'directors', named after the switching technology used in mainframe ESCON cabling. Like Fibre Channel switches they provide routing, alias names, name server and zoning functions. Fibre Channel direc- tors are designed to avoid any single point of failure, having for instance two backplanes and two controllers. Current directors (2003) have between 64 and 256 ports. Designing a SAN often raises the question whether several complementary switches or a single director should be preferred. As described, directors are more fault-tolerant than switches, but they are more expensive per port. Therefore, designers of small entry-level SANs commonly choose two complementary Fibre Channel switches, with mutual traffic fail-over in case of a switch or a I/O path failure (Figure 3.31). Designers of larger Fibre Channel SANs often favour directors due to the number of ports currently available per device and the resulting layout simplicity. However, this argument in favour of directors becomes more and more obsolete since today switches with a greater number of ports are available as well. SANs running especially critical applications, e.g. stock market banking or flight control,

would use complementary directors with mutual traffic failover, even though these directors already avoid internal single points of failure. This is similar to wearing trousers with a belt and braces in addition: protecting against double or triple failures. In less critical cases, a single director or a dual complementary switch solution will be considered sufficient. If we disregard the number of ports and the cost, the decision for a switch or a director in an Open Systems Fibre Channel network primarily comes down to fault-tolerance of

 

an individual component. For the sake of simplicity we will use the term 'Fibre Channel switch' throughout this book in place of 'Fibre Channel switch or Fibre Channel director'. A hub simplifies the cabling of an arbitrated loop. Hubs are transparent from the point of view of the connected devices. This means that hubs send on the signals of the connected devices; in contrast to a Fibre Channel switch, however, the connected devices do not communicate with the hub. Hubs change the physical cabling from a ring to a star-shape. Hubs bridge across defective and switched-off devices, so that the physical

ring is maintained for the other devices. The arbitrated loop protocol is located above this cabling. Hubs are divided into unmanaged hubs, managed hubs and switched hubs. Unman- aged hubs are the cheap version of hubs: they can only bridge across switched-off devices. However, they can neither intervene in the event of protocol infringements by an end device nor indicate the state of the hub or the arbitrated loop to the out- side world. This means that an unmanaged hub cannot itself notify the administrator if one of its components is defective. A very cost-conscious administrator can build up a small SAN from PC systems, JBODs and unmanaged hubs. However, the upgrade path to a large Fibre Channel SAN is difficult: in larger Fibre Channel SANs it is questionable whether the economical purchase costs compensate for the higher administration costs. In contrast to unmanaged hubs, managed hubs have administration and diagnosis functions like those that are a matter of course in switches and directors.Managed hubs monitor the power supply, serviceability of fans, temperature, and the status of the individual ports. In addition, some managed hubs can, whilst remaining invisible to the connected devices, intervene in higher Fibre Channel protocol layers, for example, to deactivate the port of adevice that frequently sends invalid Fibre Channel frames. Managed hubs, like switches and directors, can inform the system administrator about events via serial interfaces, Telnet, HTTP and SNMP (see also Chapter 8). Finally, the switched hub is mid-way between a hub and a switch. In addition to the properties of a managed hub, with a switched hub several end devices can exchange data at full bandwidth. Fibre Channel switched hubs are cheaper than Fibre Channel switches, so in some cases they represent a cheap alternative to switches. However, it should benoted that only 126 devices can be connected together via hubs and that services such as aliasing and zoning are not available. Furthermore, the protocol cost for the connection or the removal of a device in a loop is somewhat higher than in a fabric (keyword 'Loop Initialisation Primitive Sequence', 'LIP'). Finally, so- alled link extenders should also be mentioned. Fibre Channel supports a maximum cable length of several ten kilometres (Section 3.3.2). A link extender can

increase the maximum cable length of Fibre Channel by transmitting Fibre Channel frames using MAN/WAN techniques such as ATM, SONET or TCP/IP (Figure 3.32).When using link extenders it should be borne in mind that long distances between end devices significantly increase the latency of a connection. Time-critical applications such as database transactions should therefore not run over a link extender. On the other hand,Fibre Channel SANs with link extenders offer new possibilities for applications such as back-up, data sharing and asynchronous data mirroring.

Fibre Channel SAN is a comparatively new technology. In many data centres in which Fibre Channel SANs are used, it is currently (2003) more likely that there will be several islands of small Fibre Channel SANs than one large Fibre Channel SAN (Figure 3.33).Over 80% of the installed Fibre Channel SANs consist only of up to four Fibre Channel switches. A server can only indirectly access data stored on a different SAN via the LAN and a second server. The reasons for the islands of small Fibre Channel SANs are that they are simpler to manage than one large Fibre Channel SAN and that it was often unnecessary to install a large one.

Originally, Fibre Channel SAN was used only as an alternative to SCSI cabling. Until now the possibility of flexibly dividing the capacity of a storage device etween several servers (storage pooling) and the improved availability of dual SANs have been the main reasons for the use of Fibre Channel SANs. Both can be realized very well with several small Fibre Channel SAN islands. However, more and more applications are now exploiting the possibilities offered by a Fibre Channel SAN. Applications such as back-up

 

(Chapter 7), remote data mirroring and data sharing over Fibre Channel SAN and storage virtualization (Chapter 5) require that all servers and storage devices are connected via a single SAN. Incidentally, the connection of Fibre Channel SANs to form a large SAN could be one field of application in which a Fibre Channel director is preferable to a Fibre Channel switch (Figure 3.34). As yet these connections are generally not critical. In the future, however, this could change (extreme situation: virtualization over several data centres). In our opinion these connection points between two storage networks tend to represent a single point of failure, so they should be designed to be particularly fault-tolerant.

BASICS OF SCSI STORAGE NETWORKS

SCSI basics

The Small Computer System Interface (SCSI) was for a long time the technology for I/O buses in Unix and PC servers, is still very important today and will presumably remain so for a good many years to come. The first version of the SCSI standard was released in 1986. Since then SCSI has been continuously developed in order to keep it abreast with technical progress.

As a medium, SCSI defines a parallel bus for the transmission of data with additional lines for the control of communication. The bus can be realized in the form of printed conductors on the circuit board or as a cable. Over time, numerous cable and plug types have been defined that are not directly compatible with one another (Table 3.1). A so-called daisy chain can connect up to 16 devices together (Figure 3.3).The SCSI protocol defines how the devices communicate with each other via the SCSI bus. It specifies how the devices reserve the SCSI bus and in which format data is transferred. The SCSI protocol has been further developed over the years. For example, a server could originally only begin a new SCSI command when the previous SCSI command had been acknowledged by the partner; however, precisely this overlapping of SCSI commands is the basis for the performance increase achieved by RAID. Today it is even possible using asynchronous I/O to initiate several write or read commands to a storage device at the same time.

The SCSI protocol introduces SCSI IDs (sometimes also called target ID or just ID) And Logical Unit Numbers (LUN) for the addressing of devices. Each device in the SCSI us must have an unambiguous ID, with the host bus adapter in the server requiring its wn ID. Depending upon the version of the SCSI standard, a maximum of 8 or 16 Ids re permitted per SCSI bus. Storage devices such as RAID disk subsystems, intelligent isk subsystems or tape libraries can include several subdevices, such as virtual hard isks, tape drives or a media changer to insert the tapes, which means that the IDs would e used up very quickly. Therefore, so-called LUNs were introduced in order to address ubdevices within larger devices (Figure 3.4). A server can be equipped with several SCSI Controllers. Therefore, the operating system must note three things for the differentiation f devices – the controller ID, SCSI ID and LUN .The priority of SCSI IDs is slightly trickier. Originally, the SCSI protocol permitted nly eight IDs, with the ID '7' having the highest priority. More recent versions of he SCSI protocol permit 16 different IDs. For reasons of compatibility the IDs '7' to 0' should retain the highest priority, so that the IDs '15' to 8' have a lower priority Figure 3.5).Devices (servers and storage devices) must reserve the SCSI bus (arbitrate) before they may send data through it. During the arbitration of the bus, the device that has the highest

 

priority SCSI ID always wins. In the event that the bus is heavily loaded, this can lead to devices with lower priorities never being allowed to send data. The SCSI arbitration procedure is therefore 'unfair'.

SCSI and storage networks

SCSI is only suitable for the realization of storage networks to a limited degree. First, a SCSI daisy chain can only connect a very few devices with each other. Although it is theoretically possible to connect several servers to a SCSI bus, this does not work very well in practice. Clusters with so-called twin-tailed SCSI cables and a stand-by server have proved their worth in increasing the availability of data and the applications based upon it (Figure 3.6). Both servers can access the shared storage devices, with only one server having active access to the data at any time. If this server fails, then the stand-by server actively accesses the storage device and continues to operate the application. Second, the maximum lengths of SCSI buses greatly limit the construction of storage networks. Large disk subsystems have over 30 connection ports for SCSI cables, so that several dozen servers can access them (Figure 3.7), and many of the advan-

tags of storage-centric IT architectures can be achieved with this layout. However

due to the dimensions of disk subsystems, tape libraries and servers and the length limits of SCSI buses, constructing the configuration shown in Figure 3.7 using real devices is a challenge. Although it is possible to extend the length of the SCSI buses with so-called link extenders, the use of a large number of link extenders is unwieldy. Despite these limitations, SCSI is of great importance even for storage-centric IT systems. Techniques such as Fibre Channel SAN and iSCSI merely replace the SCSI bus by a network; the SCSI protocol is still used for communication over this network. The advantage of continuing to use the SCSI protocol is that the transition of SCSI cables o storage networks remains hidden from applications and higher layers of the operating system. SCSI also turns up within the disk subsystems and NAS servers used in storage networks.

 

FREE TUTOR ON STORAGE LUN MASKING AND AVAILABILITY OF DISK SUBSYSTEMS

FREE TUTOR ON STORAGE LUN MASKING  AND AVAILABILITY OF DISK SUBSYSTEMS

So-called LUN masking brings us to the third important function – after instant copy and remote mirroring – that intelligent disk subsystems offer over and above that offered by RAID. LUN masking limits the access to the hard disks that the disk subsystem exports to the connected server.A disk subsystem makes the storage capacity of its internal physical hard disks available
to servers by permitting access to individual physical hard disks, or to virtual hard disks created using RAID, via the connection ports. Based upon the SCSI protocol, all hard disks – physical and virtual – that are visible outside the disk subsystem are also known
as LUN (Logical Unit Number).Without LUN masking every server would see all hard disks that the disk subsystem pro-vides.

A disk subsystem without LUN masking to which three servers are connected. Each server sees all hard disks that the disk subsystem exports outwards.As a result, considerably more hard disks are visible to each server than is necessary.

AVAILABILITY OF DISK SUBSYSTEMS
In particular, on each server those hard disks that are required by applications that runon a different server are visible. This means that the individual servers must be verycarefully configured. In Figure 2.23 an erroneous formatting of the disk LUN 3 of server1 would destroy the data of the application that runs on server 3. In addition, some operating systems are very greedy: when booting up they try to draw to them each harddisk that is written with the signature (label) of a foreign operating system.Without LUN masking, therefore, the use of the hard disk must be very carefullyconfigured in the operating systems of the participating servers. LUN masking brings order to this chaos by assigning the hard disks that are externally visible to servers. As  result, it limits the visibility of exported disks within the disk subsystem.shows how LUN masking brings order to the chaos of Figure 2.23. Each server now sees only the hard disks that it actually requires. LUN masking thus acts as a filter between the exported hard disks and the accessing servers.It is now no longer possible to destroy data that belongs to applications that run on another server. Configuration errors are still possible, but the consequences are no longer so devastating. Furthermore, configuration errors can now be more quickly traced since the information is bundled within the disk subsystem instead of being distributed over all servers.We differentiate between port-based LUN masking and server-based LUN masking.Port-based LUN masking is the 'poor man's LUN masking', it is found primarily in low-end disk subsystems. In port-based LUN masking the filter only works using the granularity of a port. This means that all servers connected to the disk subsystem via the
same port see the same disks.Server-based LUN masking offers more flexibility. In this approach every server sees only the hard disks assigned to it, regardless of which port it is connected via or which other servers are connected via the same port.

 AVAILABILITY OF DISK SUBSYSTEMS
Disk subsystems are assembled from standard components, which have a limited fault-tolerance. In this chapter we have shown how these standard components are combined in order to achieve a level of fault-tolerance for the entire disk subsystem that lies sig-
nificantly above the fault-tolerance of the individual components. Today, disk subsystems can be constructed so that they can withstand the failure of any component without databeing lost or becoming inaccessible. We can also say that such disk subsystems have no
'single point of failure'.The following list describes the individual measures that can be taken to increase the availability of data:
• The data is distributed over several hard disks using RAID processes and supple-mented by further data for error correction. After the failure of a physical hard disk,the data of the defective hard disk can be reconstructed from the remaining data and
the additional data.46 INTELLIGENT DISK SYSTEMS• Individual hard disks store the data using the so-called Hamming code. The Hamming code allows data to be correctly restored even if individual bits are changed on the hard disk. Self-diagnosis functions in the disk controller continuously monitor the rate of bit errors and the physical variables (temperature sensors, spindle vibration sensors).
In the event of an increase in the error rate, hard disks can be replaced before datais lost.• Each internal physical hard disk can be connected to the controller via two internal I/O channels. If one of the two channels fails, the other can still be used.• The controller in the disk subsystem can be realized by several controller instances. If one of the controller instances fails, one of the remaining instances takes over the tasks of the defective instance.• Other auxiliary components such as power supplies, batteries and fans can often beduplicated so that the failure of one of the components is unimportant. When connect-ing the power supply it should be ensured that the various power cables are at leastconnected through various fuses. Ideally, the individual power cables would be supplied
via different external power networks; however, in practice this is seldom realizable.• Server and disk subsystem are connected together via several I/O channels. If one of the channels fails, the remaining ones can still be used.• Instant copies can be used to protect against logical errors. For example, it would be possible to create an instant copy of a database every hour. If a table is 'accidentally'
deleted, then the database could revert to the last instant copy in which the database is still complete.• Remote mirroring protects against physical damage. If, for whatever reason, the original data can no longer be accessed, operation can continue using the data copy that was generated using remote mirroring.This list shows that disk subsystems can guarantee the availability of data to a very high degree. Despite everything it is in practice sometimes necessary to shut down and switch off a disk subsystem. In such cases, it can be very tiresome to co-ordinate all project groups to a common waiting window, especially if these are distributed over different
time zones.Further important factors for the availability of an entire IT system are the availabilityof the applications or the application server itself and the availability of the connection between application servers and disk subsystems. Chapter 6 shows how multipathing can improve the connection between servers and storage systems and how clustering canincrease the fault-tolerance of applications.