Basics Hardware components for Fibre Channel SAN

Basics Hardware components for Fibre Channel SAN

Within the scope of this book we can only introduce the most important product groups. It is not worth trying to give an overview of specific products or a detailed description of individual products due to the short product cycles. This section mentions once again some product groups that have been discussed previously and introduces some product groups that have not yet been discussed. It is self-evident that servers and storage devices are connected to a Fibre Channel network. In the server this can be achieved by fiting the host bus adapter cards (HBAs) of different manufacturers, with each manufacturer offering different HBAs with differing performance features. In storage devices the same HBAs are normally used. However, the manufacturers of storage devices restrict the selection of HBAs. Of course, cables and connectors are required for cabling. In Section 3.3.2 we discussed different copper and fiber-optic cables and their properties. Various connector types are currently on offer for all cable types. It may sound banal, but in practice the installation of a Fibre Channel SAN is sometimes delayed because the connectors on the cable do not fit the connectors on the end devices, hubs and switches and a suitable adapter is not

to hand. A further, initially improbably, but important device is the so-called Fibre Channel-to- SCSI bridge. As the name suggests, a Fibre Channel-to-SCSI bridge creates a connection between Fibre Channel and SCSI (Figure 3.30). These bridges have two important fields of application. First, old storage devices often cannot be converted from SCSI to Fibre Channel. If the old devices are still functional they can continue to be used in the Fibre Channel SAN by the deployment of a Fibre Channel-to-SCSI bridge. Second, new tape libraries in particular often initially only support SCSI; the conversion to Fibre Channel is often not planned until later.With a Fibre Channel-to-SCSI bridge the newest tape libraries can be operated directly in a Fibre Channel SAN and Fibre Channel connections retrofitted as soon as they become available. Unfortunately, the manufacturers have not agreed upon consistent name for this type of device. In addition to Fibre Channel-to-SCSI bridge, terms such as SAN router or storage gateway are also common. The switch is the control centre of the fabric topology. It provides routing and aliasing, name server and zoning functions. Fibre Channel switches support both cut-through routing and the buffering of frames. In new switches a number of ports between eight and about 250 and a data transfer rate of 200 MByte/s should currently (2003) be viewed as standard. In Fibre Channel SANs that have already been installed, however, a large base of switches exists that still work at 100 MByte/s.

 

 

Resilient, enterprise-class switches are commonly referred to as 'directors', named after the switching technology used in mainframe ESCON cabling. Like Fibre Channel switches they provide routing, alias names, name server and zoning functions. Fibre Channel direc- tors are designed to avoid any single point of failure, having for instance two backplanes and two controllers. Current directors (2003) have between 64 and 256 ports. Designing a SAN often raises the question whether several complementary switches or a single director should be preferred. As described, directors are more fault-tolerant than switches, but they are more expensive per port. Therefore, designers of small entry-level SANs commonly choose two complementary Fibre Channel switches, with mutual traffic fail-over in case of a switch or a I/O path failure (Figure 3.31). Designers of larger Fibre Channel SANs often favour directors due to the number of ports currently available per device and the resulting layout simplicity. However, this argument in favour of directors becomes more and more obsolete since today switches with a greater number of ports are available as well. SANs running especially critical applications, e.g. stock market banking or flight control,

would use complementary directors with mutual traffic failover, even though these directors already avoid internal single points of failure. This is similar to wearing trousers with a belt and braces in addition: protecting against double or triple failures. In less critical cases, a single director or a dual complementary switch solution will be considered sufficient. If we disregard the number of ports and the cost, the decision for a switch or a director in an Open Systems Fibre Channel network primarily comes down to fault-tolerance of

 

an individual component. For the sake of simplicity we will use the term 'Fibre Channel switch' throughout this book in place of 'Fibre Channel switch or Fibre Channel director'. A hub simplifies the cabling of an arbitrated loop. Hubs are transparent from the point of view of the connected devices. This means that hubs send on the signals of the connected devices; in contrast to a Fibre Channel switch, however, the connected devices do not communicate with the hub. Hubs change the physical cabling from a ring to a star-shape. Hubs bridge across defective and switched-off devices, so that the physical

ring is maintained for the other devices. The arbitrated loop protocol is located above this cabling. Hubs are divided into unmanaged hubs, managed hubs and switched hubs. Unman- aged hubs are the cheap version of hubs: they can only bridge across switched-off devices. However, they can neither intervene in the event of protocol infringements by an end device nor indicate the state of the hub or the arbitrated loop to the out- side world. This means that an unmanaged hub cannot itself notify the administrator if one of its components is defective. A very cost-conscious administrator can build up a small SAN from PC systems, JBODs and unmanaged hubs. However, the upgrade path to a large Fibre Channel SAN is difficult: in larger Fibre Channel SANs it is questionable whether the economical purchase costs compensate for the higher administration costs. In contrast to unmanaged hubs, managed hubs have administration and diagnosis functions like those that are a matter of course in switches and directors.Managed hubs monitor the power supply, serviceability of fans, temperature, and the status of the individual ports. In addition, some managed hubs can, whilst remaining invisible to the connected devices, intervene in higher Fibre Channel protocol layers, for example, to deactivate the port of adevice that frequently sends invalid Fibre Channel frames. Managed hubs, like switches and directors, can inform the system administrator about events via serial interfaces, Telnet, HTTP and SNMP (see also Chapter 8). Finally, the switched hub is mid-way between a hub and a switch. In addition to the properties of a managed hub, with a switched hub several end devices can exchange data at full bandwidth. Fibre Channel switched hubs are cheaper than Fibre Channel switches, so in some cases they represent a cheap alternative to switches. However, it should benoted that only 126 devices can be connected together via hubs and that services such as aliasing and zoning are not available. Furthermore, the protocol cost for the connection or the removal of a device in a loop is somewhat higher than in a fabric (keyword 'Loop Initialisation Primitive Sequence', 'LIP'). Finally, so- alled link extenders should also be mentioned. Fibre Channel supports a maximum cable length of several ten kilometres (Section 3.3.2). A link extender can

increase the maximum cable length of Fibre Channel by transmitting Fibre Channel frames using MAN/WAN techniques such as ATM, SONET or TCP/IP (Figure 3.32).When using link extenders it should be borne in mind that long distances between end devices significantly increase the latency of a connection. Time-critical applications such as database transactions should therefore not run over a link extender. On the other hand,Fibre Channel SANs with link extenders offer new possibilities for applications such as back-up, data sharing and asynchronous data mirroring.

Fibre Channel SAN is a comparatively new technology. In many data centres in which Fibre Channel SANs are used, it is currently (2003) more likely that there will be several islands of small Fibre Channel SANs than one large Fibre Channel SAN (Figure 3.33).Over 80% of the installed Fibre Channel SANs consist only of up to four Fibre Channel switches. A server can only indirectly access data stored on a different SAN via the LAN and a second server. The reasons for the islands of small Fibre Channel SANs are that they are simpler to manage than one large Fibre Channel SAN and that it was often unnecessary to install a large one.

Originally, Fibre Channel SAN was used only as an alternative to SCSI cabling. Until now the possibility of flexibly dividing the capacity of a storage device etween several servers (storage pooling) and the improved availability of dual SANs have been the main reasons for the use of Fibre Channel SANs. Both can be realized very well with several small Fibre Channel SAN islands. However, more and more applications are now exploiting the possibilities offered by a Fibre Channel SAN. Applications such as back-up

 

(Chapter 7), remote data mirroring and data sharing over Fibre Channel SAN and storage virtualization (Chapter 5) require that all servers and storage devices are connected via a single SAN. Incidentally, the connection of Fibre Channel SANs to form a large SAN could be one field of application in which a Fibre Channel director is preferable to a Fibre Channel switch (Figure 3.34). As yet these connections are generally not critical. In the future, however, this could change (extreme situation: virtualization over several data centres). In our opinion these connection points between two storage networks tend to represent a single point of failure, so they should be designed to be particularly fault-tolerant.

FREE TUTORS ON FIBRE CHANNEL SAN Point-to-point topology

FREE TUTORS ON FIBRE CHANNEL SAN Point-to-point topology

The previous section introduced the fundamentals of the Fibre Channel protocol stack This section expands our view of Fibre Channel with the aim of realizing storage network with Fibre Channel. To this end, we will first consider the three Fibre Channel topologies point-to-point, fabric and arbitrated loop more closely (Sections 3.4.1. to 3.4.3). We will then introduce some hardware components that are required for the realization of a Fibr Channel SAN (Section 3.4.4). Building upon this, the networking of small storage network islands to form a large SAN will be discussed (Section 3.4.5). Finally, the question interoperability in Fibre Channel SANs will

Point-to-point topology

The point-to-point topology connects just two devices and is not expandable to three or more devices. For storage networks this means that the point-to-point topology connects a server to a storage device. The point-to-point topology may not be very exciting, but it offers two important advantages compared to SCSI cabling. First, significantly greater cable lengths are possible with Fibre Channel than with SCSI because Fibre Channel supports distances up to ten kilometres without repeaters, whilst SCSI supports only up to 25 metres. Second, Fibre Channel defines various fiber-optic cables in addition tocopper cables. Optical transmission via fiber-optic is robust in relation to electromagnetic interference and does not emit electromagnetic signals. This is particularly beneficial in technical environments. Fibre Channel cables are simpler to lay than SCSI cables. For example, the SCSISAN shown in Figure 3.7 can very simply be realized using the point-to-point topology.Application servers for the control of production can be set up close to the production machines and the data of the application server can be stored on the shared storage systems, which are located in a room that is protected against unauthorized access and physical influences such as fire, water and extremes of temperature. 3.4.2 Fabric topology The fabric topology is the most flexible and scalable of the three Fibre Channel topologies.

A fabric consists of one or more Fibre Channel switches connected together. Servers and storage devices are connected to the fabric by the Fibre Channel switches. In theory a fabric can connect together up to 15.5 million end devices. However, Fibre Channel SANs connected to several hundreds of end devices are currently (2003) still the exception. Most installations use two to four switches. There are, however, a few 'power users',who operate significantly larger Fibre Channel SANs. End Devices (servers and storage devices) connected to the various Fibre Channel switches can exchange data by means of switch-to-switch connections (inter-switch links,ISLs). Several inter-switch links can be installed between two switches in order to increase the bandwidth. A transmitting end device only needs to know the Node ID of the target device; the necessary routing of the Fibre Channel frame is taken care of by the Fibre Channel switches. Fibre Channel switches generally support so-called cut-through routing: cut-through routing means that a Fibre Channel switch forwards an incoming frame before it has been fully received. The latency describes the period of time that a component requires to transmit a signal or the period of time that a component requires to forward a frame. Figure 3.23 compares the latency of different Fibre Channel SAN components. Light requires approximately 25microseconds to cover a distance of ten kilometres. A ten kilometre-long Fibre Channel cable thus significantly increases the latency of an end-to-end connection. For hardware

components the rule of thumb is that a Fibre Channel switch can forward a frame in two to four microseconds; a Fibre Channel host bus adapter requires two to four milliseconds to process it. Additional Fibre Channel switches between two end devices therefore only increase the latency of the network to an insignificant degree.One special feature of the fabric is that several devices can send and receive data simultaneously at the full data rate. All devices thus have the full bandwidth available to them at the same time. Figure 3.24 shows a Fibre Channel SAN with three servers and three storage devices, in which each server works to its own storage device. Each of the three logical connections over the Fibre Channel SAN has the full bandwidth of 200MByte/s available to them.A prerequisite for the availability of the full bandwidth is good design of the Fibre Channel network. Figure 3.25 shows a similar structure to that in Figure 3.24, the only difference is that the single switch has been replaced by two switches, which are connected via one inter-switch link (ISL). It is precisely this inter-switch link that represents the limiting factor because all three logical connections now pass through the same inter-switch link. This means that all three connections have, on average, only a third of the maximum bandwidth available to them. Therefore, despite cut-through routing, switches have a certain number of buffers (frame buffers) available to them, with which they can temporarily bridge such bottlenecks. However, the switch must still reject valid frames if the flow control does not engage quickly enough. In addition to routing, switches realize the basic services of aliasing, name server and

zoning. As described in Section 3.3.6, end devices are differentiated using 64-bit WorldWide Node Names (WWNNs) or by 64-bit World Wide Port Names (WWPNs) and addressed via 24-bit port addresses (N-Port ID). To make his job easier the administratorcan issue alias names to WWNs and ports.

throughput of the three connections is limited by the ISL The name server supplies information about all end devices connected to the Fibre Channel SAN (Section 3.3.7). If an end device is connected to a switch, it reports to this and registers itself with the name server. At the same time it can ask the name server which other devices are still connected to the SAN. The name server administers end devices that are currently active; switched off end devices are not listed in the name server. Finally, zoning makes it possible to define subnetworks within the Fibre Channel net-work. This has two main advantages. First, zoning limits the visibility of end devices. With zoning, servers can only see and access storage devices that lie in the same zone. Zoning therefore helps to protect sensitive data. Furthermore, incompatible Fibre Channel host bus adapters can be separated from each other by different zones. Second, individual ports of a multiport disk subsystem – and thus a certain bandwidth – can be reserved for important applications. The bandwidth of inter-switch links (ISLs) cannot be reserved in

this manner since switches currently (2003) do not support this, or at least not officially. Although the Fibre Channel standard defines service classes that reserve a certain band- width (Classes 1, 4 and 6), these service classes are not implemented in most current Fibre Channel devices. There are many variants of zoning, for which unfortunately no consistent terminology exists. Different manufacturers use the same term for different types of zoning and differ-

ent terms for the same type of zoning. Therefore, when selecting Fibre Channel switches do not let yourself get fobbed off with statements such as 'the device supports hard zon- ing'. Rather, it is necessary to ask very precisely what is meant by 'hard zoning'. In the following we introduce various types of zoning.

In zoning, the administrator brings together devices that should see each other in Fibre Channel SAN into a zone, whereby zones can overlap. Zones are described by World Wide Node Names, World Wide Port Names, port addresses or by their alias names. The description on the basis of WWNNs and WWPNs has the advantage that zoning is robust

in relation to changes in cabling: it does not need to be changed for a device to be plugged into a different switch port. By contrast, zoning on the basis of port addresses must be altered since every port in the switch has a different port address.Soft zoning restricts itself to the information of the name server. If an end device asks the name server about other end devices in the Fibre Channel network, it is only informed of the end devices with which it shares at least one common zone. If, however, an end device knows the address (Port ID) of another device, it can still communicate with it. Soft zoning thus does not protect access to sensitive data. Soft zoning is problematic in relation to operating systems that store the WWNs of Fibre Channel devices that have been found in an internal database or in which WWNs are announced in configuration files because this means that WWNs remain known to the operating system even after

a system reboot. Thus in soft zoning operating systems continue to have access to all known devices despite changes to the zoning, regardless of whether they lie in a common zone or not. Hard zoning offers better protection. In hard zoning only devices that share at least one common zone can actually communicate with one another. Both hard zoning and soft zoning can be based upon port addresses or WWNs. Nevertheless, port-based zoning is sometimes known as hard zoning. Some more modern Fibre Channel switches support LUN masking – described in Section 2.7.3 in relation to disk subsystems – within the switch. To achieve this they read the first bytes of the payload of each Fibre Channel frame. Although reading part of the Fibre Channel payload increases the latency of a Fibre Channel switch, this increase in latency is so minimal that it is insignificant in comparison to the latency of the host bus adapter in the end devices.

So-called virtual storage area networks (virtual SAN, VSAN) represent a further inno- vation. In this technique, several ports or WWNs and thus several end devices of a Fibre Channel fabric, are grouped together to form a virtual fabric. This means that several virtual Fibre Channel fabrics that are logically separate from one another can be operated over one physical Fibre Channel network. In addition, separate fabric services such asname server and zoning are realized for each virtual storage network. In addition to pure zoning, virtual storage networks thus not only limit the mutual visibility of end devices but also the mutual visibility of the fabric configuration. This is particularly advantageous in installations which aim to offer storage services for various customers over a consol-

idated infrastructure. Here, in particular, it is not desirable for a customer to be able to read which end devices belonging to other customers are still connected in the storage network, or even change their configuration, over the name server.

FREE TUTOR ON STORAGE LUN MASKING AND AVAILABILITY OF DISK SUBSYSTEMS

FREE TUTOR ON STORAGE LUN MASKING  AND AVAILABILITY OF DISK SUBSYSTEMS

So-called LUN masking brings us to the third important function – after instant copy and remote mirroring – that intelligent disk subsystems offer over and above that offered by RAID. LUN masking limits the access to the hard disks that the disk subsystem exports to the connected server.A disk subsystem makes the storage capacity of its internal physical hard disks available
to servers by permitting access to individual physical hard disks, or to virtual hard disks created using RAID, via the connection ports. Based upon the SCSI protocol, all hard disks – physical and virtual – that are visible outside the disk subsystem are also known
as LUN (Logical Unit Number).Without LUN masking every server would see all hard disks that the disk subsystem pro-vides.

A disk subsystem without LUN masking to which three servers are connected. Each server sees all hard disks that the disk subsystem exports outwards.As a result, considerably more hard disks are visible to each server than is necessary.

AVAILABILITY OF DISK SUBSYSTEMS
In particular, on each server those hard disks that are required by applications that runon a different server are visible. This means that the individual servers must be verycarefully configured. In Figure 2.23 an erroneous formatting of the disk LUN 3 of server1 would destroy the data of the application that runs on server 3. In addition, some operating systems are very greedy: when booting up they try to draw to them each harddisk that is written with the signature (label) of a foreign operating system.Without LUN masking, therefore, the use of the hard disk must be very carefullyconfigured in the operating systems of the participating servers. LUN masking brings order to this chaos by assigning the hard disks that are externally visible to servers. As  result, it limits the visibility of exported disks within the disk subsystem.shows how LUN masking brings order to the chaos of Figure 2.23. Each server now sees only the hard disks that it actually requires. LUN masking thus acts as a filter between the exported hard disks and the accessing servers.It is now no longer possible to destroy data that belongs to applications that run on another server. Configuration errors are still possible, but the consequences are no longer so devastating. Furthermore, configuration errors can now be more quickly traced since the information is bundled within the disk subsystem instead of being distributed over all servers.We differentiate between port-based LUN masking and server-based LUN masking.Port-based LUN masking is the 'poor man's LUN masking', it is found primarily in low-end disk subsystems. In port-based LUN masking the filter only works using the granularity of a port. This means that all servers connected to the disk subsystem via the
same port see the same disks.Server-based LUN masking offers more flexibility. In this approach every server sees only the hard disks assigned to it, regardless of which port it is connected via or which other servers are connected via the same port.

 AVAILABILITY OF DISK SUBSYSTEMS
Disk subsystems are assembled from standard components, which have a limited fault-tolerance. In this chapter we have shown how these standard components are combined in order to achieve a level of fault-tolerance for the entire disk subsystem that lies sig-
nificantly above the fault-tolerance of the individual components. Today, disk subsystems can be constructed so that they can withstand the failure of any component without databeing lost or becoming inaccessible. We can also say that such disk subsystems have no
'single point of failure'.The following list describes the individual measures that can be taken to increase the availability of data:
• The data is distributed over several hard disks using RAID processes and supple-mented by further data for error correction. After the failure of a physical hard disk,the data of the defective hard disk can be reconstructed from the remaining data and
the additional data.46 INTELLIGENT DISK SYSTEMS• Individual hard disks store the data using the so-called Hamming code. The Hamming code allows data to be correctly restored even if individual bits are changed on the hard disk. Self-diagnosis functions in the disk controller continuously monitor the rate of bit errors and the physical variables (temperature sensors, spindle vibration sensors).
In the event of an increase in the error rate, hard disks can be replaced before datais lost.• Each internal physical hard disk can be connected to the controller via two internal I/O channels. If one of the two channels fails, the other can still be used.• The controller in the disk subsystem can be realized by several controller instances. If one of the controller instances fails, one of the remaining instances takes over the tasks of the defective instance.• Other auxiliary components such as power supplies, batteries and fans can often beduplicated so that the failure of one of the components is unimportant. When connect-ing the power supply it should be ensured that the various power cables are at leastconnected through various fuses. Ideally, the individual power cables would be supplied
via different external power networks; however, in practice this is seldom realizable.• Server and disk subsystem are connected together via several I/O channels. If one of the channels fails, the remaining ones can still be used.• Instant copies can be used to protect against logical errors. For example, it would be possible to create an instant copy of a database every hour. If a table is 'accidentally'
deleted, then the database could revert to the last instant copy in which the database is still complete.• Remote mirroring protects against physical damage. If, for whatever reason, the original data can no longer be accessed, operation can continue using the data copy that was generated using remote mirroring.This list shows that disk subsystems can guarantee the availability of data to a very high degree. Despite everything it is in practice sometimes necessary to shut down and switch off a disk subsystem. In such cases, it can be very tiresome to co-ordinate all project groups to a common waiting window, especially if these are distributed over different
time zones.Further important factors for the availability of an entire IT system are the availabilityof the applications or the application server itself and the availability of the connection between application servers and disk subsystems. Chapter 6 shows how multipathing can improve the connection between servers and storage systems and how clustering canincrease the fault-tolerance of applications.