Storage Network Technology

Fibre Channel supports three topologies: switched fabric, point-to-point, and arbitrated loop. Point-to-point refers to connecting two devices without the benefit of a network. Although fairly uncommon, it is sometimes used when there is no need to share devices but distance or performance is a problem. It has become less common since the advent of SCSI Ultra3 and SCSI Ultra 360.

Switched fabric, usually referred to simply as fabric, uses a Fibre Channel switch to provide full-bandwidth connections between nodes in the network. Fabrics may consist of one or many switches, depending on scale, availability, and cost considerations. Fibre Channel fabrics also provide other network services. Switches implement naming, discovery, and time services as part of the fabric. This is different from many other network architectures. Unlike Ethernet or IP, these are required to be implemented in the switch. DNS for IP networks does not require that it be part of a switch or router and is often a separate device. In Fibre Channel fabrics, these services are integrated into the fabric.

The terms fabric and switch are often used interchangeably. They are not the same and should not be used as such. Fabric refers to the topology and services, but not a device. A switch is a network device that implements the fabric.

The third Fibre Channel topology is called Fibre Channel Arbitrated Loop, or just loop. Arbitrated loops were conceived of as an inexpensive way to implement a SAN by eliminating the relatively expensive switch, along with its integrated services. In an Arbitrated Loop, all nodes are connected in a loop, with frames passing from one node to the next. All nodes on the loop share the available bandwidth, which inhibits the scalability of Arbitrated Loop. The more nodes that are transmitting on the network, the less bandwidth is available for any individual node. As fabric switches have become less costly, Arbitrated Loop has fallen from favor. It is used mostly inside storage devices such as arrays.

Fibre Channel Addressing

All Fibre Channel nodes carry an address called a World Wide Name (WWN). The WWN is a unique 64-bit identifier. Much like an Ethernet MAC address, part of the WWN is unique to manufacturer of the equipment, and the rest is usually a serialized number. However it is created, the WWN is unique and specific to a physical port.

The I/O Stack

A 64-bit address is very large. Needng to have a frame carry two of these one for the source and one for the destination makes routing packets between ports cumbersome. To combat this problem, Fibre Channel also uses an alternative addressing scheme within Fibre Channel switched fabrics. Each port is assigned a 24-bit port address within the fabric when the port logs in. There are two advantages to this. One, it is faster to route packets on a smaller address (and takes less processor time). Second, the addresses are dynamically assigned and managed by the fabric operating system. This also makes it easy and faster for the OS to deal with changes in the fabric.

Changing the World Wide Name

Although it would seem that the WWN is immutable, there are some instances in which it can be changed. WWNs are often placed in nonvolatile memory (NVRAM) and as such can be changed, given the right utility. A utility such as this would need to be available at boot time, before the port was fully initialized. In the early days of Fibre Channel, it was not uncommon to find host bus adapters with this capability.

Why would anyone want to do such a thing? Having the same WWN would have a similar effect to having the same MAC address in an Ethernet environment. At least one of the devices would not be able to log into the network, or devices would become confused as to the origin of a frame.

The reason this facility sometimes exists is that the NVRAM that holds the World Wide Name can become corrupted, making the port unusable. This used to happen with host bus adapters placed in poorly shielded computers. It is a dangerous utility to have around and should be locked away, where no one can get at it.

Some devices do this on purpose. In certain multiport devices, some ports are kept offline in case a port fails. If one does, the spare port becomes active with the original port's WWN. This makes it look like the original port to the network. This method of failover has the advantage of not requiring hosts or applications to do anything. On the other hand, I/O is usually lost, and some applications may fail during the changeover.

Storage Area Network

Extending SANs over MAN and WAN

SANs based on Fibre Channel are isolated installations. Because Fibre Channel is a switched, not routable, protocol, it cannot be routed over a wide area network or metropolitan area network. There are many good reasons to want to connect SANs over a long distance, data protection being chief among them. Copying blocks of data in near real time over a distance is a key component of data protection. Fibre Channel can reach to 2 kilometers, which is enough to get across a campus or over a river. However, doing so requires the laying or leasing of a dedicated fiber optic cable. That can be extremely costly. The solution is to find ways to interact with public networks available from telecommunications providers. That provides a balance between cost and function.

There are several ways to extend a SAN beyond its own cable limits by using public networks. The most popular is to change the transport by using the IP network to carry an FC frame. A protocol called FC-IP was developed to do this. The FC frame, including its payload, is encapsulated in an IP packet. The frame can now be routed over a public network. At its destination, the FC frame is stripped out and placed once again onto the Fibre Channel network. There are several SAN appliances that perform this function, as well as blades that integrate into Fiber Channel switches. A similar protocol called FC-BB sends FC frames over SONET.

Another way to get FC packets over a WAN or MAN is to crack open the frame; pull out the data; and place it in another type of packet, such as iSCSI. Converting to and from another type of protocol can be processor intensive and can lead to interesting address-mapping issues, but several products do this.

Finally, raw bits can be sent over an optical network. Optical switches convert electrical signals to wavelengths of light and send them out over a fiber optic cable. Several optical switches have blades available that convert physical Fibre Channel signals to optical and then send them over optical fiber. Flow control and other network functions have to be handled by the Fibre Channel switch. The optical network basically acts as very long, fat, fast cable.

Extending IP SANS

The problems of routing FC frames do not exist for IP SANs. IP SANs use a routable transport protocol, and IP switches already have interfaces for WANs and MANs. What remains to be seen is whether applications can tolerate the latency inherent in a WAN or MAN.

Key Points 👍

• Direct Attach Storage was the original storage architecture. The term DAS came later. One of the first disk systems was called Direct Access Storage Devices, or DASD, and was popular in the IBM mainframe environment. The choices of storage architecture changed in the 1990s with the introduction of Network Attached Storage (NAS) and Storage Area Networks (SAN).
• Hard drives are the primary online storage media, with high speed and high capacity. Tape, CD-ROM/RW, DVD-ROM/RW, and magneto-optical systems are used mainly for backup and archive, as well as software distribution.
• Aggregation into large systems provides benefits in speed, logical capacity, and data protection. Removable media libraries and jukeboxes reduce the chance of error, increase availability, and allow multiple computers to access different media simultaneously.
• When data is accessed directly as blocks, it is called block I/O. If it is accessed through a file system, it is referred to as file I/O.
• RAID is a way of increasing performance and data protection by writing and reading data to multiple disks at the same time. Major RAID functions including striping, the writing of different data to many disks simultaneously, and mirroring, which is the writing of the same data to several disks.
• SCSI is a high-performance standard for transferring data to and from devices. It is used extensively for mass storage, and it encompasses both hardware specifications and a software protocol.
• ATA is the most popular storage technology today. It is used in most desktop and laptop computers. It is an in-the-box technology, almost never used to attach storage externally. There is a new serial implementation called SATA or Serial ATA.
• Network Attached Storage (NAS) devices are highly optimized file servers. They use standard protocols to communicate with a large number of clients. They provide high performance, can be quite scalable, and are easy to install and inexpensive to maintain. NAS uses a file head, sometimes called the NAS head, to provide a file system, management, and an interface to the network.
• A SAN is a storage architecture that performs block I/O over a network. SANs have advantages over DAS in terms of distance capabilities, address space, the ability to support many-to-many device configurations, better cable plan management, greater scalability, and higher availability.
• Fibre Channel (FC) is a high-speed, low-latency technology that marries networks with I/O channels. It is often used for SANs. Fibre Channel supports three topologies: fabric, arbitrated loop, and point-to-point. Fabric is the most common and allows for full-bandwidth connections between all nodes in the network. It also implements naming, discovery, and time protocols as part of the fabric.