Homebrew SAN – Expanding capacity and bandwidth

So far my SAN configuration is holding up very well. Performance is better than most similar setups that I have read about on other blogs and forums. It can however become a much more powerful storage backbone, and that’s where I am taking it next.

Current configuration:

Server:

I am currently running a dual-quad Intel Xeon server board (S5000XVNSATA) with a pair of 2.33GHz E5345 processors and 4GB RAM. This seems to be overkill at the moment, I am using a tiny percentage of this system’s capacity with Openfiler. However with the addition of 10gE and aggregated gigabit networking (quad links) I am anticipating putting much more demand on the system itself.

RAID Array:

At the moment I am using a 3Ware 9550SXU-8LP, this is a fantastic top end controller card with 8 SATA II ports (3Gb/sec). Should these ports be saturated, in theory we should see 300MB/sec from each port or 2400MB/sec from the array. Right now I only have one drive per port, totalling 8 SATA II drives. These are Hitachi server drives, 500GB capacity each.

Chassis:

A large Antec Titan chassis fits 10 drives total, two system drives and the RAID. A 1200W PSU provides the juice.

New configuration:

Server:

I will keep the current server board and processors.

RAID Array:

I will keep the current controller card, but add port multipliers to each port and expand the array to 24 SATA II drives of either 500GB or 1TB per drive depending on the cost of the drives. I intend to use Hitachi Ultrastar A7K2000 drives. This puts 3 drives on each controller port. This will give a theoretical saturation of each port, and at 1TB per drive will provide 24TB storage at well over 1GB/sec, in fact it may deliver close to 2GB/sec depending on the configuration and type of data. I will configure the array as RAID 5 as it is currently.

Chassis:

I have chosen the Chenbro RM51924B, a 5U 24-Bay High Density Storage Server Chassis with a 3+1 1350W redundant PSU.

Networking:

This is where it gets interesting. There will be two networks, one will be dedicated to ISCSI traffic, and one to general LAN traffic and management. The main high bandwidth pipe will be 10Gb Ethernet, or 10gE over CX-4 copper connections to a Dell Powerconnect 6224 managed switch. This switch provides 4 x 10gE connections and 24 gigabit connections.

Up to three client workstations can enjoy a 10gE link to the SAN, but there will also be a second high bandwidth pipe for ISCSI traffic thanks to aggregated quad gigabit (803.2ad). This should give us up to 400MB/sec over gigabit network infrastructure at far less cost than the 10gE, which due to the limited ports of the Dell switch, will be limited to the SAN server and three client connections.

The motherboard on-board dual gigabit NICs will be used for normal network traffic and will be connected to a separate gigabit switch.

Homebrew SAN – Openfiler

Next up is Openfiler. Openfiler can run as a VM or natively as it includes it’s own operating system. In my case I started running it on Xenserver as a VM.

I found Openfiler to be a bit more difficult to configure than Xenserver. This was mostly due to the fact I had never set up or even used iSCSI before.

The first task was for Openfiler to see the RAID volume in the first place. In order for Openfiler to see it, Xenserver has to first see it, and mount it as a virtual storage repository, this is where the CLI work took place as Xenserver did not automatically mount the two RAID partitions even though it did recognize the 3ware controller card. The two partitions were found easily enough and VMFS repositories were created for Xenserver to make the volume available to it’s VM’s, specifically Openfiler.

It gets a bit confusing when you dive into what’s really going on here. Xenserver mounts the two RAID partitions and formats them as EXT3, a standard Linux filesystem. However, it manages any number of virtual partitions in this space which can themselves be used as RAW volumes by VM’s.

In the case of Openfiler, one assigns the volume to a group, which is assigned iSCSI file system and then assigned a iSCSI target. Once the volume is mounted via a iSCSI initiator, in Windows for instance, Windows see’s the space and will want to format it NTFS.

Now, the limitation of NTFS of course is that it’s not a clustered file-system. Only one Windows mount can be accomodated at any time. iSCSI is meant to allow multiple simultaneous mounts, and if the volume is formatted in a clustered filesystem such as GFS, it will operate as a true SAN… allbeit in a Linux only environment.

My read/write tests gave me 238MB/sec over teamed dual Gigabit LAN, but I have a feeling that the link aggregation wasn’t really working as I know for a fact that my switch doesn’t support it.

Homebrew SAN – Virtualization

For my IT challenged readers, I will attempt to explain each step in building my storage server in layman terms.

First up is virtualization. This has been a hot topic in IT circles for some time, especially since the performance and capacity of relatively inexpensive off the shelf server hardware has expanded in leaps and bounds.

It used to be that for every mundane but mission critical server task, a seperate dedicated physical computer was required, this was because the server was often taxed to capacity with it’s given task. Also, the software systems running these tasks often require exclusive access to the servers physical resources.

What has happened more recently is that the performance of a single physical server with multiple CPU cores and lots of fast RAM is equal or greater to a whole rack full of older servers. The idea behind virtualization is to run a very lightweight, fast operating system on the server which mirrors it’s underlying physical hardware to as many “virtual machines” or “VM’s” as the system can handle.

As far as the VM is concerned, it is a physical machine, and whatever operating system, services and software it’s running is totally unaware that it may be sharing resources with other VM’s.

There are many advantages to server virtualization. A few off the top of my head are listed below.

1. Multiple VM’s can run on one physical high performance server, and the VM Server itself as well as the VM’s can all be monitored and managed remotely.

2. Resources can be shifted on the fly between VM’s while they are running.

3. A VM can crash and reboot without upsetting the whole system or the underlying physical server.

4. A VLAN or Virtual LAN is a virtual ethernet network internal to the server connecting all the VM’s to each other and the outside physical LAN using only the server’s physical network ports (NIC). As many virtual IP addresses as necessary are assigned to VM’s and these IP addresses are totally accessible from the outside just as if they were separate physical machines. This free’s up valuable physical network infrastructure.

5. Electricity and space are saved in the server room, and much less heat is generated.

Citrix Xenserver

I started my build with Citrix Xenserver. This is a popular open source server virtualization solution. Some Linux experience is a major plus as I had to mount volumes from the CLI (command line interface) which might be tricky for the uninitiated.

Once installed, the Windows based XenCentre is fantastic and intuitive. It’s easy to get VM’s configured and running in very little time.