Homebrew SAN – Expanding capacity and bandwidth

So far my SAN configuration is holding up very well. Performance is better than most similar setups that I have read about on other blogs and forums. It can however become a much more powerful storage backbone, and that’s where I am taking it next.

Current configuration:

Server:

I am currently running a dual-quad Intel Xeon server board (S5000XVNSATA) with a pair of 2.33GHz E5345 processors and 4GB RAM. This seems to be overkill at the moment, I am using a tiny percentage of this system’s capacity with Openfiler. However with the addition of 10gE and aggregated gigabit networking (quad links) I am anticipating putting much more demand on the system itself.

RAID Array:

At the moment I am using a 3Ware 9550SXU-8LP, this is a fantastic top end controller card with 8 SATA II ports (3Gb/sec). Should these ports be saturated, in theory we should see 300MB/sec from each port or 2400MB/sec from the array. Right now I only have one drive per port, totalling 8 SATA II drives. These are Hitachi server drives, 500GB capacity each.

Chassis:

A large Antec Titan chassis fits 10 drives total, two system drives and the RAID. A 1200W PSU provides the juice.

New configuration:

Server:

I will keep the current server board and processors.

RAID Array:

I will keep the current controller card, but add port multipliers to each port and expand the array to 24 SATA II drives of either 500GB or 1TB per drive depending on the cost of the drives. I intend to use Hitachi Ultrastar A7K2000 drives. This puts 3 drives on each controller port. This will give a theoretical saturation of each port, and at 1TB per drive will provide 24TB storage at well over 1GB/sec, in fact it may deliver close to 2GB/sec depending on the configuration and type of data. I will configure the array as RAID 5 as it is currently.

Chassis:

I have chosen the Chenbro RM51924B, a 5U 24-Bay High Density Storage Server Chassis with a 3+1 1350W redundant PSU.

Networking:

This is where it gets interesting. There will be two networks, one will be dedicated to ISCSI traffic, and one to general LAN traffic and management. The main high bandwidth pipe will be 10Gb Ethernet, or 10gE over CX-4 copper connections to a Dell Powerconnect 6224 managed switch. This switch provides 4 x 10gE connections and 24 gigabit connections.

Up to three client workstations can enjoy a 10gE link to the SAN, but there will also be a second high bandwidth pipe for ISCSI traffic thanks to aggregated quad gigabit (803.2ad). This should give us up to 400MB/sec over gigabit network infrastructure at far less cost than the 10gE, which due to the limited ports of the Dell switch, will be limited to the SAN server and three client connections.

The motherboard on-board dual gigabit NICs will be used for normal network traffic and will be connected to a separate gigabit switch.