If your IT department is being squeezed by the downturn in the economy but still needs to add let's say a storage area network (SAN) then why not do it yourself (DIY). Sure you could contact your favorite hardware vendor like Dell, HP, IBM etc., but when you compare the costs of an off the shelf product compared to the DIY project you'll realize big savings, in our case around 50%.
My company currently has a need for SAN. Based on our needs we decided to go with a 24 TB SATA in 4U form. Total cost for parts, taxes and shipping $9,500 Canadian. Comparable vendor systems start at around $30,000.
We generally buy Supermicro boards and chassis as they're good quality at a good price. Our operating system (OS) of choice is FreeBSD. Here's the basic specifications we used.
A 24 disk storage array consisting of the following hardware:
- SUPERMICRO MBD-X7DWN+ Dual LGA 771 Intel 5400 Enhanced Extended ATX Server Motherboard
- SUPERMICRO SC846TQ-R900B storage chassis
- 2 x Intel Xeon E5410 Quad Core 2.33GHZ LGA771 12MB 1333FSB Harpertown Retail Processor
- 2 x 4GB RAM (see all compatible Crucial DIMMs for this board, and see notes below about RAM requirements)
- Areca ARC-1680ix-24 24-port RAID controller
- Areca ARC-6120 Battery Backup Module
- 1 x 2GB cache memory module for Areca RAID card
- 24 1TB SATA II drives (current candidate is Western Digital Green Power 1TB, model Caviar GP WD10EACS)
- 1 x Western Digital Caviar Black 500GB boot drive (this resides on the motherboard controller)
- 1 x 4GB or 8GB compact flash card as maintenance boot device (resides in compact flash slot of motherboard)
- 1 x compact flash adapter (it might be worth it to get several as failsafe bootables)
NOTES:
- Sufficient memory needs to be onboard for fsck of 20+TB, and large filesystems can consume a large amount of RAM or not work at all without it. 1GB RAM per 1TB storage is a common rule, so running 8GB is a lower limit, not an upper limit.
- We also need reliable GPT support for this size of an array, which is not there yet on many OSs.
- X7DWN+ is not listed on Supermicro's BSD compatibility matrix, but people report OK use under Free BSD 7.0 REL
- A useful discussion about compatible generic memory for the Areca controller
- Areca manuals, benchmarks, firmwares etc.
- Drive compatibility with the RAID controller needs to be established. For compatible Western Digital drives this requires adjusting the TLER values in each drive's bios prior to using it in the RAID array. Note the DOS utility rewrites the BIOS of all attached drives, so this should only be done an a system dedicated to this task. Other non-RAID certified drives may need similar adjustments.
- Western Digital reports the 1TB GP as compatible with the Areca.
- As of 12 November 2008, only a handful of 1TB drives are certified by Areca and several common ones are explicitly NOT compatible. In the Western Digital line only the WD RE2 GP RAID Edition 1TB is certifed, with SAS controller firmware V1.45 80915 or later. Unfortunately such RAID-certified drives are nearly double the cost of non-RAID variants.
- Western Digital Green Power (WD GP) drives are only 5400 RPM but reports suggest that differences in read/write performance between hard drive brands becomes less important as more drives are added to the array.
- A test report of an Areca 1680-type SAS controller
- This 2GB memory module reported to be compatible with a 1680 controller. Worth getting.
- Without the battery backup the Areca cards turn off write-caching completely to help prevent data loss.
OTHER NOTE:
- use http://www.shopbot.ca/ to find the lowest price on gear above for price matching purposes. Some items in the SAN parts list are already linked to shopbot.
Thanks for sharing this info Marc. I'm curious why you didn't use Open Solaris for this given ZFS and DTrace. I would have thought ZFS would be particularly useful for a SAN.
We generally use FreeBSD but have not decided 100% on the OS as of yet. We may yet use Open Solaris.
I think most of the value in storage area network solutions these days is in the software, not the hardware. Most of the big name players are moving or already using industry standard hardware and the magic is in their software.
Justin Lofton
VP of Engineering
Tredent Data Systems, Inc.
Storage Area Networking Experts
Mark, what OS and filesystem did you eventually settle upon? I noticed from your other posts that you prefer FreeBSD, so I was hoping to find out if you have started working with FreeBSD and ZFS. We're currently using Solaris with the Areca ARC-1680ix-24 controller, and it works well with ZFS. However, with Oracle's decision this month to sell future Solaris support contracts only to sites that use Oracle/Sun hardware, we have to move off Solaris since all of our hardware is white box and we're not due for a hardware refresh for another 4-5 years.
Our first choice of an alternative architecture is FreeBSD 8.0 with ZFS, using the same Areca ARC-1680ix-24 controller, but it seems very few if any users are trying this combination of OS, filesystem, and controller, so I'm looking for experience reports before we start our testing in a couple months. We heavily use the iSCSI and SMB share properties of ZFS, and plan on using FreeBSD's istgt and Samba to replace that functionality.
Our testing will consist of running the Solaris and FreeBSD configurations side-by-side for six months (we have cold spares of the controllers that we'll bring out of storage for the testing), mirroring the static files over with rsync, then gradually ramping up the application load onto the FreeBSD configuration each month, with copious tape backups every step of the way. We'll report our test results to the FreeBSD community, as this kind of large-scale storage configuration seems an underreported area in FreeBSD forums.
FreeBSD with ZFS is the way to go. ZFS on FreeBSD was not stable enough when we originally set ours up but it is now.