I apologize to everyone for the looooonnnggg wait for this post. I have swapped out a few hardware components as well as OS for this NAS build. It wasn’t all pretty… and at times it felt frustrating… However, my NAS box and I have lived to tell the story.
In this post I will walk you through each step of the building process, so you won’t make the same mistakes I did. I have written two other posts detailing the thought-process on some of the choices, but for your convenience I’ll sum everything up in this one post.
The Goal
The story here is simple – we are building a DIY storage appliance that is not only FASTER but CHEAPER than the Drobo. I have owned a 2nd generation Drobo for about 8 months. For the most part, it did its job. However, all the weakness of the device surfaced when I dug into the world of Network Attached Storage or simply NAS. I have written about my attempts to put the Drobo on my LAN … http://frankleng.me/2010/02/11/droboshare-alternatives-tonidoplug-gigabit-switch/ I did not want to spend another $200 for the Droboshare! That setup worked for the most part, but performance was not even close to production-worthy.
I mean try to live with a 4TB NAS that transferred at 3-5MB/s… which is actually on par with Droboshare’s throughput… need I say more?
What I needed was:
- Energy efficient, lower powered machine. (It’s a NAS after all, not a media center)
- Able to let me swap a drive when it fails, and not lose data.
- Able to expand, swapped out the smallest drive and replace with a larger drive.
- Costs a lot less than $350USD. (that’s the price of the Drobo, not including drives)
- Easily manageable, settings can be tweaked and system status can be monitored.
- Speed, Speed, Speed. Must be able to sustain Gigabit throughput, even for larger files.
The Facts
Most BYOD (Bring Your Own Drives) storage devices on the market today do NOT offer the features that the Drobo does. The Drobo was never designed as a NAS, but rather a DAS (Directly Attached Storage), and it did that job fairly well. So if you are an average user who favour ease-of-use than anything else… then read no more… you are better off with the Drobo. However, if you are like me and you want that blazing fast performance for the smallest price tag… you have found the right place. =)
NAS devices are computers too. The basic rules do apply – the faster the chipset the faster the throughput; the more RAM the better, etc.
Lastly, and most importantly NAS devices’ throughput depend largely on caching and the CPU’s ability to translate between different storage and communication protocols.
Intel D510MO
The Hardware
CPU
It is really quite difficult to find a blazing fast computer by today’s standard without killing your electricity bill. Most high end processors run 100W + and even lower end chips easily go over 40W (That’s almost the same as the Drobo’s power consumption with 4 drives inserted… so it’s quite a lot for a single chip!) Thankfully, Intel created the Atom series of processors. Originally made for netbooks and mobile devices, the Atom series chips are Intel’s smallest and greenest – consume only about 10W. You can buy Atom from most local computer stores, but note that they are bundled with a motherboard and not sold separately.
I picked the latest generation, and the fastest Dual-Core Atom chip -D510 @ 1.66GHZ. This chip gives the equivalent performance of a Single-Core Celeron running at 900-1000MHZ but uses 1/4 of the power, TDP rated at 13W. Because the chip is so new, very few motherboard manufacturers have them shipped to stores. I grabbed the Intel D510MO motherboard bundle for $80 CAD.(It was the cheapest option I found, and had the same features as the more expensive ASUS and Supermicro boards).
RAM
The D510MO takes DDR2 memory, and I happened to have two 1GB sticks. Obviously the more the better, DDR2 RAM sticks are super cheap nowadays and you don’t need the fancy ones. 2GB RAM – $30
Modded PCI SATA Controller
Add-ons
I want to talk about add-on cards because you will need one for this build. Like most Mini-ITX boards, the D510MO only has 2 onboard SATA ports and it’s not enough to have a more robust ZFS setup. Therefore, I searched online and bought a PCI-X/PCI SATA II controller card and gave me two more internal SATA ports. Later, I modded one of the the eSATA ports to SATA by replacing the connector head. The card was the SYBA SD-SATA2-2E2I – $36. There is a 4 SATA port version of the card, but I wasn’t able to find one at the store. I also liked the idea of having an eSATA port.
CHENBRO ES34069
Case
I went through 2 cases for this build. I originally bought the Chenbro ES34069 because it had 4 swappable drive bays. However, the proprietary power supply, the lack of space in the case and noise from the case fans(fans were not user replaceable as far as I could tell) eventually made me return the purchase.
The case requirement here is simple, find a small case (mATX or mini-ITX) that has 4 drive bays. Do note that besides the Chenbro case, there really isn’t an alternative that has swappable bays. So go spend the $200 and buy it if you want ease of access. The case I ended up using was the Antec NSK-2480 – $100. It has plenty of space for the two 5.25′ and 3.25′ bays and a 380W standard power supply. It has the best cooling arrangement I have ever seen in a media center case. Head over to SilentPcReview for some professional opinions.
Antec NSK2480
I also considered the NSK 1380 case, but spacing arrangement inside the case was nowhere near what the NSK 2480 had. However, the first does have much smaller dimensions.
Antec NSK1380
Drives
First of all, you will need a set of drives to store data and one other drive to host the OS. I have purchased 4 WD Green 2TB 5400rpm drives because they are quiet, efficient and more than fast enough for a NAS. However, do know that you will need at least 3 drives to take full advantage of ZFS. Also, the current version of ZFS does not allow you to add physical drives to an existing ZFS pool. Although you will be able to REPLACE one of the four drives for a bigger one, adding a 5th drive without rebuilding is not possible at the moment. This means if you started out with 4 drives, you will always have 4 unless you recreate your logic pool and lose all the data in the process.
I guess this is the only spot where Drobo may have an edge over our ZFS build. You can add more drives into the Drobo at any time until the slots are full. I believe Drobo accomplishes this by pre-populating its storage pool with virtual devices, and a virtual total size. That way, any new drive can just slide into these pre-made virtual slots without affecting the storage pool as a whole. Lastly, the ZFS development team has plans to resolve this limitation in the near future. So let’s stay tuned.
For the OS drive I decided to use USB sticks. They are much cheaper and much more energy efficient than having another disk spinning at 5400rpm to keep the system running. I used a USB port expansion cable to make use the connector on the motherboard and keep the sticks inside the case. Check out the picture and you’ll know what I mean. =p
Adapter with dual USB sticks
Accessories
The D510MO board does not have an IDE port to hook up a CD-ROM drive. It does support booting from USB. However, most OSes do not offer a .usb image for install. Therefore, I purchased an IDE to USB adapter so I could boot from my DVD drive. My IDE to SATA adapter was not recognized by BIOS for some reason… so please be aware if you are thinking about getting one of these things. It’s safer to get a SATA optical drive instead. I see them on sale now for under $20. Anyway, since FreeNAS supports USB images, there was no need for any additional accessories. (Note: Other OSes mentioned above will require a optical drive to install.)
Total cost: $246 (shipping included, taxes not included)
Next up… the software. Turn the page. =p
Popularity: 100% [?]
Thank you for the interesting article. I have have been thinking about the Drobo FS vs Linux based NAS for a while, sounds like FreeNAS may be a winner.
thanx for dropping by. not sure exactly what hardware are inside the DroboFS but Engadget mentioned that it has a dual-core CPU, more RAM than the original Drobo and a revised Linux kernel. so I'd say it's not too much different than what I've got. =p
but for $699 (empty DroboFS) the DIY setup costs 1/3 of that.
can i add drives with different size to the array ?
you sure can. its just like a drobo.
Thanks for your reply.
Will it be like raid 1 or 5 ?
it's like Raid 5, but much more improved. ex. no more RAID 5 write hole
Thanks alot , Great guide !
Seems a pretty good device
runs really well. been on ever since it was completed and never had to reboot once and never had any problems.
It's also super quiet! no more crazy fan noise from the Drobo. I keep the box in my bedroom and I don't ever notice it. =p
Another question that is on my mind , if the main HHD (the one that holds the freenas) , can i move an existing array to another machine or freenas installation ?
you mean if the OS drive fails?
After your freeNAS is all configured. go to system backup it'll save everything into a XML file. you can simply import it into another freeNAS installation. takes seconds.
also I use a USB stick as OS drive. It's much much more energy efficient than a HDD, and it's more reliable.
The USB edition of FreeNAS only writes to the USB stick when the system is shutting down or saving changes. so the lifetime of the stick is preserved.
thanks a lot again , you are very helpful
Hey, I looked for the software followup post but didn't see it. What OS are you running on this box?
Hi Howard, it's on the 2nd page. http://frankleng.me/2010/05/01/zfs-powered-nas-...
I'm going to test again with Nexenta soon, and see how it performs against the FreeNAS build.
Hello Frank,
I also have a wild comment to try out.
As a Sys Admin BSD/Linux/Solaris, I find this solution very interesting.
The only concern or wish I have for home use is for a case similar to those used by Thecus for example or by Drobo.
)
The one you chose at the end from Antek, can it be placed vertically or provide hot-swappable bays ?
If so, do you know of a slick alternative and by this I mean:
1. cool design
2. small size
3. vertical position
4. hot swappable bays
5. low-noise (I also want to sleep in the same room
Can you advise here with an alternative ?
Hi Mihai. The Chenbro case I mentioned in the post will suit your needs a bit better. but it is a pricey case, and its exhaust fan was a bit loud. so I decided to return it.
The Antek case cannot be placed vertically because there is no rubber foot to stand it up, and it's not very stable vertically either. I have tried it.
Another interesting solution is go for the Intel Entry Storage machine. -http://www.smallnetbuilder.com/nas/nas-reviews/30482-intel-entry-storage-system-ss4200e-review
I think manufactures are using the term “hot-swap” very losely these days, as opposed to calling them “cold-swap”. I guess SATA by nature is hot-swappable and ZFS will still run if you unplug a drive while the system is live. Personally. I don't think it's a particular safe practice to put in a drive like you do with the Drobo. For a home setup, I would just power off and replace the drive as I usually do, aka cold swap. =)
hope that helps!
Hello Frank,
Thanks for the quick answer.
I took a look at the Intel storage solution, but I still want ZFS and my own homemade, geeky setup, not something prebuilt
Do you have any other recommendation similar to the Chenbro case ? (but with fan control and a power supply that can be easily replaced in case of a failure ?)
Also, for the mainboard what would you suggest in case I'd want to be able to connect both a max. of 4 sata hdds and 2 SSDs (I was thinking at having one for ZIL and one for ARC).
the only case I can think of off the top of my head is a barebone 1U or 2U rack with swappable bays. obviously those don't come cheap.
I also thought about going with a tower case and just buy 5.25' to 3.5' adapters with a hot-swappable tray inside.
I was looking at these and was very surprised at how inexpensive they are:
http://www.dealextreme.com/details.dx/sku.19704...
no problem. let me know how ur build turns out.
Hi Frank,
What is the process for swapping out one HDD for a larger one?
the process is almost exactly like the drobo.
If the drive is not dead, you first issue a “scrub” command in ZFS to allow it to check the integrity of the data on the ZFS pool. just a precaution to make sure everything is intact as it should be.
Then should down the computer, swap out the drive you want to replace with a larger drive.
Go back into FreeNAS and issue the ZFS replace command to instruct the system that you are replacing a member of the ZFS with a larger drive.
The system will again check for data integrity and start copying redundancy data onto the new drive.
FYI: I’m using OpenSolaris with ZFS and I’m able to add additional drives to my ZFS pool. You can’t ever remove a drive from a pool (well, you can replace it), but its easy to add one.
I can be wrong. but the implementation of ZFS on Solaris and FreeBSD (what FreeNAS runs on) are two completely different beasts. The FreeBSD version will always be a bit behind Solaris.
Again, I’m not completely certain about this as I do not have any more room in the NAS to actually add more drives in. However, from my research so far it is not possible to just add a new device into an existing ZFS pool on FreeNAS.
Awesome write up. I’ve been looking at the Drobo stuff and wondering if I could build something similar and this is by far the best comparison I’ve come across.
The ability to add drives is still making me lean towards Drobo, but at least I’ve got a good idea of what the alternatives are.
Many thanks.
thanx forkless. glad to know u find it useful.
I’m sure ZFS and FreeNAS will catch up soon to the drobo. I think how the drobo software works is that they prefill the system with virtual drives, so once a drive is inserted it takes over the virtual device. thus allowing u to add drives. and there are only a fixed amount of drive bays, so this method works.
At the end it’s about keeping our data safe. I still like the ZFS approach a lot better. it’s open source, super stable. vs. the Drobo’s proprietary system that nobody else uses. if your drive is in trouble, you can save it. when the drobo is in trouble… you need to buy another one just to recover the data.
Go back into FreeNAS and issue the ZFS replace command to instruct the system that you are replacing a member of the ZFS with a larger drive.