ZFS powered NAS, ultimate alternative to Drobo + Droboshare [Complete Guide]

There is a newer version of this build. Check it out!

I apologize to everyone for the looooonnnggg wait for this post. I have swapped out a few hardware components as well as OS for this NAS build. It wasn’t all pretty… and at times it felt frustrating… However, my NAS box and I have lived to tell the story.
In this post I will walk you through each step of the building process, so you won’t make the same mistakes I did. I have written two other posts detailing the thought-process on some of the choices, but for your convenience I’ll sum everything up in this one post.

The Goal
The story here is simple – we are building a DIY storage appliance that is not only FASTER but CHEAPER than the Drobo. I have owned a 2nd generation Drobo for about 8 months. For the most part, it did its job. However, all the weakness of the device surfaced when I dug into the world of Network Attached Storage or simply NAS. I have written about my attempts to put the Drobo on my LAN …   http://frankleng.me/2010/02/11/droboshare-alternatives-tonidoplug-gigabit-switch/ I did not want to spend another $200 for the Droboshare! That setup worked for the most part, but performance was not even close to production-worthy.
I mean try to live with a 4TB NAS that transferred at 3-5MB/s… which is actually on par with Droboshare’s throughput… need I say more?

What I needed was:

  1. Energy efficient, lower powered machine. (It’s a NAS after all, not a media center)
  2. Able to let me swap a drive when it fails, and not lose data.
  3. Able to expand, swapped out the smallest drive and replace with a larger drive.
  4. Costs a lot less than $350USD. (that’s the price of the Drobo, not including drives)
  5. Easily manageable, settings can be tweaked and system status can be monitored.
  6. Speed, Speed, Speed. Must be able to sustain Gigabit throughput, even for larger files.

The Facts
Most BYOD (Bring Your Own Drives) storage devices on the market today do NOT offer the features that the Drobo does. The Drobo was never designed as a NAS, but rather a DAS (Directly Attached Storage), and it did that job fairly well. So if you are an average user who favour ease-of-use than anything else… then read no more… you are better off with the Drobo. However, if you are like me and you want that blazing fast performance for the smallest price tag… you have found the right place. =)

NAS devices are computers too. The basic rules do apply – the faster the chipset the faster the throughput; the more RAM the better, etc.

Lastly, and most importantly NAS devices’ throughput depend largely on caching and the CPU’s ability to translate between different storage and communication protocols.

Intel D510MO

The Hardware

CPU
It is really quite difficult to find a blazing fast computer by today’s standard without killing your electricity bill. Most high end processors run 100W + and even lower end chips easily go over 40W (That’s almost the same as the Drobo’s power consumption with 4 drives inserted… so it’s quite a lot for a single chip!) Thankfully, Intel created the Atom series of processors. Originally made for netbooks and mobile devices, the Atom series chips are Intel’s smallest and greenest – consume only about 10W. You can buy Atom from most local computer stores, but note that they are bundled with a motherboard and not sold separately.

I picked the latest generation, and the fastest Dual-Core Atom chip -D510 @ 1.66GHZ. This chip gives the equivalent performance of a Single-Core Celeron running at 900-1000MHZ but uses 1/4 of the power, TDP rated at 13W. Because the chip is so new, very few motherboard manufacturers have them shipped to stores. I grabbed the Intel D510MO motherboard bundle for $80 CAD.(It was the cheapest option I found, and had the same features as the more expensive ASUS and Supermicro boards).

RAM
The D510MO takes DDR2 memory, and  I happened to have two 1GB sticks. Obviously the more the better, DDR2 RAM sticks are super cheap nowadays and you don’t need the fancy ones. 2GB RAM – $30

Add-ons

Modded PCI SATA Controller

I want to talk about add-on cards because you will need one for this build. Like most Mini-ITX boards, the D510MO only has 2 onboard SATA ports and it’s not enough to have a more robust ZFS setup. Therefore, I searched online and bought a PCI-X/PCI SATA II controller card and gave me two more internal SATA ports. Later, I modded one of the the eSATA ports to SATA by replacing the connector head. The card was the SYBA SD-SATA2-2E2I – $36. There is a 4 SATA port version of the card, but I wasn’t able to find one at the store. I also liked the idea of having an eSATA port.

CHENBRO ES34069

CHENBRO ES34069

Case
I went through 2 cases for this build. I originally bought the Chenbro ES34069 because it had 4 swappable drive bays. However, the proprietary power supply, the lack of space in the case and noise from the case fans(fans were not user replaceable as far as I could tell) eventually made me return the purchase.

The case requirement here is simple, find a small case (mATX or mini-ITX) that has 4 drive bays. Do note that besides the Chenbro case, there really isn’t an alternative that has swappable bays. So go spend the $200 and buy it if you want ease of access. The case I ended up using was the Antec NSK-2480 – $100. It has plenty of space for the two 5.25′ and 3.25′ bays and a 380W standard power supply. It has the best cooling arrangement I have ever seen in a media center case. Head over to SilentPcReview for some professional opinions.

Antec NSK2480

Antec NSK2480

I also considered the NSK 1380 case, but spacing arrangement inside the case was nowhere near what the NSK 2480 had. However, the first does have much smaller dimensions.

Antec NSK1380

Antec NSK1380

Drives
First of all, you will need a set of drives to store data and one other drive to host the OS. I have purchased 4 WD Green 2TB 5400rpm drives because they are quiet, efficient and more than fast enough for a NAS. However, do know that you will need at least 3 drives to take full advantage of ZFS. Also, the current version of ZFS does not allow you to add physical drives to an existing ZFS pool. Although you will be able to REPLACE one of the four drives for a bigger one, adding a 5th drive without rebuilding is not possible at the moment. This means if you started out with 4 drives, you will always have 4 unless you recreate your logic pool and lose all the data in the process.

I guess this is the only spot where Drobo may have an edge over our ZFS build. You can add more drives into the Drobo at any time until the slots are full. I believe Drobo accomplishes this by pre-populating its storage pool with virtual devices, and a virtual total size. That way, any new drive can just slide into these pre-made virtual slots without affecting the storage pool as a whole. Lastly, the ZFS development team has plans to resolve this limitation in the near future. So let’s stay tuned.

For the OS drive I decided to use USB sticks. They are much cheaper and much more energy efficient than having another disk spinning at 5400rpm to keep the system running. I used a USB port expansion cable to make use the connector on the motherboard and keep the sticks inside the case. Check out the picture and you’ll know what I mean. =p

Adapter with dual USB sticks

Accessories
The D510MO board does not have an IDE port to hook up a CD-ROM drive. It does support booting from USB. However, most OSes do not offer a .usb image for install. Therefore, I purchased an IDE to USB adapter so I could boot from my DVD drive. My IDE to SATA adapter was not recognized by BIOS for some reason… so please be aware if you are thinking about getting one of these things. It’s safer to get a SATA optical drive instead. I see them on sale now for under $20. Anyway, since FreeNAS supports USB images, there was no need for any additional accessories. (Note: Other OSes mentioned above will require a optical drive to install.)

Total cost: $246 (shipping included, taxes not included)

Next up… the software.  Turn the page.  =p

  • http://twitter.com/AaronJAnderson Aaron Anderson

    :::At last, the limitation to adding drives has to do with the FreeBSD port of ZFS. It will be fixed in the upcoming version of FreeNAS based on the FreeBSD 8.0 kernel:::

    I am very excited about this… is there information anywhere about this?

  • http://twitter.com/AaronJAnderson Aaron Anderson

    At last, the limitation to adding drives has to do with the FreeBSD port of ZFS. It will be fixed in the upcoming version of FreeNAS based on the FreeBSD 8.0 kernel.

  • http://www.frankleng.me Frank Leng

    no word on this yet. and no confirmation if this has made its way into the latest stable port of ZFS.

    my understanding is that ZFS has always been difficult to expand. so it’ll be interesting to see what the FreeNAS team can accomplish with the FreeBSD 8.1 release.

    The easiest way to expand the array is still by swapping out your drives. so to be safe please assume swapping is the only way.

  • http://twitter.com/AaronJAnderson Aaron Anderson

    I think I’m going to go with a big stack of 2tb disks, an MDADM raid5 array on Ubuntu and the iscsitarget package.

    I’m using unRAID now… the performance is good if you use a Cache drive. And it’s nice to have any size disk in there… but iSCSI is a must for me.

  • Steve

    Hi Frank, interesting read, I’ve been using the D510 mobo for about 6 months as a ‘NAS’. I was using a ReadyNAS previously but found the software buggy and I wanted a more flexible system. I’m running Ubuntu as I use it as a ‘citrix’ server to (using FreeNX), in addition to that I use it as a CCTV system (kmotion), an IDS with the alienvault distro running in a virtualbox VM and cacti for monitoring stuff! So the system is working hard :-)

    Can you tell me which PCI riser card you bought as I need to get one?

    Thanks

    Steve

  • http://www.facebook.com/HarshReality Steven Machado

    Dont suppose youd be willing to hang your FreeNAS config file here would you? I have almost the exact same setup and would be curious as to the subtle differences in our rigs.

  • http://www.frankleng.me Frank Leng

    sorry. my config file has all the workgroup credentials in there so I can’t post.
    tho all the critical configs/tweaks are mentioned in the blog post.

  • Gradiator

    Thanks for the article. It’s been tremendously helpful.

    I first came across your blog while searching for a DroboPro alternative and, regrettably in hindsight, dismissed a FreeNAS solution as too time-consuming (in terms of assembly)… After all, the whole Drobo appeal is that you plug it in, and it just works. Further research, however, revealed a growing number of Drobo users dissatisfied with their product’s lack of stability. I went back to square one and once again stumbled upon your site–intent this time on giving FreeNAS the consideration it deserves (mostly out of appreciation for its ZFS support, whose data scrubbing, insane theoretical capacity, and copy-on-write principles I consider necessary; in particular, Drobo fails to provide any scrubbing features, which I now hold of paramount importance).

    I am having trouble finding the right enclosure/motherboard combo for the
    unit–which is where I decided to start searching for parts, as I require
    at least 8 drives and figured this would be the bottleneck in terms of
    selection. All the cases seem to come with built-in RAID controllers
    (offering, obviously, less comprehensive solutions than FreeNAS).

    After a cursory search, I found the 790FX-GD70 motherboard by MSI:

    http://www.newegg.com/Product/Product.aspx?Item=N82E16813130223

    Would it be feasible to gut an 8-bay RAID box and throw this motherboard in to make it FreeNAS? It might be considered “overkill” for NAS, with all the onboard features. In particular, I’d feel a little funny having what’s essentially an external hard drive chug along wielding more power than my actual computer… Regardless, I was wondering more about everyone’s thoughts in regards to 8-bay FreeNAS boxes in general. What sort of pitfalls should be avoided? Are there any other special conerns of which one should be aware?

    If I even remotely sound like I know what I’m doing, please understand that I don’t. I’ve never built a computer before (though I’m hoping now to build all my future computers, given how useful the information I’ve uncovered on this little search has been)… I do, however, have the determination to conceive and implement the right solution–virtually regardless of investment time and capital… It’s somewhat of an obsession.

    Thanks for your time.

  • Dork

    You should have added something ECC compatible if you going with ZFS.

  • Dork

    And you could have used a CF reader with 8gb running at 3.3v

  • Dork

    AMD 7xx/8xx chipset allows use of ecc + super low power 20-25w dual/quad cores processors.

  • Anonymous

    is it still not possible to add new drives without rebuilding everything?

  • http://twitter.com/AaronJAnderson Aaron Anderson

    Don’t think so.

  • http://aaroneiche.com Aaron Eiche

    Hi Frank,

    As others have mentioned, I think your throughput bottleneck isn’t the NM10. I think it’s the PCI card. I think the PCI limit is slowing down your whole disk array. Unless I’ve got it wrong, the max speed on 133mhz pci card is 266MB/s. Hardly the Sata II spec of 3Gbps. Because ZFS has to stripe the data across all 4 drives, you’re slowed down to the max throughput on the PCI card. You’ll definitely get better performance out of the mini pcie slot.

    I bring it up because my current NAS (which I’ve just transitioned into a 1U server case I got on-the-cheap) suffers from the same issue. I figure for a lot of things (backups) the speed isn’t a tremendous issue. My next board though is going to need 4 sata connectors to avoid just this issue.

  • Anonymous

    would a PicoPSU be enough to handle the board and 4 x 2 TB drives?

  • http://twitter.com/AaronJAnderson Aaron Anderson

    I wouldn’t run that many disks….

  • CarloS

    Hello, someone is still using the Zotac ION-ITX-G motherboard?
    Did you get good performance from it?
    Thanks

  • Frank Leng

    no experience with it. tho the reviews I have read on Zotac boards haven’t been great. esp considering the ongoing BIOS support and updates.

  • http://www.frankleng.me Frank Leng

    I just got a standard 90 degree right angle PCI riser. but didn’t need it after I switched cases.

  • http://www.frankleng.me Frank Leng

    my recommendation would be to get the droboPro if you want ease of use and have the cash for it. Drobo’s pro products are actually quite decent in terms of features and ease-of-use. but they are incredibly expensive comparing to a DIY setup.

    You do no need to have a RAID card if you are going with FreeNAS and ZFS. simply buy a motherboard with 8 SATA ports and you are good to go.

    hope that helps.

  • http://www.frankleng.me Frank Leng

    good point. I’ll look into that

  • http://www.frankleng.me Frank Leng

    Hey Aaron,
    that is a very good point. the card I bought supports PCI-X which is 166mhz I think. but I after reading your post made me realize that the X standard was exclusive to older server boards… and the NM10 didn’t support it.

    I searched everywhere for a mini-PCIE SATA adapter and did not have any luck. I think my next build will be something with at least 4 ports.

    thanx again for the tip. I’ll update the article accordingly!

  • http://www.frankleng.me Frank Leng

    Hi. like Aaron said. I wouldn’t push it that hard. tho it could work. esp if you use WD Green drives or the equivalent.
    those should use less than 20W on peak access. I believe normal drives are 20-30W.
    so you are still within the limit of the 120W PSU. Google says they also make a 150W version. I would go for the 150 just to be safe.

    good luck

  • http://www.frankleng.me Frank Leng

    have heard some horrible stories about Zotac boards. esp when they rarely update their BIOS and warranty is sub-par
    I would stay away from it unless it has everything you need out of the box, and it’s cheaper than everything else.

  • http://www.facebook.com/massayoshi Flávio Massayoshi Oota

    does this eSATA-SATA mod require soldering? or can I just use an eSATA-SATA cable?

  • Anonymous

    I was looking at using the new AMD Zacate board (MSI E350IS-E45). Is this board compatible with FreeNas?

  • http://twitter.com/laconismo Guilherme

    Hi Frank. I think I found a mini-PCIE sata adapter with 2 ports.
    Check this out!
    http://cgi.ebay.co.uk/Mini-PCI-e-PCI-Express-Internal-SATA-II-Controller-Card-/220721563113#ht_2355wt_800

  • http://www.frankleng.me Frank Leng

    all x86 boards should work. so anything that’s compatible with AMD, Intel and VIA.
    FreeNAS is built to work with basic I/O standards. nothing really fancy.

    hope that helps

  • http://www.frankleng.me Frank Leng

    it requires soldering. but an E-SATA to SATA cable will get around that.

  • http://www.frankleng.me Frank Leng

    Looks like the real deal. but it costs the same as the Atom board it self. ~$70USD. :(

  • Anonymous

    Yea, I just got everything set up with my new zacate board and 8gb of Ram. I just did a test and was getting 100 Mbps…awesome. I am glad I did this instead of drobo or any other nas server.

  • http://profiles.google.com/webbernet Frank Leng

    sounds like u r using the USB image and not an actual install.

    I mentioned this in my post.

    “load /cf/boot/loader.conf (if you are using the USB embedded FreeNAS version). Otherwise, load /boot/loader.conf”

    hope that helps

  • http://www.frankleng.me Frank Leng

    sounds like u r using the USB image and not an actual install.

    I mentioned this in my post.

    “load /cf/boot/loader.conf (if you are using the USB embedded FreeNAS version). Otherwise, load /boot/loader.conf”

    hope that helps

  • Anonymous

    Yea I it is embedded on a USB drive but it is not in the cf folder either. Does it matter I used Unetbootin to embed it on my usb drive?

  • Anonymous

    You can purchase it on MallExtreme, look for HiPhone 4G Quad Band Dual Cards Dual Cameras WiFi Bluetooth Java 3.5 – inch Capacitive Touch Screen
    . I’m not sure if this blog allows direct linking, if it does, just click here

    http://www.mallextreme.com/hiphone-4g-quad-band-dual-cards-dual-cameras-wifi-bluetooth-java-3-5-inch-capacitive-touch-screen_p25378.html

  • Looking_for_ultimate_NAS

    Actually, I wouldnt consider the solution(s) offered an alternative.. since the whole point of Drobo appliances is they are simple “appliances”; where you just plug it in and pop in any old sata disk; replacing disks only when it tells you they are bad or when you want to increase the partition size.

    The solution offered here is similar to someone saying an alternative to a microwave you buy at costco would be to buy parts and try to put together a microwave yourself.

  • http://www.frankleng.me Frank Leng

    Other than buying the parts, the whole setup process takes less than a few hours. but the savings are quite significant.

    More importantly, I the drobo and the DIY NAS are not even on the same level of performance. if you buy the original drobo + droboplug. you spend twice as much. for less than 1/3 of the throughput. you can do the math. even the droboFS cannot sustain the same level of throughput as the DIY NAS. this is a common limitation of these so-called NAS appliances. they have at most 512mb of RAM. the box we build has at least 2GB for caching transfer.

    again you can do the math. in my opinion. if I can build a much better, less expensive microwave than common commercial offerings. I would certainly do so.

  • http://twitter.com/naaronne Nicholas Aronne

     Hi Frank,

    I wanted to know if you thought that the System Tweaks you recommended for performance, would still yield any results on FreeNAS 8.  I am looking to improve my I/O and before I took the bold and blinded approach to just changing my loader.conf.  I was hoping I could get your 2 cents on it.

    thanks in advance.

  • http://www.frankleng.me Frank Leng

     Hi Nicholas, FreeNAS 8 has many of these tweaks built-in. Many harmful configs have also been removed. So I doubt those tweaks I mentioned would yield any results.

    I also question if FreeNAS 8 would need any tweaks out of the box. the only tweaks I would recommend you look into are the ones for CIFS/SMB, not the system tweaks for FreeNAS.

  • NASGuy

    Thanks for the great write-up. 2 questions:
    1. Is it still not possible to add a drive? I mean, let’s say you have a setup with 3 HDDs (with data) and wish to add a 4th. When will it be possible to do this without losing data? FreeNAS 8.1? Already?2. Here’s a weird one. Like the Drobo, I wish for the ability to plug this thing directly into a computer (say, via USB/Firewire/Thunderbolt (at a stretch)) and get maximum performance, as well as having it on the network for machines not physically close. Is there any way to set this up, or is it network only?

    On another note, the Array R2 Mini-ITX NAS Case looks perfect for this.

  • NASGuy

    Wow, well the formatting went crazy with that comment. Sorry about that, can’t edit it now though.

  • http://www.frankleng.me Frank Leng

    FreeNAS 8.0 already supports adding new devices to an existing array.
    However it requires significant amount of RAM for the ZFS subsystem to
    perform at optimal conditions. that is to say that 2GB is the absolute
    minimum you should have for a FreeNAS 8.0 system. In fact, FreeNAS
    recommends 4-8GB of RAM.
    Release 8.1 is currently being tested. and you should get either 8.0
    now or wait for 8.1. 7.x releases are NOT upgradable to 8.x.

    Gigabit ethernet is already maxing out your drives performance unless
    you go SSD. There is no way of making your NAS a standard USB/FireWire
    storage device. someone might figure out a way to emulate them down
    the road. but I do not see any benefit in having that. At least not in
    terms of speed.

    Hope that answers ur questions.

    Let me know how ur FreeNAS 8 experience turns out.

  • NASGuy

    Great, thanks! I don’t mind going for about 4GB RAM if it improves things — hell, it’s dirt cheap nowadays anyway. You mentioned that releases can’t eb upgraded to 8.x, but I assume that 8.0 can be upgraded to 8.1, right?

    Why is it that the RAM requirements are so extremely different, anyway?

  • http://www.frankleng.me Frank Leng

    Totally agree. RAM is ridiculously cheap right now. and you do not need the high performance sticks for a NAS anyway.

    Release 8.0 will be upgradable to 8.1 and the system will automatically preserve the configs like it did in 0.7.x. so no worries there.

    Lastly, FreeNAS runs on a FreeBSD kernel called NanoBSD. It has a much more complete port of ZFS (from OpenSolaris) compared to the 0.7.x releases. Specifically, it now supports snapshots, and adding drives to an existing ZFS array.

    If you look at the stuff from OpenSolaris on ZFS technology, it uses tremendous amount of cache/buffer to ensure the system is functioning at optimal levels which data is being moved and duplicated yet still maintaining throughput.
    Regular NAS do not have this problem because they are simply not doing as much work as a ZFS system.

    Ex. DroboFS gives you about 25mb/s at peak speeds when transferring files smaller than 2GB. that speed dies down tremendously when you have files larger than 2GB. Its hardware simply can’t keep up. Now try to have multiple users access the box at the same time… nightmare.
    Now throw in 4GB of RAM on a ZFS system. As long as you have enough network bandwidth, your speed will be consistent.
    People are also getting 80mb/s on ZFS. and that was on FreeNAS 0.7.x
    :)

  • NASGuy

    Thanks again, Frank! Nice explanation right there!

    I’ve decided on a build with 8GB RAM, an Array R2 Mini-ITX case and an Asus E35M1-I Deluxe Mobo. Case looks great and supports 6 HDDs, motherboard has 5 SATA-600 ports and runs the (apparently) excellent AMD Fusion E-350. And hell, RAM is so cheap that it’s worth going for the recommended amount!

    Just thought I’d mention for others looking into decent NAS hardware, as I think it’s a pretty good build (if on the expensive side). I’m particularly satisfied with the case.

  • http://www.frankleng.me Frank Leng

    Not a problem.

    Love your selection of the board and the case. tho the cost probably would be my only concern. lol

  • sims

    Hi Frank, thanks for sharing your experience.

    Much as I am tempted to go into creating a solution like the one you have created, time pressures do not allow me to do so. And neither am I happy with the rather slow performance (Comparitively at least) that I hear Drobo’s suffer from.

    So would you know if there is an option of buying a pre-built computer with such features as the ones you mention. I guess I mean to ask if there is a pre-built FreeNAS computer of this kind that I can buy from somewhere?

  • http://www.frankleng.me Frank Leng

    Not that I’m aware of. I don’t think FreeNAS has a commercial license.

    You can take a look at unRAID
    http://www.lime-technology.com/

    hope that helps

  • eric

    Hi Frank,

    Where did you get your case from?

  • sims

    thank you for the link. too expensive for my needs.
    need to find somebody who can do what you have done :)

    thanks for sharing!

See also: