ZFS powered NAS, ultimate alternative to Drobo + Droboshare [Complete Guide]

There is a newer version of this build. Check it out!

I apologize to everyone for the looooonnnggg wait for this post. I have swapped out a few hardware components as well as OS for this NAS build. It wasn’t all pretty… and at times it felt frustrating… However, my NAS box and I have lived to tell the story.
In this post I will walk you through each step of the building process, so you won’t make the same mistakes I did. I have written two other posts detailing the thought-process on some of the choices, but for your convenience I’ll sum everything up in this one post.

The Goal
The story here is simple – we are building a DIY storage appliance that is not only FASTER but CHEAPER than the Drobo. I have owned a 2nd generation Drobo for about 8 months. For the most part, it did its job. However, all the weakness of the device surfaced when I dug into the world of Network Attached Storage or simply NAS. I have written about my attempts to put the Drobo on my LAN …   http://frankleng.me/2010/02/11/droboshare-alternatives-tonidoplug-gigabit-switch/ I did not want to spend another $200 for the Droboshare! That setup worked for the most part, but performance was not even close to production-worthy.
I mean try to live with a 4TB NAS that transferred at 3-5MB/s… which is actually on par with Droboshare’s throughput… need I say more?

What I needed was:

  1. Energy efficient, lower powered machine. (It’s a NAS after all, not a media center)
  2. Able to let me swap a drive when it fails, and not lose data.
  3. Able to expand, swapped out the smallest drive and replace with a larger drive.
  4. Costs a lot less than $350USD. (that’s the price of the Drobo, not including drives)
  5. Easily manageable, settings can be tweaked and system status can be monitored.
  6. Speed, Speed, Speed. Must be able to sustain Gigabit throughput, even for larger files.

The Facts
Most BYOD (Bring Your Own Drives) storage devices on the market today do NOT offer the features that the Drobo does. The Drobo was never designed as a NAS, but rather a DAS (Directly Attached Storage), and it did that job fairly well. So if you are an average user who favour ease-of-use than anything else… then read no more… you are better off with the Drobo. However, if you are like me and you want that blazing fast performance for the smallest price tag… you have found the right place. =)

NAS devices are computers too. The basic rules do apply – the faster the chipset the faster the throughput; the more RAM the better, etc.

Lastly, and most importantly NAS devices’ throughput depend largely on caching and the CPU’s ability to translate between different storage and communication protocols.

Intel D510MO

The Hardware

CPU
It is really quite difficult to find a blazing fast computer by today’s standard without killing your electricity bill. Most high end processors run 100W + and even lower end chips easily go over 40W (That’s almost the same as the Drobo’s power consumption with 4 drives inserted… so it’s quite a lot for a single chip!) Thankfully, Intel created the Atom series of processors. Originally made for netbooks and mobile devices, the Atom series chips are Intel’s smallest and greenest – consume only about 10W. You can buy Atom from most local computer stores, but note that they are bundled with a motherboard and not sold separately.

I picked the latest generation, and the fastest Dual-Core Atom chip -D510 @ 1.66GHZ. This chip gives the equivalent performance of a Single-Core Celeron running at 900-1000MHZ but uses 1/4 of the power, TDP rated at 13W. Because the chip is so new, very few motherboard manufacturers have them shipped to stores. I grabbed the Intel D510MO motherboard bundle for $80 CAD.(It was the cheapest option I found, and had the same features as the more expensive ASUS and Supermicro boards).

RAM
The D510MO takes DDR2 memory, and  I happened to have two 1GB sticks. Obviously the more the better, DDR2 RAM sticks are super cheap nowadays and you don’t need the fancy ones. 2GB RAM – $30

Add-ons

Modded PCI SATA Controller

I want to talk about add-on cards because you will need one for this build. Like most Mini-ITX boards, the D510MO only has 2 onboard SATA ports and it’s not enough to have a more robust ZFS setup. Therefore, I searched online and bought a PCI-X/PCI SATA II controller card and gave me two more internal SATA ports. Later, I modded one of the the eSATA ports to SATA by replacing the connector head. The card was the SYBA SD-SATA2-2E2I – $36. There is a 4 SATA port version of the card, but I wasn’t able to find one at the store. I also liked the idea of having an eSATA port.

CHENBRO ES34069

CHENBRO ES34069

Case
I went through 2 cases for this build. I originally bought the Chenbro ES34069 because it had 4 swappable drive bays. However, the proprietary power supply, the lack of space in the case and noise from the case fans(fans were not user replaceable as far as I could tell) eventually made me return the purchase.

The case requirement here is simple, find a small case (mATX or mini-ITX) that has 4 drive bays. Do note that besides the Chenbro case, there really isn’t an alternative that has swappable bays. So go spend the $200 and buy it if you want ease of access. The case I ended up using was the Antec NSK-2480 – $100. It has plenty of space for the two 5.25′ and 3.25′ bays and a 380W standard power supply. It has the best cooling arrangement I have ever seen in a media center case. Head over to SilentPcReview for some professional opinions.

Antec NSK2480

Antec NSK2480

I also considered the NSK 1380 case, but spacing arrangement inside the case was nowhere near what the NSK 2480 had. However, the first does have much smaller dimensions.

Antec NSK1380

Antec NSK1380

Drives
First of all, you will need a set of drives to store data and one other drive to host the OS. I have purchased 4 WD Green 2TB 5400rpm drives because they are quiet, efficient and more than fast enough for a NAS. However, do know that you will need at least 3 drives to take full advantage of ZFS. Also, the current version of ZFS does not allow you to add physical drives to an existing ZFS pool. Although you will be able to REPLACE one of the four drives for a bigger one, adding a 5th drive without rebuilding is not possible at the moment. This means if you started out with 4 drives, you will always have 4 unless you recreate your logic pool and lose all the data in the process.

I guess this is the only spot where Drobo may have an edge over our ZFS build. You can add more drives into the Drobo at any time until the slots are full. I believe Drobo accomplishes this by pre-populating its storage pool with virtual devices, and a virtual total size. That way, any new drive can just slide into these pre-made virtual slots without affecting the storage pool as a whole. Lastly, the ZFS development team has plans to resolve this limitation in the near future. So let’s stay tuned.

For the OS drive I decided to use USB sticks. They are much cheaper and much more energy efficient than having another disk spinning at 5400rpm to keep the system running. I used a USB port expansion cable to make use the connector on the motherboard and keep the sticks inside the case. Check out the picture and you’ll know what I mean. =p

Adapter with dual USB sticks

Accessories
The D510MO board does not have an IDE port to hook up a CD-ROM drive. It does support booting from USB. However, most OSes do not offer a .usb image for install. Therefore, I purchased an IDE to USB adapter so I could boot from my DVD drive. My IDE to SATA adapter was not recognized by BIOS for some reason… so please be aware if you are thinking about getting one of these things. It’s safer to get a SATA optical drive instead. I see them on sale now for under $20. Anyway, since FreeNAS supports USB images, there was no need for any additional accessories. (Note: Other OSes mentioned above will require a optical drive to install.)

Total cost: $246 (shipping included, taxes not included)

Next up… the software.  Turn the page.  =p

  • Hector

    Hello again frank,
    I wanted to ask something about the size of the “array” , if I’m using the technique you excellently explained and i have 4 drives : 2x750G and 2x2T , what will be the size of the pool/storage ?

  • .Alasdair

    Hi,

    Firstly, it’s a great article, so thanks! When you say that you have to be able to swap out the smallest drive with a bigger one, how exactly do you do that with FreeNAS/ZFS?

    I’ve been looking into it and found that it will recognise the increase in space only if all 4 drives are swapped out for bigger ones, and there’s a guide to get it to do it with one but you have to partition all the drives into 5 different arrays, which seems like a lot of hassle.

    Have you actually done that yourself?

    Thanks.

  • Hector

    So if its the same as RAID5 , how is is similar to Drobo ?

  • http://www.frankleng.me Frank Leng

    The drobo is the same as RAID5 and RAIDz in that regard. 4 1TB disks will get you roughly 2.7TB of actual usable space. 4 2TB will give you roughly 5.5TB instead.

    These are the same for RAID5, RAIDz and drobo. I don’t think the other offerings out there will give you any extra.

    Drobo provides a space calculator here if you want to test it out.
    http://www.drobo.com/calculator/drobo-s/index.php

  • http://www.frankleng.me Frank Leng

    that will give you roughly 3TB of usable space.
    those 750GB drives will set you back quite a bit in term of capacity.

  • http://www.frankleng.me Frank Leng

    Yep I have done it myself, and successfully swapped out my old 1TB drives in the NAS before getting the 2TB Greens in the mail.
    check out this excellent guide for swapping drives on the fly:
    http://wiki.mattrude.com/index.php/Freenas/ZFS_and_FreeNAS_expansion#Swapping_Drives_On_The_Fly

    The guide does use ZFS commands to accomplish the task in Terminal. However, all the commands are available in the WebGUI as well. you just won’t be able to interact with it very well because it doesn’t always show the output of the commands.
    so I would suggest you SSH into the NAS and replace the drive that way. You do not need to rebuild the pool, you could if you want to. but it takes longer.

  • .Alasdair

    Thanks for the reply, the article that you’ve linked to was in fact the one I was talking about when I said I found a way that involved partitioning all the drives several times – and the amount of effort required to swap just one drive for a larger one is considerable.

    Without this partitioning, the only way you can increase the storage available is if the storage was limited to start with (e.g. you have 1x500GB and 3x1TB drives, then replacing the 500GB with 1TB will increase storage) or if you replace all drives.

    Take the example that I have 1x2TB and 3x1TB and I swap out one of the 1TB drives for another 2TB, the usable storage will not increase at all (unlike doing this in a drobo which will gain you nearly 1TB in space), which is quite disappointing – the only way to use this additional space would be to follow that article through from the top… and if you take a look at it, it is nothing like just swapping the drives like the drobo can do.

    I’ve been playing around with different scenarios using Parallels and have failed to find a way other than I have described.

  • http://www.frankleng.me Frank Leng

    The drobo can do this because it creates many virtual devices and allow each physical drive to fit into a virtual envelope. when you format the drobo. you do it to the maximum possible capacity. so you can format it to 8TB when you only have 4TB of usable space. and you can get away with it that way.
    In ZFS you cannot.

    However, I don’t think you need to repartition over and over. what I did was replace the drive while both are physically attached. then issue a replace command. and resilver the pool.
    see this example:
    http://harryd71.blogspot.com/2008/08/how-to-resize-zfs.html

    I’ll be sure to record what I do next time I’m upgrading my drives. right now I’m maxed out at four 2TB drives.

  • .Alasdair

    Ahh, that explains it then – that method also only works if you replace all 4 drives unfortunately, still – even if I am more limited to what I can do with it, the (literally) 4MB/s I get over FW800 is something that I won’t be missing when all the parts I’ve ordered arrive!

    Thanks again for the article, I’ve decided to use the updated Chenbro (ES34169) and an ASRock mobo with ION chipset and 4 SATA (+2 eSATA) ports :)

  • Hector

    Hi Frank , according to Drobo’s calculator :
    Drives: 2.0 TB + 2.0 TB + 750 GB + 750 GB = 5.5 TB
    Actual size = 5 TB

  • http://www.frankleng.me Frank Leng

    Hector, you have to look at the space in blue marked as “available for data”. because part of the total capacity will have to be reserved for data redundancy. This is why you will get about 3TB of actual storage space. This is true for RAID 5, RAIDz (ZFS) and the drobo. the drobo does no magic. There will always be a trade-off between capacity and protection.

    hope that helps.

  • http://www.frankleng.me Frank Leng

    yep. FreeNAS can boot in numerous generic VM setups.
    tho I cannot comment on the performance of the VM vs. native setups.

  • http://www.frankleng.me Frank Leng

    sorry I didn’t see this until now.
    1. Yes, FreeNAS is perhaps the most actively supported open NAS solution out there. and its ZFS implementation is constantly being improved and worked on. FreeNAS 8.0 will be out soon and bring many enhancements to the system. That being said, the current version 7.2 is a rock solid platform that can run on various type of hardware. For a production machine, esp for photo/videography work. I would recommend getting a full power system instead of an Atom based setup. Consider a low end Celeron, it’s more than powerful enough to run as a NAS and fairly efficient.

    2. if there is a mobo or cpu failure. replace the hardware and you can boot the image from USB on any other machine. You may need to resilver the ZFS pool because the SATA port address will change. but it’s no big deal, a few commands will take care of it. If you bought a drobo… the only way to replace it is to buy another drobo…

    hope that helps.

  • Todd

    So do you have to swap out all drives when you want to “upgrade” to more capacity? I was hoping to start with just two larger drives, say 2 TB and couple those with a pair of drives I have on hand, say for example a pair of 80 GB drives. That way I could later swap out one of the 80s for another 2 TB.

    Is that or is that not possible?

    Thanks again Frank for this great article and for following up with everyone’s comments. This is a great resource.

  • http://www.frankleng.me Frank Leng

    thanx Todd for the kind words.
    no you certainly do not need to swap out all four drives.
    It’s as easy as issuing a “replace” command while having both the old and new drives attached, then “resilver” the storage pool to populate the new drive with data.

    the resilvering will take majority of the time. there is no way to get around that. drobo does the same thing except you don’t have to issue any commands.

  • .Alasdair

    It depends how you want to do it – if you add 2x2TB and 2x80GB to a Raid5 (single parity) vdev and then add it to a zpool, you will only get 234GB of usable space (yes that number is right – I have just done it on a virtual machine to test).

    You could add them as Raid mirrors instead to get 2TB+80GB, but if you upgrade the 80GB drives, you’ll be using more space for overhead than is needed.

    If you do use the Raid5 example, then you will have to upgrade BOTH 80GB drives before you will be able to get the full amount of space, upgrading one will simply keep the space level at 234GB (I have just done it using the method that Frank suggests; plug in new drive, issue replace, then resilver – and the available space only changed after upgrading BOTH drives), but let’s face it, you won’t want to have just 234GB of available data.

    Now please don’t get me wrong, Frank’s article is great and it is because of it that I have decided to build my own, but I do have to disagree with the part where he says that he can ‘swap out the smallest drive and replace it with a larger one’ – this will work if you have 3x1TB drives and a 500GB drive, then swapping out the 500GB for 1TB will increase your space – but if you decided to swap it for a 2TB instead, you won’t get any additional space at all. Similarly, if you have 2x1TB and 2x500GB, you will only get an increase if you upgrade both of them to 1TB drives.

    I have spent 3 days of searching around for this and testing with a virtual machine and I have come across countless forum posts asking for the feature, but no genuine way of doing it without partitioning all the drives into 4 parts like the wiki article that Frank linked to describes.

  • Todd

    Thanks again Frank. It seems like you and Alasdair have different opinions about how upgrading works. Frank I like your answer better. ;-) Seem like I can avoid the headaches if I just get four 2TB drives from the get-go.

  • http://www.frankleng.me Frank Leng

    haha of course. just make sure you have the latest possible version of ZFS and FreeNAS running.
    The previous guide that Alasdair mentioned could be using an older version of ZFS. The one I have is Version 13 and support replacing/resilvering.
    All old versions of ZFS can be upgraded to V13 by issuing an “upgrade” command in FreeNAS.

  • VENAM1

    Hi Frank…

    I know this is a super-old post but I just stumbled upon it as I am looking to do the DIY NAS thing…

    I was wondering how you got 4 HDD’s into the Antec case though?

    Did you stuff 2 of them into the 5.25″ bays?

    Maybe I missed the trick but I was hoping to hear how you tackled that…

    Thanks for the great write-up!

  • VENAM1

    Whoops! Nevermind…;)

  • Anonymous
  • http://www.frankleng.me Frank Leng

    Hi Antheo
    I did take a look at some of the ready made solutions from NetGear. but the performance/price ratio just didn’t make sense to me. even the drobo seemed better.
    so this build was strictly a DIY project. =p

  • Todd

    Hey Frank, I would still be interested to know what the power consumption is on your build.

  • Zamo

    Hi Frank, awesome article, I think I’m going to apply it somewhere in the following months, my question as a total ZFS newbie, why should one have at least 3 drives to fully enjoy the benefits of ZFS ?

    I’m thinking of a 2 x WD 1TB setup, with one drive acting as the main storage point, and the other one acting as a mirror. I think I can squeeze all the stuff I have to backup within 1TB.

  • http://www.frankleng.me Frank Leng

    LOL.
    two adapters were used. very cheap. I got two online for $3 each.

  • http://www.frankleng.me Frank Leng

    Hi Zamo,
    what I meant was that if you only use 2 drives, ZFS offers you nothing more than RAID-1.
    Like I mentioned in the article, I used 4 drives because ZFS in FreeBSD(FreeNAS) does not allow you to add new devices to an existing storage pool. Other words, it’s better to fill your case with the most number of drives and then upgrade them one by one in the future to expand your capacity. (as much as budget allows of course)

    If you only use 2 drives, you would have to upgrade them BOTH in order to achieve a greater capacity. The more drives you have, the less drive you need to upgrade to increase total capacity.

  • Anonymous

    do you mean that with only 2 drives you have to replace them both at the same time to increase the storage space but with 4 drives you can replace them one by one?

  • http://www.frankleng.me Frank Leng

    yep, that’s exactly what I said above.
    the logic is that – when you have a RAID 1 mirror (ZFS or not) the only way to upgrade is by replacing BOTH drives. The drives must be the same size in order to mirror each other.
    With 4 or more drives, you can mix and match, and make use of all drives more efficiently.
    hope that helps.

  • http://www.facebook.com/profile.php?id=540766240 Evan Fredericks

    Great guide. I’ve been looking for a good explanation of the benefits of ZFS and this has helped out alot.

  • http://www.facebook.com/profile.php?id=540766240 Evan Fredericks

    Frank, can you from experience deny what alasdair is saying about the 2TB+2TB+80GB+80GB ~ 240GB usable data (i.e. the usable data contribution of each drive is not limited by the capacity of the smallest drive)? I’m going to be doing a similar setup to this scenario (2TB + 2TB + 750GB + 500GB) so would I be right to expect this to yield ~2.95 TB of usable data as opposed to 500*3~1.5TB of data.

  • Charleschao

    hey frank and todd,
    i’m also interested in the Gigabyte GA-D525TUD mobo, i notice that it does have 4sata but two different/seperate controllers. is there an issue when using this for raid?
    after figuring out which mobo to get the next step is the case, which is the hard part for me. I like the chenbro design/style..can’t find anything else like it. know of something ?

  • http://twitter.com/bbchucks charleschao

    the ASRock mobo, is it ASRock A330ION ? Does the software support this type of mobo?

  • http://twitter.com/bbchucks charleschao

    Hi,
    thanks for this write, you’ve inspired me to build my own instead of spending 700+ bux for the DS1010+ Synology!
    I’m looking to get the GA-H55N-USB3 board with CHENBRO ES34169 case (only 120W). with 4x/2TB seagate drives.
    Do you know if 120W would be sufficient?
    Thanks for your help!

  • Anonymous

    I am building the same machine here: with i3/4GB/Chenbro https://secure.newegg.com/WishList/MySavedWishDetail.aspx?ID=14474185
    $604.94

  • http://www.frankleng.me Frank Leng

    Hi Charles,
    that’s a very good question. the 120W PSU was the main reason why I used the WD Greens. Each drive will use about 35W during peak load. and the mITX motherboard with the Atom will use about 12W. so it’ll be just enough. tho it will be rare that you ever hit that maximum load with a NAS.

    However, the problem is that the Chenbro ES case uses a proprietary power supply. so it will be difficult to find a replacement for it. Also, the Chenbro PSU I had was making a high pitch noise… so I lost confidence in it and return the whole case. but doesn’t mean you won’t find the Chenbro case useful. It’s a tiny little box with a lot of functionality and it is still the best fit for a mITX board.

  • http://www.frankleng.me Frank Leng

    all the ION board should work. FreeNAS is generic enough to support most if not all Atom boards out there.

  • http://www.frankleng.me Frank Leng

    I haven’t tried it personally but it should give you about 2TB of space by my calculations. I could be wrong tho.either way, you might not want to buy 2x 80GB drives and 2x 2TBs and only end up with 2TB of space.

  • http://www.frankleng.me Frank Leng

    good points Alasdair. I think you might have done more research on this than I have.
    What I started with was 3x 2TBs and 1TB. I later replaced the 1TB with a 2TB drive using resilvering and replacing.
    Now you have a good point that upgrading sometimes does not boost your capacity and that is VERY true for RAID 5, RAIDz, and the drobo. That is something we all have to watch for.

  • nbrebel

    Hello Frank and all who have contributed thus far.
    After reading your post and lots of other topics on the NAS subject I am a lot more confident I can get mine done.
    Just a few questions
    1. I have 120G ide drives lying around and wanted to know if I can use a ide to usb converter or a sata to ide converter to allow me to build this system using these driver to mak my full capacity and later change them out. I will be buying at least 2 new T drives of course. Will this work? So 2 2T drives + 3 ide 120G drives. Reason I ask is becuase I have them and can use them as a maxing facility and change them out later.

    2. In addition to the sata ports can i use the sata to usb converter to add drives will freenas allow me to do this.

    3.System is for my home use and may be in short term as a normal pc or HTPC until suitable is started. Want to also use it as a PDVR.

    http://www.newegg.com/Product/Product.aspx?Item=N82E16813131659

    will using that processor allow for faster peformance and features allow better data transfer along network?

    Please advise as to a suitable option if what I suggested above will not work and the questions I asked.
    Thanks

  • http://www.frankleng.me Frank Leng

    hey. first of all the two 120GB drives won’t give you more than 100GB of extra usable space. but you can still use them with either USB – IDE or SATA – IDE adapter. having those drives in the ZFS pool will give you more flexibility later when you need to swap out one of the drives for a bigger one.

    The motherboard you picked looks like a very good candidate for a PVR system. the power usage will be a lot more than an Atom setup. but the AM3 setup will give you significantly more processing power.
    As far as network and disk performance you shouldn’t have much to worry about.

  • http://spuder.wordpress.com/ spencer0

    Very nice write up, I have spent hours looking into drobo vs unraid vs freenas and just give up every time. After reading this I think I will make a FreeNas.

    Do you need to start with blank drives? All my external drives currently have data on them.
    What about reliability, if a drive fails is it as simple as swapping in a new one?

  • Pingback: Terabytes of network storage « Spencer's Technical Diaries

  • http://www.frankleng.me Frank Leng

    Hey Evan. glad this helped. good luck!

  • http://www.frankleng.me Frank Leng

    link didn’t work.
    but in my opinion an i3 is a bit of an overkill. are you running it as a media center?

  • http://www.frankleng.me Frank Leng

    Hey Spencer.
    you do need empty drives because they need to be formatted into a ZFS storage tank.
    when a drive fails. you can still access the data. but ZFS will tell you to add one more drive for the array to become stable again. so just add one to the SAME port. and issue the “resilver” command in freeNAS. this process will take some time as it does in all systems like drobo, unraid, or RAID 10.

    hope that helps. :)

  • http://www.techspread.de Patrick

    I prefer the Lian Li PC-Q08 as case for a DIY-NAS.

    I’m going to build a NAS based on unRAID or FlexRAID because I want to be able to add a new HDD without reorganisation of the data.

  • http://www.frankleng.me Frank Leng

    Hey Patrick. the case looks nice. 6 internal 3.25 bays. tho I am still not a big fan of the internal layout for thermal considerations. The arrangement in the Antec case is still far more superior in my opinion. This gives you a much cooler system. I run mine fanless. :)

    unRAID is essentially a RAID 3/4. from my experience with RAID 3/4 the performance is very slow, even slower than a single drive setup under heavy load. However, unRAID is quite simple to setup.

    FlexRAID is a very interesting solution, and the recently releases solved A LOT of the problems they had before. I’m still not sure how worthy it is since it is a software layer that sits on an existing file system and OS.
    So you need to have a working file system, then install FlexRAID on top of it… not sure exactly how it works. but sounds a bit fishy to me, esp. when ZFS supports all of its features at the FS level.

    At last, the limitation to adding drives has to do with the FreeBSD port of ZFS. It will be fixed in the upcoming version of FreeNAS based on the FreeBSD 8.0 kernel.

  • http://twitter.com/AaronJAnderson Aaron Anderson

    What would you do if you wanted both iSCSI & the ability to add one additional drive to an array?

  • http://www.frankleng.me Frank Leng

    give Nexenta a shot. the older version I tried was quite buggy, but it may be quite capable now. I will do a review on it soon.
    http://www.nexenta.org/

  • http://www.frankleng.me Frank Leng

    also, another thing is the NexentaStor project. http://www.nexentastor.org/
    It’s advertised as a ready to use storage appliance OS.

See also: