I’ve been working on filling all the bays in my Fractal Node 804 case for months now. I actually filled it awhile ago, but accidentally killed 4 out of 5 brand-new WD drives by using the wrong SATA power connector cable from my modular PSU (note: turns out even though modular PSUs share cable form factors, the pin-outs can and do vary enough that you should NEVER use cables that don't specifically match the pin-out of that specific PSU). Thankfully, WD let me RMA the drives (although their customer service department is awful), and I finally was able to install 5x 10TB HDDs alongside the 5x 8TB HDDs I already had. I took this downtime opportunity to install 2x 1TB SSDs that I picked up recently as well. The grand total of installed drives: 5x 8TB HDDs, 5x 10TB HDDs, 2x 1TB SSDs, 1x 256GB mSATA SSD, and 1x slimline, slotloading DVD-RW drive.
I've mentioned in passing that I use ZFS. Adding the new drives to my existing pool, and creating a new ssd-only mirrored pool, left me looking like this:
I have 2 vdevs in my main storage pool (pool0), each consisting of a 5-drive raidz array, one with 5x 8TB drives and the other with 5x 10TB drives. I considered adding the SSDs as cache to my main pool, but I think that the actual value would be minimal. Instead, I copied my Plex database to the new ssd-pool0 mount point, and I pointed my Plex Docker container there instead of it's previous location on pool0. So far it seems like the GUI is more responsive with the database on SSDs instead of spinning disks, but I'm not commited to that long-term. One of the benefits of using Docker is that changing the location is a simple data copy and config file change to update the Plex container.
Already I've noticed a huge jump in data access speeds - I was operating at around 92% full before the additional drives were installed, and zfs gets progressively "less happy" above 80% full. (I've mentioned to my wife multiple times that I realize that I don't have normal people storage problems - my "less than 10% free space" is still 10+ times what most people have in total storage!) With the new drive additions, I'm now at around 40% full. A benefit I didn't expect is that with more drives to write to at once, my iowait times have gone down significantly, and my memory usage has also gone down.
Maxing out the Fractal Node 804 bays does mean that in order to expand in the future, I'll either have to look at building a DAS or upgrading my vdevs disk-by-disk. Not sure which method I'll pursue, or if I'll look at an entirely new design by then - I'm hoping that the 40TB of usable free space I now have will last at least a couple years. In the meantime...I'm going to start looking at my old NAS to see what fun things I can use it for. It has 4x 3TB internal HDDs and 3x 3TB + 1x 4TB HDDs connected via an external RAID enclosure, and it's basically empty right now. Let me know if you have any ideas...