Where Did All My Storage Go???

Where Did All My Storage Go???

I run a home NAS with (currently) 5x8TB HDDs in a raidz zpool. When I migrated from my old cobbled-together NAS, I moved ~18TB of data across to the new ~32TB pool. I was hopeful that the remaining ~14TB would last me for at least 12-18 months before I'd need to add another vdev to my pool (I have the bays available, but I don't have drives). Last week I noticed that I only had 2TB free! That wasn't supposed to happen this fast! I finally had a little time to do some investigating to see how I could buy myself some more time.

Without getting into the details, I have several scripts that run on a schedule: One will copy my Plex database, compress and date stamp it, and save it to an "archive" directory, another deduplicates all the data in my zpool, hardlinking duplicates, and another kicks off a daily delta sync of all changes (new, edited, deleted) on my filesystem to a directory in my Google Drive. I rarely have to go in and manage anything manually on my server, it's 99% automated and other than the few times I need to manually add or pull data from it, I don't even log onto the machine directly or remotely - I interact with the services I host from the server. BUT.

That 1% gets me every time! I forgot that my Plex database backup runs 3x a week. I didn't know that my database had tripled in size over the past couple months, after I turned on preview thumbnail generation for my entire library. I hadn't checked on the "archive" directory where the date-stamped backups are held. Everything was running smoothly, backups to Google Drive were happening, so I really wasn't worried until I noticed that I was running out of space.

You may already know where this is going - I had over 10TBs of database backups sitting on my server still. 2 and a half months' worth of compressed data that I didn't intend to keep locally. Once I realized that, I deleted all the local copies of the compressed backups made prior to May 2019, and got back a ton of usable space. I should probably script that cleanup for the future.