I am actually backing up my entire server once a day incrementally. The problem with this approach is that I only have one day's worth of revisions.
So, I considered building a NAS specifically for Gameserver backups, as I sometimes require hourly granularity.
Only a few questions remain:
- What filesystem should I use?
- What hardware should I use?
- What OS should I use?
- How should I perform backups?
Filesystem
Since I'm using Linux on my server, I've thought about using ZFS as the filesystem. It's a reliable filesystem with numerous features.
The only downside is that it's resource-intensive. I would need at least 8GB of RAM for a 4TB pool.
This would solely be a backup NAS, so slow performance wouldn't be a major concern.
However, all of my Gameservers together use around 150GB of space, meaning backing it up every hour will create a lot of stress on the System.
With 24 revisions per day, 14 revisions per week, and 12 revisions per month, with enabled deduplication the ZFS ARC cache would fill up quickly.
Consequently, this could result in slow performance, possibly achieving less than 150GB per hour in speed.
Hardware
Initially, I experimented with an old desktop system featuring an old i5 3rd Gen processor I purchased a few years back. The issue is that it's a consumer-grade system and rather sluggish. Additionally, the mainboard only supports 2 DIMMs.
It dates back to around 2013, and I only have 2x 4 gigabyte sticks of normal non-ECC memory, totaling 8GB of RAM.
I housed everything in a Rajintek case I acquired from work. It provides 10x 3.5" bays, which should suffice for the next few years.
Initial test with a consumer system
I added an LSI-9207 flashed in IT-Mode for connecting the HDDs.
Another issue went unnoticed for the first few days: the mainboard lacks fan headers, resulting in drive temperatures of around 56°C.
I'm using 3x 4TB WD Red drives, known for their reliability, but they're not the fastest and tend to heat up quickly. For cache and metadata, I use 2x 128GB SanDisk SSDs.
Then, I remembered I still had an old server mainboard lying around. While it might consume more energy than the i5 3300, it would be more reliable.
It features a BMC, supports ECC RAM, and houses a Xeon E3-1220v2. It also boasts 4 DIMMs, potentially increasing my total system memory to 32GB.
So, I disassembled everything and reassembled it with the server components. I put everything together, installed the CPU, and added the RAM. However, the 24-Pin ATX cable falls short by about 1cm.
It's too short...
Subsequently, I turned to eBay and found a suitable PSU with a cable long enough, which also addresses an upcoming problem and offers sufficient SATA power connectors.
After a few days, the power supply arrived and I was finally able to finish the build.
It was time to disassemble everything again, install the new power supply and wire everything up.
The final build!
OS
I opted to use TrueNAS Scale for this project, given that it's based on Debian and features a user-friendly web interface.
It's easy to navigate, offering an array of features.
I set up the ZFS pool and added all the drives to it. I created a separate backup user for my servers to transfer their data. In order for the bottleneck to be located on the hard disks, I have configured the compression to zstd-4, to save further storage space I have also enabled deduplication.
Why waste valuable storage space?
Backup
I use rsync to copy everything to a "base" folder.
The base folder is then tarballed into hourly, daily, weekly, and monthly folders.
For my Gameservers, I added the following command to the cronjob:
rsync -a -v --stats --progress --delete /gameserver/ buser@gaab-backup.gaab.intra:/mnt/backup/gameserver/base/$(hostname -s)/
Given that they run via PXE and NFS as root, I can easily roll out updates for the Gameserver users.
For creating the tarball, I created a script executed every hour, every day and every week (With minor adjustments of course):
#!/bin/bash
ssh buser@gaab-backup.gaab.intra "tar -cvf /mnt/backup/gameserver/hourly/$(date '+%H').tar /mnt/backup/gameserver/base/"
Since ZFS handles compression, there's no need to worry about that aspect.
All copying tasks run hourly, and the tarball is created at half past the hour.
Conclusion
The system works great, securely backing up all my Gameservers without the fear of accidental deletion.
After further adjustments, I managed to maintain the average energy consumption at around 50W, which is impressive for a 24/7 setup.
Disaster recovery is straightforward: I can copy the files back to the server, making a deletion or restoring files pretty easy.
Thanks to ZFS, creating snapshots of the entire pool and sending them to another server is also uncomplicated.
Also, adding another server to the backup is a breeze; I just need to include the SSH key for the backup user and add the rsync command to the cronjob.
Overall, I'm very satisfied with the result. I also manage to meet the 3-2-1 Rule!
