Hope we’re all well!
I’ve been tinkering with my backup solution lately, and now Syncthing has been used for long enough to be proven reliable (in my situation), I thought I would try and create the perfect solution.
I see that @kinsham was in a similar situation a few months ago. I used file versioning initially (in ST) and the system went a little squirrelly. So I tried turning that off and so far it seems to be running very smoothly, Backblaze and ST are playing nicely with each other.
Of course though, this means that a very helpful feature of ST can’t be used. For me, using TrueNAS, I’m falling back to Snapshots. If I delete a file or need to restore a previous version, I can go to TrueNAS and restore a cloned version of the snapshot that contains the correct file or version of the file.
As you can imagine, this is a bit of a pain. What’s more of a pain, as @kinsham said, was that Backblaze do not offer a “hold one version of the file” option. The Lifecycle options are only as shown below:
As you can see, I’ve got it on ‘last version’ option. I did try to set “Keep prior versions…” to ‘0’, but that wasn’t accepted.
Just for anyone else’s reference, I have disabled ‘Use -fast list’ in TrueNAS and the C transactions seem to be reasonably low.
If anyone has any comments, I very much welcome them.
What do you mean exactly by this? Did you encounter any problems or malfunctions when using file versioning? As far as resource usage goes, file versioning should have no impact on that, with the exception of taking (possibly quite a lot of) disk space, of course .
Ah, I sadly didn’t make any notes of my squirrelly definition, but I do recall it relates to BackBlaze C Transactions ramping up to ridiculous levels.
I might turn it on again and see if it breaks or causes too many transactions again, it’s a PITA to trial though.
Just for extra clarity, I was referring to the file versioning option within ST:
It’s not a performance usage issue, but a “backblaze acting up” issue.
Yes, disk space is the only issue, but I’ve tried my best to have enough spare
No problem . You may want to be careful with simple versioning though, as the cleanup behaviour is buggy (see https://github.com/syncthing/syncthing/issues/7988) and can/will delete versioned copies too early. I’d recommend staggered versioning personally.
Cheers for that, I was looking at staggered, partly because of what I read, but also Tom at Lawrence Systems does it that way too.
May be simple versioning should be removed as an option, until it’s bugless?
Hmm… I think you are trying to push a square object (Syncthing) into a round hole (doing backup).
I recommend using a backup tool like for example Restic https://restic.net/ instead. Yes, it works with B2 as well.
Well, NASA did it You are right though.
Thank you for intro’ing me to restic, not heard of that before. Looking at it briefly it mentioned snapshots, which is what TrueNAS does:
It also lets me replicate snapshots to other servers, so I have 3 copies locally on the LAN and my intention is to have an off-site one with BackBlaze. The cost is the only kick in the pants.
the replication feature is good though, I started a new replication last night (200GB) from one machine to another. It finished in 30 minutes.
I don’t think that’s true here, unless I am misunderstanding what the setup is: Syncthing makes sure the server has all the data from other devices, local snapshots and backblaze back that data up. That’s basically textbook usage of the right tools for the right job.
Ideally, the bug should just be fixed, especially since the cause and solution are known already. Someone only needs to roll up their sleeves and sit at the code. This includes myself too .
Many people already use the versioning, so it can’t just be “disabled”. The default settings are actually safe, as cleanout is deactivated as follows.
It’s only if you specify the number of days here that the bug manifests itself.
This is on the radar of my new colleague, should be solved at some point.
Thanks for that Simon, I admit doubts starting to set in about what I’m doing. I thought snapshots were at least sensible, in case SyncThing goes strange and starts deleting files. So far, the only times it’s gone wrong, is when the user (me) makes a mistake with the config.
Neither you or @calmh should beat yourself up about it, you and the other devs are doing a fantastic job and don’t receive the praise you should. I’ve looked at code and it’s mind boggling and that fact you maintain it to the level you have is already an amazing feat. As I’ll be eventually using it for business, I will be making monthly donations
One downside to running ST on TrueNAS Core is the delayed update schedule, iXsystems/FreeBSD tend to update plugins very slowly (for good reason). I have been meaning to do a manual install of ST to see if the update schedule is a little quicker. Because of this, it appears the default looks like this:
After an update, it remains the same version:
I have a TrueNAS Scale install that I’ve been playing with, I might run it on that and see what happens
Just as a small update, I did manage to get Syncthing working on TrueNAS Scale. It was a bit of a fiddle, but it is do’able.
I’m curious, what was the required fiddling? My understanding is that it’s “just” Docker so should be fairly straightforward? Is there something we can improve on our side?
You and your colleagues have done a fantastic job, with TrueNAS Scale we’re still at the early stages of the product and many have advised not to trust for production workflows.
My issues were not the fault of ST, but rather the lack of info relating to Scale. I think part of the problem is a few things that are different between the Core ST and Scale ST, for instance the user id is different, which doesn’t help when trying to migrate between the two.
Do appreciate my background knowledge is fairly minimal, so don’t pay too much attention to my comments - I’m just an average Joe
I think you’re right that Syncthing isn’t necessarily the best tool for this particular use case. As much as I love it.
My current backup setup is:
- Dedicated “backup server” for all users
- Syncthing: Syncg users with server continuously
- Dullicati: Periodic incremental backups of Syncthing folders for all users locally and on BackBlaze
This has proven very convenient, reliable, and stable for months now.
This could work fine without a dedicated backup server as well.
Hope this can help inspire someone.
That was the solution I eventually settled on. I didn’t know about Duplicati so used rclone in a cron script to update B2. I’ll take a look at that. Thanks for the tip.
Thanks for adding this Jean, out of interest are you using TrueNAS or something else?
At the moment, I’m really happy with using TrueNAS’s built in snapshots and having BackBlaze back them up. Allows me to roll back locally if there’s a problem (so no extra backblaze cost) and as replication goes to another machine, I have 2 lots of snapshots to locally lean on if things go a little wrong.
I use openmediavault on Debian, mainly for the convenient web interface. But the actual OS doesn’t matter much when everything is set up via containers.
I do both remote and local backups via Duplicati. I did use rsync with hard links initially for the local backups, but the large number of files made this impractical and wasted a ton of space. Thanks to deduplication, the Duplicati backups is about 10x smaller than the source data.
Ah, thanks. TrueNAS Core uses BSD jail’s so it’s not so straight forward, Scale does do containers though which is nice.
It does sound a bit like TrueNAS snapshots - only backs up the difference. So add 1MB file and the next snapshot is just 1MB in size.