Multiple use cases - a long read.

Hopefully this helps someone down the road determine if syncthing is for them. I was skeptical. But so far (1.5 weeks later) I’m convinced. it’s a great tool, maintained by a group of obviously highly skilled developers, and nice helpful people on the forums.

I have four distinct use cases:

  1. User data sync (dotted black arrows);
  2. Data backup into a “live” environment (red one way arrows);
  3. Photo and video editing without network lag (yellow-ish arrow);
  4. Phone camera backup (solid black arrows).

Use case 1: user data sync

  • I kept running out of space on onedrive, and don’t want to pay for more. Same for any of the other tools that sync user data (think Windows folders: documents, downloads, picture etc.). I found data transfers to be slow (we live rurally and internet service is high speed but not FAST - uploads take a while). I have a 42 TB primary server that I want to use.
  • I find with syncthing (running as synctrayzor on all windows machines) the transfer is nearly instant so that within 10-15 seconds all items are present on all machines. Using a central server I’m able to version on the server and not on the client machines, thus saving space on limited size OS SSDs.
  • synctrayzor only runs when the user is logged in, and is user specific. setting up separate folders on the server allows syncing between the same user on different machines, but with carefully controlled sharing and careful folder names (!!) you can be sure not to cross between logins.
  • You also gain the ability to share ACROSS users on windows machines if you want to - for instance my wife and I have separate logins for windows, but I could share the server “photos” folder under me with her login as well.
  • Additionally if I want to I can drop something into the server folder and next time the client logs in, the sync will pull down the file onto the local drive. I’ve used this when the dance teacher shared music with the parents for the routines to practice at home. I put the file on the server and both the kids have access to it on whatever laptop they’re using that day.

Use case 2: data backup into “live” environment

  • I used to run my backups from the always on VM (contained on the primary server) to push files around. It worked great for years, but I was never entirely happy as there were occasional permission issues, disconnected drives or share issues, and it just felt “clunky”.
  • I am running syncthing in linuxserver’s docker container on my unraid servers. Primary server is 42TB, backup is 22TB. As you can see from the diagrams above, there is a bit of folder nesting going on. But since I want everything that matters captured (the entire “private” share, for instance), I have a one way send on the primary to a send/receive folder on the secondary (more on that in a second).
  • The backup server must be “live”, in that I should be able to turn it on whenever I want and have a functioning version of plex, and access to all my files (that matter).
  • The backup server is set to wake up in the early morning hours. When it does, the syncthing docker automatically starts, scans, syncs, and then shuts itself off once the drives spin down 30 minutes later. Every day. So I have a live version but it’s not burning any electricity most of the time.
  • Version is set to a longer time on the backup than on the primary (30 days vs 5 days) so if something goes wrong I’ll still have it on the backup.
  • I used send/receive on the backup server so that I could control from the primary if something goes wonky on the backup. the primary is ALWAYS right. So rather than having to log into the backup to “revert changes”, I can do it form the primary by “override changes”. Forgive me if my verbiage isn’t right, I’ve only seen it once or twice.

use case 3: photo and video editing

  • we take lots of pictures, and lots of video. My big pain was always waiting for video to load over the network (sure it’s gigabit, but still have to wait). the phones now back up and sync to my workstation hard drive (a separate 2TB physical spinner) where I do all my editing and review. Thumbnails generate almost instantly and there is no lag when playing video. Renames are instant, then reflected across the remainder of the syncs. then I copy back to the proper network share location (fully organized by person, date, etc. in folders…) so they are available to family elsewhere running plex. Once it’s moved into it’s final place, … (see use case 4)
  • version cleanup is set to 180 days on the local hard drive because I have the space. It’s a second to last ditch rescue in case something goes horribly wrong.

use case 4:

  • I wanted a robust backup solution for the phone camera and video. I had used “foldersync” for years on the android devices and it was always pretty good, but not “fantastic”. Sometimes it would fail, or something would go wrong (like android would put it to sleep), and it relied on the home wifi connection to work, and the two way sync was scary. So now I have syncthing running on the androids and it’s so far been flawless. Once I"ve finished editing the pictures (use case 3), and move them into the final proper spot on the server, they get cleared off the phones.
  • we also use google photos so all the history is still there on the devices, just not locally.

offsite backup

  • I am paranoid. I’ve lost data and pictures once before, in 2001. the next day I bought a new computer with a raid5 card and 4 hard drives. spent a fortune. Ever since I’ve always looked for great ways not to lose the stuff that matters. In addition to everything I’ve described I backup nightly to a cloud backup provider. Their program runs on the Windows VM and does its thing every night. In the morning I check my email and see that it ran. It’s important.

a final note

  • The main point is that syncthing makes it easy to set up, easy to monitor, and easy to automate across all sorts of devices. I have 4 use cases, and i’m sure there are many many more. But so far syncthing is able to handle everything I’ve thrown at it, and the errors I experienced were because I didn’t know what I was doing. I had issues with nesting and syncing the syncthing folders. sooo much great info in this thread: complex sync structure - bad? that it doesnt’ need to be repeated here.
  • I’ve also set the rescan interval to one day (86400 seconds) rather than the defaul 1 hour. Most things are captured by the watching, and anything that’s not will be caught up later. Plus it lets the drives spin down to save some energy - important when 19 drives are busy spinning a way under the stairs…

I wish you all the best of luck with it, and have a great day! Tiwing.

9 Likes

This was a very detailed and interesting write-up :clap:.

Do you create those mock-up images yourself, or do you use particular software for them? I also have quite a complex structure, and it would be nice to have it recorded somewhere like that.

Just a comment on this one. If you actually record videos on the computer itself, then I would recommend using a temporary folder for the recordings before adding them to Syncthing. Otherwise it will try to sync files that are still being recorded, and thus changing constantly. Although, to be fair, I have been helping someone who records tons of videos, and we just keep everything in Syncthing for the sake of simplicity. Fortunately, the PCs and the network are fast enough to deal with it.

I would say that there is nothing paranoid there :wink:. I have had a computer fried by a lightning bolt once. Offsite/offline/remote backups are a must these days.

I use powerpoint a bit for work … it’s just that. very manual, but does the job well enough. Excel has all the same functionality too.

My comment was actually intended to be more about “review” of files rather than actual editing - 1/3 of video taken from the phones are garbage. So I review them locally and delete / rename locally thereby syncing back to the primary server and deleting from the phones.

But Good point on actual editing and thanks for it… I do use Shotcut sometimes to do overlays, combine different angles, or simply add fades. I do most actual editing on files saved on the local SSD because it’s so fast. I haven’t done any editing since starting syncthing but I’ll likely export to the SSD and then copy the final, good, file into the syncthing structure.

Cheers

I dislike the sync-client for Nextcloud on Linux. Most painful is, that there’s no client running in background when I’m and all other family members are NOT logged in.

Therefore I sync all family data from our personal devices to our own Nextcloud instance’s external storage running on our cloudserver at a german Hoster with Syncthing :slight_smile:

Thank you for the detailed visual and written description.

Can you add in what operating systems are in use on the various clients and servers?

as requested

3 Likes

update 3 weeks later:

  • added two external windows 10 clients (parents) over the internet. Syncing one way of pictures, videos, documents etc in Windows to my primary server which then syncs to my backup.
  • I also set my Mum’s laptop to sync to her login on Dad’s laptop, and Dad’s to sync to his user on Mum’s laptop. So we have achieved “local backup” and “offsite backup” for them as well, with no additional hardware purchased. All connections have been flawless for weeks.

The only occasional glitch is “over the internet” stuff that gets interrupted for one reason or another, but after a day or so it all clears up and goes back to perfect, exactly what it should be doing. Next up is setting up my sister and my wife’s family… woohooo!!

2 Likes