Specify sync folders?

Hi there, I’m new to Syncthing.

I haven’t found an answer in the docs or support for something that I need to do. I’m a photographer adding several GB of data every day that I need to “push” from my laptop (where new data is downloaded to) to a local Synology NAS (NAS #1) as well as an offsite Synology NAS (NAS #2). I’ve installed Syncthing on the laptop and both NAS devices and have that working.

Here’s what I’m having trouble wrapping my head around: because the folder names on all three devices aren’t necessarily the same, I’m wondering if there is an option to specify both a source and destination folder?

For instance:

  • On laptop: e:\foldername1\foldername2\foldernametosync
  • On NAS #1: \foldername3\foldername4\foldernametosync
  • On NAS #2: \foldername5\foldername6\foldernametosync

I don’t see a way to point a source folder to a destination folder. Am I overthinking this?

Please have a look at https://docs.syncthing.net/intro/getting-started.html if you haven’t already.

Basically, only folder IDs need to match, while labels and paths can be all different. I’m guessing that you’re trying to set both local and remote paths while sharing the folder. Syncthing doesn’t work that way. You first share a folder on one device, then on the other you accept it under your preferred path.

You first share a folder on one device, then on the other you accept it under your preferred path.

Ah. I haven’t gone as far as actually sharing a folder, since I wanted to understand what I was doing first. If I can specify the destination, I think that will do exactly what I need.

Thanks very much.

I have another question that I want to understand the answer to before I set this up.

The two NAS devices should keep all data in two-way sync, since they act as data repositories. But my laptop should only upload data (i.e. one-way) to the NAS devices, but not download data from them. The laptop is basically just a working version of the data until I’m done with it at which point I delete the local copy once it’s on the NAS to make space on the laptop. (The NAS devices each have about 35 TB of data, and growing.)

So on the laptop, should I specify the folder type as “send only”, and on the NAS devices specify “send & receive”?

If your laptop is set to Send Only and the NAS to Send & Receive, and then you delete files on the laptop, then those deletions will also be pushed to the remote devices.

You will need to copy the files somewhere else on the NAS if you want to preserve them there. You could also utilise one of the built-in versioning options. Alternatively, you could also play with ignore patterns to keep the files from being deleted, but it would likely require a lot of manual work, and be more prone to errors than the other methods.

1 Like

Here’s what I do for a similar setup:

  • Regular 2-way sync:
    • Laptop: e:\folder\foldertosync1
    • NAS #1: /folderA/foldertosync1

A bash script in cron or Synology “Scheduled Task” on NAS #1 then copies files as hardlinks from foldertosync1 to foldertosync2.

  • Regular 2-way sync:
    • NAS #1: /folderA/foldertosync2
    • NAS #2: /folderB/foldertosync2

Unix hardlinks link an additional filename entry to an existing physical file (“inode”), so no extra disk space is used. If either filename is deleted, the link count is reduced by one but the physical file remains on disk until there are no filename entries linked to it. (Hardlinks only work on the same mount point, so the two folders on NAS #1 should ideally be in the same parent folder.)

Files created on Laptop would be synced to NAS #1 in foldertosync1, then would be hardlinked into foldertosync2, causing them to be synced to NAS #2.

When files are deleted from Laptop, they will be deleted from NAS #1 foldertosync1, but remain in foldertosync2.

The bash script could be something like:

cp -Rlu /folderA/foldertosync1/* /folderA/foldertosync2/

cp -Rlu recursively copies files and folders [“R”] as hardlinks [“l”] and avoids copying files that already exist unless they are newer [update, “u”].

It could run every 30 minutes or every hour, as appropriate - some interval long enough not to capture Syncthing temp files created for incomplete files (especially if you have large video files that may take some time to transfer).

In that case, you might use a bash script that deletes the temp files – on Linux/POSIX they would be .syncthing.original-file.ext.tmp (see Understanding Synchronization — Syncthing v1 documentation).

Syncthing will ignore and not sync .syncthing.something.tmp files, but they would clutter the target folder /folderA/foldertosync2 and will not get deleted without additional intervention.

Something like this ought to do it:

#!/bin/bash
cp -Rlu /folderA/foldertosync1/* /folderA/foldertosync2/
find /folderA/foldertosync2 -iname '.syncthing.*.tmp' -exec rm -f '{}' \;

The ' surrounding '.syncthing.*.tmp' prevents the shell from prematurely interpreting the wildcard, '{}' substitutes each filename from find and \; adds a trailing ; to terminate the command – it has to be escaped with \ to prevent it being interpreted by the shell (the ; may not be strictly necessary, but I don’t think it hurts to have it there).

The two lines could also be separate one-line commands run several minutes apart (i.e., run the first line to copy at the top of every hour and the second line at 2 minutes after the hour, or set both more frequently if needed).

NOTE: I strongly advise that both Syncthing folders on both NAS devices be configured with at least Trashcan versioning to save yourself from $*%# moments. (I prefer Simple or Staggered versioning; and if you have plenty of disk space, set “clean out after” to 300 days or more. See File Versioning — Syncthing v1 documentation for details.) And test with expendable files first!

Someone with more experience than I could probably improve on this.

1 Like

This is NOT a good solution… it preserves the directory structure, but duplicates files at every level. Clearly I should test solutions first!

Someone with better find experience might be able to suggest a better one-liner.

One option would be to loop through each entry in each directory (folder) and test before copying.

The other option (my preference – brute-force over fallible logic!) would be to just run the first copy command, and follow it with a command to delete the .syncthing.*.tmp files in the target.

I’ll test something like this and then correct the above response. [Edited above post with fix tested with temp files.]

1 Like

Hmmm. I don’t want that to happen.

Alternatively, you could also play with ignore patterns to keep the files from being deleted, but it would likely require a lot of manual work, and be more prone to errors than the other methods.

I agree that that sounds messy.

I’ll have to think this through.

I’ll test something like this and then correct the above response. [Edited above post with fix tested with temp files.]

Thanks for the thoughts and suggestions.

I wonder if I would be better off using third-party sync software to push the data to the NAS devices, and then letting Syncthing keep files synchronized between the two NAS devices? That would let Syncthing work on its strengths automatically while I just have to get the data to either NAS for Syncthing to take over.

For years I’ve been using SyncBack Pro to push new data from my laptop to both NAS devices, so I’ve already set up and tested that step. Synology also offers their own software to synchronize two NAS devices, but it becomes unreliable with the volume of files that I have.

Its best to use what you’re familiar and comfortable with.

True, what I’ve suggested above is a bit of a hack, so if you’re not comfortable with that level of “making things work” then definitely do what is comfortable to you.

Since you already have one piece of the chain solved, keep using that and let Syncthing handle the other piece, and as you say, do what it does best.

I still suggest you turn on Trashcan versioning – but bear in mind it will only version files on NAS #2 that have been deleted on NAS #1, since you would be using SyncBackPro to push the files to NAS #1. Its still a worthwhile belt and suspenders as long as you recognize the limitations.

I’m going to try using SyncBack Pro for the laptop-to-NAS step and Syncthing for the NAS-to-NAS step.

I still suggest you turn on Trashcan versioning – but bear in mind it will only version files on NAS #2 that have been deleted on NAS #1, since you would be using SyncBackPro to push the files to NAS #1.

Thanks for the suggestion. I’ll read up on that.

Is there a way to set Syncthing to not delete any files that are deleted on one NAS from the other NAS? I realize that’s outside of the scope of a pure mirror-type synchronization but I’d rather use more storage space to avoid potential unintended data loss.

The best option is to setup Trashcan versioning and set up a long period for cleanout (or 0 = “never” if you have plenty of diskspace).

The WebGUI has a nice UI for inspecting and restoring versions in the trashcan.

Check the versions button on the folder every few weeks until you are comfortable that all is well.

Excellent. I appreciate your help with this. It helps wrap my head around how to use Syncthing for this purpose.

:+1:

I should have mentioned that you should enable versioning on both NAS units.

Read up on the variations - Trashcan, Simple Versioning, or Staggered. Trashcan is probably what you would want, as it will save all deleted files until they reach the maximum age set by “Clean out after”, which you can set to some pathological number – or 0. https://docs.syncthing.net/users/versioning.html

Reading up now. Thank you.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.