Send only / Receive only not removing orphaned files

Support
1

Apologies in advance - this is likely something very simple that I’m misunderstanding.

I have a folder on my laptop that I want to one-way sync to my NAS:

  • The folder on my laptop is ‘send-only’ and I also use ignore rules
  • On the NAS the folder is set as ‘receive-only’

My expectation is that the NAS will always be a 100% identical mirror of the folder on my laptop.

But in reality if I delete files from my laptop, or if reduce the files via ignore rules, the changes do not propagate to the NAS. The NAS always ends up with more files and folders than my local folder. I did not change the ‘ignoredelete’ or any advanced setting, there are no errors, and permissions are not an issue (if I set the folder as send-receive then files get deleted accordingly, but I never want the NAS to be able to affect data on my laptop for this folder).

Rescanning doesn’t help either. The only way to get them to sync properly (eg delete the orphaned files from the NAS) is to remove/delete the entire folder from the NAS and then add it again. But I would need to do this each time something gets deleted on the laptop which seems highly impractical and I’m pretty sure that I’m not setting this up correctly.

This is the folder on my laptop:

image
image615×397 34.3 KB

And this is the folder on my NAS. Here I would expect to see the 67,534 files (since the laptop is the ‘control’, send-only folder). But instead this is what I have:

image
image615×430 38.7 KB

2

EDITED TO ADD: I got this wrong, please see downthread.

If you want the files on your NAS to be a 100% identical mirror, I would suggest that you configure the Folder for Send & Receive on both sides. Otherwise, when you delete a file on the Send Only side, you’ll be Out of Sync until it’s resolved. See Folder Types — Syncthing documentation .

If your use case is backup rather than file sync, use Syncthing Versioning ( File Versioning — Syncthing documentation ) or use something on your NAS to back up files there.

While I don’t use a NAS, I do have a Syncthing Device with a ton of storage on it that’s backed up continuously, somewhat of a similar use case to what I think you’re trying to do.

3

The folder in question is my linux home folder - a conflict or data loss/overwrite can potentially render me unable to use my laptop so I cannot use a 2-way sync unfortunately.

Maybe the naming conventions is what confuses me? I never think of syncthing as any sort of backup tool - I am aware of the versatile file versioning options, however I’ve often seen it being discouraged from using syncthing as a (primary) backup tool, so I have a whole other process dedicated for backing up that uses tools specifically made for the job.

My understanding was that syncthing’s secret sauce is its ability to… sync things. A one-way sync seems right up syncthing’s alley…?

Having Send-only and Receive-only requiring consistent manual interaction to resolve differences sort of defeats the purpose of having an automated tool. This seems like a missed opportunity - is there no advanced setting or any way to ‘always override’?

4

I see from your screenshots that there are some database files included in the mirror. Depending on the app, those SQLite files should be consistent, but might not contain all the data when Syncthing copies them to your NAS because many database engines buffer data in RAM – i.e. if you shut everything down immediately after Syncthing reports everything is up to date, it’s very likely your laptop and NAS are not going to match.

It might not be a deal breaker, but still something to keep in mind.

Syncthing is primarily intended for bidirectional sync. For one-way sync meeting the requirements of your setup, rsync would be an excellent option. For more ideas: https://en.wikipedia.org/wiki/Comparison_of_file_synchronization_software

For your requirements, configure Syncthing like so:

  • laptop: Send Only
  • NAS: Send & Receive

Because your laptop is “Send Only”, Syncthing doesn’t apply any changes from your NAS. And with your NAS being “Send & Receive”, it’ll tell your laptop about local changes and you’ll get an [Override Changes] button on your laptop. But if your NAS doesn’t make any local changes, your laptop will quietly push all changes to your NAS.

5

Hello again gadget :slight_smile:

Many thanks for the heads-up re: databases - hadn’t crossed my mind so far. The ones in the screenshot are buried within some obscure parts of the browsers and their extensions which is no big deal; I did do a few full restores already and didn’t notice any issues, but I will keep it at the back of my mind from now on.

Funnily enough I’ve replaced all those kinds of tools over the years with just syncthing and I love it for that.

Even in the chart you linked, rclone and syncthing are 2 of the tools that check every box, and I’m surprised to learn that 1-way-sync is not really a thing in syncthing (it is in rclone)

What confuses me is that syncthing is supposed to be a synchronisation tool and not a backup tool.

Yet the current 1-way sync mechanism behaves like a combination between sync and backup, and neither can truly fulfil a robust sync or a backup scenario. I’d be curious about which practical use cases are served by the way in which syncthing currently handles 1 way syncs.

I did try that as well, and no joy. I don’t even get the red button to revert changes that I occasionally see on other folders.

This issue is further emphasised by any subsequent changes to the .stignore (of the send-only folder) after it had already synced onto the NAS (whether it’s receive only or send&receive). Once the files make it over to the other side, there’s no getting rid of them anymore unless you delete them manually.

At this point whatever I end up with on the NAS is not a sync and it’s not a backup either - to be honest I’m not quite sure what it is, but I know it’s neither and I can’t rely on it.

I think a flag to always match destination to source (for send only and vice versa for receive only) would help cover these scenarios entirely. Syncthing already has the mechanism to manage this process fully, my guess is that it was probably a good reason not to allow a true 1-way sync?

6

Yes, this is on purpose.

For me, this is a typical use case for .stignore:

  • A NAS with huge disks and a lot of data. Here I use .stignore to ignore for example cache files/directories etc, so that those are not synched to/from other devices.
  • A laptop or phone with quite limited storage. Here I use .stignore for two purposes: 1. Ignore cache files/directories (like above). 2. Sync files of most interest.

Now, given the use case above, picture this made up situation:

  • Syncthing would work as YOU expect w.r.t. .stignore, that files on other devices would be deleted if .stignore was edited to ignore more stuff.
  • On the laptop, I edit the .stignore to ignore more stuff, because the disk is nearing full and I need to sync less stuff.
  • To actually “reclaim” the disk space used by the stuff I additionally ignored, I need to manually delete those files from the laptop.

If Syncthing had worked as you expected, a lot of stuff would have been deleted from the NAS as a result. That would be very bad. This is why Syncthing never deletes stuff just because stuff has been ignored.

Quite frankly, I consider the use of “.stignore” to be an advanced topic you should delay doing until you have a good understanding of the basics. :slight_smile:

Read more here: Ignoring Files — Syncthing documentation

7

rclone, rsync and others like it either push or pull each time they’re invoked, they don’t do both at the same time.

Correct, although it hasn’t deterred many from trying to use it as a dedicated backup tool. :smirking_face:

From the POV of backups, even the 1-way sync is definitely not a reliable backup solution because something as simple as swapping drives or malware could result in data loss. At a bare minimum, Syncthing’s versioning would need to be enabled.

However, backing up is only 1/2 of the equation. At my employer, I’m backing up over 100 million files with new additions and changes happening every second, so finding and restoring a specific file out of Syncthing’s .stversions would not be all that fun.

A good backup solution also offers some protection against data corruption due to bit rot, hardware issues, etc.

At work, I’ve got a cluster (hub-and-spoke topology) that pushes changes to a downstream cluster (mesh topology). The mesh nodes get all of their updates from the upstream hub-and-spoke cluster that’s configured as send-only.

Whenever there are any rare local changes on the mesh nodes, an [Override Changes] button appears on the upstream node that can be used to bring the rogue node back in sync.

In addition, the mesh nodes store all of their synced files in a RAM drive, so rebooting a node purges its copy of the synced files.

I could use rsync for the 1-way sync, triggering it via incron, but handling a rebooted node is more cumbersome. With Syncthing, it’s just a simple script to recreate the .stfolder marker directory → start Syncthing → clear the local changes error (via Syncthing’s REST API) → let Syncthing catch up.

hmmm… should review Syncthing’s runtime log, and perhaps crank up the debugging options for a deeper look at what’s happening.

Yes, changes to ignore patterns only apply to new files and directories. For 1-way sync, tools such as rsync are more convenient because by default rsync assumes the contents of the source are the ultimate truth.

I use rsync to move files from various servers to a central NAS where it’s picked up by a backup system.

1 Like
8

Thank you both for the very insightful replies, I now have a better understanding of the reasoning behind this mechanism.

Allow me to preface this with the fact that I don’t think syncthing is doing something wrong - I very much appreciate the default-defensive stance syncthing takes. In all the years I’ve used syncthing, I have never lost data - even as a result of poor configuration on my part; if anything, I have the opposite issue - having to manually deal with redundant data (a very tedious process, or potentially deleting and re-downloading huge folders again for the sake of removing some orphaned files).

The roadblock I am hitting is the inability to use syncthing to properly 1-way-sync a folder (at my own peril) - not even as an advanced setting.

Thanks for outlining your use case - I also have identical instances where I do exactly what you describe - and I agree it would be quite damaging if syncthing was to delete things by default.

Totally agree with you (also many thanks for the broad context and use cases you provided, they added much valuable insights) - on my NAS, syncthing is sitting on a btrfs volume with built-in regular snapshots, backups, and in addition to that I also have periodic backups to external mediums and to the cloud as well - all of which are external to syncthing and don’t rely on any sort of file versioning or ‘protection’ from syncthing’s part (although I am very appreciative that syncthing does offer these options, more love and power to syncthing). So in this scenario, syncthing’s defensiveness is costing me additional storage on every other medium, with no added benefit.

I understand the potential dangers of a true 1-way-sync solution - and I’m confident that I have mitigated (to my satisfaction at least) any potential shortfalls outside of syncthing (which I thought was the preferred practice, since syncthing is not a backup solution).

and

I am not saying let’s replace how syncthing works at all - I am only saying that the ‘Send Only’ and/or ‘Receive Only’ are not exactly true to their word, not without an additional flag to also ‘enforce deletes’ (and thus also propagate subsequent .stignore deletions).

In line with the defensive-first ideology of syncthing, this flag could be restricted to ‘Receive Only’ folders - so in my case I would make the laptop’s folder Send Only, and the NAS to be Receive Only + ‘enforce deletes’. This way the laptop can’t arbitrarily (or accidentally) delete things on the NAS without the NAS’ prior and explicit consent.

I am aware that syncthing has an advanced setting for ‘ignore deletes’ - I am proposing a similar advanced flag for the rebels among us to also be able to ‘enforce deletes’.

Like you said @gadget, rsync could indeed be used for this - but it seems like syncthing is 95% of the way there already; I was hoping that I wouldn’t have to involve and maintain another medium for this use-case alone.

9

.. in particular this:

Umm… This is not correct. If you have a folder which has type “Send Only“ in which you delete a file which is not ignored, the delete will be sent to other devices. Every other device will apply the delete, unless the receiving device is ignoring that file or if the folder type on the receiving device is “Send Only“.

But that is exactly what “Send Only“ does! Folder Types — Syncthing documentation says:

The intention is for this to be used on devices where a “reference copy” of files are kept - where the files are not expected to be changed on other devices or where such changes would be undesirable.

In send only mode, all changes from other devices in the cluster are ignored. Changes are still received so the folder may become “out of sync”, but no changes will be applied.

So this is THE way for you to have your “1-way-sync a folder“. (I actually believe that the wording in the documentation is not the best, but the intention is to protect a folder from outside changes.)

Looking at the original post, what you wanted to achieve is this:

To achieve the goal of “NAS will always be a 100% identical mirror“, do like this:

  1. Laptop: Configure the folder as “Send Only”. Rationale: Changes on the laptop are sent to the NAS, as expected. If some changes are done on the NAS, a red button “Override Changes“ will be shown on the laptop. Touching that button will revert the changes on the NAS. This will once again bring the NAS back to being a 100% identical mirror.
  2. Laptop: Do NOT use ignore patterns. Rationale: If you use ignore patterns, the laptop will not send some changes to the NAS. This would contradict the goal of “100% identical mirror“
  3. NAS: Configure the folder as “Send & Receive”. Rationale: See above for the laptop.
  4. NAS: Do NOT use ignore patterns. Rationale: If you use ignore patterns, the NAS will ignore some changes received from the laptop. This would contradict the goal of “100% identical mirror“
2 Likes
10

I think you’re starting to see my point about the 1-way behaviour being confusing :slight_smile:

Partly - maybe, but for the purpose I described (1 way sync), it is far from ‘exactly’:

  1. Inconsistent .stignore behaviour
  2. Requires manual intervention to resolve differences

The beauty about syncthing is that you can leave it completely unattended and have the confidence that everything works like clockwork. Except for this scenario, which needs me to always click the same 1 button every single time, without exception.

Seems you also joined the camp of mixed-signals .stignorers :slight_smile:

Your recommendation is to essentially not use .stignore and instead of syncing 3.5 GB of meaningful data that sometimes changes (for context - we’re talking about the /home folder here), I should sync ~150 GB of ever-changing live and temporary files to the NAS. I’m not sure I follow.

If syncthing did not have the concept of Send-only/Receive-only and was strictly a bidirectional synchronisation tool, the intentions of the software would be crystal clear. But for ‘not a backup tool’, syncthing sure goes a long way into trying to cater to backup scenarios, while pure sync scenarios seem to take a back seat.

Syncthing is great. And perhaps the scenario I am describing may not be the most common, but it is a valid synchronisation scenario nevertheless, and it could be handled a bit better than what we currently have.

11

Oh, THAT is your use case! Then I recommend you do like this:

  1. Remove the Syncthing folder you have now, both on the laptop and on the NAS. On the NAS, also delete the entire directory. (FYI: When I write “folder” I always mean what you see on the left side of the WebUI and when I write “directory” I always mean the thing in a filesystem in which you can have other directories and files.)
  2. Laptop: Configure the folder as “Send Only”.
  3. Laptop: Check the box “Add ignore patterns“. This will give you the opportunity to enter ignore patterns BEFORE the folder is created.
  4. Laptop: Share the folder with the NAS.
  5. NAS: Accept the share.
  6. NAS: Configure the folder as “Send & Receive”.
  7. NAS: Do NOT use ignore patterns.

This should do it.

Alternative step 6: NAS: Configure the folder as “Receive Only”. This would give you a red “Override changes” on the NAS whenever Syncthing detects local changes.

Haha! Higher up I wrote:

Quite frankly, I consider the use of “.stignore” to be an advanced topic

This is even more true when it comes to editing the ignore patterns after a folder has been shared and accepted by others…

1 Like
12

I think I finally get what you are talking about. Sorry for being thick.

I believe that you DO want to make changes to the ignore rules on the laptop, and you want an option in Syncthing so that Syncthing on the laptop would announce the ignored stuff as deleted. Correct?

13

I want to thank you for willing to lend a hand, these few exchanges have been very pleasant and eye-opening, learning experiences.

The steps you just suggested now are exactly what I eventually trialed and error’ed into as well (by the way, for step 6 I go with the alternative - “Receive Only” - works exactly as you described) :slight_smile:

It works, but it is a semi-automated solution. Since it’s the home folder that I’m syncing, sometimes I notice on the NAS (looking at the most recently updated file) that garbage files (temporary/cache/cookies/databases) sometimes slip past my original ignore filters, so I have to adjust the .stignore on the fly.

Yes, exactly.

Currently, each time I adjust the .stignore on the laptop, I have to remove the folder (not directory) from the NAS, add it again, wait for the local scan to finish, then wait for the comparison to take place, and only then do I get the ‘Revert local changes’ button. It is literally always the same tedious, repetitive process with exactly the same outcome - all I’m saying is why not have it (optionally) automated?

This is why I am suggesting that “Receive Only” folders have an additional checkbox that reads:

Enforce delete

So that syncthing can do this whole process unattended - which is basically a 2 part process:

  1. Automatically (and immediately) propagate .stignore changes (eg. add/delete existing files, as per the SOURCE)
  2. Automatically (and immediately) discard/revert any local changes and always prefer SOURCE
14

The same to you!

I understand what you want, but IMO the option which would be needed to achieve what you (and sometimes me too!) want is not the one you proposed, but instead this:

Announce ignored files and directories as if they are deleted

This option would be relevant when the folder type is “Send & Receive” or “Send Only”.

The reason for this option (instead of the one you proposed) is that when you edit the ignore patterns on the laptop to ignore more stuff, the “Global State” does not change. (This is completely in line with the Syncthing design philosophy.) So the NAS will never know that the laptop is now ignoring more stuff, and hence the NAS would not be able to do anything to achieve your goal. That is why the option is needed on the sender side.

Thoughts?

15

Yeah, I’m all in - although I think there may be slightly more pushback on your suggestion because it has potential to globally affect any and all devices that share that folder, so data loss scenarios may be more frequent.

The reason I limited it to “Receive Only” Folders is because I could send-only that same folder from my laptop to multiple destinations and control how each destination handles it individually - I think this goes more in line with how syncthing usually does things.

But I’m 100% with you, the announcement has to come from the source folder in order for any of this to work, so I think we’re using different words to say the same thing :laughing:

1 Like
16

Thanks very much for the correction sir!

1 Like
17

Yeah, you might be right about the risk of unintended data loss, but it is not clear to me what the alternative is. But anyhow, I also believe that we are now standing on common ground. :slight_smile:

The core of the issue is this:

What do for example @calmh and @imsodin say about this problem and the suggestion solutions? Is this worthy of a ticket in github? I have searched, but could not find one which mentions this already.

1 Like