Selective Sync Feature(most important)

Feature
39

Since I don’t use SyncThing at the moment, I have to agree, if deleting files which are in the .ignore file caused a change (i.e. they weren’t actually ignored), I would be very surprised.

I would expect that if there is a device with a selective sync folder, then deleting files from it would not delete them on the fully synched devices.

The best example I can give is that with ResilioSync you don’t need t sync an entire folder, but can double-click a placeholder file to sync and open that file. When you are done with the file, you can delete it from your local disk and a new placeholder file will be created. This is useful because you may have folders with 100s of GB of data, but you only want to sync 1 or 2 files you are working on at the moment.

On the other hand it’s not to say that any such functionality has to exactly copy Resilio or DropBox - anything would be better than nothing, so long as it is clearly defined and documented.

40

Selective Sync would be wonderful to have but it the task to add it would be too big I think implementing something like syncthing-pyselective would mostly be good enough.

Having something like that directly in the ignore patterns page would be great. You could switch between the raw text version (how it currently look like) or the GUI version that would look like pyselective where you can select what folders and files you wish to sync or not sync.

2 Likes
41

Dear Danell, Thanks a lot for mentioning the project!

Here I would like to highlight that there is at least one long term bug in Syncthing which prevent flawlessly using selective sync feature even realized by pure ignore patterns:

42

As far as I understand it that’s mostly a cosmetic issue. Given the three devices connected A → B → C it’s impossible for C to get something from A that’s ignored on B, because it doesn’t have a connection to any device with that file. Selective sync wouldn’t change this. You can’t download a file from a device that doesn’t have it.

1 Like
43

I’m the author of that issue, so I can confirm that it’s 100% cosmetic, although it does get annoying, since because of it the affected devices are never marked as “Up to Date” in the GUI. Another similar issue is https://github.com/syncthing/syncthing/issues/8735, although this one is more serious, as it leads to Receive Only folders being stuck in a “Local Additions” state, even though they are actually in sync.

3 Likes
44

I posted a another thread with a feature request that I think would be a good solution for this, at least on Mac - File Provider.

Would be amazing if Syncthing supported this. I know this functionality is not super popular with Dropbox and Google Drive users at the moment, mostly because those apps had 1st-party implementations of various features that now work differently (or no longer work at all) with File Provider. But I don’t think that necessarily means it wouldn’t be a fit for Syncthing.

Would require the folder being synced to be located in ~/Library/CloudStorage but I think for certain workflows that’s an acceptable compromise, if the File Provider selective sync functionality could be implemented on a per-folder basis in Syncthing.

45

Is this actually the case? Could you provide a specific example when this happens?

I’ve just accidentally ran into this scenario again. A file was accidentally synced (as I made a mistake in the ignore pattern). Then I fixed the ignore pattern. Then I deleted the file on the remote end where I didn’t want it and triggered a rescan there. After that the file was still not deleted on the device where I fixed the ignore pattern.

So I was wrong about this. I’m not sure what lead me to think it works in the way I’ve described in my previous comment. Maybe when I ran into this scenario several years ago I still did not get the ignore pattern right the second time so the deletion was just propagated as usual.

1 Like
46

Yes, that would be too complicated to edit on mobile. My use case is I have a huge music library on my computer that’s shared with my phone, and on the phone I can select or deselect musicians I want synced with the phone, and can unselect them to save space, etc. Manually editing a text file on a phone in order to enter a bunch of band names would not be fun.

The purpose of sync is that each folder is updated when I update it on the computer, though (new albums, better mp3 tags, etc.)

I would like to get rid of Resilio Sync because it mangles datestamps and git repos, etc. but I don’t think syncthing works for my use cases, either. I don’t know what else exists.

1 Like
47

I think we can all/most agree that creating ignore patterns can get complicated. It scared me away.

Perhaps a GUI aid of some sort to make The process of creating complex ignore patterns easier. I think everybody would love to be able to create ignore patterns using a GUI rather than knowing how to type it in via trial and error. I haven’t been using syncthing for more than six to eight months or so and in that time I have seen some people make relatively simple requests that turn into complex ignore patterns that I would never imagine ever being able to create on my own.

Anything at all that makes ignore patterns easier to implement and manage would be a fantastic addition to the product.

That feature would cure many people’s problems with not having a selective sync function.

And just one more thing. AI can help in this case. I just asked ChatGPT To explain to me about syncthing ignore patterns. This is what it came up with and it’s pretty good Even though the examples are very very basic:

"In Syncthing, ignore patterns allow you to specify certain files or folders that should be excluded from synchronization between devices. This can be useful in situations where you have files or directories that you do not want to share or sync across devices, or that are unnecessary to include in the synchronization process.

Ignore patterns are specified using regular expressions, which are a way of defining search patterns that can match specific text strings. Syncthing allows you to create a list of ignore patterns that are applied to all synchronized folders.

Here are some examples of ignore patterns:

To ignore all files with the extension “.log”, you can use the pattern “.log". To ignore all files in a folder called “temp”, you can use the pattern "/temp/”. To ignore all files and folders that start with a dot, you can use the pattern “.". You can also use more complex patterns that combine multiple criteria. For example, to ignore all files with the extension “.log” and all files in a folder called “temp”, you can use the pattern "temp/, *.log”.

It’s important to note that ignore patterns are only applied to files and folders that have not been explicitly shared between devices. If a file or folder has been explicitly shared, it will be synchronized regardless of any ignore patterns.

Overall, ignore patterns can be a useful tool for customizing the synchronization process in Syncthing and ensuring that only the files and folders that you want to share are synchronized between devices."

That was ChatGPT’s explanation of ignore patterns in Syncthing.

1 Like
48

One last thing: ChatGPT AI was able to put this together for me. I simply asked the AI engine how to ignore all music files and this is what it came up with:

how would I ignore all subfolders containing music files

ChatGPT To ignore all subfolders containing music files in Syncthing, you can use the following ignore pattern:

markdown Copy code

**/*.mp3

**/*.wav

**/*.flac

This pattern will match all files with the extensions .mp3, .wav, and .flac, regardless of their location within the synchronized folder structure. The **/ syntax matches any number of subdirectories, so this pattern will match files in any subfolder of the synchronized folder.

To apply this ignore pattern, you can add it to the “Ignore Patterns” field in the folder settings for the folder you want to exclude the music files from. Here are the steps to do this:

Open the Syncthing web interface and navigate to the folder you want to exclude music files from. Click on the folder settings icon (gear icon) on the right side of the screen.

Scroll down to the “Ignore Patterns” section. Add the above ignore pattern to the list of patterns, separating each pattern with a comma.

Click the “Save” button to save the changes.

After adding this ignore pattern, Syncthing will exclude all subfolders containing music files from the synchronization process for this folder.

49

The danger with AI is that they can be confidently wrong with their statements. This is often mixed with thruth which makes errors hard to spot.

Most of what ChatGPT said looks correct, though some things are confusing or just plain wrong:

Wrong. Ignore patterns are per-folder within the syncthing UI, not global.

This is confusing, because /temp/* only ignores files in a top-level folder temp, but the description makes it sound like any folder named is matched. temp/* would make more sense with the given description. The * is also unnecessary here.

Syncthing uses one ignore pattern per line. Comma separation is incorrect. The first sentence also sounds like it refers to a logical “and” of patterns, which syncthing does not do.

This makes absouletly no sense. Unless it’s talking about negated ignore patterns (! syntax), this is just wrong. It also doesn’t mention the negation syntax at all.

This may have been a markdown formatting issue, but IIRC this literally ignores a file named .. .* is what I would consider correct here.

This is okay, though

*.mp3
*.wav
*.flac

is simpler and does the same thing.

Using AI is fine, though please always fact check what it spits out and refer to official documentation for reliable information.

4 Likes
50

Thank you very much for your feedback and corrections. I have updated my notes. I try and document everything I learn about Syncthing in another program called Evernote.

I know that the aI chat bots are far from perfect. It gives me ideas and places to go and finalize the solution that I’m looking for so I use it as an aid not as a rule.

I find chat GPT very helpful for laying out Linux commands for me. They are always accurate. I once asked it how to make a program auto start at boot time and it gave me line by line instructions that worked perfectly so sometimes in the right situation it’s a good thing,

I usually compare the output from chat GPT to BingAI and Google Bard. between the three of them I usually get the answer I need.

51

@Nummer378 has already pointed out a few problems in ChatGPT’s answer. Here’s another one:

Ignore patterns do not use regular expressions. At least not what is usually considered “regular expressions”. Ignore patterns work more like globbing otherwise known from shell programming. So the AI is mixing things up.

I would suggest you checkout the official documentation which is much more reliable than some possibly (in this case definitely) mixed-up output from an AI.

If you use AI for programming (and making up an ignore pattern kind of is) you have to assume the answer is wrong.

3 Likes
52

My two cents: A proper selective sync feature would not be based on an ignore list. That would be a selective ignore feature.

A folder configured as selective sync would have all files and folders ignored by default. Ignored files would be replaced by a file with an extension that is unique to and opens to sync thing and retrieves the file.

Files retrieved are added to the synced files list. Right click to remove from this or all devices. There should be really no other UI needed as a good implementation of this feature would simply use the existing OS.

53

I agree that this would be an extremely helpful feature for Syncthing to get. As a Resilio user for the past 5 years or more, this is really the only obvious thing missing in Syncthing, but it’s an important thing. I like Sam G’s point about how this would work, but I’m not sufficiently knowledgeable to offer a pertinent opinion about the best means of implementation.

54

Agreed:

It’s simply my opinion that the ignore patterns being like shell programming or a descendant of shell programming makes it difficult for people that are not programmers.

Syncthing can easily, I think implement something a bit easier for the average user. The docs can be intimidating sometimes.

I have a similar program that’s Windows only that lets you right click on your selection of folders and add an exclusion or an inclusion filter just with the click and then it’s easy to see the syntax so you can learn more about how to do it yourself.

55

What about that ? lib/ignore Have a nice treeview for handling ignores - Next Gen Ignores by ibizaman · Pull Request #5132 · syncthing/syncthing · GitHub Sadly the author abandoned it(?)

56

I think the main problem is that there is really NO interface to the operating system GUI. The only interface is the Web browser. So if Syncthing creates empty/dummy files that “open in syncthing” for them to be on-demand downloaded, there is really no program for them to open in.

For me this is still a really important feature. I think the ignore list capability means that a good bit of the underlying code may be ready for this kind of feature. But the fundamental problem of a lack of integration with the GUI of the OS means it really can only be managed from the web interface. I would propose the following: (Of course, I have no idea how to do it…)

  • A folder can be identified as a selective sync folder. It should be assured at least when identifying a folder as selective sync that there is at least one device in the cluster that does not identify it as selective. That device would always have all the files in the folder.
  • Empty/Dummy files are NOT created in the selective sync folder. There’s no application to open them in. So there’s no point in them being there.
  • In the web browser on the device with the selective sync folder, the user would be able to view the folder’s directory tree and “enable” sync for files or subfolders.
  • When files or subfolders are selected for syncing, they are transferred from the pool and placed appropriately in the directory tree in the filesystem.
  • Files can be edited and would be synced back to the pool.
  • When someone is “done” with the local copy, has made any changes they desire, and ready to clean up, they would simply go back into the web UI, and “deselect” the subdirectory or files. Syncthing would then verify those files are in fact stored elsewhere in the pool, preferably on at least one device that is not selectively syncing, and then Syncthing would delete the local copy, and NOT leave the copy for the user to delete manually. This is REALLY important, because it would be too easy for a user to unsync files, and then delete other files they’re still syncing by mistake… And then, of course, they’re gone from the whole pool.
  • There would have to be a substantial amount of testing particularly if individual file syncing was permitted… I.e. if new files are created, those are also presumably selectively synced (not ignored). An alternative is to say selective sync only works on subfolders. And it’s all or nothing (the whole subfolder or not).

Additionally I would advocate this selective sync, sync list be SEPARATE from the existing ignore list. Even if it’s possible the ignore list capability is the underlying capability used to implement, the selective sync list should be completely managed from the WebUI, and the ignore list sort-of combined with the selective sync list which is perhaps hidden (Don’t need to show the user how the sausage is made.)

The benefit of this kind of solution is it doesn’t require any change to the architecture. There is no need to create “helper” applications for each OS. It’s all managed through the WebUI.

Anyway, just my thoughts after using Resilio for years and switching to Syncthing for some of the reasons mentioned here.

Let me know if you guys think this makes sense…

1 Like
57

I think it’s fairly clear what needs to be done here, and I think all of the maintainers agree on that.

The problem is that nobody is interested in spending time on this, and all previous efforts by external contributors have stalled.

58

If I knew how to code, I’d offer to assist with the additional feature because I like the idea.

One day we’ll get there.