Selective Sync Feature(most important)

Dear Danell, Thanks a lot for mentioning the project!

Here I would like to highlight that there is at least one long term bug in Syncthing which prevent flawlessly using selective sync feature even realized by pure ignore patterns:

As far as I understand it thatā€™s mostly a cosmetic issue. Given the three devices connected A ā†’ B ā†’ C itā€™s impossible for C to get something from A thatā€™s ignored on B, because it doesnā€™t have a connection to any device with that file. Selective sync wouldnā€™t change this. You canā€™t download a file from a device that doesnā€™t have it.

1 Like

Iā€™m the author of that issue, so I can confirm that itā€™s 100% cosmetic, although it does get annoying, since because of it the affected devices are never marked as ā€œUp to Dateā€ in the GUI. Another similar issue is https://github.com/syncthing/syncthing/issues/8735, although this one is more serious, as it leads to Receive Only folders being stuck in a ā€œLocal Additionsā€ state, even though they are actually in sync.

3 Likes

I posted a another thread with a feature request that I think would be a good solution for this, at least on Mac - File Provider.

Would be amazing if Syncthing supported this. I know this functionality is not super popular with Dropbox and Google Drive users at the moment, mostly because those apps had 1st-party implementations of various features that now work differently (or no longer work at all) with File Provider. But I donā€™t think that necessarily means it wouldnā€™t be a fit for Syncthing.

Would require the folder being synced to be located in ~/Library/CloudStorage but I think for certain workflows thatā€™s an acceptable compromise, if the File Provider selective sync functionality could be implemented on a per-folder basis in Syncthing.

Is this actually the case? Could you provide a specific example when this happens?

Iā€™ve just accidentally ran into this scenario again. A file was accidentally synced (as I made a mistake in the ignore pattern). Then I fixed the ignore pattern. Then I deleted the file on the remote end where I didnā€™t want it and triggered a rescan there. After that the file was still not deleted on the device where I fixed the ignore pattern.

So I was wrong about this. Iā€™m not sure what lead me to think it works in the way Iā€™ve described in my previous comment. Maybe when I ran into this scenario several years ago I still did not get the ignore pattern right the second time so the deletion was just propagated as usual.

1 Like

Yes, that would be too complicated to edit on mobile. My use case is I have a huge music library on my computer thatā€™s shared with my phone, and on the phone I can select or deselect musicians I want synced with the phone, and can unselect them to save space, etc. Manually editing a text file on a phone in order to enter a bunch of band names would not be fun.

The purpose of sync is that each folder is updated when I update it on the computer, though (new albums, better mp3 tags, etc.)

I would like to get rid of Resilio Sync because it mangles datestamps and git repos, etc. but I donā€™t think syncthing works for my use cases, either. I donā€™t know what else exists.

1 Like

I think we can all/most agree that creating ignore patterns can get complicated. It scared me away.

Perhaps a GUI aid of some sort to make The process of creating complex ignore patterns easier. I think everybody would love to be able to create ignore patterns using a GUI rather than knowing how to type it in via trial and error. I havenā€™t been using syncthing for more than six to eight months or so and in that time I have seen some people make relatively simple requests that turn into complex ignore patterns that I would never imagine ever being able to create on my own.

Anything at all that makes ignore patterns easier to implement and manage would be a fantastic addition to the product.

That feature would cure many peopleā€™s problems with not having a selective sync function.

And just one more thing. AI can help in this case. I just asked ChatGPT To explain to me about syncthing ignore patterns. This is what it came up with and itā€™s pretty good Even though the examples are very very basic:

"In Syncthing, ignore patterns allow you to specify certain files or folders that should be excluded from synchronization between devices. This can be useful in situations where you have files or directories that you do not want to share or sync across devices, or that are unnecessary to include in the synchronization process.

Ignore patterns are specified using regular expressions, which are a way of defining search patterns that can match specific text strings. Syncthing allows you to create a list of ignore patterns that are applied to all synchronized folders.

Here are some examples of ignore patterns:

To ignore all files with the extension ā€œ.logā€, you can use the pattern ā€œ.log". To ignore all files in a folder called ā€œtempā€, you can use the pattern "/temp/ā€. To ignore all files and folders that start with a dot, you can use the pattern ā€œ.". You can also use more complex patterns that combine multiple criteria. For example, to ignore all files with the extension ā€œ.logā€ and all files in a folder called ā€œtempā€, you can use the pattern "temp/, *.logā€.

Itā€™s important to note that ignore patterns are only applied to files and folders that have not been explicitly shared between devices. If a file or folder has been explicitly shared, it will be synchronized regardless of any ignore patterns.

Overall, ignore patterns can be a useful tool for customizing the synchronization process in Syncthing and ensuring that only the files and folders that you want to share are synchronized between devices."

That was ChatGPTā€™s explanation of ignore patterns in Syncthing.

1 Like

One last thing: ChatGPT AI was able to put this together for me. I simply asked the AI engine how to ignore all music files and this is what it came up with:

how would I ignore all subfolders containing music files

ChatGPT To ignore all subfolders containing music files in Syncthing, you can use the following ignore pattern:

markdown Copy code

**/*.mp3

**/*.wav

**/*.flac

This pattern will match all files with the extensions .mp3, .wav, and .flac, regardless of their location within the synchronized folder structure. The **/ syntax matches any number of subdirectories, so this pattern will match files in any subfolder of the synchronized folder.

To apply this ignore pattern, you can add it to the ā€œIgnore Patternsā€ field in the folder settings for the folder you want to exclude the music files from. Here are the steps to do this:

Open the Syncthing web interface and navigate to the folder you want to exclude music files from. Click on the folder settings icon (gear icon) on the right side of the screen.

Scroll down to the ā€œIgnore Patternsā€ section. Add the above ignore pattern to the list of patterns, separating each pattern with a comma.

Click the ā€œSaveā€ button to save the changes.

After adding this ignore pattern, Syncthing will exclude all subfolders containing music files from the synchronization process for this folder.

The danger with AI is that they can be confidently wrong with their statements. This is often mixed with thruth which makes errors hard to spot.

Most of what ChatGPT said looks correct, though some things are confusing or just plain wrong:

Wrong. Ignore patterns are per-folder within the syncthing UI, not global.

This is confusing, because /temp/* only ignores files in a top-level folder temp, but the description makes it sound like any folder named is matched. temp/* would make more sense with the given description. The * is also unnecessary here.

Syncthing uses one ignore pattern per line. Comma separation is incorrect. The first sentence also sounds like it refers to a logical ā€œandā€ of patterns, which syncthing does not do.

This makes absouletly no sense. Unless itā€™s talking about negated ignore patterns (! syntax), this is just wrong. It also doesnā€™t mention the negation syntax at all.

This may have been a markdown formatting issue, but IIRC this literally ignores a file named .. .* is what I would consider correct here.

This is okay, though

*.mp3
*.wav
*.flac

is simpler and does the same thing.

Using AI is fine, though please always fact check what it spits out and refer to official documentation for reliable information.

4 Likes

Thank you very much for your feedback and corrections. I have updated my notes. I try and document everything I learn about Syncthing in another program called Evernote.

I know that the aI chat bots are far from perfect. It gives me ideas and places to go and finalize the solution that Iā€™m looking for so I use it as an aid not as a rule.

I find chat GPT very helpful for laying out Linux commands for me. They are always accurate. I once asked it how to make a program auto start at boot time and it gave me line by line instructions that worked perfectly so sometimes in the right situation itā€™s a good thing,

I usually compare the output from chat GPT to BingAI and Google Bard. between the three of them I usually get the answer I need.

@Nummer378 has already pointed out a few problems in ChatGPTā€™s answer. Hereā€™s another one:

Ignore patterns do not use regular expressions. At least not what is usually considered ā€œregular expressionsā€. Ignore patterns work more like globbing otherwise known from shell programming. So the AI is mixing things up.


I would suggest you checkout the official documentation which is much more reliable than some possibly (in this case definitely) mixed-up output from an AI.

If you use AI for programming (and making up an ignore pattern kind of is) you have to assume the answer is wrong.

3 Likes

My two cents: A proper selective sync feature would not be based on an ignore list. That would be a selective ignore feature.

A folder configured as selective sync would have all files and folders ignored by default. Ignored files would be replaced by a file with an extension that is unique to and opens to sync thing and retrieves the file.

Files retrieved are added to the synced files list. Right click to remove from this or all devices. There should be really no other UI needed as a good implementation of this feature would simply use the existing OS.

I agree that this would be an extremely helpful feature for Syncthing to get. As a Resilio user for the past 5 years or more, this is really the only obvious thing missing in Syncthing, but itā€™s an important thing. I like Sam Gā€™s point about how this would work, but Iā€™m not sufficiently knowledgeable to offer a pertinent opinion about the best means of implementation.

Agreed:

Itā€™s simply my opinion that the ignore patterns being like shell programming or a descendant of shell programming makes it difficult for people that are not programmers.

Syncthing can easily, I think implement something a bit easier for the average user. The docs can be intimidating sometimes.

I have a similar program thatā€™s Windows only that lets you right click on your selection of folders and add an exclusion or an inclusion filter just with the click and then itā€™s easy to see the syntax so you can learn more about how to do it yourself.

What about that ? lib/ignore Have a nice treeview for handling ignores - Next Gen Ignores by ibizaman Ā· Pull Request #5132 Ā· syncthing/syncthing Ā· GitHub Sadly the author abandoned it(?)

I think the main problem is that there is really NO interface to the operating system GUI. The only interface is the Web browser. So if Syncthing creates empty/dummy files that ā€œopen in syncthingā€ for them to be on-demand downloaded, there is really no program for them to open in.

For me this is still a really important feature. I think the ignore list capability means that a good bit of the underlying code may be ready for this kind of feature. But the fundamental problem of a lack of integration with the GUI of the OS means it really can only be managed from the web interface. I would propose the following: (Of course, I have no idea how to do itā€¦)

  • A folder can be identified as a selective sync folder. It should be assured at least when identifying a folder as selective sync that there is at least one device in the cluster that does not identify it as selective. That device would always have all the files in the folder.
  • Empty/Dummy files are NOT created in the selective sync folder. Thereā€™s no application to open them in. So thereā€™s no point in them being there.
  • In the web browser on the device with the selective sync folder, the user would be able to view the folderā€™s directory tree and ā€œenableā€ sync for files or subfolders.
  • When files or subfolders are selected for syncing, they are transferred from the pool and placed appropriately in the directory tree in the filesystem.
  • Files can be edited and would be synced back to the pool.
  • When someone is ā€œdoneā€ with the local copy, has made any changes they desire, and ready to clean up, they would simply go back into the web UI, and ā€œdeselectā€ the subdirectory or files. Syncthing would then verify those files are in fact stored elsewhere in the pool, preferably on at least one device that is not selectively syncing, and then Syncthing would delete the local copy, and NOT leave the copy for the user to delete manually. This is REALLY important, because it would be too easy for a user to unsync files, and then delete other files theyā€™re still syncing by mistakeā€¦ And then, of course, theyā€™re gone from the whole pool.
  • There would have to be a substantial amount of testing particularly if individual file syncing was permittedā€¦ I.e. if new files are created, those are also presumably selectively synced (not ignored). An alternative is to say selective sync only works on subfolders. And itā€™s all or nothing (the whole subfolder or not).

Additionally I would advocate this selective sync, sync list be SEPARATE from the existing ignore list. Even if itā€™s possible the ignore list capability is the underlying capability used to implement, the selective sync list should be completely managed from the WebUI, and the ignore list sort-of combined with the selective sync list which is perhaps hidden (Donā€™t need to show the user how the sausage is made.)

The benefit of this kind of solution is it doesnā€™t require any change to the architecture. There is no need to create ā€œhelperā€ applications for each OS. Itā€™s all managed through the WebUI.

Anyway, just my thoughts after using Resilio for years and switching to Syncthing for some of the reasons mentioned here.

Let me know if you guys think this makes senseā€¦

1 Like

I think itā€™s fairly clear what needs to be done here, and I think all of the maintainers agree on that.

The problem is that nobody is interested in spending time on this, and all previous efforts by external contributors have stalled.

If I knew how to code, Iā€™d offer to assist with the additional feature because I like the idea.

One day weā€™ll get there.

Why not have #include file in the folder then edit it from the computer?

Because if youā€™re not at home, and you want to retrieve something on your phone, you canā€™t edit the file on the computer.

1 Like