Dear Danell, Thanks a lot for mentioning the project!
Here I would like to highlight that there is at least one long term bug in Syncthing which prevent flawlessly using selective sync feature even realized by pure ignore patterns:
As far as I understand it thatās mostly a cosmetic issue. Given the three devices connected A ā B ā C itās impossible for C to get something from A thatās ignored on B, because it doesnāt have a connection to any device with that file. Selective sync wouldnāt change this. You canāt download a file from a device that doesnāt have it.
1 Like
Iām the author of that issue, so I can confirm that itās 100% cosmetic, although it does get annoying, since because of it the affected devices are never marked as āUp to Dateā in the GUI. Another similar issue is https://github.com/syncthing/syncthing/issues/8735, although this one is more serious, as it leads to Receive Only folders being stuck in a āLocal Additionsā state, even though they are actually in sync.
4 Likes
I posted a another thread with a feature request that I think would be a good solution for this, at least on Mac - File Provider.
Would be amazing if Syncthing supported this. I know this functionality is not super popular with Dropbox and Google Drive users at the moment, mostly because those apps had 1st-party implementations of various features that now work differently (or no longer work at all) with File Provider. But I donāt think that necessarily means it wouldnāt be a fit for Syncthing.
Would require the folder being synced to be located in ~/Library/CloudStorage but I think for certain workflows thatās an acceptable compromise, if the File Provider selective sync functionality could be implemented on a per-folder basis in Syncthing.
Is this actually the case? Could you provide a specific example when this happens?
Iāve just accidentally ran into this scenario again. A file was accidentally synced (as I made a mistake in the ignore pattern). Then I fixed the ignore pattern. Then I deleted the file on the remote end where I didnāt want it and triggered a rescan there. After that the file was still not deleted on the device where I fixed the ignore pattern.
So I was wrong about this. Iām not sure what lead me to think it works in the way Iāve described in my previous comment. Maybe when I ran into this scenario several years ago I still did not get the ignore pattern right the second time so the deletion was just propagated as usual.
1 Like
Yes, that would be too complicated to edit on mobile. My use case is I have a huge music library on my computer thatās shared with my phone, and on the phone I can select or deselect musicians I want synced with the phone, and can unselect them to save space, etc. Manually editing a text file on a phone in order to enter a bunch of band names would not be fun.
The purpose of sync is that each folder is updated when I update it on the computer, though (new albums, better mp3 tags, etc.)
I would like to get rid of Resilio Sync because it mangles datestamps and git repos, etc. but I donāt think syncthing works for my use cases, either. I donāt know what else exists.
3 Likes
I think we can all/most agree that creating ignore patterns can get complicated. It scared me away.
Perhaps a GUI aid of some sort to make The process of creating complex ignore patterns easier. I think everybody would love to be able to create ignore patterns using a GUI rather than knowing how to type it in via trial and error. I havenāt been using syncthing for more than six to eight months or so and in that time I have seen some people make relatively simple requests that turn into complex ignore patterns that I would never imagine ever being able to create on my own.
Anything at all that makes ignore patterns easier to implement and manage would be a fantastic addition to the product.
That feature would cure many peopleās problems with not having a selective sync function.
And just one more thing. AI can help in this case. I just asked ChatGPT To explain to me about syncthing ignore patterns. This is what it came up with and itās pretty good Even though the examples are very very basic:
"In Syncthing, ignore patterns allow you to specify certain files or folders that should be excluded from synchronization between devices. This can be useful in situations where you have files or directories that you do not want to share or sync across devices, or that are unnecessary to include in the synchronization process.
Ignore patterns are specified using regular expressions, which are a way of defining search patterns that can match specific text strings. Syncthing allows you to create a list of ignore patterns that are applied to all synchronized folders.
Here are some examples of ignore patterns:
To ignore all files with the extension ā.logā, you can use the pattern ā.log". To ignore all files in a folder called ātempā, you can use the pattern "/temp/ā. To ignore all files and folders that start with a dot, you can use the pattern ā.". You can also use more complex patterns that combine multiple criteria. For example, to ignore all files with the extension ā.logā and all files in a folder called ātempā, you can use the pattern "temp/, *.logā.
Itās important to note that ignore patterns are only applied to files and folders that have not been explicitly shared between devices. If a file or folder has been explicitly shared, it will be synchronized regardless of any ignore patterns.
Overall, ignore patterns can be a useful tool for customizing the synchronization process in Syncthing and ensuring that only the files and folders that you want to share are synchronized between devices."
That was ChatGPTās explanation of ignore patterns in Syncthing.
1 Like
One last thing: ChatGPT AI was able to put this together for me. I simply asked the AI engine how to ignore all music files and this is what it came up with:
how would I ignore all subfolders containing music files
ChatGPT To ignore all subfolders containing music files in Syncthing, you can use the following ignore pattern:
markdown Copy code
**/*.mp3
**/*.wav
**/*.flac
This pattern will match all files with the extensions .mp3, .wav, and .flac, regardless of their location within the synchronized folder structure. The **/ syntax matches any number of subdirectories, so this pattern will match files in any subfolder of the synchronized folder.
To apply this ignore pattern, you can add it to the āIgnore Patternsā field in the folder settings for the folder you want to exclude the music files from. Here are the steps to do this:
Open the Syncthing web interface and navigate to the folder you want to exclude music files from. Click on the folder settings icon (gear icon) on the right side of the screen.
Scroll down to the āIgnore Patternsā section. Add the above ignore pattern to the list of patterns, separating each pattern with a comma.
Click the āSaveā button to save the changes.
After adding this ignore pattern, Syncthing will exclude all subfolders containing music files from the synchronization process for this folder.
The danger with AI is that they can be confidently wrong with their statements. This is often mixed with thruth which makes errors hard to spot.
Most of what ChatGPT said looks correct, though some things are confusing or just plain wrong:
Wrong. Ignore patterns are per-folder within the syncthing UI, not global.
This is confusing, because /temp/* only ignores files in a top-level folder temp, but the description makes it sound like any folder named is matched. temp/* would make more sense with the given description. The * is also unnecessary here.
Syncthing uses one ignore pattern per line. Comma separation is incorrect. The first sentence also sounds like it refers to a logical āandā of patterns, which syncthing does not do.
This makes absouletly no sense. Unless itās talking about negated ignore patterns (! syntax), this is just wrong. It also doesnāt mention the negation syntax at all.
This may have been a markdown formatting issue, but IIRC this literally ignores a file named .. .* is what I would consider correct here.
This is okay, though
*.mp3
*.wav
*.flac
is simpler and does the same thing.
Using AI is fine, though please always fact check what it spits out and refer to official documentation for reliable information.
5 Likes
Thank you very much for your feedback and corrections. I have updated my notes. I try and document everything I learn about Syncthing in another program called Evernote.
I know that the aI chat bots are far from perfect. It gives me ideas and places to go and finalize the solution that Iām looking for so I use it as an aid not as a rule.
I find chat GPT very helpful for laying out Linux commands for me. They are always accurate. I once asked it how to make a program auto start at boot time and it gave me line by line instructions that worked perfectly so sometimes in the right situation itās a good thing,
I usually compare the output from chat GPT to BingAI and Google Bard. between the three of them I usually get the answer I need.
@Nummer378 has already pointed out a few problems in ChatGPTās answer. Hereās another one:
Ignore patterns do not use regular expressions. At least not what is usually considered āregular expressionsā. Ignore patterns work more like globbing otherwise known from shell programming. So the AI is mixing things up.
I would suggest you checkout the official documentation which is much more reliable than some possibly (in this case definitely) mixed-up output from an AI.
If you use AI for programming (and making up an ignore pattern kind of is) you have to assume the answer is wrong.
4 Likes
My two cents: A proper selective sync feature would not be based on an ignore list. That would be a selective ignore feature.
A folder configured as selective sync would have all files and folders ignored by default. Ignored files would be replaced by a file with an extension that is unique to and opens to sync thing and retrieves the file.
Files retrieved are added to the synced files list. Right click to remove from this or all devices. There should be really no other UI needed as a good implementation of this feature would simply use the existing OS.
3 Likes
I agree that this would be an extremely helpful feature for Syncthing to get. As a Resilio user for the past 5 years or more, this is really the only obvious thing missing in Syncthing, but itās an important thing. I like Sam Gās point about how this would work, but Iām not sufficiently knowledgeable to offer a pertinent opinion about the best means of implementation.
2 Likes
Agreed:
Itās simply my opinion that the ignore patterns being like shell programming or a descendant of shell programming makes it difficult for people that are not programmers.
Syncthing can easily, I think implement something a bit easier for the average user. The docs can be intimidating sometimes.
I have a similar program thatās Windows only that lets you right click on your selection of folders and add an exclusion or an inclusion filter just with the click and then itās easy to see the syntax so you can learn more about how to do it yourself.
1 Like
What about that ? lib/ignore Have a nice treeview for handling ignores - Next Gen Ignores by ibizaman Ā· Pull Request #5132 Ā· syncthing/syncthing Ā· GitHub Sadly the author abandoned it(?)
1 Like
I think the main problem is that there is really NO interface to the operating system GUI. The only interface is the Web browser. So if Syncthing creates empty/dummy files that āopen in syncthingā for them to be on-demand downloaded, there is really no program for them to open in.
For me this is still a really important feature. I think the ignore list capability means that a good bit of the underlying code may be ready for this kind of feature. But the fundamental problem of a lack of integration with the GUI of the OS means it really can only be managed from the web interface. I would propose the following: (Of course, I have no idea how to do itā¦)
- A folder can be identified as a selective sync folder. It should be assured at least when identifying a folder as selective sync that there is at least one device in the cluster that does not identify it as selective. That device would always have all the files in the folder.
- Empty/Dummy files are NOT created in the selective sync folder. Thereās no application to open them in. So thereās no point in them being there.
- In the web browser on the device with the selective sync folder, the user would be able to view the folderās directory tree and āenableā sync for files or subfolders.
- When files or subfolders are selected for syncing, they are transferred from the pool and placed appropriately in the directory tree in the filesystem.
- Files can be edited and would be synced back to the pool.
- When someone is ādoneā with the local copy, has made any changes they desire, and ready to clean up, they would simply go back into the web UI, and ādeselectā the subdirectory or files. Syncthing would then verify those files are in fact stored elsewhere in the pool, preferably on at least one device that is not selectively syncing, and then Syncthing would delete the local copy, and NOT leave the copy for the user to delete manually. This is REALLY important, because it would be too easy for a user to unsync files, and then delete other files theyāre still syncing by mistakeā¦ And then, of course, theyāre gone from the whole pool.
- There would have to be a substantial amount of testing particularly if individual file syncing was permittedā¦ I.e. if new files are created, those are also presumably selectively synced (not ignored). An alternative is to say selective sync only works on subfolders. And itās all or nothing (the whole subfolder or not).
Additionally I would advocate this selective sync, sync list be SEPARATE from the existing ignore list. Even if itās possible the ignore list capability is the underlying capability used to implement, the selective sync list should be completely managed from the WebUI, and the ignore list sort-of combined with the selective sync list which is perhaps hidden (Donāt need to show the user how the sausage is made.)
The benefit of this kind of solution is it doesnāt require any change to the architecture. There is no need to create āhelperā applications for each OS. Itās all managed through the WebUI.
Anyway, just my thoughts after using Resilio for years and switching to Syncthing for some of the reasons mentioned here.
Let me know if you guys think this makes senseā¦
3 Likes
I think itās fairly clear what needs to be done here, and I think all of the maintainers agree on that.
The problem is that nobody is interested in spending time on this, and all previous efforts by external contributors have stalled.
1 Like
If I knew how to code, Iād offer to assist with the additional feature because I like the idea.
One day weāll get there.
1 Like
Why not have #include file in the folder then edit it from the computer?
Because if youāre not at home, and you want to retrieve something on your phone, you canāt edit the file on the computer.
3 Likes