I have been tasked with investigating using SyncThing to transfer files to our clients and was hoping to ask some questions.
Here are my requirements.
- I need to be able to setup a shared sync for some files and a separate sync per client for files specific to each. The non-shared files are never shared between any group of clients. Also, there must be no way that one client can gain access to another client’s files. They need to be authenticated in some fashion to ensure that no one else can access these files and download them. We don’t even want each client to have any way to detect any other client. They should only be able to sync with us.
- Is this possible to do without some extremely complicated setup such as setting up a separate sync process per client and one for core files?
My understanding is that I would setup some kind of unique certificate to authenticate each client. Is that correct?
- I need for any changes on the client side to be ignored and overwritten by the sync.
- From what I have read, SyncThing is not really designed for one-way sync, so if a client accidentally or on purpose were to change or delete any file that was synchronized, it would cause issues such as failures that are difficult at the least to resolve and would likely require some kind of manual intervention.
- At some point my bosses were thinking that we could also allow clients to send us files via SyncThing. These files can never be shared with other clients. Would enabling that cause issues with #2 or possibly require an additional sync process to allow that?
I appreciate any feedback that anyone can provide on this.
I was hoping to edit my post since I think saying “The non-shared files are never shared between any group of clients.” is confusing, unfortunately it appears that time has taken that away from me. Instead I should have said something like “The files specific to a client are never shared with any other.”
If the directories can be managed separately, that would not be a problem at first. The question, however, would be what proportion of manual interventions do you imagine? When it comes to exchanging files between clients anyway, things get more complicated. Therefore, this framework should be known.
How is this to be understood in the overall context? If these files could be in different directories, that wouldn’t be a problem either.
I wasn’t even thinking of setting up clients to allow them to obtain the shared files from each other, but instead for everyone to only get files from us.
Also no problem. It would be good, to read some facts in the documentation
And yes, all file groups are separated into different folders.
I have heard about that, but other forum posts indicate that this would not work well in our case. From https://forum.syncthing.net/t/how-to-properly-set-up-one-way-syncing-from-windows-folder-to-linux-box/15818:
Syncthing is not designed for one way syncing. You can use a send only or receive only folder types, but that still requires that the files are not touched in any way on the receiving side, as you’ll end up with local changes which will effectively stop the process.
I believe there are other posts about it as well.
Perhaps things have changed in regards to this behaviour?
The last bit is not correct: Local changes do not stop “the process”. Local changes stop changes to those local changes, but other files are still being synced.
Syncthing cannot ensure that files are not modified/deleted or new ones added on your recipient clients. A user application can’t do that, you’ll need to setup appropriate permissions. If you are in control of the recipients, you can absolutely setup directories that are read-only except for Syncthing.
Whether Syncthing is the ideal tool for a centralised, one-way setup is a different question. It definitely wasn’t designed for that purpose, then again it’s pretty versatile and you can do a lot of things with it - depending on how much effort you put into it.
I actually never thought of it like it blocked the entire sync process, but rather that those modified files will no longer be kept in sync. I assume this is what you meant as well. If so, that would not work for us.
Unfortunately, because it is our client’s servers and network, I believe we cannot be guaranteed to have and keep control over it.
Sounds like you need a tool, that can access some remote storage that isn’t under your control, overwrite everything in there to look like you want it to, regardless of what happened/is present there, and then Syncthing definitely isn’t for you. Sounds like a case for rsync.
Isn’t rsync a Linux tool?
I probably should have mentioned that this is all running on Windows.
I just threw in whatever was at the top of my head, which is linux not windows. I think rsync might even run on windows. And if it doesn’t, there’s unison. And there’s your favorite search engine which will clearly tell you what the “equivalent of rsync on windows” is. This isn’t a question about Syncthing anymore.
You could still use Syncthing to transfer the files though. Just keep a separate folder for each client, with both ends send-receive. Then changes from your client will get back to the server, but you can use any tool to overwrite them with your desired state again on the server, which will also get propagated back to the client. Just beware not to cause conflicts when your client changes stuff simultaneously to the server. But even those can be handled by the server, just deleting the conflict file and overwriting the original again.
I appreciate your response @imsodin. My remaining concern is that some people here may want to continue down the path of using Syncthing, regardless of this concern.
Is there anything you, or anyone else, can mention about the multiple client setup having access to a limited set of files as I mentioned above? For instance, would a client be able to manually add another client’s folder after they are setup to sync their own?
I appreciate that @acolomb, but one concern would be that before we revert the changes made by a client, it is distributed to others. Keep in mind that some files are common, while other are not, but we would be the only ones with the reliable master copy.
For instance, if a client got a virus that infected the common files, Syncthing could then end up distributing them to the rest of our clients. (I tried to put this as a separate response, but it won’t let me - 4 hour delay for newbies. It really sucks that I cannot reply to all the reponses. I do greatly appreciate them.)
You can’t prevent the other side from “adding” stuff.
You can choose to ignore that on your side.
Also, if the files of different clients reside in the same folder (syncthing folder, not physical folder), you can’t hide different files from different clients.
In general, I agree that syncthing is not the software you are looking for.
I still agree with Audrius/Obis stance of these aren’t the droids/softwares you are looking for. Still I recently came in contact with a proposition where my initial reaction was “You want to use Syncthing? You mustn’t use Syncthing!”. And while I maintain it’s not the ideal tool, they made it work quite nicely. Mainly my point is: A lot of things are possible
If you can keep files from a single client in a separate directory (as in on disk) on your central device(s), there’s no issue about clients interfering with each other. You can just have separate Syncthing folders for each client and go with @acolomb’s suggestion.
If you need to sync the same directory on disk to multiple clients, you might still get away with setting the central folder to send-only. Then have some tool that immediately triggers “override” when any client does any changes.
Anyway, sounds like we agree Syncthing isn’t the ideal tool and the rest are corporate concerns that in my point of view don’t benefit the community much. You can always contact kastelo.net for corporate assistance.
I wrote you must use a separate folder pair per client. Then your server can distribute the common files to each of them, if keeping one copy per client is feasible space wise. Deduplicating file systems underneath could help with that.
In such a setup, your clients will be sufficiently isolated as there is no common state. The server would need to be configured explicitly to share a folder among different clients, so that is totally under your control.
So in general, I think it’s not a perfect match but could well be utilized in such a a way, needing additional components though. Whether your clients’ IT would accept there is a folder on their machine that your company has full and only control over, that’s the part I have more doubts about.
Thank you for that @acolomb.
I think that having multiple copies of the common files, even if deduplicated, would probably introduce complexities that people would not know how to address. Perhaps some system to attempt to undo it could work though.
It sounds like you are saying that with that setup it would be difficult, if not impossible for one client to connect to and download another client’s set of files. Did I understand that correctly?
If so, that sounds like it would alleviate my concerns with one client managing to obtain a copy of another’s files.
(Yes! Finally able to reply. )
I do agree with you @imsodin that it doesn’t seem that corporate concerns would benefit the community much.
I was just trying to determine if there was anything else we should consider with any of the ideas if it was decided to pursue this further.
I greatly appreciate everyone’s input on this and want to extend my thanks to the community.
If anyone wants to contribute any further info, I will be happy to accept it.
Yes, access to files that are not explicitly shared with a specific remote device (client in your case) should be impossible. Unless a very serious bug were discovered in Syncthing or you fail to protect the server’s Web GUI appropriately.