I need to make individual users sync directories with my server(s), where I would process files they sync and provide them resulting files through another synced directory. In this Syncthing environment node users do not have enough training to install an applications that requires configuration. They need to have something that “just works”. Is it possible to install a node using a configuration file I provide for the installer to use during installation? Is there an easy way? It is not obvious how to do that using the CLI; it is not maintained in any case. Also, I would need to get the node ID and it’s public decryption key to use them on my end and possibly for use by other nodes. I need to protect node users from each other and from hackers but unfortunately they are unable to protect themselves from me; or I do not know how to protect them from me. I am on Linux and the other nodes can be on Windows or Linux. I am going to suggest this for development if you say this would be a sane request.
It would be nice for a situation like this to let me use Syncthing itself to generate configuration files inside a folder on my machine and then provide that folder to the user to give it to the installer on his node machine?
I can generate the relevant configuration files using Syncthing, as it is today. I can install Syncthing in a clean VM and copy the relevant files for the new node. Is there hidden information not in the configuration files? What would be the relevant files for a Linux node and for a Windows node?
There still remains the need for Syncthing to use those configuration files during installation on the remote node; or as a workaround one can write a script to replace the configuration files with the ones provided to it. Will this work? Ideally, the script would just install Syncthing silently and copy the configuration I provide. I am not familiar with things like “Rest API”. I am a hardware person and not a software person. Has anybody done this already? I could not find a relevant post.
I don’t think what you are trying to do really fits in well with syncthing. It’s an application that needs to be setup, operated and monitored. When something goes wrong there will be zero feedback to your users/customers, which is not a good solution. Plus the benefit of syncthing is it’s decentralised peer to peer nature - that can form clusters on devices, etc, and your use case doesn’t sound like would use any of the selling points of syncthing. Your use case is centralised and point to point.
I think you might be better of building something yourself, something that is simple enough for your users to know how to operate. I obviously don’t understand your use case, but sounds like it’s a request response thing that could perhaps be a webpage?
This environment has very bad internet service where nodes get disconnected often. A relay can help if the source node gets disconnected before the file reaches the destination, which can get disconnected also. The file can be in the middle in a third node that would relay it to the destination, or several nodes can relay it. That way the throughput will be higher than without Syncthing. At least that is my understanding of what would happen. Am I hallucinating?
The relay doesn’t store files, its purely for connectivity purposes, it reads upto 4k bytes from one socket and writes it to the other socket. It doesn’t even know what the data is, as its encrypted via tls.
Thanks from waking me up from my dream. I will look at the simpler alternatives. Nothing I know about alleviates the interruption issue though. I am deep in the file processing rather in the connectivity, upload/download or syncing. I want a ready made syncing solution. I may have to come back. Thanks.
you want a script to install ST and config the user
the install script requests a pre-prepared config file. This is sent to a new user (or retrieved by the install script) and it auto-configs the users end
a users files are dumped into an intermediate server (why??)
you grab those files, suck them behind your own firewall, do something with them, then return them to the intermediary server
those post-manipulated files are then auto-sucked back to the users
Also
you want users NOT to see each others files (??)
obviously it needs to be secure against baddies
Also also
how long are users to remain users?
is this an office environment? Open internet?
do you know who the users are? (i.e.are they random people, or customers/members?)
I can see how much of this can be done scripted. since you are using Linux thats good.
My first thought are a few options… but if you can answer the questions above I can guide you.
First thoughts:
Create a shell script. Have new users grab it via URL, install ST, grab the Identification code, report that back to your server, set up the directory on your end, register the remote device on your side, auto test
sshfs … Same as above. Script to download. Have script create RSA key and report that back to you. Script your end to add a new user. Auto test.
FreefileSync. Use the auto-trigger to do much the same thing.
FTP. Are all users in environments that can use FTP? In which case, there are options there…
This sort of mirrors my own needs. I prefer syncthing, but its far from effortless…
My users (friends/family) are not all “clueless newbs” but they have limited patience and not a lot of time…
The scripted, pre-install experience could be a lot better, allowing me, the “techy” informed user, to easily set up the non-techy, halfway across the world user.
Yes, it would be amazing if there were a server-package that could do this - install it, hook it to an existing sync-thing share, provide a url to the “end” user.
Result of the script/app being that syncthing is installed on their computer/phone and linked to the share I wanted to link them to.
That sounds promising. I’m not sure if it meets the OP’s requirements, but…
[EDIT]:
Nevermind, not what I had in mind at all. I don’t want to tell my users to download a relatively complex file sync manager solution in addition to the somewhat complex syncthing.
TBH, I’m not sure this provides much beyond WinSCP + sshfs/sftp.
Generating such a script should be easy enough from within our frontend code. I guess if someone took the time to develop the feature, we wouldn’t be opposed to include it with the Syncthing packages. For starters, I’d recommend implementing it as a separate GUI version, similar to our already included next-gen GUI. Basically a very simple page offering to choose one or more of the existing shared folders, enter a device name and upon the click of a button, offer an appropriate PowerShell or whatever script to install Syncthing with those folders pre-configured, or a ready-made ZIP file to use as portable executable.
I think the complexity can be kept low by not trying to integrate it right away. When it works nicely and has a sane implementation, we can still port it over to the main GUI with little effort.
Absolutely. Most users know the computer only as far as moving the mouse and pushing a button.
These days, many users probably don’t care to remember phone numbers and cannot add numbers or multiply; the computer or phone does that for them.
I like FreeFileSync, WinsCPY and Filezilla. I think those are great products and I commend those who contributed to them; however, most users will need to go through a learning curve to configure and use them, and it can be a steep one for some. Also they can be dangerous tools in the hands of the untrained. If those tools will do what we ask, I am not averse of using them.
Users are already familiar with the Windows Human Interface that is Windows Explorer (or Linux’s GUI or CLI if they want), we can use that interface rather than force them to learn to use or configure another kind of “Windows Explorer”; in my opinion.
I am told that Webdav has security concerns and, as far as can I tell, once a directory is synced on the server by Webdav, the back-end outside of Webdav must not access those directories; the back-end outside of Webdav can only access those directories by being a Webdav client. I understand why that is if file locking is used for multi-user access; but I do not understand it if users are not going to try to open files simultaneously. In my application they do not open files in synced directories simultaneously.
Sftp has lingering bad reputation from the days of insecure ftp and command line; but Sftp using things like SSHFS-Win configured for sftp only (no ssh login) on the server is not to be laughed about but is an incomplete solution for my application; and sadly development has stalled. Syncthing is way ahead of it; and what I am asking for seems to me to be a small step forward.
This group is on the right track and I am glad I kindled interest. Unfortunately I am a hardware person designing semiconductor integrated circuits and I do not have the software skills necessary nor the time to go and master it to contribute. You may consider me as one of those who move the mouse and press buttons, (that is not true, I use Linux and I regard MS Windows as a big virus. I wish people stop using it)
I don’t think spreading FUD about WebDAV or SFTP helps in this regard, though. What security concerns are you talking about specifically? Who gives those solutions any reputation? I do have a WebDAV share running for example where the same files in the “backend” are synchronized with others through Syncthing, no doubt about that being a possibility, but it may be implemented very differently among WebDAV servers.
Syncthing is fundamentally different from both of them, as it works peer to peer in a decentralized manner. Every device is created equal.
If you have room in your concept for a central server that is reachable by all other nodes, then using WebDAV or SFTP are perfectly valid solutions. It really depends on the workflow you’re trying to establish.
Your first sentence sounds like a political campaign for Trump or Biden. Nobody is trying to spread FUD about anything and nobody has anything to gain or loose from FUD. I arrived at my erroneous conclusions reading forums and blogs; and if you read the subject heading of my post you will find that I am seeking advice and guidance; and some people have graciously already expressed willingness to provide. That is my intention and not FUD, so please do not shoot me.
I have tried to explain that I am not a software or IT guru; I am a fool doing semiconductor design using computers. I do special language programming, general language programming, shell and other scripting necessary for the design and verification process; but the IT and other programming was always done by somebody else. I explained I am looking for a ready made solution and that I do not have the time to go through a learning curve and learn what I do not know about IT. If I am able to know the ins and outs so as to be able to answer your questions the way you put them then I would not be seeking advice. I am very highly technical (Rocket Science) and I always appear to know what I am talking about. This is what I want to do:
1 – Have users get connected to my server in isolation from each other.
2 – Users must not be expected to configure software; they can only move a mouse and press a button; they can be office workers not interested in computers. (We used to call this kind of thing “fool proof”.)
3 – Users cannot and must not access each other’s work. Thus file locking is not required.
4 – User will upload files to folders on my server (Linux).
5 – My rocket science application written in C++ will take uploaded files, process them then place the resulting files in an appropriate download directory for the respective user to download. Each function or kind of processing required necessitates the use of different upload/download directories.
6 – Each user will have multiple upload directories and multiple download directories. Directory trees acceptable. Structure decided by my server.
7 – Users are familiar with Windows Explorer; do not make them learn another “Windows Explorer”. WinSCP, etc are excellent, but bloated with functionality we do not need, require much configuration, require knowledgeable users and are dangerous in the hands of a fool; plus they are another “Windows Explorer”. WinSCP has scripting capability, can run headless, do SFTP and do Windows network drive. SFTP with SSHFS-Win is much less bloated, mush simpler and safer in the hands of a fool.
8 – Security is paramount.
9 – I think Syncthing would be eminently suitable, especially with the suggestions I made and the contributions of others on this thread.
9a – If I configure a user node in a clean VM on my server, and pass the resulting configuration files to a user to place them in his configuration, which files are these and where are they? I can ask the user to run Syncthing install on his node with a simple configuration using the install gui. It does not matter what he does because the .batch file I provide him will replace the configuration files with a configuration I provide. (Will this work today.)
9b – Instead of using a clean VM like 9a to produce configuration files, I can edit a copy of my own XML configuration to suit the other nodes. What should I be doing? (I am not familiar with XML.) Many of the nodes will have identical or similar configuration. (IDs and encryption keys are unique.)
9c – The procedures of item 9a and 9b should probably be in the documentation.
10a – If you give me a ready made solution using Webdav, SFTP or $^#$@^ I will try it.
10b – Minimal configuration required. Assume I can only use the mouse and press a button. (Assume I am a little more than a fool.)
Completely understandable, but from personal and professional experience as a sysadmin, the solution you’re asking for is going to require more effort on your part than it sounds like you’re able to provide.
At first it sounded like it was a work project, but you also mentioned your users being friends and family. Those are two entirely different groups that require different solutions – i.e., it’s unlikely that remotely managing all of your friends’ and family’s devices is an option.
Very straight forward. Lots of options with varying degrees of isolation. The more sandboxing is desired, the more effort it requires on your part.
For every foolproof solution, there’s eventually going to be a more skilled “fool”.
End-users do some of the most unexpected things. Apple and Google spend billions hardening iOS and Android not only from hackers, but also the users themselves.
The difficulty I foresee is that although you’ll be aware when a Syncthing connection is offline for an extended period of time, you won’t necessarily know why. And if your users cannot configure software, they aren’t likely to be able to help much with the diagnostics. That means keeping the end-user part as simple as possible is ideal, which pretty much rules out Syncthing or anything else like it.
Easily done with most types of file server software.
Is your server accessible from the internet without a VPN and/or other special requirements?
Sounds straight forward.
Pretty normal requirement for a file server.
Noted.
That’s a pretty broad scope…
Are username + password authentication sufficient, or is something more secure required?
Does ensuring that file transfers are only over SSL/TLS or some other form of encryption cover it?
Does the data have to be protected while at rest?
Your particular use case requires a level of ease-of-use and minimal personal effort that might not be as good a fit for Syncthing.
At this point, if you hit [Ctrl + C] after Syncthing completes its initialization process, your $HOME/rocketscience directory would look something like this…
…, a self-contained configuration with a Syncthing executable. You could even include the Windows version too for convenience.
Both Linux and Windows desktop users have a “Desktop” folder, so it’s probably the best place to locate your data folder (e.g. ~/Desktop/rocketsciencedata). It’s consistent, and it’ll feel natural to your users.
Once you have Syncthing configured as desired, $HOME/rocketscience is now your template.
For each user:
Shut down Syncthing.
Copy $HOME/rocketscience to be bundled for the specified user.
Delete the cert.pem and key.pem files.
Start up Syncthing. A new set of cert.pem and key.pem files – which is what the unique device ID is derived from – will automatically be generated.
Go to step 1.
Deployment could be simple as zipping up $HOME/rocketscience and sharing the .zip file with the intended user (how you securely do that is up to you to figure out). It might even be useful to make it a self-extract zip archive. Or if you want to get fancy, Inno Setup.
The parts about the file locations and config file are already well documented. For the rest of 9a and 9b the number of variables involved could fill a book and really has nothing to do with Syncthing because it’s a classic issue of remote software deployment and configuration.
Ansible, Puppet, Chef, Salt and a laundry list of other change management and/or configuration tools are actually better for what you’re seeking to do.
A web-based file manager meets all of your requirements except for the network transfer efficiency and resiliency provided by Syncthing.
Which brings up the question of have you considered using Dropbox, pCloud or some other suitable cloud storage service?
You could share a folder with each user for uploading/downloading via a web browser. Or if you’re a bit more adventurous, the desktop clients for Dropbox and pCloud (there’s no official Google Drive client for Linux). Like Syncthing, their client apps do block-level transfers and work on less than reliable network links.
Depending on how much data is involved it might be “free” or fairly low cost.
The conundrum is that there cannot be “fools” on both sides of the equation…
If the network connections were reliable, sure, but mounting network shares across what you’ve described isn’t going to work well.
Thanks everybody, your input is invaluable. I will need some time to digest it. I am a one man team building a prototype of an application and it is too early to tell anybody what I am doing. I have no support of any kind from anybody except this forum now. I know that this prototype will not be of production quality; I hope to get to the point where other people will be interested. By the way gadget, it is smaudet who is doing things for his family.
In a small family where one or both of the parents is the official/unofficial IT support, and the rest of the family members are willing participants, installing and managing software on all of the devices (likely purchased by the parents) is generally uncomplicated.
In an enterprise environment, all of the managed devices are owned by the enterprise, not the users. The users understand and accept it without (much) resistance. But I have yet to meet anyone who welcomed the chance to install employer-mandated software on their personally-owned devices.
With extended family and friends, if you have to sell them on installing and using a certain software package, you’ll be responsible if it doesn’t work and/or there’s a perception that the software is causing problems with something else even if it’s unrelated.
Of course every situation is different. Perhaps all of your family and friends asked you to find a way to share photos without using Google Photos, Yahoo!, Amazon, Facebook or other cloud-based options. It’d be a group decision with buy-in from everyone, so fewer chances for issues.
This post, and the observations in it Thank You for Multiple Connections ], seems to confirm my feelings as expressed in this post, even though complete files are not stored in relays.
