I’d like to see that data in slightly more detail. From my understanding, it says that >50% of clusters have no more than three devices. That means about 49% could have >=4 devices already, which is not a minority. Looking from the other side, only 5% have 11 or more devices, but maybe 4 is the oddity and ~40% have five and more devices? Just not quite sure if that is a valid justification for leaving out possible features.
Consider this: Everyone who tries Syncthing starts out with one or better two devices, and maybe never puts it to better use. They certainly wouldn’t care, especially since for the all-connected-already case, it looks almost like before.
For my own usage (25 devices, ~30 folders as far as I can oversee), keeping track of which devices share (or worse don’t share) a folder was the major sore spot. The data is already available, I’m just making it discoverable in the GUI, so not that much additional code complexity to manage. I talked to several Syncthing-using people I know personally about this idea, and everyone thought it would be a very useful addition.
So I think for everyone who really puts Syncthing to work (meaning a cluster that you cannot draw on paper from the top of your head), there is currently either:
The Introducer feature. Kind of centralized and not easy to see what’s going on. Delegates trust to the introducer. Not needed for the “average” three-device-cluster.
External management tools. The 2.2k devices largest known cluster probably uses this. Costs time and / or money. Definitely not needed for the “average” three-device-cluster.
I’d like to provide a middle ground through unobtrusive hints in the GUI. Folder sharing still has to be mutually acknowledged, like me calling up a colleague for this new project folder I created or him calling me to verify the sharing notification that just popped up. That yields a natural, decentralized mesh structure with explicit trust from each user. Just that the GUI helps with closing the gaps if desired.
The top only have ~11 devices and ~11 folders. How often do you really add a new device or a new folder? Is it really necessary that each device shares all folders which all other devices? If you go down this (more resource intensive) road you can take care of if by simply selecting all devices during first sharing (e.g. a share for all employees).
Also if you take e.g. a company with 100 notebooks on mobile connections, what good does it do if they are constantly burning through expensive data plans instead of just syncing with a master Syncthing node (which is backed up and 99% online anyways)?
And you also need to consider that Syncthing, although it is an excellent piece of software, only can be used in (work) environments where each node has similar rights to view/change the data, e.g. in a company you would not share accounting data this way. So while there may be exciting use cases like gathering weather data, etc. I think most of these setups are not a mesh system.
This is not about to be the most often used feature in Syncthing Granted, for really small clusters it does not matter. It gets interesting when it starts to grow, devices get added and the new participants get clear hints on what the existing cluster looks like and what links are missing. Also, previous cluster members are reminded that the newcomer could be connected directly. Before that point in time, the feature is completely invisible. Maybe it will even positively affect the statistical numbers by making it easier to grow clusters.
If you re-read the first post in this thread, I precisely had in mind clusters with an intentionally incomplete mesh topology, and avoiding automatic introduction.
The problem I’m trying to solve is what happens when devices get added later, not on initial sharing.
Some users choose Syncthing especially for its decentralized nature, where it’s possible to sync opportunistically whenever two devices meet, even without Internet access. For lots of people, “connected to a network” does not imply “connected to the Internet”.
Frankly I don’t understand why you seem to be arguing against this feature. Especially now, eight months after I proposed it with an explicit request for discussion, and after I put in several days of developing a working prototype. I’m willing to dig in all the way and finish it, too. So this is certainly not your typical please-work-hard-to-support-my-exotic-use-case request.
Having said that, I think the UI has room for improvement and I welcome suggestions on the looks and wording to make it more helpful than confusing. I now lack the external perspective of someone who hasn’t thought about lots of details
I did not read your first entry back in 2018, as I’m not around so much as last years. Also over the years there have been many suggestions that have never been worked on. In my opinion this PR only serves a very specific use case (yours?) and I don’t want you to spend your time and effort (thanks for that) working on something that will hardly be used once implemented. Your words like “cluster members” reminded me of a file sharing software Direct Connect from many years ago where users could setup P2P hubs. Is this what you are trying to achieve?
I was writing something longer but essentially it comes down to this: Whether or not this serves the canonical use-case is mostly a topic to debate, when it comes to a general sentiment of “should something happen”. If there is someone doing the work, then the question left to ask are: Is it in line with the purpose of Syncthing and is it maintainable? I don’t yet see anything speaking against the former and the latter is to be determined in review. Details like whether the distinction should be hidden for small numbers of devices are definitely up for discussion. To me that also seems pretty unproblematic, as it just uses a bit of vertical screen estate, while horizontal space is the one that’s usually missing.
I agree that this is useful information that we are not showing, but I kind of disagree where this is being exposed.
I’d create a new menu item with a modal called “nearby/neighbouring devices”, which shows this information and allows adding devices and sharing shares, without cluttering the existing workflow.
Also, less accidental select all’s that people can click.
I am definitely oppose storing this info in the config, as I already made the mistake with the pending devices/folders.
I knew people using Direct Connect at my university, but haven’t myself. It’s not what I’m aiming for, no.
My personal Syncthing use just naturally led to a structure where it would be a useful addition. Exchanging pictures and documents with family members (partly intersecting groups, Dropbox alternative), recommending it to other people where (except myself) one to four people work on a project with shared data for some time, and sharing project data among co-workers for a small business. Each person has one to three devices, plus I run two NAS servers. So my overall device and folder count is above average, yes. But most people in those small groups have less, yet they would benefit from getting a hint when I set up a new laptop for myself, or add a smartphone to the mix.
My terminology is “cluster members” from kind of an admin perspective, but the goal is not having to tell two people: “I put a text file in this folder, please copy each others’ ID from there to enable direct sharing among you”. Instead I could tell them to open the GUI, edit the folder and follow the suggestions on the Sharing tab. Whatever other uses they might have put Syncthing to for their independent personal needs.
Adding someone new to my laptop’s Syncthing and on the NAS after I already did the initial setup with them on my smartphone will also become easier because the IDs are already listed, one click away instead of copy-and-paste.
It kind of follows the spirit of listing locally discovered device IDs when clicking “Add Device” by the way. Very useful IMHO.
As the suggestions are always in one folder’s context, can you envision an easy to grasp structure to show the relations? Or just a list of “proposed actions” to be checked off one by one?
Keep in mind that the existing workflow is not really altered up to the point when a candidate has actually been detected.
My longer term plan is to even mirror the function to the Edit Device dialog: “Folders you might want to share with this device, because they are already shared via detours.”
Agreed, that’s a little much. Better ideas anyone? I’ll keep thinking about that part.
I strongly agree. That was just because of @calmh’s first suggestion and easier for a prototype. I will try to move the pendingDevice stuff along once it’s clear where all that should live peacefully. That’s why it’s definitely a draft PR.
I think there could be a section per folder, where each section would list neighbour devices that have that folder and all available information about them.
So how about an earlier idea, show a cue symbol in a folder’s expanded section to open the list of suggested additional links? Example: (Freepik from www.flaticon.com is licensed by CC 3.0 BY).
That would avoid creating a totally separate entry point and will be easier to discover.
Aside from the GUI issues, do you have any pointer where to start with moving the persisted info to the database? Is there any other part of Syncthing storing similar data, as an example of how to structure the code?
Sorry I’m very new to both Go and LevelDB and so far reading the lib/model/ and lib/db/ code has not been very enlightening…
@AudriusButkevicius, how would you start for correcting the “mistake” of pending* stuff not living in the database?
You could take a key name in there and store the set of devices etc. If the thing you need to store is more complex than one of the already supported basic data types you would ideally need to declare a protobuf message for it. One place to do that could be in the protobuf stuff in lib/db, although it’s not particularly beautiful. You might also cheat and just store the cluster config message, which is already serializable.
Thanks for the hints, will look into it as time permits.
I think it’s better to store the candidate list per-folder, whereas the ClusterConfig message concerns one device. So the API handler would have to traverse them all. Or do you have other future uses in mind for cached ClusterConfig messages, @calmh?
The draft PR already outlines the needed information structure in the config entries (XML / JSON). Please review that part (mainly lib/config/observed.go) so I don’t have to fundamentally change what data is stored. I’d more or less mirror that structure to a database schema, with the same JSON representation for the API endpoint.
We don’t have anything else in the database in json format and I’m not super excited about adding that to the mix. Hence if you need something complex I think a protobuf message is the appropriate thing. The db package already declares some custom messages used in the database, so that could be one place for it.
Sorry that wasn’t very clear I suppose. I didn’t mean to put JSON serialized data into the DB. As you said, it will be recorded as a protobuf message in there.
I meant I will keep the semantic data structure the same, so that my new API endpoint (/rest/db/candidates maybe?) will output the same JSON structure as the candidateDevices list that the prototype code sends as part of the folder configuration.
So I hope to agree on this structure:
"candidateDevices": [
{
"deviceID": "...",
"certName": "",
"addresses": null,
"introducedBy": null // This device is already known
},
{
"deviceID": "...",
"certName": "",
"addresses": [
"dynamic" // accumulated from all ClusterConfigs
],
"introducedBy": [ // unknown device, keep track of all introducers
{
"time": "2019-06-05T10:21:22+02:00", // timestamp of last ClusterConfig from introducer
"deviceID": "...",
"name": "My Foo Device",
"address": "" // (leftover from reusing struct ObservedDevice, can be dropped)
}
]
}
],
For making this info available over the API, independent of the configuration, there is now a draft for the API endpoint documentation.
Moving the pending devices / folders information to the API as well is drafted in a separate pull request. I plan to finish that first, then build upon it for the candidate suggestions.
Sorry if bumping this old thread causes trouble for anyone to remember the context… But just a quick heads-up, this feature idea is still very much alive and I’m still working on it as time permits.
Regarding the GUI aspects, I sketched some more ideas how to show these “candidate devices” in an unobtrusive way. An additional category heading in the edit folder modal (see prototype PR) was too heavy on the eyes, so I tried to just highlight candidate devices within the “Unshared Devices” category appropriately and list them first. Completely unknown candidate devices still need their own category, but are now at the very bottom. See screenshot below:
The same principle applied to the “Edit Device” modal dialog, showing folders for which this device is a candidate (so unknown devices are out of scope):
Another suggestion from @AudriusButkevicius was about moving this info to a separate dialog altogether. I like the general idea, but maybe in addition to what is sketched above. It would allow browsing through (and acting on) all these suggested links in one central place, without clicking through all folder edit modals. Below are two mock-ups of such a dialog, where detailed information (from ClusterConfigs) about each suggestion can be expanded:
Granted, for really small clusters it does not matter. It gets interesting when it starts to grow, devices get added and the new participants get clear hints on what the existing cluster looks like and what links are missing. Also, previous cluster members are reminded that the newcomer could be connected directly. Before that point in time, the feature is completely invisible. Maybe it will even positively affect the statistical numbers by making it easier to grow clusters.
(
