Manual intervention for Introducer feature

So I’ve been experimenting with this idea and tried to understand the Syncthing GUI code well enough to hack up a prototype.

The data structures in the web GUI seem rather inefficient, or at least with my very limited JavaScript knowledge, I struggle with walking through arrays to find the right objects. Sometimes there are strange function invocations of my code with an empty object as argument, but I’m sure these things can be sorted out during PR review :slight_smile:

Now what I haven’t looked at yet is the backend code. But one question came up just by looking at the GUI code: How would I get access to the information mentioned above, specifically the list of foreign devices from the cluster config message? It needs to be cached somewhere for sure, but would I introduce a new API endpoint to read it from the GUI or what other options do I have?

1 Like

Probably! :slight_smile: The GUI is a somewhat naive/half-hearted thing to begin with, which has since grown … organically, let’s say.


There’s been a lot of versions now. I had grand plans on a revised GUI but it’s a lot of work to undertake and … time/energy is finite as always.

For better or worse some things are persisted into the config, afaik, and are visible somewhere in the config editor. Unaccepted devices and folders are found there. We might treat devices that are announced but unknown similarly, maybe.

Historically we also expose a lot of such things as events when they happen, but that requires the GUI to be online at the time (-ish).

2 posts were split to a new topic: Rogue introducing

As for exposing the suggested devices as events, that would seriously limit this feature’s usefulness, as I imagine the process would be stretched over a longer time period.

So pushing the info to the config and therefore caching even over restarts seems like a good idea on second thought. Initially I was wondering what value this information has if it’s not up to date, but read from possibly ancient config entries. However, the opposite might be more limiting: only seeing “suggested” devices that have been announced by other nodes that we have been connected to since our last restart…

So probably handling the information similar to the <pendingDevice> entries is the way to go. But probably better to list them per folder, using new <candidateDevice> tags inside each <folder> section? They could look just like the current <device> tag, listing an ID string and therefore covering both known and unknown devices to suggest.

Now when would this information be cleaned up after living in the config? Not very nice to have long-gone devices being suggested forever…

  • Maybe some kind of timestamp for automatic expiry?
  • Or just remember where we got the respective ClusterConfig message from and weed out the list whenever that device connects again?
  • Allowing to consciously ignore device suggestions would be a logical next step as well.

I’m kind of getting comfortable around Syncthing’s code slowly as time permits, just looking for some guidance on architecture before I make some draft PR.

Ping @calmh, is this going in the right direction?

Maybe, I don’t know. Stuffing this (and the existing pending devices) into the config is kind of an ugly hack, also updating the config is not free as there are a lot of things listening to those updates and potentially acting on it. Could also persist it to the database somehow and expose it via a new REST method of some sort. Some thought and experimentation is probably required to find the best way.

So I have a prototype almost working. The backend successfully records the candidate devices for categories 2 (known but not directly shared) and 4 (unknown but shares indirectly) mentioned above. For now they are saved to each folder’s configuration because that is rather easy to prototype and examine during development. For better scalability, this needs to be reworked to live in the database, though. Same goes for the pendingDevice stuff, but not for ignoredDevice I guess.

For a frontend prototype, I’m trying to implement the splitting of “shared with” and “unshared” device groups (hidden if empty) in the folder edit dialog as a first step, which will be a first, separate PR. The “suggested devices” category will then fit right in between, and looks pretty good already.

Now a major blocker for me is the inefficiency of finding device objects in the $scope.devices array. Could that be made associative (a map just like $scope.folders) instead? I cannot oversee the consequences right now, but it seems like a lot of depending code to revisit / adjust. At least when copying back the device list to $scope.config.devices it would need to be degraded to a simple array again.

@calmh, could use some help at this point for untangling the GUI code. Or who else has a deeper understanding of the “grown” GUI code?

It’s a sorted list, so that might help with more efficient lookup. But it’s probably easier to just have another object, e.g. $scope.deviceMap if that’s required for what you intend to do.

I don’t xD

Generally even if it’s work in progress, pushing a branch/PRing and then ascing on the specific code makes it probably easier to answer stuff (unless it’s clear high-level questions).

@imsodin Thanks for your comments. I actually solved it by reworking the code to only calculate the lists once when opening the modal dialog, and temporarily storing the result within $scope.currentFolder.

So the first part, separation of the shared devices list into two categories, is now here:

Now I’ll adjust the suggested devices prototype and make a draft PR once it’s usable.

Do we still have to support ancient browsers (e.g. IE)? If not, we could just use this, instead of building the deviceMap:

$scope.devices.find(d => d.deviceId === n.deviceID)

Also, the default function param value in line 1727 in syncthingController.js is not supported by ancient browsers.

I tried to stay close to the original coding style, where rather verbose function (x) { return x.attr; } constructs are often used. Plus I’m not really experienced in JavaScript, so no idea what didn’t work ages ago :wink:

IMO code readability is more important than keeping compatibility with ancient software at all costs.

@wweich, sorry I missed your main point with the code snippet. I found it quite inefficient to walk the devices array for a linear search, because that happens inside another loop. When the number of known devices grows, I suspect the hashing starts to pay off soon, but that really depends on the browser’s JavaScript implementation. Which old browsers are probably especially bad at.

My current prototype is now available as a PR. Looking forward to your feedback and suggestions!

What exactly is the use case here? According to usage data the average user has 3 devices. How can there be any useful suggested devices? And I think if a computer is part of a larger Syncthing network the GUI is probably not used for admin. I’m all for expanding Syncthing but in my opinion this adds a lot of code for a very limited audience.

I’d like to see that data in slightly more detail. From my understanding, it says that >50% of clusters have no more than three devices. That means about 49% could have >=4 devices already, which is not a minority. Looking from the other side, only 5% have 11 or more devices, but maybe 4 is the oddity and ~40% have five and more devices? Just not quite sure if that is a valid justification for leaving out possible features.

Consider this: Everyone who tries Syncthing starts out with one or better two devices, and maybe never puts it to better use. They certainly wouldn’t care, especially since for the all-connected-already case, it looks almost like before.

For my own usage (25 devices, ~30 folders as far as I can oversee), keeping track of which devices share (or worse don’t share) a folder was the major sore spot. The data is already available, I’m just making it discoverable in the GUI, so not that much additional code complexity to manage. I talked to several Syncthing-using people I know personally about this idea, and everyone thought it would be a very useful addition.

So I think for everyone who really puts Syncthing to work (meaning a cluster that you cannot draw on paper from the top of your head), there is currently either:

  • The Introducer feature. Kind of centralized and not easy to see what’s going on. Delegates trust to the introducer. Not needed for the “average” three-device-cluster.
  • External management tools. The 2.2k devices largest known cluster probably uses this. Costs time and / or money. Definitely not needed for the “average” three-device-cluster.

I’d like to provide a middle ground through unobtrusive hints in the GUI. Folder sharing still has to be mutually acknowledged, like me calling up a colleague for this new project folder I created or him calling me to verify the sharing notification that just popped up. That yields a natural, decentralized mesh structure with explicit trust from each user. Just that the GUI helps with closing the gaps if desired.

The top only have ~11 devices and ~11 folders. How often do you really add a new device or a new folder? Is it really necessary that each device shares all folders which all other devices? If you go down this (more resource intensive) road you can take care of if by simply selecting all devices during first sharing (e.g. a share for all employees).

Also if you take e.g. a company with 100 notebooks on mobile connections, what good does it do if they are constantly burning through expensive data plans instead of just syncing with a master Syncthing node (which is backed up and 99% online anyways)?

And you also need to consider that Syncthing, although it is an excellent piece of software, only can be used in (work) environments where each node has similar rights to view/change the data, e.g. in a company you would not share accounting data this way. So while there may be exciting use cases like gathering weather data, etc. I think most of these setups are not a mesh system.

This is not about to be the most often used feature in Syncthing :slight_smile: Granted, for really small clusters it does not matter. It gets interesting when it starts to grow, devices get added and the new participants get clear hints on what the existing cluster looks like and what links are missing. Also, previous cluster members are reminded that the newcomer could be connected directly. Before that point in time, the feature is completely invisible. Maybe it will even positively affect the statistical numbers by making it easier to grow clusters.

If you re-read the first post in this thread, I precisely had in mind clusters with an intentionally incomplete mesh topology, and avoiding automatic introduction.

The problem I’m trying to solve is what happens when devices get added later, not on initial sharing.

Some users choose Syncthing especially for its decentralized nature, where it’s possible to sync opportunistically whenever two devices meet, even without Internet access. For lots of people, “connected to a network” does not imply “connected to the Internet”.

Frankly I don’t understand why you seem to be arguing against this feature. Especially now, eight months after I proposed it with an explicit request for discussion, and after I put in several days of developing a working prototype. I’m willing to dig in all the way and finish it, too. So this is certainly not your typical please-work-hard-to-support-my-exotic-use-case request.

Having said that, I think the UI has room for improvement and I welcome suggestions on the looks and wording to make it more helpful than confusing. I now lack the external perspective of someone who hasn’t thought about lots of details :slight_smile:

I did not read your first entry back in 2018, as I’m not around so much as last years. Also over the years there have been many suggestions that have never been worked on. In my opinion this PR only serves a very specific use case (yours?) and I don’t want you to spend your time and effort (thanks for that) working on something that will hardly be used once implemented. Your words like “cluster members” reminded me of a file sharing software Direct Connect from many years ago where users could setup P2P hubs. Is this what you are trying to achieve?

I was writing something longer but essentially it comes down to this: Whether or not this serves the canonical use-case is mostly a topic to debate, when it comes to a general sentiment of “should something happen”. If there is someone doing the work, then the question left to ask are: Is it in line with the purpose of Syncthing and is it maintainable? I don’t yet see anything speaking against the former and the latter is to be determined in review. Details like whether the distinction should be hidden for small numbers of devices are definitely up for discussion. To me that also seems pretty unproblematic, as it just uses a bit of vertical screen estate, while horizontal space is the one that’s usually missing.

I agree that this is useful information that we are not showing, but I kind of disagree where this is being exposed.

I’d create a new menu item with a modal called “nearby/neighbouring devices”, which shows this information and allows adding devices and sharing shares, without cluttering the existing workflow.

Also, less accidental select all’s that people can click.

I am definitely oppose storing this info in the config, as I already made the mistake with the pending devices/folders.