Manual intervention for Introducer feature

Feature
1

Hi all,

I’ve seen several threads here that circle about the Introducer feature sometimes re-adding devices that should not be. That gave me an idea for a new feature, a “soft introducer”.

My background is actually trying to maintain a mesh without total everyone-knows-everyone topology. Every device owner gets to choose what other nodes to connect to. There IS a central cluster node that everyone should have in their list. However, if everyone sets the introducer flag, the mesh will be completed automatically – not my goal as it could lead to performance problems with large clusters.

So I was thinking about putting a list of device IDs as a text file within some share, so everyone can pick their desired connections. However, the data is already there in BEP’s cluster config message.

So I propose for discussion:

  1. Memorizing the known other device IDs for a folder after a cluster config message (not persistent).

  2. Adding a link to each share in the GUI, with tooltip “List other devices you might want to share this with”. It will be hidden if all devices are already in our connection set. It should be non-intrusive so the user can easily ignore it if a complete mesh topology is not desired (so not a notification box).

  3. Show a list of “unknown” devices with the same share when clicking the link. Choosing an entry there will a) add the device ID and b) set the folder’s flag to be shared with that device. Note: The other end will have to acknowledge the connection request after getting a notification box and possibly verifying the device ID over an independent channel.

  4. For devices that are already known to us, but do not directly share a folder we both have, maybe also suggest enabling the respective “share with” flag. Haven’t thought out a GUI concept for this yet.

All of this is not meant to automate trust management, like introducers do. It just tries to simplify device ID distribution and manual setup of network topology.

Looking forward to your comments, regardless of who will have time to possibly implement which parts. I think it could be useful even to less sophisticated users that would not mock about with introducers, but don’t like reading long device ID strings over the phone.

1 Like
2

UI wise I wouldn’t add a new link for that, but put it in the “Share With Devices” section of the folder edit dialog (moving it to its own “Sharing” tab, which should already happen now to be consistent with the device dialog). There your point 4. is already present: a list of all known devices to share with. Now a second list of advertised but not yet added devices would appear there with some kind of explanation what it will do (add this new device). I actually think this would be quite nice, but you already mentioned the “time to possibly implement” problem :wink:

3

Actually the current list shows all known devices to share with. So maybe it should highlight those which are known to also have that folder ID shared with someone, but not us.

4

You’re right, I missed that aspect. So basically there are these categories of devices:

  1. Paired devices that we share the folder with.
  2. Paired devices that share the folder with a remote device, but not us.
  3. Paired devices that do not share the folder in any way as far as we know.
  4. Unpaired devices that share the folder with a remote device.

Now you’d have to somehow formulate/present that in a way, that the average user understands it - I think that would be hard. I don’t think visualizing 2. is important. The main benefit in my opinion is, to have all possible devices presented. Devices 1-3 are already present, devices 4 are not.

5

So your category 1 is the subset of checked items in the current list.

Categories 2 and 3 are currently impossible to distinguish as both are in the list as unchecked items. I see category 2 as one of the major improvements with my suggestion, because it helps to create “missing” links in the cluster for performance or better opportunistic sync behavior.

Technically there is another distinction within categories 2 and 3, because if we share a folder with a device, it might not do so the other way round. But for me it would be okay to not visualize such “pending acceptance” states, at least not in a first step.

So here’s my suggestion (any empty section should be hidden):

First section “Shared With (uncheck to revoke):” All the devices of category 1 with checked boxes. Unchecking and saving removes the folder sharing flag.

Second section “Suggested Devices (check to offer share and improve cluster connectivity):” Devices of category 2 with empty checkboxes, friendly name only. Checking and saving sets the folder sharing flag. Sorted below that, devices of category 4 with a plus sign instead of the checkbox, with the device ID and friendly name. That makes it clear that we have not verified who that ID stands for. Clicking the device name opens the “Add Device” dialog, with the respective folder sharing flag pre-selected.

Third section “Unrelated Devices (check to give them share access):” Devices of category 3 with unchecked boxes. Checking and saving sets the folder sharing flag.

So most of these UI changes can be done already without knowing about the category 4 devices. Moving the device list to a separate tab (like the folder list in the edit device dialog) should be done in any case for consistency. Should I create an issue for that?

Splitting the list as outlined above is beneficial to show the user whether checking a box will make data initially available to someone, or just close a missing link. Which is important information from a data security point of view, avoiding accidental sharing to the wrong guys (and gals).

I seem to remember @calmh has issued some critical remarks about having the cluster config message at all, so I’d kind of like to hear an opinion whether this proposal would make the collected information useful or stand in the way of rather yanking it from the protocol. How likely do you think the “memorizing the known other device IDs” part could be implemented, as a prerequisite for category 4 devices?

Finally, the same GUI concept should be applied to the “Shared Folders” list within the “Edit Device” dialog. But let’s get started on one thing first.

6

Ping? @calmh or another project leader, what do you think?

I’ll try to find some time at the end of this year to experiment with an implementation. Need to learn Go first, though… :roll_eyes:

7

I think something like this could be useful, but I don’t have too much of an opinion on the UI at this point. Certainly indicating somewhere that there are further possible devices to add / share with seems reasonable.

There’s no plan to remove the cluster config stuff, afaik. The one possible improvement I could see in the future is the ability to send a new such message without tearing down the connection, for when important properties change.

8

So I’ve been experimenting with this idea and tried to understand the Syncthing GUI code well enough to hack up a prototype.

The data structures in the web GUI seem rather inefficient, or at least with my very limited JavaScript knowledge, I struggle with walking through arrays to find the right objects. Sometimes there are strange function invocations of my code with an empty object as argument, but I’m sure these things can be sorted out during PR review :slight_smile:

Now what I haven’t looked at yet is the backend code. But one question came up just by looking at the GUI code: How would I get access to the information mentioned above, specifically the list of foreign devices from the cluster config message? It needs to be cached somewhere for sure, but would I introduce a new API endpoint to read it from the GUI or what other options do I have?

1 Like
9

Probably! :slight_smile: The GUI is a somewhat naive/half-hearted thing to begin with, which has since grown … organically, let’s say.

2012-11-19-0430-software-engineering-now-with-cats

There’s been a lot of versions now. I had grand plans on a revised GUI but it’s a lot of work to undertake and … time/energy is finite as always.

For better or worse some things are persisted into the config, afaik, and are visible somewhere in the config editor. Unaccepted devices and folders are found there. We might treat devices that are announced but unknown similarly, maybe.

Historically we also expose a lot of such things as events when they happen, but that requires the GUI to be online at the time (-ish).

10

2 posts were split to a new topic: Rogue introducing

11

As for exposing the suggested devices as events, that would seriously limit this feature’s usefulness, as I imagine the process would be stretched over a longer time period.

So pushing the info to the config and therefore caching even over restarts seems like a good idea on second thought. Initially I was wondering what value this information has if it’s not up to date, but read from possibly ancient config entries. However, the opposite might be more limiting: only seeing “suggested” devices that have been announced by other nodes that we have been connected to since our last restart…

So probably handling the information similar to the <pendingDevice> entries is the way to go. But probably better to list them per folder, using new <candidateDevice> tags inside each <folder> section? They could look just like the current <device> tag, listing an ID string and therefore covering both known and unknown devices to suggest.

Now when would this information be cleaned up after living in the config? Not very nice to have long-gone devices being suggested forever…

  • Maybe some kind of timestamp for automatic expiry?
  • Or just remember where we got the respective ClusterConfig message from and weed out the list whenever that device connects again?
  • Allowing to consciously ignore device suggestions would be a logical next step as well.

I’m kind of getting comfortable around Syncthing’s code slowly as time permits, just looking for some guidance on architecture before I make some draft PR.

12

Ping @calmh, is this going in the right direction?

13

Maybe, I don’t know. Stuffing this (and the existing pending devices) into the config is kind of an ugly hack, also updating the config is not free as there are a lot of things listening to those updates and potentially acting on it. Could also persist it to the database somehow and expose it via a new REST method of some sort. Some thought and experimentation is probably required to find the best way.

14

So I have a prototype almost working. The backend successfully records the candidate devices for categories 2 (known but not directly shared) and 4 (unknown but shares indirectly) mentioned above. For now they are saved to each folder’s configuration because that is rather easy to prototype and examine during development. For better scalability, this needs to be reworked to live in the database, though. Same goes for the pendingDevice stuff, but not for ignoredDevice I guess.

For a frontend prototype, I’m trying to implement the splitting of “shared with” and “unshared” device groups (hidden if empty) in the folder edit dialog as a first step, which will be a first, separate PR. The “suggested devices” category will then fit right in between, and looks pretty good already.

Now a major blocker for me is the inefficiency of finding device objects in the $scope.devices array. Could that be made associative (a map just like $scope.folders) instead? I cannot oversee the consequences right now, but it seems like a lot of depending code to revisit / adjust. At least when copying back the device list to $scope.config.devices it would need to be degraded to a simple array again.

@calmh, could use some help at this point for untangling the GUI code. Or who else has a deeper understanding of the “grown” GUI code?

15

It’s a sorted list, so that might help with more efficient lookup. But it’s probably easier to just have another object, e.g. $scope.deviceMap if that’s required for what you intend to do.

I don’t xD

Generally even if it’s work in progress, pushing a branch/PRing and then ascing on the specific code makes it probably easier to answer stuff (unless it’s clear high-level questions).

16

@imsodin Thanks for your comments. I actually solved it by reworking the code to only calculate the lists once when opening the modal dialog, and temporarily storing the result within $scope.currentFolder.

So the first part, separation of the shared devices list into two categories, is now here:

Now I’ll adjust the suggested devices prototype and make a draft PR once it’s usable.

17

Do we still have to support ancient browsers (e.g. IE)? If not, we could just use this, instead of building the deviceMap:

$scope.devices.find(d => d.deviceId === n.deviceID)

Also, the default function param value in line 1727 in syncthingController.js is not supported by ancient browsers.

18

I tried to stay close to the original coding style, where rather verbose function (x) { return x.attr; } constructs are often used. Plus I’m not really experienced in JavaScript, so no idea what didn’t work ages ago :wink:

IMO code readability is more important than keeping compatibility with ancient software at all costs.

19

@wweich, sorry I missed your main point with the code snippet. I found it quite inefficient to walk the devices array for a linear search, because that happens inside another loop. When the number of known devices grows, I suspect the hashing starts to pay off soon, but that really depends on the browser’s JavaScript implementation. Which old browsers are probably especially bad at.

20

My current prototype is now available as a PR. Looking forward to your feedback and suggestions!