Plugin framework

I’ve been thinking of a plugin framework of sorts, to enable some weird behaviors that people keep asking for, or more generally for doing things we haven’t anticipated in the main code. I figured I’d write up something like a proposal before starting to hack on it to see what others think.

Plugins are written in Javascript, interpreted by Goja (https://github.com/dop251/goja) or possibly Otto (https://github.com/robertkrimen/otto). Another option is Starlight (https://github.com/starlight-go/starlight) with Python-like syntax. (The principles are the same regardless of what syntax we might choose in the end, probably it’d boil down to which package is better in other respects.)

Plugin scripts are loaded from somewhere in the config dir ($CONFIGDIR/plugins/foo.plugin.js) and apply to all folders, or somewhere in the folder (.stfolder/foo.plugin.js) and then apply to that folder only. We provide a number of methods that can be used to register functionality in the plugin. For example:

onShouldIgnore(function (fileInfo) {
	// "this" on folder callbacks is the FolderConfiguration object
	// The fileInfo parameter is a FileInfo object.
	
	if (this.folderID == "small-files-folder" && fileInfo.size > 1<<30) {
		// we ignore files larger than a gigabyte
		return true
	}
	
	if (this.folderID == "noexec-folder"
		&& fileInfo.type == 0 // FILE_INFO_TYPE_FILE
		&& fileInfo.permissions & 0111 != 0) {
		// we ignore executable files
		return true
	}
	
	if (this.folderID == "stupid-folder" && fileInfo.name.length % 2 != 0) {
		// we ignore files with names of uneven length
		return true
	}
	
	return false
})
// assume predeclared constants
// FolderIdle = 0
// FolderScanning = 1
// ...
onFolderStateChanged(function (oldState, newState) {
	if (this.folderID != "hooks-folder"
		|| newState != FolderIdle
		|| oldState != FolderSyncing) {
		return
	}
	
	// We just completed syncing on hooks-folder, let's commit the changes.
	// Let's hope there's an "os" module provided and that Syncthing sets the
	// working directory to the folder path.
	os.Execute("git add -A ; git commit -m Syncthing")
})

One could imagine other potential callbacks; onDeviceConnected (can accept or refuse connections), onIncomingUpdate to tweak or transform incoming fileinfo updates, onConfigLoaded to transform the configuration, etc.

We would need to provide some functionality for use in the plugin:

  • “os” or something to execute commands
  • “fs” to perform file operations (copy files perhaps)
  • “console.log” which should go to our log
  • maybe, possibly, an HTTP module so that webhooks could be called
  • possibly other things as we go along

I think all of this could be implemented in a fairly self contained manner, passing along a plugin object that we’d call callbacks on at strategic points in the existing code.

Clearly there’d be a performance cost to some of these things (like onIncomingUpdate which might get called thousands of times when exchanging indexes etc), but ideally that cost would be near zero when plugins aren’t used.

Also clearly this is a huge footgun. If we did have onIncomingUpdate for example and allowed changes to FileInfos there that’s very powerful – you could implement things like ignore deletes and similar, but if you change things that shouldn’t be changed (file name or size, etc) you will obviously break syncing.

Similarly onConfigLoaded could be used to implement mandatory policies (all folders shall have ignore-permissions and modtime-window=2s set, for example, or all devices get an allowed-networks setting) but means what you see is no longer what you get, necessarily…

6 Likes

This would enable things such as New folder type: Transfer I suppose. Very nice idea overall.

I think people who even get to understanding the concepts of whats what, ie whats a fileinfo, whats onIndexUpdate, how to write a meaningful plugin, can already customize Syncthing to do what they want in Go.

I don’t think having a sandboxed script engine counts as a framework for plugins.

Real plugins should probably be more like hashicorp plugins, running out of process, in whatever language. Then you don’t need to provide train-bandit.js or whatever other latest greatest hipster bs, and don’t need to add access to file support.

I am betting my socks, this will still not be enough, and people will want more and more bells and whistles upto the point where Syncthing will become kafka over ipfs stored on the blockchain.

My personal view is that people who want something that Syncthing doesn’t do, should not expect to be able to do those things with Syncthing.

Granted, I see a usecase for external versioning being a thing, and perhaps external ignores, but not some crazy fiddle with the index to implement your own folder type.

3 Likes

I disagree with you. I think plugin subsystem would be nice to fill the gap between BEP-proto possibilities and BEP-proto implementation (Syncthing)

Thats exactly the reason why I think its a horrible idea. If you want the protocol, implement it, instead of butchering an existing application to do what it’s not supposed to, and then complain when it doesn’t bend itself to do what you want.

I thought about it more like a hooks-style system. For “transfer folders”, one would hook into some file completion event and have the plugin script move the completed file somewhere else. Really simple stuff like that, I wouldn’t make it possible to muck with internal data structures, more like filesystem operations triggered on some defined points.

You can already do that via the api tho?

Sure, but having to manage a separate process reading the events API and starting stuff is certainly less convenient than placing a script somewhere and having it executed by the already running Syncthing process at the appropriate time. I just think it lowers the barrier to implementing custom behavior.

Certainly this is about ease of use. Many things are currently possible if you write a daemon that listens to events and talks to the API, but not things like custom ignore logic, and writing that thing would be like a weekend project for someone who is already a developer, more or less impossible for someone who isn’t.

Finding a JavaScript example on the web and dropping it into a folder is a five minute operation for a much larger category of people, and enables behaviours which is not currently possible.

Having plugins as sub-processes that talk over some simplified RPC channel like json over stdin is some sort of middle ground. There’s more boilerplate required by the user - some sort of event loop, json parser, switch logic based on the incoming request, etc. I don’t even know how to do that on stock Windows - am I supposed to like install python or nodejs or create a visual studio project or something?

I don’t think the argument about Syncthing not being designed so these things holds much water. Syncthing didn’t appear as a fully formed ideal application, it’s gradually grown to be what it is due to changing requirements and requests. I think we’d be in a better and simpler place today if we had less built in features and more expandability like this. For better or worse the “do one thing and do it well” ship has sailed and we’re already well into “Swiss army sync tool” territory.

4 Likes

To avoid some of the worms in the can, perhaps you can:

  • Only allow plugins if explicitly set in settings, no implicit registration by existing in the right folder etc.
  • Make sure that it is easy to select “Disable all plugins” without losing important settings.
  • Make sure registered plugins show up in relevant logs often enough to be visible in error reports

That way, any support case would start with “Have you disabled plugins?” and could be dismissed if the log indicates some funny plugins doing magic.

I think you could have a long and interesting discussion about how to implement them and in what language. It’s a pity the Plugin package in Go is still not available on Windows, otherwise that could have been a nice fit. (Says a guy who recently started learning Go just to understand the Syncthing source better.)

In principle I am on the same page as Audrius. Also because I really, really want people to do their own cool apps and usecases based on Syncthing or the BEP. However as also brought up already by Jakob somewhat differently: That’s wishful thinking.

I do think having a nice, flexible hook system is useful. I am very sceptical about allowing any changes to Syncthing’s state though. As in I feel like this should be restricted to “high-level” decisions like doing something with a file, indicate if it should be ignored, …, but not changing file-infos. The entire thing is a foot-gun (with great potentially if pointed away from foots), but that is a mega-foot-gun pointed directly at the heart of Syncthing. And if one goes to handling that kind of complexity and understanding of Syncthing, you really ought to be learning Go and PR/fork/plugin, to not have to deal with JS (or whatever scripting language the system would use).

So basically I think it has potential, but I am also not personally motivated by it. I actually do have a one or two ideas for Syncthing extensions, which could be implemented by this framework, but I’d rather create a wrapper/plugin/… if I ever get to actually act on those ideas. Then again, I know this isn’t targeted at a co-maintainer :slight_smile:

I think this would be really handy, and I can think of a couple of simple things that would be pretty easy to do with a plugin. I can see the possibility for opening up a can of worms, so I understand the scepticism. But as @imsodin says, maybe keeping it to ‘high-level’ functions (at least at first!) would limit the potential damage that could be caused.

Maybe there should be a ‘wishlist’ of things that users would like to do with a plugin framework. This would allow development to focus on creating a framework that is both sensible and reflects what users want.

I don’t think it’s a good use of energy. Plugin quality will always be low, plugins will be abandoned regularly and it’ll become a graveyard, people will create vulnerable plugins. If people want to implement features for their own needs, they have the source code. Also if you start building things with plugins support in mind, it adds more things to consider at all times, more regression testing when an update breaks old plugins. I think we shouldn’t go too far off the UNIX philosophy of doing one thing and doing it well. I think synching is perfect as it is so far, it has stricken a good balance of features and simplicity.

3 Likes

I want to make an app or do business from this , can anyone tell me how to make an app

If you want to “do business” with making apps, being able to make one sounds like an essential requirement. We can help you with specific problems, but please reconsider what it is you actually want to achieve. Your request wasn’t quite clear and I think it would be better to open a separate topic for it.

I partially agree - I think Syncthing should be kept as pure as possible - feature hell is real and we need to fight it.

However, on the other hand I have a use case that would benefit from some kind of plugin / adapter system: I would like For Syncthing to be able to access a dropbox file / folder. I am unsure how this can be done (help / guidance appreciated), and did not found any existing solution.

This could be implemented as a Virtual Device (that works as an adapter between the Syncthing API and the Dropbox API) OR as a Virtual Folder (as a plugin?).

You install both Syncthing and Dropbox, and you sync the file/folder within Dropbox via Syncthing. I’ve done this for years because I prefer Syncthing but some collaborators prefer Dropbox.

1 Like

Sadly, I cannot install the Dropbox client on my machine.

The reason for dropbox is quite convoluted: My KDBX (KeyPassXC (password manager) database file) is synced by Dropbox, because the Android app does not have Syncthing integration yet (AFAIK).

So I would like some adapter that can share a synced folder into / from dropbox. Currently it seems to me, that Syncthing only has a REST API and no way to add such a “custom dropbox folder” - or a custom “virtual dropbox device”.

You could use Rclone to sync your Dropbox locally and then sync that over Syncthing.

1 Like