Report free space on remote devices

I suppose this could be solved by a cron job on the client, piping “df” to a file in the synced folder…

And then improved from there, independently from Syncthing. I know everyone in the thread probably thought of this, I am just pointing out that sometimes you can add features to a system as pluggable additions, instead of expanding the core product.

I think Syncthing is at a level of maturity when we should be very careful with adding features/complexity. I think of Jira and Trello, both at one time favourites of mine, but now impossible to introduce to a non tech-savvy user.

I am well aware of this “solution”. Might work for syncthing running on linux server and being managed by linux admin. But try setting that up on several android devices (some of them being managed by average consumers/users). That is just terrible.

I want syncthing to be solution that i can use to share files with my family, not just with my tech savvy co-workers.

On the other hand, these basic users will not care that there is no support for nested mountpoints and similar complex configurations they will never have on their devices.

I agree with the general statement, but:

Try getting free space information from Go on Android :smiley:

Earlier in this thread i had the same concern, but it was confirmed that syncthing already knows how to do that:

Yeah, we do something. But it might fail (because not implemented), in which case we do nothing. Best case the library does return something on android, maybe even a relevant number for /data (where the DB is), but I’d be surprised if it works.

We may discuss all you want about the potential technical solutions and issues, but it looks to me that there is a more pressing matter: is Syncthing expected to provide this feature? On that matter, only the Syncthing team can answer.

If the answer is no, then the next question is: how to extend Syncthing to provide additional features that the team won’t have to spend effort maintaining?

I really think that these questions should be answered first. Because if we have to rely on a plugin, then we can split the effort:

  • have a thread about supporting plugins (if there is no such support yet)
  • redirect this thread towards how to implement a plugin for sharing space information

Does it sound reasonable?

1 Like

That sounds very reasonable. There’s been some discussion about whether Syncthing even should send this information - one might not want remotes to know about local system details. Then again, when you are sharing data with said device, sharing a limit on how much data one can share seems like much less sensitive information.

Anyway, there’s the REST api. You cat get a list of remote devices and free disk space (not sure about the latter, better do that in the plugin), then you can do whatever you want with that, to share it with other devices and reconfigure things. So a plugin/estension/companion/… is possible.

I think we’re getting carried away here.

Why do we need to support plugins?

Why can’t you just use a different application that does what you want?

If one doesn’t exist, write one, instead trying to piggy back on syncthing and support plugins.

I don’t think we need to run syncthing into a rootkit that allows people to build random applications on top via some shonky plugin infrastructure.

Think of syncthing more like you think of “rsync”, which does one job and does it well (hopefully), and less of like “excel”, where you can bolt any random stuff if you want to.

There is no plugin system for rsync.

If you want to add more features to syncthing, fork it, create your own version with the features you want.

1 Like

There is no plugin system for rsync.

Unlike Syncthing the Rsync is not shipped with web GUI, therefore no one can question it’s UX.

Syncthing has a Rest Api, and the web UI can easily be extended (see e.g. synctrayzor). So I think it has everything to build on top of it.

Sorry for the long answer, but we speak about global perspectives to know where Syncthing stands, so it is important to consider several aspects.


Forks

Forking is not about extensibility. You fork when you want to do your own stuff, which can evolve as far away as you want from the basis. If you want someone who operates a “custom” Syncthing to still interact with a “standard” Syncthing, you need to ensure compatibility in some way. Forking does not care about that: it completely cuts the dependency. Compatibility then have to be maintained manually.

If you want to deal with forked variants of Syncthing in a compatible way, you are gonna have to deal with a mess. If you want to avoid that, then Syncthing should provide the acceptable boundaries, which is why extensibility should be supported by Syncthing itself.

Additional layer

We may think that people just have to implement an additional layer using Syncthing as a backend, but that means that all the features not related to the extension itself, but still provided by Syncthing, must be implemented too to make them available. Just to delegate them to Syncthing.

Moreover, depending on the kind of extension you provide, it might be hard to stack it with others: extension A does not know what extension B is providing, just exploiting the standard Syncthing stuff it supports.

People who are used to implement the decorator pattern probably know what I mean: you can stack them, but only if they remain within the adaptation of existing features. New features are lost by adding yet another layer aware only of the basic stuff.

Extensibility

Plugins are a way to go, although it is rather integrated and thus implies that the team spend some effort architecturing Syncthing in a pluggable way. But it ensures that whatever plugin we build has to be within the boundaries of what Syncthing accepts. Boundaries that can evolve at the rythm of Syncthing itself, so you keep control on that.

Another way to make it extensible is to establish a common interface/protocol that can be implemented by other apps to properly interact with Syncthing. But it means that any stuff that one needs to adapt must be exposed through this interface/protocols. Is the REST API adapted to this kind of use? Can we easily use it for instance to add the support of remaining space discussed here?

And there is probably other ways, each with its advantages and drawbacks. I don’t know what would be the most reasonable choice here. All I am pointing at is that some extensibility is required, and forking is a different matter.

From my point of view, Syncthing provides the entry point. And the whole point is to allow Syncthing to get custom, context-dependent adaptations, without having to impose the effort of each of them on the main team, and this is what extensibility is all about. There is of course the user interface to adapt, but also the processing, which makes it hard to just consider it as an additional layer.

Comparison with rsync

Rsync is not extensible. It is a standalone application that you can only run through what it supports directly. If you look at its man page, there is around 150 options that you can provide to customize its behaviour. All these options are maintained directly by rsync.

You can use rsync as a backend (behind an additional layer) to enrich its inputs and outputs, but you cannot adapt rsync itself to deal with contextual specificities unless it is already covered by the provided options.

For example:

  • -z/--compress compresses the data.
  • --compress-level provides the compression level to use
  • --skip-compress allows to avoid spending time compressing files having a specific suffix

What if you want to change the compression algorithm? What if you want to skip the compression based on prefix instead of suffix? What if… You just cannot do it because rsync is not adaptable enough, either because of the lacking option or because of a lacking extensibility.

This is exactly what you risk by missing the extensibility aspect of Syncthing.

Conclusion

I totally agree that Syncthing should not deal with the many, context-specific features that everyone wants. I totally agree that it should focus on its job. But if you are not extensible, that focus becomes also the limit of Syncthing. Then people wanting more have to use something else. This “something else” might be based on Syncthing, but Syncthing remains the backend and can be replaced at any time because of its limits. However, if it is extensible, then Syncthing remains the main stuff, and plugins are secondary. The great stuff is related to Syncthing itself, not to some extra layer hiding Syncthing and dealing with its limits.

If you are OK with Syncthing to be a limited tool like rsync, that runs behind the door and can be replaced at any time when the limits are too annoying, then not caring about extensibility is reasonable.

If you want to be in the front, then adaptability becomes a requirement, otherwise you cannot deal with the fluctuating requirements of people (nobody remains within a restricted boundary for long). Then come the question: spend effort supporting all the stuff yourself, or make it extensible for other people to provide the additional stuff making Syncthing greater?

If Syncthing has a setting for “Minimum Free Disk Space” and can return errors stating “disk has insufficient free space” then it can certainly state how much free space there actually is.

1 Like

I think a plugin infrastructure would be a significant engineering effort. Someone is welcome to take a stab at it, probably starting with a design proposal outlining the possibilities, limits and use cases for it.

There are some things I could see moved to a plugin based system – ignore handling, for example, where people often seem to have odd ideas. Versioning. Various pre- and post-sync hooks.

Free space reporting as a plugin would require that plugin to do the relevant OS stuff (get the free space), send that over the protocol in some extensible mechanism (what would that look like), and also implement some sort of UI (so it’s visible). That also seems like a lot of work, but if someone wants to do it, why not.

As for doing something in Syncthing “proper”, we currently don’t even have a protocol message to notify peers about folder status at all – whether it’s syncing, stopped due to error, etc. That would seem more fundamental than the free space stuff.

1 Like

Lets not spiral this “plugin” idea out of control :slight_smile:

What about adding way for syncthing to optionaly send small arbitrary JSON object to its peers along with status report? Other peers will then just display the JSON as is on the web UI.

What would be the most reasonable way to implement this? Sending one JSON per device or one JSON per share? Or both?

That would be nice starting point. It would be possible to include free space info to that JSON. Or perhaps later plugins (if they ever get implemented) might use that to include own data in that JSON.

I agree with @Harvie that this stuff should go one step at a time. Prefer to feel safe implementing a small thing that unlocks a small ability, rather than feel regretful spending time building a big stuff that you never make it fully work with the rest. I have done that plenty and it is not worth doing. Working iteratively is longer, but way safer. If some internal stuff are good candidates for externalization in dedicated plugins, they can be used as starters.

I only had one job using Go some years ago, so I know the basics but not enough to provide high quality code. However, on the conceptual level I can help. If someone wants to start a thread on the plugin setup, feel free to tell me. I can share general expertise on that matter. I let the “remaining space” topic to the guys here.

Can somebody please point us to the code which contains some kind of “status reports” currently exchanged between syncthing instances?

There is no such thing. You can find the messages and such that are in fact exchanged in lib/protocol (and the overview/specs at Block Exchange Protocol v1 — Syncthing v1 documentation).

2 Likes