There’s an upcoming change to the database engine, with some related build changes, which warrants a version bump to 2.0. I was going to let that be that, but while thinking about I think we could take the opportunity to widen the scope a little and make a couple of further breaking changes at the same time.
The current API is a hand rolled “mess” (it has grown organically over more than a decade). We would benefit from cleaning it up and switching to a structured, machine readable API. I’m thinking gRPC and Connect, which provides a REST layer that can be called from curl or anything else that does not wish to speak gRPC.
The logging could benefit from a refresh and moving to log/slog and structured entries.
Features or experiments that didn’t turn out as we expected could be removed and cleaned up (the weak hashing is first on the chopping block here; copy ownership from parent is at risk; ignore deletes might want to start saying its goodbyes).
In essence, things I’ve wanted to do for a long time but which we “can’t” because it breaks integrations or users. This will of course result in a longer beta period for the v2 branch, as it is more work and will take longer to do. In the end it will result in a leaner, meaner and more modern Syncthing, which I think is a desirable end result.
I could see a couple of stretch goals as well, which I’m not promising I’d get to:
Revamping the configuration format from XML to something more modern
Revaming the ignore format in the same fashion
These might be for a v3…
tl;dr
Expect a long lived v2 branch and regular betas over the next several months. Expect it to be a rocky road with the upgrade not being a trivial drop-in for the current version in all cases.
Oh, I need to check because I remember copyOwnershipFromParent used within my ZFS fileserver setup. I recall there were permission problems on the SMB shares it hosts with acl and synced files. But maybe there is a more correct or better way to handle this and I just need to migrate. My fear currently is that I need the cyclic run of a cron script doing chown’s on SMB shared+synced ZFS filesystem like in old times without Syncthing.
Untrusted Devices / Encrypted folders is in the documentation as being a beta release feature. Respectfully, that’s a feature I would like to see continued in Syncthing 2.0.
Count me non-surprised
And I concur. I have a soft spot for stable and backwardscompatible stuff, but sometimes breaks are necessary. And it’s not even like these proposals are that disruptive. Don’t get me wrong, they are massive especially in the work required, I am not belittling them, just seem like things that can be reasonably easily adapted to by most people. I am impressed (and thankful) for the energy that’s going into syncthing lately from @calmh, not that you didn’t do much before but what’s happening now is… quite something
Respectfully, that’s a feature [encrypted folders] I would like to see continued in Syncthing 2.0.
I think that experiment did turn out well. It does have some known warts though, which we should squash before calling it anything other than beta. Doing that on a v2 might be nice, as the easy solutions there will also be breaking. Which is also fine without a major bump on a beta feature, but if we are doing a major one anyway, we might as well combine the disruption.
To add to the things that aren’t necessarily breaking, but in this case large-ish and risky thus would benefit from longer testing: Revamping the configuration handling to something less racy would also be very nice.
So far I haven’t seen anything proposed that would require a break in the protocol, so could be compatible I think. Maybe there’s a reason not to be though (aka break on purpose), e.g. if there will be known, problematic issues. And in any case I’d expect v1.x not to be supported, so shouldn’t be used for that reason alone once v2.x fully rolled out.
That’s also possible without requiring the v2 break though, right? Big change, yes. Breaking compatibility, hopefully no.
While you’re onto that, might as well put the events API into scope. I once started a restructuring to protobuf-based types (though still JSON-serialized), but the existing data structures proved difficult to resemble with protobuf messages in some cases. Not sure it’s worth it sticking to protobuf descriptions if we’re only ever going to send it as JSON. But the advantage of that has been a consistent schema across JSON and Go structures, independently documented. The REST alternative implementations may very well be applicable to the events API.
Do you still see a chance to bring it back properly with a new specialized folder type? Receive-only-no-delete would simply not care about local additions, and not carry out deletions on the file system. Voilà, a clean state without Syncthing ever propagating remote deletes locally. Need to make sure that remote additions of a previously deleted file are accepted by replacing the local remnants, but that seems absolutely doable.
If that’s a perspective, we might want to hold back on dumping ignoreDelete until the new variation has materialized.
Absolutely. That was slightly confusingly worded, and has also slightly evolved since then. It will be a manual-click-required upgrade because we can’t guarantee that it works out of the box in all scenarios – command line options are slightly changed, disk space requirements have increased a little, maybe the API or config format will have time to change before release.
That said, if you’re “just” running Syncthing, without special integrations or funky setup modifications, it is absolutely a drop-in replacement. You may not even notice any difference at all.
For me, it appears that at this moment the main breaking point is likely going to be the lack of support for single - command line switches. They were the only supported format up to v1.18.x (maybe earlier, but I’m basing this information on the Docs), so all installations that had been set up earlier than that and utilise custom command line switches will require a manual intervention or otherwise they will break on upgrade.
I’ve personally got plenty of such older installations.
Are you going to do breaking REST-API changes as part of v2? I’m not generally against that but I would avoid “breaking” (or rather improving ) too many things at a time. Especially an API needs to be designed with care. We don’t want to end up with something messy again. I don’t know much about gRPC but if it is still popular and actually good (providing a better performance then the usual HTTP/REST/JSON) it might be a good fit for Syncthing.
As maintainer of an integration (Syncthing Tray) I would say it would make more sense to provide a new API first as an addition and remove the old API only in the next major release. Otherwise it’ll break all integrations for quite a while as it will be a significant amount of work to replace/change all REST-API calls, especially when changing to gRPC. The new API could also be introduced as experimental first. Then the official UI and integrations like Syncthing Tray can use it experimentally (e.g. with a fall back to the old API). Then tweaking the new API based on feedback from users and frontend developers would still be possible for a while - even if it would break something. (I don’t know how feasible having two APIs would be from the point of Syncthing’s codebase, though.)
I’m also wondering how you plan to handle changing the REST-API when it comes to Syncthing’s own web-based UI. It probably needs modernization anyway (as it depends on quite old versions of some JavaScript libraries) so it would perhaps make sense to combine the API rework with a frontend rework. Those two things could be done in one major release but those are then probably also enough big changes for one major release. That would speak postponing REST-API changes to v3 and not do it in v2 with all the other changes that have accumulated so far.
