Question about: Deleted items are no longer kept forever in the database, instead they are forgotten after six months.

Deleted items are no longer kept forever in the database, instead they are forgotten after six months.

My apologies for bothering the community, but could somebody explain what this actually means in practice? If I have two devices A and B that have the same state and then they disconnect, and I delete a file on A, and they connect after eight months, what happens?

If this is something that is of consequence to you, I suggest setting the interval in question to something that fits your device connection interval.

Otherwise, the answer is either ā€œnothingā€ or ā€œthe previously deleted file is synced backā€, depending on things.

1 Like

Thanks so much for the info!

I decided to try to up this interval just to be safe, but I couldn’t find it in ā€œAdvancedā€ > ā€œOptionsā€ or the regular ā€œSettingsā€. Where would I find it in the UI? I’m guessing that I’m just being blind and I can’t find it, my apologies.

It is not in the GUI, you need to set the env var or command line option as mentioned in the update notes.

1 Like

Thank you for the clarification! Is there any hope for it to be added to the UI? Modifying the start scripts on all my machines could be a little cumbersome. (I guess I have about six months to figure out how to set it.)

Not really, it needs to be the same for the entire cluster to make sense, so making it an easily changeable value per device is likely to result in more confusion.

Of course, your setup is your setup, but I’d venture that for syncing things once every eight months, Syncthing might not be the perfect tool. (ā€œContinuousā€ and ā€œreal-timeā€ are in our tagline since forever ;))

(Also, for guaranteed immediate consistency of deletes, you do not have six months from today. You have six months from when the devices were last in contact, effectively.)

I’ve been using it this way since forever, so it seems to be the perfect tool for me :woman_shrugging: modifying the startup scripts is more difficult since for some setups they’re provided by the distribution. So if I’m meant to set this option everywhere, I think that would be a good point for a UI option. My apologies if I’m misunderstanding something, however.

Unfortunately, at this point, this is your only option. I don’t like this behaviour either, so I just modify the source code and set it not to delete anything regardless of how long time has passed.

In our case, we’ve got a setup of maybe a dozen devices or so, and while most of them run 24/7 and are always connected, there is one that is turned on maybe once or twice a year. It doesn’t share many folders either, however what it does sync is very important, so we cannot risk the folder contents getting messed up when the device is finally turned on after being offline for many months…

I’d also add that even though we’re talking about ā€œdeletionsā€, for Syncthing, if you move or rename files or folders, the previous locations are also treated as ā€œdeletionsā€, so I think in some cases the impact may be even more serious.

1 Like

I think this is really the only negative about this release. I do think it’s gonna cause a lot of support issues. It would be interesting to collect data on this kind of thing.

You could put a ā€œlast connection ageā€ in the Mon.syncthing.net dataset and see how many instances have such outdated connections. I suspect quite a bit.

I am not a Go coder myself but I would still think this 6 month limit could be dynamically adjusted based on the oldest connection (and successful metadata sync) date among the list of devices. Perhaps the limit is 6 months or if a device hasn’t been seen for 8 months, then it becomes 8 months… And when that device finally reconnects, then it becomes 6 months again and 2 months of deletes can be purged.

Anyway. For now I’m not upgrading because it will be more difficult to manage my cluster of 15 machines (some of which have been offline a while but maybe not 6 months.)

It’s open source y’all :person_shrugging:

Would you guys be more chill if the cutoff were 12 months instead of six then? Or is this a case of ā€œmy workflow requires devices to be offline for arbitrary amounts of time!ā€?

That sounds nice but is difficult to implement, because all devices would have to agree on the same limit, and not all devices see each other, nor all the time (so this ā€˜oldest connection age’ would differ between devices).

It’s major version increment after all, and I’m surprised there’s so little breaking changes. Maybe recommend on main page universal transfer tools like rclone/rsync? Might as well suggest using backup tools for the job (you might know some, Jakob Borg :grin:)

3 Likes

I figured each device would need to agree only with the other devices they can see.

Anyway. It’s a real use case. It’s apparently more difficult to manage than I thought because now I have to change all the instances to the same limit. Even if they don’t talk to each other. And I don’t have direct access to all the machines.

so I basically am prohibited from upgrading unless I’m okay to have to manage a cleanup of my data when an old machine comes back online.

So here’s the thing, there are two individually reasonable requirements here that are unfortunately in conflict:

  • A device should be able to be offline for an arbitrary amount of time without losing any sync changes
  • The Syncthing database should have a finite size for a finite set of files

Previously we optimized for requirement one and allowed an infinitely large database. You can still get that behavior by setting the retention period to zero (forever). Syncthing not optimizes for the latter and sets the interval at six months. I’m not very particular about that interval, I think it could be 12 months instead by default. But, I do think there should be a cutoff at a reasonable interval, and those who want to do something less reasonable can configure accordingly.

If you can’t bother to set a command line switch or environment variable, sure.

1 Like

It’s not about bother or not. It’s about I don’t have control of all of the machines in the cluster.

In that case nothing we do will help, so it is indeed your only recourse to never upgrade.

Edit: If I sound a bit exasperated it’s because when these things boil down to ā€œI can’t change anything, you must put in the engineering effort to make sure the defaults are perfectly suitable for my use caseā€ I do, in fact, get a bit exasperated.

2 Likes

I think it is more than reasonable to expect users to make certain changes in their config on a major release like this (other command line flags changed as well). Just stay on 1.x if you want everything to stay exactly the same, or change the interval to something absurdly large on 2.x, your choice.

(@calmh is there a particular reason though why this setting is node-wide instead of folder-specific? I assume all nodes sharing a folder will have to agree on the interval, but it should be fine to have different intervals for folders shared by the same devices, and also you might want to pick different intervals for different folders depending on your use case. I.e. if I have a folder that has a lot of deletions I might want a smaller interval, and a larger for another. Note I do not have any specific use case myself for this, I am fine with the current default)

There is no technical reason it couldn’t be more fine grained, it’s just not implemented that way.

I get it. And I also don’t mean to sound ungrateful. I’m not at all. This is just a problem for me and I wish I could upgrade. I guess I have to see if there’s some manageable/controlled approach I can take to upgrade one machine at a time and if the machines I don’t control don’t upgrade then I guess I’m still okay.