Enhancing staggered versioning to keep fewer older versions after 1 year and later

Feature
1

Currently, staggered versioning keeps:

  • 1 version per 30 seconds in the first hour
  • 1 version per hour in the first day
  • 1 version per day in the first month
  • 1 version per week in the first year

The versions are cleaned when they reach the maximum age, which is set to 365 days or 1 year by default.

This works great if you keep the default values, but I personally would be interested in disabling version cleanup at all for the most important folders. The problem is that with the above settings and cleanup disabled, staggered versioning will still keep 1 version per week even after 1 year and later, which all together can take up a lot of space.

I think it would be useful to extend the current cleanup to consider longer periods of time as well. I was thinking of something in the manner of:

  • 1 version per month after 1 year
  • 1 version per 6 months after 2 years
  • 1 version per year after 3 years

or alternatively in a slower pace of:

  • 1 version per month after 1 year
  • 1 version per 3 months after 2 years
  • 1 version per 6 months after 3 years
  • 1 version per 9 months after 4 years
  • 1 version per year after 5 years

I’ve checked the code, and it doesn’t seem to be too difficult to implement (by myself), but I’d like to ask first whether something like this would even be desired and what’s your opinion about it in general. I’m also not 100% sure whether the two propositions above are the best. Maybe just a single change to keep 1 version per month (instead of 1 per week) after 1 year would be better?

The potential change wouldn’t affect anyone who uses the current defaults, as with them versions are cleaned after 1 year anyway, but it could potentially affect those, who use longer periods of time already, which may be problematic, hence the question. The alternative to the above is either to leave the current staggered versioning alone and run custom scripts to clean older versions manually, or to re-implement the whole staggered versioning logic with an external versioning script, customising it to own needs in the process.

In sum, the problem that I’m trying to solve here is to keep fewer older versions when cleanup is disabled (or set to something much longer than the default of 365 days), so that they take less space.

What do you think?

1 Like
2

I think additional configuration for the staggered versioner would make sense. I.e allowing people to specify the intervals/counts etc.

1 Like
3

Well, I was just considering adding yet another hard-coded interval, but there’s already an enhancement proposal for what you mentioned. Having everything configurable would obviously be much better and more flexible as well. I’ll see what I can do. I’d still like to add at least one more interval for even more flexibility.

4

I’ve been playing around with the staggered versioner code, however I’m a little confused about the test file dates used in syncthing/staggered_test.go at fb4209e382ffa08d53f96569d91f5c4fe172d622 · syncthing/syncthing · GitHub. I understand that they are meant to test the various intervals but is there any logic to why these specific dates are used? Some are obvious but others (e.g. 44 days ago or 358 days ago) seem completely random.

In other words, if I wanted to add new data to the test that exceeds the current maximum of 367 days ago, what logic should I follow when setting the test file dates?

5

Don’t gaze too deeply into the abyss, just make something that makes sense.

1 Like