Syncthing 2.0.0-beta.11

Announce Pre-release
101

Been watching the beta 6 thrash the drives all night, even the C drive / SSD has been at 100% most of the time. It got me thinking about performance again. Given the increased number of database files and the overall improvement of IO, it seems to now be creating bottlenecks. I realise concurrency is an option, but I really don’t want to enable it as I prefer everything to sync when it needs to sync, not when it’s the folders turn.

So thinking out loud, is it possible for St (maybe as a user option) to scan folders in the way concurrency does now on loading St, but not be in the ‘waiting to scan’ phase after scanning / indexing.

Essentially, folders are scanned incrementally rather than in bulk ‘on load’, but the folder is then either up to date or carries on syncing when the scan / indexing on that folder is complete. When all the folders are scanned, it becomes a normal St

102

How about if I fix #5353 instead?

1 Like
103

That would help, but I was think more along the first time indexing. I’ve come to the PC this morning and it’s still working hard doing the first pass through

image
image961×492 24.9 KB

image
image558×118 2.27 KB

104

recalcGlobalForFolder:

Maybe I’m missing my first coffee, but the very first query could be written without the EXISTS clause as you’re allowing self joins.

recalcGlobalForFile:

A lot of flag wrestling. Partial indexes or dedicated columns might help?

105

You’re going to have to be more specific. :slight_smile:

106

Wrapped my head around the query and it’s fine as is :sweat_smile:

SELECT f.name FROM files f
WHERE NOT EXISTS (
	SELECT 1 FROM files g
	WHERE g.name = f.name AND g.local_flags & ? != 0
)
GROUP BY name

My point about flag magic still stands, though. All these local_flags & ? predicates are not able to use an index.

107

I started testing beta5 and updated to beta6. Should beta6 automatically convert my single file index-v2.db into the newer one database per folder format?

108

Nope. beta5 → beta6 is breaking. Nuke your DB to proceed.

109

Indeed. However the median user has 3 devices, 95% of devices have 9 peers or less. We have an index on the name, so I’m thinking these queries should be fairly straightforward even in the absence of an index, since they all result in a small handful of entries, and maintaining the index isn’t free in disk space or CPU?

110

Maybe we can give the sqlite query planner a little nudge if we know that:

https://sqlite.org/lang_corefunc.html#unlikely

SELECT f.name FROM files f
WHERE NOT EXISTS (
	SELECT 1 FROM files g
	WHERE unlikely(g.name = f.name) AND g.local_flags & ? != 0
)
GROUP BY name

This would tell the query planner that name has high selectivity and should be preferred.

Edit: this should be checked via EXPLAIN if it really improves execution time

1 Like
111

A partial index on a flag with low selectivity might also work.

112

Nuked database, had to manually create index-v2 folder. It works now.

113
114

Advice needed, i’m still running through the first pass of beta 6, it started shortly after 6 was released!!. Do you want me to let it carry on, or would you like me to clear the index 2 folder and start again with beta 8?

Edit: Renamed beta6 / index 2 and starting again on 8

115

I’m excited for the 2.0 changes, especially weak hashing removal! I manage a bunch of low-end devices (10+ years old and not the best when they were new either; ~50GB synced across ~200k files) that I expect will see very noticeable performance improvements from this.

I have a question about configurable delete retention: How do devices behave when they have differing intervals? For example, suppose interconnected Devices A, B, and C have the default 6-month interval and (also connected) Device D is set to 12 months. Device C is taken offline; then Device A deletes a file; then 9 months later, Device C reconnects. Based on my understanding of Syncthing internals, the file’s most recent change is its deletion, so I would expect Device D to “win” and the file to eventually be deleted from all devices - even if Device C manages to sync it to A and B first. Is this correct?

(My preferred setup would be default 6-month intervals on my low-end devices and a much longer or infinite interval configured on my high-end devices, but only if Device D is the winner in the above scenario. If not, I’ll be setting all of my devices to 0/infinity. I do rarely have devices come back online after being out for an unpredictable number of years.)

116

99% the delete would take, as it’s accepted and processed when received and only forgotten on a slow periodic task. But if it fails for permission reasons or whatever for 8 hours, or you lose the timing lottery, or we change things in the intervening year, … better set it to be what you want a guarantee for.

1 Like
117

Thank you for the reply, but I don’t think I quite follow. I’ll modify my scenario just slightly:

  • Devices A, B, and C forget deleted files after 180 days. Device D forgets after 360 days.
  1. Jan 1: C disconnects from network
  2. Jan 2: A deletes file. Deletion is synced to B and D
  3. Jul ~1: A and B forget file
  4. Sep 30: D disconnects from network
  5. Oct 1: C reconnects to network. File is synced to A and B since it is “new” to them
  6. Oct 2: D reconnects to network
  7. ???

If I understand correctly, there is no reason for any device to forget the file around Oct 1. So this would be equivalent to losing the timing lottery, right?

(Now that I think about it, it makes sense to me that it would behave the same as v1.x does when you add an entirely new device that has a file which was recently deleted by the existing devices. Which I’m also not sure what would happen)

118

Garbage collection runs every 8 hours (by default). It’s possible, but not very likely, for C to get information about an old deleted file and not have time to process/sync that deletion before garbage collection kicks in, if that happens very soon after receiving the file entry.

119

Actually I think something interesting will happen when the delete is forgotten, as those devices will then consider themselves out of sync as they are missing the file that the long gone device was now the only one to announce. This would serve as a reminder to bring that device back from the dead.

So yeah if this is a situation you expect, set the flag accordingly.

1 Like
120