Rehash time after upgrading to 0.11.1

Support
1

I upgraded my work laptop to 0.11.1 this morning and did the same with my aging home PC this morning. The work laptop is a Lenovo T530 so has a decent spec - Core i7, 8GB RAM, SSD. The home machine doesn’t fair so well - Core i3-2100 (yup, it’s that old), 8GB RAM, 7200rpm HDD.

The upgrade on the work laptop was pretty much unnoticeable. The upgrade has crippled the home machine - I’m guessing the rehash of all the files is grinding the CPU and HDD.

Is there any way to get some sort of progress indication of how the rehash is going?

2

The gui shows how much data has been hashed so far; you can compare with what you know the total to be.

3

Hmm… just looking around and I see that the old Syncthing db directory (C:\Users\A\AppData\Local\Syncthing\index) has 226 files whilst the “new index-v0.11.0.db” only has 75.

Is it fair to say the number of files in the new db folder will match the old and so I’m only about one third of the way through?

(damn, it’s been slogging away for about an hour now)

Also, if I look in the web UI, the three large shared folders (one for docs, one for music, one for RAW photos) show (eg) 19000 total files and 18999 out of sync. Is that a concern or does it just mean it’s rehashing the lot?

Thanks Andrew

4

Thanks for the prompt response: so I look at the difference between the Global State and the Local State per shared folder?

5

Maybe, I don’t know. Not necessarily, at least. It depends.

The 19000 files are the one it has rehashed so far. If you know there are 25000 files or so according to explorer, then you know how far it’s come.

It’s expected that every file is out of sync at this stage - that’ll sort itself out when the devices sync after the initial scan. The actual sync will be very fast since presumably nothing or very little needs to actually change.

6

Progress note: One of the three larger folders “completed initial scan” and now the computer is completely useable again and the noise from the HDD is much reduced. Perhaps having three folders being queried all at once saturated the drive.

The scan of the remaining two folders appears to be going more quickly now as well.

7

Yeah, doing too many things at once to a single spindle is usually not a recipe for high performance. :unamused:

8

Perhaps we should share the hashing routines across folders.

9

I had the same issue. Rehashing was going to take about a week on one machine by my estimate.

So on that machine I removed each repository in the GUI, and then added one at a time, after each had completed rehashing. Painfully manual, but then it only took about an hour.

I’m guessing the “problem” is that it’s rehashing more than one folder at a time. Instead it should go through sequentially.

Should I raise an issue for this?

1 Like
10

Probably yes, but the sad part is that most of the people already gone past that stage, so I guess this is something we should remember to do next time we change the database format.

11

Maybe we should default to a single hasher per folder. The current setup is kinda optimized to maximize resource utilization when doing a single folder. If there are many, and they’re on the same disk, this is not optimal. Or maybe runtime.NumCPU() / numberOfFolders + 1.

12

How about an iterative performance-based approach to choosing how many folders to hash at once?

Like so:

  • Begin hashing 1 folder.
  • Monitor some basic stats on throughput (number of files, size of files, elapsed time, CPU usage)
  • After 30 seconds, increase to 2 folders.
  • After another 30 seconds, compare the throughput from the first 30 seconds to the next 30 seconds.
  • Was it a lot quicker and CPU usage acceptable? If so, increase to 3 folders and repeat.
  • If it was a lot slower, drop back down to one folder. If it was about the same, stay at 2 folders.

Within a few minutes an equilibrium is reached. You could base the starting parameter on NumCPU instead of starting from 1 folder.

13

That’s rather complex. This

will have the effect that the total load will be about the same for a single up to a handful of folders. That covers almost everyone, per https://data.syncthing.net.

1 Like
14

Thanks for the quick fix - you do not hang around :smile: