HELP: Syncthing is redownloading already existing files!

Support
1

How can I make Syncthing recognize that the local files already exist?

History:

  • We transferred initial data via rsync
  • Then we connected Syncthing (one send-only, one receive-only)

Now Syncthing tries to re-download files that are already present! I compared the checksums and they are identical, also permissions (777) and times are the same.

Do I need to start over or what can I do? We are talking about several TB of data half way round the world. So I really do not want to re-transfer of all of it…

2

How do you know it’s redownloading? It usually has the concept of “processing files” which doesn’t necesserily mean redownloading.

3

The transfer window says so…

10
Screen Shot 2019-01-12 at 18.21.10.png2880Ă—1800 713 KB

4

Or is this just very slow comparison of metadata and blocks?

5

No, this is downloading. Have you verified that the path is right and that the files are definately there, as it looks like it’s downloading to a new location?

6

maybe pause devices, rescan on both sides and resume again helps?

7

Yes, checked everything, path, times and md5 checksum. I also thought of rescanning but the button is greyed out.

I decided to start fresh and wiped everything. I’ll see how it goes then.

8

OK, started fresh. Both folders rescanned, checked permissions, paths, etc. Still Syncthing tries to re-download already existing files! At the same time rsync is totally happy… I’m a bit at loss as to what the cause of this is… Both nodes ignore permissions and both have large blocks enabled.

9

What startles me:

  • I transferred all files from the master server via rsync
  • I setup Syncthing on both sides
  • Both sides finished their scans
  • Now I connect the two nodes
  • The master server now shows a lot of out of sync items (in principle all files that were already transferred to the slave via rsync)

52
Screen Shot 2019-01-13 at 09.10.52.png1112Ă—1020 85.7 KB

PS: This is still shortly after the connection is made and out of sync just keeps growing and growing.

10

The situation on the slave is as follows:

  • a lot of failed items
  • if I check the file is present on the master

09
Screen Shot 2019-01-13 at 09.19.09.png1114Ă—1082 93.1 KB
51
Screen Shot 2019-01-13 at 09.19.51.png1796Ă—584 48.7 KB 

# slave
-rwxrwxrwx+ 1 sc-syncthing root 2576384 Sep 27  2017 Marketing/Anzeigen/00_Anzeigen_Must Have.indd
MD5(Marketing/Anzeigen/00_Anzeigen_Must Have.indd)= 259a7dbcbdeb9029b19d585fbd94c81b

# master
-rwxrwxrwx+ 1 sadmin users 2576384 Sep 27  2017 Marketing/Anzeigen/00_Anzeigen_Must Have.indd
MD5(Marketing/Anzeigen/00_Anzeigen_Must Have.indd)= 259a7dbcbdeb9029b19d585fbd94c81b

Any idea?

11

One more thing: why does it report the global state to be different? stignore is identical.

12

Well the file count definately does not match, do it’s probably just downloading the missings files and whatever your rsync did it didn’t do for all filez as the reported file count differs.

13

Yeah, that is true, we did not rsync everything. The very old archived files we expected Syncthing to pull over the coming days/weeks (file order “newest first”). But shouldn’t the global state numbers match?

14

They should, but perhaps the downloads are taking up all the traffic so index exchange is not finished yet.

Let it settle and see what happens. Also, failed items don’t mean anything good so you should address those.

15

OK, I’ll give it some more hours/days and report back.

16

No change of the situation. I restarted Syncthing a couple of times etc. I can pause Syncthing, run rsync (which is happy) and then start Syncthing again and it will start downloading existing files again… So either the WebGUI is reporting something that does not really happen or something else is screwed up here. I’m at loss at to what I should do now?

Kill Syncthing, let rsync do its job until everything is in sync (loosing the advantage of Syncthing being able to re-use existing blocks) and then try Syncthing again? I have larger installations where Syncthing works without a glitch…

17

Well the file count doesn’t match so I don’t think rsync is doing the right thing.

18

OFFTOPIC @alh I saw you have indexed > 9TB files, how much RAM do you consume with those big fileset?

19

@xor-gate around 500MB. During scanning it might go up to 1GB but overall it is quite OK.

20

The sync is unfortunately not completed, we still have 2TB outstanding. We were hoping that Syncthing would pull these files (newest first). It does so for roughly 15 minutes after a restart but then it starts downloading existing files.