Large amount of files and big file size sync problems

Support
41

Well, the CPU load goes down to 0% and the logs don’t show any scanning progress at that point.

[EFCRJ] 2015/12/02 11:38:49.309084 events.go:144: DEBUG: log 822 FolderScanProgress map[total:9184843301643 rate:6.348927468671727e+08 folder:testfolder current:1018198070166]

[EFCRJ] 2015/12/02 11:38:49.309110 events.go:206: DEBUG: poll 1m0s

[EFCRJ] 2015/12/02 11:38:51.302459 walk.go:173: DEBUG: Walk /main/dr_003/ [] current progress 1019463177110/9184843301643 at 605.5 MB/s (11%)

[EFCRJ] 2015/12/02 11:38:51.302533 events.go:144: DEBUG: log 823 FolderScanProgress map[folder:testfolder current:1019463177110 total:9184843301643 rate:6.348927468671727e+08]

[EFCRJ] 2015/12/02 11:38:51.302559 events.go:206: DEBUG: poll 1m0s

[EFCRJ] 2015/12/02 11:38:53.308801 walk.go:173: DEBUG: Walk /main/dr_003/ [] current progress 1020664320918/9184843301643 at 605.5 MB/s (11%)

[EFCRJ] 2015/12/02 11:38:53.308906 events.go:144: DEBUG: log 824 FolderScanProgress map[rate:6.348927468671727e+08 folder:testfolder current:1020664320918 total:9184843301643]

[EFCRJ] 2015/12/02 11:38:53.308935 events.go:206: DEBUG: poll 1m0s

[EFCRJ] 2015/12/02 11:38:55.304877 walk.go:173: DEBUG: Walk /main/dr_003/ [] current progress 1021897315222/9184843301643 at 604.5 MB/s (11%)

[EFCRJ] 2015/12/02 11:38:55.304961 events.go:144: DEBUG: log 825 FolderScanProgress map[folder:testfolder current:1021897315222 total:9184843301643 rate:6.339071374632143e+08]

[EFCRJ] 2015/12/02 11:38:55.304992 events.go:206: DEBUG: poll 1m0s

[EFCRJ] 2015/12/02 11:39:41.366786 events.go:144: DEBUG: log 826 Ping

[EFCRJ] 2015/12/02 11:39:41.366886 events.go:206: DEBUG: poll 1m0s

[EFCRJ] 2015/12/02 11:40:41.367019 events.go:206: DEBUG: poll 1m0s

[EFCRJ] 2015/12/02 11:40:41.367141 events.go:144: DEBUG: log 827 Ping

[EFCRJ] 2015/12/02 11:40:41.367175 events.go:206: DEBUG: poll 1m0s

[EFCRJ] 2015/12/02 11:41:41.367381 events.go:206: DEBUG: poll 1m0s

[EFCRJ] 2015/12/02 11:41:41.367516 events.go:144: DEBUG: log 828 Ping

[EFCRJ] 2015/12/02 11:41:41.367548 events.go:206: DEBUG: poll 1m0s

[EFCRJ] 2015/12/02 11:42:41.367779 events.go:144: DEBUG: log 829 Ping

[EFCRJ] 2015/12/02 11:42:41.367842 events.go:206: DEBUG: poll 1m0s

[EFCRJ] 2015/12/02 11:42:41.367861 events.go:206: DEBUG: poll 1m0s

[EFCRJ] 2015/12/02 11:43:41.368076 events.go:144: DEBUG: log 830 Ping

[EFCRJ] 2015/12/02 11:43:41.368129 events.go:206: DEBUG: poll 1m0s

[EFCRJ] 2015/12/02 11:43:41.368148 events.go:206: DEBUG: poll 1m0s

[EFCRJ] 2015/12/02 11:44:41.368322 events.go:144: DEBUG: log 831 Ping

[EFCRJ] 2015/12/02 11:44:41.368416 events.go:206: DEBUG: poll 1m0s

[EFCRJ] 2015/12/02 11:44:41.368453 events.go:206: DEBUG: poll 1m0s

[EFCRJ] 2015/12/02 11:45:41.368601 events.go:206: DEBUG: poll 1m0s

[EFCRJ] 2015/12/02 11:45:41.368717 events.go:144: DEBUG: log 832 Ping

[EFCRJ] 2015/12/02 11:45:41.368755 events.go:206: DEBUG: poll 1m0s

I am going to try same amount of files but one folder only… just for trying something different.

42

Can you also set STDEADLOCKTIMEOUT=600 env var, and do STTRACE=sync and when it happens again, do a SIGQUIT like before.

43

And give it 10 minutes after it gets stuck.

44

Here is the log:

syncthing stdeadlock timeout 600.txt (45.4 KB)

45

As far as Syncthing knows, it’s completed hashing. That’s consistent with what we saw above;

It’ll do that if it can’t read the files in question; are you sure there isn’t a permission problem or something underlying all this?

46

Can you provide a larger portion of the log, where it prints the scanning lines etc. Also, it’s not clear at what wall time SIGQUIT happened.

47

So it’s not scanning because it seems to think it’s done scanning. I assume you have only one folder.

goroutine 1551 [select, 123 minutes]:
github.com/syncthing/syncthing/lib/model.(*roFolder).Serve(0xc8201e4000)
        /home/jenkins/workspace/syncthing-release/src/github.com/syncthing/syncthing/lib/model/rofolder.go:69 +0x11b7

How do you know the scan is not complete?

48

This is the entire log I get from the beginning (same name of the file):

syncthing stdeadlock timeout 600.txt (59.7 KB)

How do you know the scan is not complete?

The scan shows 127 files scanned (I have 256 files) and 4.9 Tib of data (I have around 9 Tib):

I put all the files under one only folder to make it as simple as possible.

It’ll do that if it can’t read the files in question; are you sure there isn’t a permission problem or something underlying all this?

How do I find out which files are already synced? that way I can test the ones not synced separately.

49

Once it’s “finished”, can you start capturing the scanner log from that point, and click rescan?

50

And can you share the folder with something? As it is, the “Unshared” status obscures whether it’s scanning, idle or what.

51

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.