Files Transferred Multiple Times?

Support
1

Hello all:

I’m running Syncthing on two locally-network Synology NAS units in order to reconcile a large data set (~4.6 million files at ~37TB) between the units, prior to moving one of the NAS units off-site. Both units have nearly the same data set (unfortunately not quite the same - probably 99% identical), and the data is not being touched, to allow the sync to complete. Only these two units are signed into the cluster, and they each only have one Shared Folder.

I’m aware that the NAS units won’t give very high performance, and that I’m using a large data set - so have set my expectations accordingly. However, I’m getting much slower sync than I would expect; the rate of transfer seems to vary considerably, but I’m typically getting ~100GB reconciled per day - and it’s been going on for months now!

I’ve finally worked out how to enable logging on the Synology installation, and have enabled verbose logging on both units. After leaving them to work for 24hrs, I’ve come to examine the logs to see what’s going on.

Unfortunately the log file for the NAS which is most-out-of-sync (65% complete) is 113GB in size - so it’s taking me a little while to extract it out of the unit. However, the log file for the other NAS is much more reasonable (350MB), so I’ve been digging through it.

On first impression, everything looks good (to my eyes). After the initial setup, I can see three entries for each file:

  • Started syncing “abcdef” / “File/Path/Here” (metadata file)
  • Finished syncing “abcdef” / “File/Path/Here” (metadata file): Success
  • Remote change detected in folder “abcdef”: added file File/Path/Here

This log file has about 1.5million lines - so I would expect 500k files (or chunks?) to be reconciled over this period - but I’m only seeing about 40k files getting reconciled.

I’ve checked to see that each set of three entries is for a unique file - and have found quite the opposite: taking a small file (~4kB - less than the chunk size), I’m seeing it come up in the log file every few hundred lines.

Now, obviously, I might see what’s going on when I eventually manage to open the log file from the other NAS - but can you please confirm that what I’m seeing here is:

  • Normal in the sense that each file (chunk?) should have three entries in the verbose log?
  • Abnormal in the sense that each file is being endlessly repeated?

Many thanks,

Pants.

2

So, you haven’t provided any logs so it’s hard to say whats up. You should pick some file you believe is out of sync and check for that in the logs, or check logs for errors related to pulling/finishing etc.

Also, you could post screenshots from both sides to see if we can spot something obvious.

3

Hi Audrius:

Many thanks for responding on this. I’ve finally managed to open the log file on the most-out-of-sync NAS, and it’s showing some interesting points which I’ll dig through before bothering you further.

If that doesn’t sort it, what logs would be most useful for me to post? The entire verbose log files are massive - so I presume that wouldn’t be productive… Would hand-trimmed versions of them still be useful?

Thanks,

Pants.

4

Well the logs talk about specific files, so I guess you should post all logs relevant to some file you are having issues with.

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.