I have a Windows 2008 server with 40 million small files on this server that need to be synchronized!


(freddie) #1

I need to sync the files of this windows server to linux!

After I added the task, this task has been in the scan! It has been 2 days! When scanning, the memory usage is very high, and even all the memory is full!

In the past two days, syncthing has performed an automatic restart at least 3 times!

Is this normal?


In the process of mass file synchronization, I encountered two new problems.
(Simon) #2

Progress updates (for the percentage info in the gui) safe all filenames to be hashed, which in your case is huge. Set progressUpdateIntervalS to -1 to disable those progress updates, that should get your memory usage down (https://docs.syncthing.net/users/config.html#options-element).

EDIT: It’s ScanProgressIntervalS which is relevant for scanning, the progress one is for pulling. Both are probably relevant to you.


(freddie) #3

Ok,me try Thank you!


(Simon) #4

@xkwy521 See my edit above: I mixed up the names, it’s scanProgressIntervalS that’s relevant to scanning, but you probably want to set both to -1 (the other is for syncing).


(Simon) #5

As a side note: @AudriusButkevicius Do you remember the argument for buffering a list of files to hash instead of walking the filesystem twice and just counting the files to be hashed on first walk? There’s the overhead of walking twice but there could then be an option to switch between walking twice (low mem, higher disk usage) and walking once (high mem, lower disk usage).


(Audrius Butkevicius) #6

To have a consistent view. If files get added or deleted between walks we’ll return 110% scans etc.


(freddie) #7

No, I wait for the sync to complete before I add and delete

I am afraid that there will be errors during the synchronization process!

thank you for the help @AudriusButkevicius @imsodin


(freddie) #8

I am prompted for this on my synced Linux backup server! How to solve? please help me!

@AudriusButkevicius @imsodin


(freddie) #9

Has been resolved, is a firewall problem


(freddie) #10

NO, not a firewall error, I will still have this error when I close the firewall!

Also, according to the above method, after setting the ‘progressUpdateIntervalS’ and ‘ScanProgressIntervalS’ parameters, the windows server that needs to be synchronized, ‘syncthing’ only scans 8,841,021 files, it has been 6 hours, and no number has been increased! How to solve this? please help me!


(Simon) #11

That error originates from decoding json, I don’t understand how tcp comes into that. Ignoring the concrete error I could imagine that your config changes timed out due to the high load of scanning. So maybe the progress settings didn’t even take effect (check the config.xml directly).

These settings mean there is no status update on how far the scan has progressed. If you have 40 million files that all need to be hashed and compared between devices, that’s going to take more than 6h. Have some patience.


(freddie) #12

I have crashed, still can’t sync

The number of scanned files is always stuck at 8841201!


(Simon) #13

was it an out of memory crash? can you post logs


(freddie) #14

Yes, the memory has run out, the server is very card!

Is there a way to reduce memory usage! Can increase the hard disk usage rate!


(Audrius Butkevicius) #15

Can you switch off syncthing, set the two settings manually by modifying the config, and start it again? I am not convinced the settings were applied based on the errors you posted.


(freddie) #16
<configuration version="28">
<folder id="***" label="***" path="D:\wwwroot\***" type="sendonly" rescanIntervalS="15552000" fsWatcherEnabled="true" fsWatcherDelayS="10" ignorePerms="false" autoNormalize="true">
    <filesystemType>basic</filesystemType>
    <device id="***" introducedBy=""></device>
    <device id="***" introducedBy=""></device>
    <device id="***" introducedBy=""></device>
    <minDiskFree unit="%">1</minDiskFree>
    <versioning></versioning>
    <copiers>0</copiers>
    <pullerMaxPendingKiB>0</pullerMaxPendingKiB>
    <hashers>0</hashers>
    <order>random</order>
    <ignoreDelete>false</ignoreDelete>
    <scanProgressIntervalS>-1</scanProgressIntervalS>
    <pullerPauseS>0</pullerPauseS>
    <maxConflicts>10</maxConflicts>
    <disableSparseFiles>false</disableSparseFiles>
    <disableTempIndexes>false</disableTempIndexes>
    <paused>false</paused>
    <weakHashThresholdPct>25</weakHashThresholdPct>
    <markerName>.stfolder</markerName>
    <useLargeBlocks>false</useLargeBlocks>
</folder>
<device id="***" name="CA-8" compression="metadata" introducer="false" skipIntroductionRemovals="false" introducedBy="">
    <address>dynamic</address>
    <paused>false</paused>
    <autoAcceptFolders>false</autoAcceptFolders>
    <maxSendKbps>0</maxSendKbps>
    <maxRecvKbps>0</maxRecvKbps>
    <maxRequestKiB>0</maxRequestKiB>
</device>
<device id="***" name="WIN-0R08VG0GBSD" compression="metadata" introducer="false" skipIntroductionRemovals="false" introducedBy="">
    <address>dynamic</address>
    <paused>false</paused>
    <autoAcceptFolders>false</autoAcceptFolders>
    <maxSendKbps>0</maxSendKbps>
    <maxRecvKbps>0</maxRecvKbps>
    <maxRequestKiB>0</maxRequestKiB>
</device>
<device id="***" name="OL-FR-4-J" compression="never" introducer="false" skipIntroductionRemovals="false" introducedBy="">
    <address>dynamic</address>
    <paused>false</paused>
    <autoAcceptFolders>false</autoAcceptFolders>
    <maxSendKbps>0</maxSendKbps>
    <maxRecvKbps>0</maxRecvKbps>
    <maxRequestKiB>0</maxRequestKiB>
</device>
<gui enabled="true" tls="false" debugging="false">
    <address>0.0.0.0:8384</address>
    <user>user</user>
    <password>***</password>
    <apikey>wKVosDiMuRauKWwxVMibALDzcbuFwxyW</apikey>
    <theme>default</theme>
</gui>
<ldap></ldap>
<options>
    <listenAddress>default</listenAddress>
    <globalAnnounceServer>default</globalAnnounceServer>
    <globalAnnounceEnabled>true</globalAnnounceEnabled>
    <localAnnounceEnabled>true</localAnnounceEnabled>
    <localAnnouncePort>21027</localAnnouncePort>
    <localAnnounceMCAddr>[ff12::8384]:21027</localAnnounceMCAddr>
    <maxSendKbps>0</maxSendKbps>
    <maxRecvKbps>0</maxRecvKbps>
    <reconnectionIntervalS>60</reconnectionIntervalS>
    <relaysEnabled>true</relaysEnabled>
    <relayReconnectIntervalM>10</relayReconnectIntervalM>
    <startBrowser>true</startBrowser>
    <natEnabled>true</natEnabled>
    <natLeaseMinutes>60</natLeaseMinutes>
    <natRenewalMinutes>30</natRenewalMinutes>
    <natTimeoutSeconds>10</natTimeoutSeconds>
    <urAccepted>3</urAccepted>
    <urSeen>3</urSeen>
    <urUniqueID>EHSTEa79</urUniqueID>
    <urURL>https://data.syncthing.net/newdata</urURL>
    <urPostInsecurely>false</urPostInsecurely>
    <urInitialDelayS>1800</urInitialDelayS>
    <restartOnWakeup>true</restartOnWakeup>
    <autoUpgradeIntervalH>12</autoUpgradeIntervalH>
    <upgradeToPreReleases>false</upgradeToPreReleases>
    <keepTemporariesH>24</keepTemporariesH>
    <cacheIgnoredFiles>false</cacheIgnoredFiles>
    <progressUpdateIntervalS>-1</progressUpdateIntervalS>
    <limitBandwidthInLan>false</limitBandwidthInLan>
    <minHomeDiskFree unit="%">1</minHomeDiskFree>
    <releasesURL>https://upgrades.syncthing.net/meta.json</releasesURL>
    <overwriteRemoteDeviceNamesOnConnect>false</overwriteRemoteDeviceNamesOnConnect>
    <tempIndexMinBlocks>10</tempIndexMinBlocks>
    <trafficClass>0</trafficClass>
    <defaultFolderPath>~</defaultFolderPath>
    <setLowPriority>true</setLowPriority>
    <maxConcurrentScans>0</maxConcurrentScans>
    <minHomeDiskFreePct>0</minHomeDiskFreePct>
</options>

This is the configuration on my window server @AudriusButkevicius


(Audrius Butkevicius) #17

Looks right. Can you capture all logs when it’s running and provide them after it crashes?


(freddie) #18

Ok, I will try and feedback