Dix
April 25, 2021, 7:12pm
1
Hello!
Please help with the synchronization of large folders, they contain more than 500 thousand files. Folders less than 200k files are synchronized normally. I read the forum, but I could not solve the problem. Synching freezes during the scanning phase. This can be seen from the resource monitor, the syncthing process does nothing with large folders. I cannot split files into different folders. Perhaps there are any recommendations for the settings.
What have I tried:
true
large
rescanIntervalS = “86400”
Common parameters:
Hosts - 2 x Windows Server 2012 R2, 16 gb RAM, 8 Core
syncthing - version 1.4.0
Folders - 1506
Files - 29.8 million, 15.9Tb
Thank you!
imsodin
April 26, 2021, 8:30am
2
That’s pretty old - please upgrade to a recent version before continuing debugging.
So what is happening? What is the overall resource usage (cpu, ram, disk, network) and Synthing’s usage?
Dix
April 26, 2021, 3:52pm
3
Hello! Thanks for trying to help.
The version was updated to 1.15.1. Problem still exists.
When scanning small folders (less than 200k files) - CPU load up to 90% -92%.
In the resource monitor in the DISK tab, I see that syncthing reads both files and databases. Disk load 100%.
When there are 3 folders with a large number of files (more than 300k), the CPU load becomes less than 15% -20%. In the resource monitor, I see only reading the database files. Disc load 10% -30%.
The network is always loaded with 15-25 Mbps from 1Gb.
RAM is always free, the syncthing process takes no more than 2Gb.
imsodin
April 26, 2021, 3:55pm
4
What’s the folder status when the scan “stalls” (ideally post a screenshot)? You can enable scanner debug logging (through environment variable STTRACE=scanner or in the web UI actions > logs) to see what it is doing.
terry
April 26, 2021, 4:21pm
5
I would suggest popping in an SSD to host your db indexes on if you can. A HDD grinding away on indexes with high volumes of files will take a very long time to complete. An SSD will take some time, but is considerably quicker.
Dix
April 26, 2021, 4:51pm
6
This is the state
I have enabled debug scanning. No new information has appeared for this folder. Paused and turned on again. And this is what I saw in the log:
[OZZTP] 2021/04/26 19:31:27.759852 folder.go:860: INFO: Failed initial scan of sendonly folder "FILE_STORAGE_20180110" (FILE_STORAGE_20180110)
[OZZTP] 2021/04/26 19:31:27.759852 model.go:583: INFO: Paused folder "FILE_STORAGE_20180110" (FILE_STORAGE_20180110) (sendonly)
[OZZTP] 2021/04/26 19:36:09.364431 model.go:421: INFO: Ready to synchronize "FILE_STORAGE_20180110" (FILE_STORAGE_20180110) (sendonly)
[OZZTP] 2021/04/26 19:36:09.364431 model.go:583: INFO: Unpaused folder "FILE_STORAGE_20180110" (FILE_STORAGE_20180110) (sendonly)
An SSD for databases is good advice. Unfortunately, it cannot be applied in this case.
imsodin
April 26, 2021, 5:50pm
7
Something went wrong there with the logs, because at the very least there must be a line containing “DEBUG: Walk” indicating that it started scanning (otherwise the folder wouldn’t be in scanning state. And then lines with “dir:”, “to hash:” etc.
However if ram is not an issue, then indeed what you are seeing is just the first pass over the filesystem, which compares items on disk with the db. If your folder is huge and on slow storage, and the db is also on slow storage, possibly even the same storage, then what you are seeing is expected. Time will mostly be spent spinning the disk.
Dix
April 26, 2021, 6:43pm
8
Later, an entry appeared in the log:
[OZZTP] 2021/04/26 19: 38: 46.234679 walk.go: 171: DEBUG: Walk FILE_STORAGE_20180110
For 2 hours there was not a single record. Does this mean that this folder has frozen?
imsodin
April 27, 2021, 12:44pm
9
That’s just a periodic progress update log. As I explained above, it being at 0 in the first face of scanning is expected.
It’s however impossible that you get from
to
without the “DEBUG: Walk” line in between mentioned before.
And to get an even more verbose output, enable all of scanner, model, db and walkfs debugging facilities.
Dix
April 27, 2021, 1:56pm
10
At the moment I have 3 folders that cannot go to the “up to date” status.
I have included scanner, model, db and walkfs. The problematic folders FILE_STORAGE_20180101 and FILE_STORAGE_20180109 have the following entries:
[OZZTP] 2021/04/27 16:14:01.949035 set.go:156: DEBUG: FILE_STORAGE_20180101 Snapshot()
[OZZTP] 2021/04/27 16:14:01.949035 model.go:935: DEBUG: model@0xc0001c6340 Completion(7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4, "FILE_STORAGE_20180101"): map[completion:100 globalBytes:149914253368 globalItems:312239 needBytes:0 needDeletes:0 needItems:0 sequence:312239]
[OZZTP] 2021/04/27 16:14:03.215110 model.go:1139: DEBUG: Index update (in): 6D2P6FZ-BONBTS7-WQPMBJH-PHATWH4-DOUFBH2-LSPDBBO-LB4A4AN-DF26HAC / "FILE_STORAGE_20180101": 1 files
[OZZTP] 2021/04/27 16:14:03.215110 set.go:118: DEBUG: FILE_STORAGE_20180101 Update(6D2P6FZ-BONBTS7-WQPMBJH-PHATWH4-DOUFBH2-LSPDBBO-LB4A4AN-DF26HAC, [1])
[OZZTP] 2021/04/27 16:14:03.216109 lowlevel.go:171: DEBUG: insert (remote); folder="FILE_STORAGE_20180101" device=6D2P6FZ-BONBTS7-WQPMBJH-PHATWH4-DOUFBH2-LSPDBBO-LB4A4AN-DF26HAC File{Name:"Комментарии согласующего Шелепина (Руководитель департамента). Указатель рассылки.doc_ae75b8ea-d83a-4f8f-92e1-3c223a38c8e8.1..doc", Sequence:161888, Permissions:0444, ModTime:2015-01-23 11:18:08 +0300 MSK, Version:{[{OZZTPFB 1619332700}]}, VersionHash:, Length:32256, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:false, BlockSize:131072, Blocks:[Block{0/32256/946798576/1aae71ce85b3d5ab4c76df7e2bb5fef7283b250829c0c5106464e7061aaadfe1}], BlocksHash:eca489e94d61428b04b62de59e1d1ee4130940783fe36ee5da5b59091cbfbe2e}
[OZZTP] 2021/04/27 16:14:03.216109 transactions.go:610: DEBUG: update global; folder="FILE_STORAGE_20180101" device=6D2P6FZ-BONBTS7-WQPMBJH-PHATWH4-DOUFBH2-LSPDBBO-LB4A4AN-DF26HAC file="Комментарии согласующего Шелепина (Руководитель департамента). Указатель рассылки.doc_ae75b8ea-d83a-4f8f-92e1-3c223a38c8e8.1..doc" version={[{OZZTPFB 1619332700}]} invalid=false
[OZZTP] 2021/04/27 16:14:03.571130 model.go:1834: DEBUG: model@0xc0001c6340 REQ(in): 6D2P6FZ-BONBTS7-WQPMBJH-PHATWH4-DOUFBH2-LSPDBBO-LB4A4AN-DF26HAC: "FILE_STORAGE_20180101" / "RO997485406CN.pdf_4a2a227a-ea29-4a9f-a4e9-3238643e6639.1..pdf" o=0 s=56590 t=false
[OZZTP] 2021/04/27 16:14:03.714138 set.go:156: DEBUG: FILE_STORAGE_20180101 Snapshot()
[OZZTP] 2021/04/27 16:14:03.714138 progressemitter.go:290: DEBUG: progress emitter: bytes completed for FILE_STORAGE_20180101: 0
There are fewer entries for the problem folder FILE_STORAGE_ 20180110:
[OZZTP] 2021/04/27 16:14:01.962037 set.go:156: DEBUG: FILE_STORAGE_20180110 Snapshot()
[OZZTP] 2021/04/27 16:14:01.963037 model.go:935: DEBUG: model@0xc0001c6340 Completion(7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4, "FILE_STORAGE_20180110"): map[completion:100 globalBytes:104205883647 globalItems:161552 needBytes:0 needDeletes:0 needItems:0 sequence:161552]
I also caught files with the modified date 01/01/1601, after copying such a file to the second server, the date changes to 07/22/2185.
Such files are present in all folders, not only in problematic ones.
It doesn’t matter to me, but maybe it matters to Syncthing? File system - NTFS. Second server folders are configured to receive only
imsodin
April 27, 2021, 2:56pm
11
If all of those folders are on the same (array of) spinning disk(s), I recommend setting maxFolderConcurrency to 1 to reduce seeking (Syncthing Configuration — Syncthing v1 documentation ).
Are you removing any lines from the logs? A folder being in status “Scanning” and no filesystem activity at all would quite curious. The only reason for that I can imagine is that the database is blocking like crazy, even on reads. As mentioned before, if your db is on a spinning disk, even worse on the same as the data, then low performance is expected. Not that low though. The contents of your database directory might be interesting, i.e. list of files with their sizes - google tells me you can use ls in powershell (and it behaves like ls -l), the output of that on the db dir should contain that info.
1 Like
Dix
April 27, 2021, 4:24pm
12
You are right, all data and DB are on the same raid spinning disks array. These servers are in production, new data that is not in problem folders is much more important, and they have time to synchronize. If I set maxFolderConcurrency to 1, then I might get stuck on the “old” problem folder. I will ask the management if we can temporarily neglect this.
The log files are very large, and I really, using text search, selected lines with the necessary folders. If you give recommendations for collecting logs, then I will follow them and attach them entirely to where you say.
The output of the ls command in PowerShell looks cumbersome, the file size is displayed in bytes, which is difficult to understand. I have modernized the output:
Name - Size (Mb) - LastWriteTime
065337.ldb - 256.37Mb - 04/25/2021 09:20:39
065338.ldb - 256.36Mb - 04/25/2021 09:20:43
065339.ldb - 256.33Mb - 04/25/2021 09:20:47
065340.ldb - 256.33Mb - 04/25/2021 09:20:51
065341.ldb - 256.36Mb - 04/25/2021 09:20:55
065342.ldb - 256.33Mb - 04/25/2021 09:20:59
065343.ldb - 256.34Mb - 04/25/2021 09:21:05
065344.ldb - 256.34Mb - 04/25/2021 09:21:09
065345.ldb - 256.36Mb - 04/25/2021 09:21:15
065346.ldb - 256.33Mb - 04/25/2021 09:21:21
065347.ldb - 256.34Mb - 04/25/2021 09:21:26
065348.ldb - 256.35Mb - 04/25/2021 09:21:31
065349.ldb - 256.38Mb - 04/25/2021 09:21:36
065350.ldb - 256.35Mb - 04/25/2021 09:21:41
065351.ldb - 256.38Mb - 04/25/2021 09:21:47
065352.ldb - 256.37Mb - 04/25/2021 09:21:53
065353.ldb - 256.37Mb - 04/25/2021 09:22:00
065354.ldb - 256.36Mb - 04/25/2021 09:22:06
065355.ldb - 256.39Mb - 04/25/2021 09:22:13
065356.ldb - 256.38Mb - 04/25/2021 09:22:18
065357.ldb - 256.37Mb - 04/25/2021 09:22:24
065358.ldb - 256.38Mb - 04/25/2021 09:22:31
065359.ldb - 256.39Mb - 04/25/2021 09:22:39
065360.ldb - 256.37Mb - 04/25/2021 09:22:45
065361.ldb - 256.35Mb - 04/25/2021 09:22:51
065362.ldb - 256.36Mb - 04/25/2021 09:22:56
065363.ldb - 256.37Mb - 04/25/2021 09:23:02
065364.ldb - 256.37Mb - 04/25/2021 09:23:11
065365.ldb - 256.35Mb - 04/25/2021 09:23:16
065366.ldb - 256.34Mb - 04/25/2021 09:23:22
065367.ldb - 256.35Mb - 04/25/2021 09:23:29
065592.ldb - 256.25Mb - 04/25/2021 22:36:43
065593.ldb - 256.29Mb - 04/25/2021 22:36:48
065594.ldb - 256.32Mb - 04/25/2021 22:36:54
065595.ldb - 256.3Mb - 04/25/2021 22:36:59
065596.ldb - 256.3Mb - 04/25/2021 22:37:09
065597.ldb - 256.29Mb - 04/25/2021 22:37:16
065598.ldb - 256.33Mb - 04/25/2021 22:37:24
065599.ldb - 256.31Mb - 04/25/2021 22:37:30
065600.ldb - 256.32Mb - 04/25/2021 22:37:36
065601.ldb - 256.32Mb - 04/25/2021 22:37:41
065602.ldb - 256.31Mb - 04/25/2021 22:37:46
065603.ldb - 256.31Mb - 04/25/2021 22:37:52
065604.ldb - 256.29Mb - 04/25/2021 22:37:57
065605.ldb - 256.3Mb - 04/25/2021 22:38:02
065606.ldb - 256.3Mb - 04/25/2021 22:38:08
065607.ldb - 256.3Mb - 04/25/2021 22:38:14
065608.ldb - 256.31Mb - 04/25/2021 22:38:19
065609.ldb - 256.28Mb - 04/25/2021 22:38:24
065610.ldb - 256.27Mb - 04/25/2021 22:38:29
065611.ldb - 256.26Mb - 04/25/2021 22:38:36
065612.ldb - 256.29Mb - 04/25/2021 22:38:44
065613.ldb - 256.28Mb - 04/25/2021 22:38:51
065614.ldb - 256.28Mb - 04/25/2021 22:38:58
065615.ldb - 256.27Mb - 04/25/2021 22:39:05
065616.ldb - 256.27Mb - 04/25/2021 22:39:12
065617.ldb - 256.27Mb - 04/25/2021 22:39:20
065618.ldb - 256.31Mb - 04/25/2021 22:39:28
065619.ldb - 256.29Mb - 04/25/2021 22:39:35
065620.ldb - 256.29Mb - 04/25/2021 22:39:41
065621.ldb - 256.28Mb - 04/25/2021 22:39:48
065622.ldb - 256.3Mb - 04/25/2021 22:39:54
065623.ldb - 256.29Mb - 04/25/2021 22:39:59
065624.ldb - 256.27Mb - 04/25/2021 22:40:05
065625.ldb - 256.28Mb - 04/25/2021 22:40:10
065626.ldb - 256.27Mb - 04/25/2021 22:40:16
065627.ldb - 256.28Mb - 04/25/2021 22:40:22
065628.ldb - 256.3Mb - 04/25/2021 22:40:28
065629.ldb - 256.27Mb - 04/25/2021 22:40:34
065630.ldb - 256.24Mb - 04/25/2021 22:40:39
065631.ldb - 256.25Mb - 04/25/2021 22:40:45
065632.ldb - 256.25Mb - 04/25/2021 22:40:51
065633.ldb - 256.28Mb - 04/25/2021 22:40:57
065634.ldb - 256.24Mb - 04/25/2021 22:41:02
065635.ldb - 256.27Mb - 04/25/2021 22:41:09
065636.ldb - 256.27Mb - 04/25/2021 22:41:14
065637.ldb - 256.28Mb - 04/25/2021 22:41:20
065638.ldb - 256.24Mb - 04/25/2021 22:41:26
065639.ldb - 256.24Mb - 04/25/2021 22:41:31
065721.ldb - 0.59Mb - 04/26/2021 11:31:22
065789.ldb - 256.23Mb - 04/26/2021 12:39:29
065790.ldb - 256.22Mb - 04/26/2021 12:39:38
065791.ldb - 256.22Mb - 04/26/2021 12:39:45
065792.ldb - 256.24Mb - 04/26/2021 12:39:53
065793.ldb - 256.28Mb - 04/26/2021 12:40:01
065794.ldb - 256.25Mb - 04/26/2021 12:40:09
065795.ldb - 256.27Mb - 04/26/2021 12:40:18
065796.ldb - 256.28Mb - 04/26/2021 12:40:26
065797.ldb - 256.26Mb - 04/26/2021 12:40:34
065798.ldb - 192.81Mb - 04/26/2021 12:40:40
065811.ldb - 3.31Mb - 04/26/2021 16:05:18
065812.ldb - 256.21Mb - 04/26/2021 16:06:08
065813.ldb - 256.22Mb - 04/26/2021 16:06:15
065814.ldb - 256.19Mb - 04/26/2021 16:06:24
065815.ldb - 256.2Mb - 04/26/2021 16:06:30
065816.ldb - 256.2Mb - 04/26/2021 16:06:38
065817.ldb - 256.18Mb - 04/26/2021 16:06:45
065818.ldb - 256.19Mb - 04/26/2021 16:06:52
065819.ldb - 256.21Mb - 04/26/2021 16:07:01
065820.ldb - 256.14Mb - 04/26/2021 16:07:06
065821.ldb - 256.14Mb - 04/26/2021 16:07:13
065822.ldb - 19.8Mb - 04/26/2021 16:07:13
065823.ldb - 256.36Mb - 04/26/2021 16:07:19
065824.ldb - 256.51Mb - 04/26/2021 16:07:28
065825.ldb - 256.54Mb - 04/26/2021 16:07:37
065826.ldb - 256.54Mb - 04/26/2021 16:07:46
065827.ldb - 256.53Mb - 04/26/2021 16:08:12
065828.ldb - 256.53Mb - 04/26/2021 16:08:35
065829.ldb - 256.53Mb - 04/26/2021 16:08:44
065830.ldb - 36.61Mb - 04/26/2021 16:08:45
065831.ldb - 256.15Mb - 04/26/2021 16:09:21
065832.ldb - 256.15Mb - 04/26/2021 16:09:27
065833.ldb - 256.13Mb - 04/26/2021 16:09:33
065834.ldb - 256.15Mb - 04/26/2021 16:09:39
065835.ldb - 256.15Mb - 04/26/2021 16:09:46
065836.ldb - 256.13Mb - 04/26/2021 16:09:53
065837.ldb - 256.14Mb - 04/26/2021 16:10:00
065838.ldb - 256.16Mb - 04/26/2021 16:10:08
065839.ldb - 256.11Mb - 04/26/2021 16:10:16
065840.ldb - 256.14Mb - 04/26/2021 16:10:23
065841.ldb - 2.48Mb - 04/26/2021 16:10:23
065842.ldb - 256.14Mb - 04/26/2021 16:13:21
065843.ldb - 256.17Mb - 04/26/2021 16:13:37
065844.ldb - 256.14Mb - 04/26/2021 16:14:07
065845.ldb - 0.67Mb - 04/26/2021 16:14:07
065945.ldb - 256.25Mb - 04/27/2021 01:46:54
065946.ldb - 256.3Mb - 04/27/2021 01:46:59
065947.ldb - 256.33Mb - 04/27/2021 01:47:05
065948.ldb - 256.33Mb - 04/27/2021 01:47:16
065949.ldb - 256.31Mb - 04/27/2021 01:47:27
065950.ldb - 256.3Mb - 04/27/2021 01:47:39
065951.ldb - 256.29Mb - 04/27/2021 01:47:55
065952.ldb - 256.3Mb - 04/27/2021 01:48:06
065953.ldb - 256.32Mb - 04/27/2021 01:48:21
065954.ldb - 256.29Mb - 04/27/2021 01:48:27
065955.ldb - 256.29Mb - 04/27/2021 01:48:34
065956.ldb - 256.31Mb - 04/27/2021 01:48:39
065957.ldb - 256.31Mb - 04/27/2021 01:48:45
065958.ldb - 256.33Mb - 04/27/2021 01:48:51
065959.ldb - 256.29Mb - 04/27/2021 01:48:57
065960.ldb - 256.32Mb - 04/27/2021 01:49:02
065961.ldb - 256.27Mb - 04/27/2021 01:49:08
065962.ldb - 256.28Mb - 04/27/2021 01:49:15
065963.ldb - 256.29Mb - 04/27/2021 01:49:21
065964.ldb - 256.27Mb - 04/27/2021 01:49:27
065965.ldb - 256.3Mb - 04/27/2021 01:49:33
065966.ldb - 256.25Mb - 04/27/2021 01:49:39
065967.ldb - 256.29Mb - 04/27/2021 01:49:45
065968.ldb - 256.26Mb - 04/27/2021 01:49:51
065969.ldb - 256.29Mb - 04/27/2021 01:49:57
065970.ldb - 256.28Mb - 04/27/2021 01:50:04
065971.ldb - 256.28Mb - 04/27/2021 01:50:10
065972.ldb - 256.31Mb - 04/27/2021 01:50:15
065973.ldb - 256.29Mb - 04/27/2021 01:50:21
065974.ldb - 256.27Mb - 04/27/2021 01:50:26
065975.ldb - 256.31Mb - 04/27/2021 01:50:32
065976.ldb - 256.26Mb - 04/27/2021 01:50:37
065977.ldb - 256.29Mb - 04/27/2021 01:50:42
065978.ldb - 256.25Mb - 04/27/2021 01:50:48
065979.ldb - 256.25Mb - 04/27/2021 01:50:53
065980.ldb - 256.26Mb - 04/27/2021 01:50:59
065981.ldb - 256.28Mb - 04/27/2021 01:51:04
065982.ldb - 256.27Mb - 04/27/2021 01:51:09
065983.ldb - 256.24Mb - 04/27/2021 01:51:14
065984.ldb - 256.26Mb - 04/27/2021 01:51:20
065985.ldb - 256.26Mb - 04/27/2021 01:51:25
065986.ldb - 256.26Mb - 04/27/2021 01:51:30
065987.ldb - 256.27Mb - 04/27/2021 01:51:35
065988.ldb - 256.28Mb - 04/27/2021 01:51:40
065989.ldb - 256.28Mb - 04/27/2021 01:51:46
065990.ldb - 256.27Mb - 04/27/2021 01:51:52
065991.ldb - 256.25Mb - 04/27/2021 01:51:58
066016.ldb - 256.23Mb - 04/27/2021 01:53:52
066017.ldb - 256.2Mb - 04/27/2021 01:53:58
066018.ldb - 256.23Mb - 04/27/2021 01:54:04
066019.ldb - 256.24Mb - 04/27/2021 01:54:09
066020.ldb - 256.26Mb - 04/27/2021 01:54:15
066021.ldb - 256.23Mb - 04/27/2021 01:54:20
066022.ldb - 256.26Mb - 04/27/2021 01:54:25
066023.ldb - 256.27Mb - 04/27/2021 01:54:31
066024.ldb - 256.3Mb - 04/27/2021 01:54:37
066025.ldb - 68.75Mb - 04/27/2021 01:54:38
066052.ldb - 12.28Mb - 04/27/2021 06:12:22
066063.ldb - 0.79Mb - 04/27/2021 06:13:37
066068.ldb - 0.21Mb - 04/27/2021 06:19:58
066075.ldb - 5.23Mb - 04/27/2021 09:36:11
066076.ldb - 256.25Mb - 04/27/2021 09:37:15
066077.ldb - 256.17Mb - 04/27/2021 09:37:21
066078.ldb - 256.22Mb - 04/27/2021 09:37:27
066079.ldb - 256.18Mb - 04/27/2021 09:37:34
066080.ldb - 256.17Mb - 04/27/2021 09:37:41
066081.ldb - 256.18Mb - 04/27/2021 09:37:49
066082.ldb - 256.18Mb - 04/27/2021 09:37:57
066083.ldb - 256.2Mb - 04/27/2021 09:38:06
066084.ldb - 256.15Mb - 04/27/2021 09:38:11
066085.ldb - 256.16Mb - 04/27/2021 09:38:16
066086.ldb - 15.8Mb - 04/27/2021 09:38:16
066087.ldb - 256.35Mb - 04/27/2021 09:38:22
066088.ldb - 256.48Mb - 04/27/2021 09:38:28
066089.ldb - 256.54Mb - 04/27/2021 09:38:34
066090.ldb - 256.54Mb - 04/27/2021 09:38:41
066091.ldb - 256.54Mb - 04/27/2021 09:38:48
066092.ldb - 256.53Mb - 04/27/2021 09:38:56
066093.ldb - 256.53Mb - 04/27/2021 09:39:06
066094.ldb - 256.36Mb - 04/27/2021 09:39:13
066095.ldb - 11.64Mb - 04/27/2021 09:39:13
066096.ldb - 256.14Mb - 04/27/2021 09:39:59
066097.ldb - 256.15Mb - 04/27/2021 09:40:05
066098.ldb - 256.12Mb - 04/27/2021 09:40:12
066099.ldb - 256.14Mb - 04/27/2021 09:40:18
066100.ldb - 256.16Mb - 04/27/2021 09:40:23
066101.ldb - 256.17Mb - 04/27/2021 09:40:28
066102.ldb - 256.15Mb - 04/27/2021 09:40:34
066103.ldb - 256.11Mb - 04/27/2021 09:40:43
066104.ldb - 256.16Mb - 04/27/2021 09:40:55
066105.ldb - 256.12Mb - 04/27/2021 09:41:13
066106.ldb - 15.43Mb - 04/27/2021 09:41:15
066107.ldb - 256.17Mb - 04/27/2021 09:45:13
066108.ldb - 256.12Mb - 04/27/2021 09:45:18
066109.ldb - 256.11Mb - 04/27/2021 09:45:22
066110.ldb - 256.23Mb - 04/27/2021 09:45:27
066111.ldb - 0.37Mb - 04/27/2021 09:45:27
066118.ldb - 7.96Mb - 04/27/2021 12:50:08
066119.ldb - 256.26Mb - 04/27/2021 12:50:17
066120.ldb - 256.21Mb - 04/27/2021 12:50:22
066121.ldb - 256.2Mb - 04/27/2021 12:50:30
066122.ldb - 256.19Mb - 04/27/2021 12:50:37
066123.ldb - 256.16Mb - 04/27/2021 12:50:45
066124.ldb - 256.17Mb - 04/27/2021 12:50:52
066125.ldb - 256.16Mb - 04/27/2021 12:50:59
066126.ldb - 256.17Mb - 04/27/2021 12:51:07
066127.ldb - 256.15Mb - 04/27/2021 12:51:13
066128.ldb - 256.12Mb - 04/27/2021 12:51:17
066129.ldb - 17.72Mb - 04/27/2021 12:51:18
066130.ldb - 256.35Mb - 04/27/2021 12:51:25
066131.ldb - 256.49Mb - 04/27/2021 12:51:32
066132.ldb - 256.52Mb - 04/27/2021 12:51:39
066133.ldb - 256.53Mb - 04/27/2021 12:51:48
066134.ldb - 256.54Mb - 04/27/2021 12:51:55
066135.ldb - 256.53Mb - 04/27/2021 12:52:03
066136.ldb - 256.51Mb - 04/27/2021 12:52:11
066137.ldb - 256.35Mb - 04/27/2021 12:52:17
066138.ldb - 10.93Mb - 04/27/2021 12:52:17
066139.ldb - 256.13Mb - 04/27/2021 12:56:21
066140.ldb - 256.13Mb - 04/27/2021 12:56:25
066141.ldb - 256.13Mb - 04/27/2021 12:56:28
066142.ldb - 256.14Mb - 04/27/2021 12:56:31
066143.ldb - 256.13Mb - 04/27/2021 12:56:34
066144.ldb - 256.15Mb - 04/27/2021 12:56:37
066145.ldb - 256.16Mb - 04/27/2021 12:56:41
066146.ldb - 256.15Mb - 04/27/2021 12:56:45
066147.ldb - 256.14Mb - 04/27/2021 12:56:49
066148.ldb - 256.11Mb - 04/27/2021 12:56:52
066149.ldb - 17.86Mb - 04/27/2021 12:56:53
066150.ldb - 256.11Mb - 04/27/2021 13:02:08
066151.ldb - 256.14Mb - 04/27/2021 13:02:14
066152.ldb - 256.15Mb - 04/27/2021 13:02:20
066153.ldb - 256.22Mb - 04/27/2021 13:02:25
066154.ldb - 16.96Mb - 04/27/2021 13:02:27
066155.log - 15.67Mb - 04/27/2021 16:10:38
066164.ldb - 256.34Mb - 04/27/2021 14:56:58
066165.ldb - 256.49Mb - 04/27/2021 14:57:03
066166.ldb - 256.52Mb - 04/27/2021 14:57:09
066167.ldb - 256.53Mb - 04/27/2021 14:57:15
066168.ldb - 256.53Mb - 04/27/2021 14:57:30
066169.ldb - 256.53Mb - 04/27/2021 14:57:37
066170.ldb - 256.51Mb - 04/27/2021 14:57:43
066171.ldb - 256.38Mb - 04/27/2021 14:57:48
066172.ldb - 256.26Mb - 04/27/2021 14:57:52
066173.ldb - 256.3Mb - 04/27/2021 14:57:57
066174.ldb - 256.31Mb - 04/27/2021 14:58:01
066175.ldb - 256.33Mb - 04/27/2021 14:58:06
066176.ldb - 256.29Mb - 04/27/2021 14:58:11
066177.ldb - 256.32Mb - 04/27/2021 14:58:15
066178.ldb - 256.29Mb - 04/27/2021 14:58:20
066179.ldb - 256.31Mb - 04/27/2021 14:58:25
066180.ldb - 256.32Mb - 04/27/2021 14:58:29
066181.ldb - 256.32Mb - 04/27/2021 14:58:34
066182.ldb - 256.32Mb - 04/27/2021 14:58:39
066183.ldb - 256.31Mb - 04/27/2021 14:58:43
066184.ldb - 256.29Mb - 04/27/2021 14:58:48
066185.ldb - 256.32Mb - 04/27/2021 14:58:53
066186.ldb - 256.31Mb - 04/27/2021 14:58:58
066187.ldb - 256.31Mb - 04/27/2021 14:59:03
066188.ldb - 256.28Mb - 04/27/2021 14:59:08
066189.ldb - 256.3Mb - 04/27/2021 14:59:13
066190.ldb - 256.31Mb - 04/27/2021 14:59:18
066191.ldb - 256.28Mb - 04/27/2021 14:59:23
066192.ldb - 256.3Mb - 04/27/2021 14:59:29
066193.ldb - 256.25Mb - 04/27/2021 14:59:34
066194.ldb - 256.26Mb - 04/27/2021 14:59:39
066195.ldb - 256.26Mb - 04/27/2021 14:59:44
066196.ldb - 256.29Mb - 04/27/2021 14:59:49
066197.ldb - 256.28Mb - 04/27/2021 14:59:54
066198.ldb - 256.32Mb - 04/27/2021 14:59:58
066199.ldb - 256.31Mb - 04/27/2021 15:00:06
066200.ldb - 256.31Mb - 04/27/2021 15:00:11
066201.ldb - 256.29Mb - 04/27/2021 15:00:17
066202.ldb - 256.26Mb - 04/27/2021 15:00:21
066203.ldb - 256.3Mb - 04/27/2021 15:00:26
066204.ldb - 256.28Mb - 04/27/2021 15:00:31
066205.ldb - 256.29Mb - 04/27/2021 15:00:35
066206.ldb - 256.3Mb - 04/27/2021 15:00:40
066207.ldb - 256.26Mb - 04/27/2021 15:00:44
066208.ldb - 256.28Mb - 04/27/2021 15:00:49
066209.ldb - 256.24Mb - 04/27/2021 15:00:53
066210.ldb - 256.25Mb - 04/27/2021 15:00:58
066211.ldb - 256.27Mb - 04/27/2021 15:01:02
066212.ldb - 256.27Mb - 04/27/2021 15:01:09
066213.ldb - 256.25Mb - 04/27/2021 15:01:15
066214.ldb - 256.26Mb - 04/27/2021 15:01:19
066215.ldb - 256.24Mb - 04/27/2021 15:01:24
066216.ldb - 256.27Mb - 04/27/2021 15:01:29
066217.ldb - 256.25Mb - 04/27/2021 15:01:33
066218.ldb - 256.25Mb - 04/27/2021 15:01:38
066219.ldb - 256.24Mb - 04/27/2021 15:01:42
066220.ldb - 256.21Mb - 04/27/2021 15:01:47
066221.ldb - 256.22Mb - 04/27/2021 15:01:52
066222.ldb - 256.16Mb - 04/27/2021 15:01:57
066223.ldb - 256.18Mb - 04/27/2021 15:02:03
066224.ldb - 256.2Mb - 04/27/2021 15:02:08
066225.ldb - 256.16Mb - 04/27/2021 15:02:14
066226.ldb - 256.19Mb - 04/27/2021 15:02:20
066227.ldb - 256.13Mb - 04/27/2021 15:02:24
066228.ldb - 256.14Mb - 04/27/2021 15:02:27
066229.ldb - 256.16Mb - 04/27/2021 15:02:29
066230.ldb - 256.14Mb - 04/27/2021 15:02:32
066231.ldb - 256.14Mb - 04/27/2021 15:02:36
066232.ldb - 256.13Mb - 04/27/2021 15:02:39
066233.ldb - 256.12Mb - 04/27/2021 15:02:44
066234.ldb - 256.14Mb - 04/27/2021 15:02:47
066235.ldb - 256.12Mb - 04/27/2021 15:02:49
066236.ldb - 256.14Mb - 04/27/2021 15:02:52
066237.ldb - 256.12Mb - 04/27/2021 15:02:55
066238.ldb - 256.14Mb - 04/27/2021 15:02:57
066239.ldb - 256.14Mb - 04/27/2021 15:03:01
066240.ldb - 256.12Mb - 04/27/2021 15:03:04
066241.ldb - 256.15Mb - 04/27/2021 15:03:07
066242.ldb - 256.19Mb - 04/27/2021 15:03:09
066243.ldb - 256.23Mb - 04/27/2021 15:03:14
066244.ldb - 256.24Mb - 04/27/2021 15:03:19
066245.ldb - 256.22Mb - 04/27/2021 15:03:23
066246.ldb - 256.23Mb - 04/27/2021 15:03:28
066247.ldb - 256.26Mb - 04/27/2021 15:03:33
066248.ldb - 256.27Mb - 04/27/2021 15:03:38
066249.ldb - 256.24Mb - 04/27/2021 15:03:42
066250.ldb - 256.28Mb - 04/27/2021 15:03:47
066251.ldb - 256.29Mb - 04/27/2021 15:03:51
066252.ldb - 179.09Mb - 04/27/2021 15:03:55
CURRENT - 0Mb - 04/26/2021 11:31:22
CURRENT.bak - 0Mb - 04/26/2021 11:31:22
LOCK - 0Mb - 04/12/2021 17:30:44
LOG - 0.66Mb - 04/27/2021 15:03:56
LOG.old - 1Mb - 04/20/2021 02:45:48
MANIFEST-065723 - 0.12Mb - 04/27/2021 15:03:55
Total 82.2 GB in 355 files
calmh
April 27, 2021, 4:27pm
13
Write dates are all recent so I wouldn’t be surprised if it’s more or less blocked on database compaction or something similar. If you’re also connected to other devices then each scanned file results in announcements, syncs, and returning announcements of the results which all hit the database. There are lots of scaling and tuning aspects to deployments of this size. Probably even the large database tuning isn’t tuned for this database size.
Generally, my gut feeling is that having a single instance with this much data and the database on non-SSD storage is absolutely not going to fly. Syncthing is very sensitive to latency on the database I/O, and that latency is going to be through the roof here, probably.
I’d split things up into more manageable pieces or just throw hardware at it.
1 Like
imsodin
April 27, 2021, 4:31pm
14
What @calmh wrote, plus for a tiny thing:
Then you can pause all old, to be scanned folders except one manually. However if this is happening on every scan, that won’t help. Which again points at what @calmh wrote
Dix
April 27, 2021, 5:33pm
15
Here is a raid array, not 1 disk. If we are talking about iops, can you give the number that is necessary for syncthing to work correctly with this amount of data?
I ran tests with syncthing running, 4 streams:
`
diskspd.exe -c200G -b64K -F4 -r -o128 -W60 -d30 -Sh testfile.dat
`
The results are as follows:
4 streams - iops = 13125.95
1 stream - iops = 10505.86
I don’t think we have some minimum requirement, but I suspect your biggest problem is that the database is on the same spindle as the data.
Ideally you’d have the database on a separate ssd, especially with that volume of data.
And 10kiops is probably fine for accessing the data, but probably kills database because of throughput and latency.
Dix
April 27, 2021, 7:26pm
17
I was promised to give a separate storage for the database (not ssd), but only in case of successful synchronization …
If the contents of the folders do not change, will syncthing finish scanning these folders?
How to calculate the required volume for the database?
Can Syncthing DB be optimized?
Do you have any other tips to improve the situation (besides setting up maxFolderConcurrency and changing the drive)?
imsodin
April 27, 2021, 8:59pm
18
That’s kind of backwards
Estimation based on current size is probably your best bet - depends on quite a few factors (how much do your files differ, how big they are, …).
There’s a lot of knobs, and if you do enough research/experimentation, there might be some real speedup. However your time doing that costs your management more than getting a small SSD for the db, and an SSD has guaranteed benefits, turning the knobs does not.
1 Like
Dix
April 28, 2021, 5:55am
19
Thanks everyone for the help
I don’t think that this has been mentioned yet, but how fragmented are your drives? If the fragmentation is severe, then you may want to try defragmenting them first (which can take forever with this amount of data though…).
You may also want to set blockPullOrder to inOrder (see https://docs.syncthing.net/users/config.html?highlight=inorder ) for all the folders, which is supposedly the most HDD friendly. I have been using this personally on all my HDD-only configurations, although how much of a noticeable difference it makes would require some actual benchmarks and such.
