xkwy521
February 27, 2019, 10:27am
1
I need to sync the files of this windows server to linux!
After I added the task, this task has been in the scan! It has been 2 days!
When scanning, the memory usage is very high, and even all the memory is full!
In the past two days, syncthing has performed an automatic restart at least 3 times!
Is this normal?
The above problem is not well solved, and I have no time to study it!
My new approach is that I compress 40 million files RAR into one file and transfer it to a new server with 16G memory.
This 16G server also has syncthing installed, and an empty folder as a synchronous use folder (folder type: send only), and selected “Watch for Changes”, and then syncthing client with the backup server " Folder type: Receive only "modes for simultaneous connections!
The “progressUpdateIntervalS” and “ScanProgressIntervalS” configurations of both servers are both “-1”
After the above setup is complete, unzip the RAR archive in this new folder!
The 16G server successfully detected the 15315 files for the first time and successfully synchronized to the backup server!
Question one:
After the synchronization is completed, “syncthing” does not continue to perform the task. The 16G server does not detect the new file increase, and always displays 15315 files!
I paused to understand compression and also did not detect the addition of new files!
At this time, 4 million files should have been unpacked!
The above is the first question!
After I pause the decompression, manually manipulate the folder’s “Rescan” button, the number of new files is successfully increased and synchronized to the backup server;
Question 2:
The backup server prompts the following error during this synchronization:
And the “syncthing” service of the backup server is frequently restarted. The longest “uptime” is 15 minutes, and then it is restarted!
You should provide logs as it crashed, restarts.
xkwy521
February 27, 2019, 10:39am
3
2019-02-27 18:33:40 syncthing v1.0.1 "Erbium Earthworm" (go1.11.5 linux-amd64) teamcity@build.syncthing.net 2019-01-18 10:34:18 UTC<br/>
2019-02-27 18:33:40 My ID: 4KWIQTG-***-GKNGCQP<br/>
2019-02-27 18:33:41 Single thread SHA256 performance is 304 MB/s using minio/sha256-simd (206 MB/s using crypto/sha256).<br/>
2019-02-27 18:33:41 Hashing performance is 172.44 MB/s<br/>
2019-02-27 18:33:44 Ready to synchronize "Zhao***e" (Zhao***e) (receiveonly)<br/>
2019-02-27 18:33:44 Overall send rate is unlimited, receive rate is unlimited<br/>
2019-02-27 18:33:44 Using discovery server https://discovery-v4.syncthing.net/v2/?nolookup&id=LYXKCHX-***-NA2MIAW<br/>
2019-02-27 18:33:44 Using discovery server https://discovery-v6.syncthing.net/v2/?nolookup&id=LYXKCHX-***-NA2MIAW<br/>
2019-02-27 18:33:44 Using discovery server https://discovery.syncthing.net/v2/?noannounce&id=LYXKCHX-***-NA2MIAW<br/>
2019-02-27 18:33:44 Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting<br/>
2019-02-27 18:33:44 ...<br/>
2019-02-27 18:33:44 TCP listener ([::]:22000) starting<br/>
2019-02-27 18:33:44 My name is "OL-FR-4-J"<br/>
2019-02-27 18:33:44 Device YL4Y5Q3-***-TIHB2AO is "WIN-0R**BSD"" at [dynamic]<br/>
2019-02-27 18:33:44 GUI and API listening on [::]:8384<br/>
2019-02-27 18:33:44 Access the GUI via the following URL: http://127.0.0.1:8384/<br/>
2019-02-27 18:33:47 Established secure connection to YL4Y5Q3-***-TIHB2AO at 62.*.*.*5:22000-192.*.*.*2:62014/tcp-server (TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384)<br/>
2019-02-27 18:33:47 Device YL4Y5Q3-***-TIHB2AO client is "syncthing v1.0.1" named "WIN-0R**BSD" at 62.*.*.*5:22000-192.*.*.*2:62014/tcp-server<br/>
2019-02-27 18:33:55 Detected 0 NAT services<br/>
2019-02-27 18:34:00 Joined relay relay://37.*.*.*9:22067<br/>
2019-02-27 18:34:59 Connected to already connected device YL4Y5Q3-***-TIHB2AO (existing: 62.*.*.*5:22000-192.*.*.*2:62014/tcp-server new: 62.*.*.*5:33136-70.*.*.*6:22067/relay-client)```
This log does not have a crash you are talking about.
xkwy521
February 27, 2019, 10:58am
5
I can’t find the syncthing Logs file for the linux server!
Please tell me! Thank you
It logs to stdout, it does not write a file on linux.
xkwy521
February 27, 2019, 2:24pm
7
2019-02-27 21:36:12 log 102084 ItemStarted map[type:file action:update folder:Zhao***e item:files/article/txt/1/1554/23284908.txt]e/files/article/txt/1/1735/.syncthing.1319420.txt.tmp: too many open files *e item:files/article/txt/1/1735/1319420.txt error:0xc00c668820]e/files/article/txt/0/740/.syncthing.30210818.txt.tmp: too many open files *e item:files/article/txt/0/740/30210818.txt error:0xc018bb1250 type:file action:update]e/files/article/txt/1/1728/.syncthing.1306346.txt.tmp: too many open files *e item:files/article/txt/1/1728/1306346.txt error:0xc019685780 type:file]e/files/article/txt/0/682/.syncthing.28733095.txt.tmp: too many open files *e item:files/article/txt/0/682/28733095.txt]e/files/article/txt/1/1546/.syncthing.37842660.txt.tmp: too many open files *e item:files/article/txt/1/1546/37842660.txt error:0xc019685ae0 type:file action:update]e/files/article/txt/0/775/.syncthing.26552428.txt.tmp: too many open files *e item:files/article/txt/0/775/26552428.txt error:0xc00c668c30 type:file action:update]e/files/article/txt/1/1843/.syncthing.21715901.txt.tmp: too many open files *e item:files/article/txt/1/1843/21715901.txt]e/files/article/txt/0/582/.syncthing.331170.txt.tmp: too many open files *e item:files/article/txt/0/582/331170.txt error:0xc00c668e50 type:file]e/files/article/txt/1/1550/.syncthing.1143325.txt.tmp: too many open files *e item:files/article/txt/1/1550/1143325.txt error:0xc00c668f20 type:file action:update]e/files/article/txt/1/1502/.syncthing.37833888.txt.tmp: too many open files *e item:files/article/txt/1/1502/37833888.txt]e/files/article/txt/1/1156/.syncthing.813856.txt.tmp: too many open files *e item:files/article/txt/1/1156/813856.txt error:0xc00c6692d0]e/files/article/txt/0/915/.syncthing.616963.txt.tmp: too many open files *e item:files/article/txt/0/915/616963.txt error:0xc0100de510]e/files/article/txt/1/1518/.syncthing.35722824.txt.tmp: too many open files *e]e/files/article/txt/1/1863/.syncthing.1430629.txt.tmp: too many open files *e item:files/article/txt/1/1863/1430629.txt error:0xc013f26ca0 type:file action:update]e/files/article/txt/0/428/.syncthing.216593.txt.tmp: too many open files *e item:files/article/txt/0/428/216593.txt error:0xc013f26e50 type:file action:update]e/files/article/txt/1/1191/.syncthing.833236.txt.tmp: too many open files *e item:files/article/txt/1/1191/833236.txt error:0xc00c6697b0 type:file action:update]e/files/article/txt/1/1086/.syncthing.754860.txt.tmp: too many open files *e item:files/article/txt/1/1086/754860.txt error:0xc013f27000 type:file action:update]e/files/article/txt/0/455/.syncthing.243980.txt.tmp: too many open files *e item:files/article/txt/0/455/243980.txt]e/files/article/txt/1/1499/.syncthing.37931686.txt.tmp: too many open files *e item:files/article/txt/1/1499/37931686.txt error:0xc00c669a60]e/files/article/txt/1/1386/.syncthing.4829943.txt.tmp: too many open files *e item:files/article/txt/1/1386/4829943.txt error:0xc00c669c10]e/files/article/txt/0/726/.syncthing.33891685.txt.tmp: too many open files *e item:files/article/txt/0/726/33891685.txt error:0xc0100df030 type:file action:update]e/files/article/txt/1/1638/.syncthing.1213108.txt.tmp: too many open files *e]e/files/article/txt/1/1728/.syncthing.1308515.txt.tmp: too many open files *e" (Zhao***e), file “files/article/txt/0/764/21229780.txt”): finisher: dst create: open /home/wwwroot/new-e/files/article/txt/0/764/.syncthing.21229780.txt.tmp: too many open files *e item:files/article/txt/1/1728/1308515.txt error:0xc016d62670 type:file action:update]e/files/article/txt/1/1499/.syncthing.37566490.txt.tmp: too many open files *e]e/files/article/txt/0/7/.syncthing.8892.txt.tmp: too many open files *e item:files/article/txt/0/7/8892.txt error:0xc009a826e0 type:file action:update]e/files/article/txt/1/1706/.syncthing.36990206.txt.tmp: too many open files *e item:files/article/txt/1/1706/36990206.txt error:0xc009a82890 type:file]e/files/article/txt/0/778/.syncthing.21644386.txt.tmp: too many open files *e item:files/article/txt/0/778/21644386.txt error:0xc016d63300 type:file action:update]e/files/article/txt/1/1502/.syncthing.22427217.txt.tmp: too many open files *e item:files/article/txt/1/1502/22427217.txt error:0xc009a82ac0 type:file action:update]e/files/article/txt/1/1282/.syncthing.18605360.txt.tmp: too many open files *e]e/files/article/txt/1/1103/.syncthing.18812716.txt.tmp: too many open files *e]e/files/article/txt/1/1852/.syncthing.1425722.txt.tmp: too many open files *e item:files/article/txt/1/1852/1425722.txt error:0xc005dde310 type:file action:update]e/files/article/txt/1/1664/.syncthing.1240145.txt.tmp: too many open files *e item:files/article/txt/1/1664/1240145.txt error:0xc0100df360 type:file action:update]e/files/article/txt/1/1156/.syncthing.812033.txt.tmp: too many open files *e]e/files/article/txt/1/1340/.syncthing.966299.txt.tmp: too many open files *e]
This is only the logging of the “events” level!
Last record of a few seconds before the service is restarted
Unable to send too long records, some were intercepted and deleted
xkwy521
February 27, 2019, 2:27pm
8
2019-02-27 21:54:10 surprise error: open /home/www/.config/syncthing/index-v0.14.0.db/1016351.ldb: too many open filese@0xc000104000 parent not missing files/article/txt/1/1694/1266406.txt-TIHB2AO e/files/article/txt/1/1694/.syncthing.1266406.txt.tmp: too many open files-TIHB2AOe" (Zhaoe), file “files/article/txt/1/1694/1266406.txt”): finisher: dst create: open /home/wwwroot/new e/files/article/txt/1/1694/.syncthing.1266406.txt.tmp: too many open filese files/article/txt/1/1694/1266406.txt e/files/article/txt/1/1620/.syncthing.1199968.txt.tmp: too many open filese" (Zhaoe), file “files/article/txt/1/1620/1199968.txt”): finisher: dst create: open /home/wwwroot/new e/files/article/txt/1/1620/.syncthing.1199968.txt.tmp: too many open filese@0xc000104000 parent not missing files/article/txt/0/984/30874956.txt-TIHB2AO e/files/article/txt/0/984/.syncthing.30874956.txt.tmp: too many open filese" (Zhaoe), file “files/article/txt/0/984/30874956.txt”): finisher: dst create: open /home/wwwroot/new e/files/article/txt/0/984/.syncthing.30874956.txt.tmp: too many open filese@0xc000104000 parent not missing files/article/txt/0/763/37468360.txt-TIHB2AO e/files/article/txt/0/763/.syncthing.37468360.txt.tmp: too many open filese" (Zhaoe), file “files/article/txt/0/763/37468360.txt”): finisher: dst create: open /home/wwwroot/new e/files/article/txt/0/763/.syncthing.37468360.txt.tmp: too many open filese@0xc000104000 parent not missing files/article/txt/0/409/29504351.txt-TIHB2AO e/files/article/txt/0/409/.syncthing.29504351.txt.tmp: too many open filese" (Zhaoe), file “files/article/txt/0/409/29504351.txt”): finisher: dst create: open /home/wwwroot/new e/files/article/txt/0/409/.syncthing.29504351.txt.tmp: too many open filese@0xc000104000 parent not missing files/article/txt/0/805/19633597.txt-TIHB2AO e/files/article/txt/0/805/.syncthing.19633597.txt.tmp: too many open filese" (Zhaoe), file “files/article/txt/0/805/19633597.txt”): finisher: dst create: open /home/wwwroot/new e/files/article/txt/0/805/.syncthing.19633597.txt.tmp: too many open filese@0xc000104000 parent not missing files/article/txt/0/4/29558742.txt-TIHB2AO e/files/article/txt/0/4/.syncthing.29558742.txt.tmp: too many open filese" (Zhaoe), file “files/article/txt/0/4/29558742.txt”): finisher: dst create: open /home/wwwroot/new e/files/article/txt/0/4/.syncthing.29558742.txt.tmp: too many open filese@0xc000104000 parent not missing files/article/txt/0/970/661719.txt-TIHB2AO e/files/article/txt/0/970/.syncthing.661719.txt.tmp: too many open filese" (Zhaoe), file “files/article/txt/0/970/661719.txt”): finisher: dst create: open /home/wwwroot/new e/files/article/txt/0/970/.syncthing.661719.txt.tmp: too many open files-TIHB2AOe item:files/article/txt/0/970/661719.txt error:0xc00b9f2a40 type:file action:update] e/files/article/txt/1/1788/.syncthing.1361644.txt.tmp: too many open filese" (Zhaoe), file “files/article/txt/1/1788/1361644.txt”): finisher: dst create: open /home/wwwroot/new e/files/article/txt/1/1788/.syncthing.1361644.txt.tmp: too many open filese@0xc000104000 parent not missing files/article/txt/0/635/1445330.txt-TIHB2AO e/files/article/txt/0/635/.syncthing.1445330.txt.tmp: too many open filese" (Zhaoe), file “files/article/txt/0/635/1445330.txt”): finisher: dst create: open /home/wwwroot/new e/files/article/txt/0/635/.syncthing.1445330.txt.tmp: too many open filese@0xc000104000 parent not missing files/article/txt/0/778/492688.txt-TIHB2AO e/files/article/txt/0/778/.syncthing.492688.txt.tmp: too many open filese" (Zhaoe), file “files/article/txt/0/778/492688.txt”): finisher: dst create: open /home/wwwroot/new e/files/article/txt/0/778/.syncthing.492688.txt.tmp: too many open filese files/article/txt/0/778/492688.txte] e/files/article/txt/1/1499/.syncthing.32676916.txt.tmp: too many open filese" (Zhaoe), file “files/article/txt/1/1499/32676916.txt”): finisher: dst create: open /home/wwwroot/new e/files/article/txt/1/1499/.syncthing.32676916.txt.tmp: too many open filese@0xc000104000 parent not missing files/article/txt/1/1703/1276133.txt-TIHB2AO e/files/article/txt/1/1703/.syncthing.1276133.txt.tmp: too many open filese" (Zhaoe), file “files/article/txt/1/1703/1276133.txt”): finisher: dst create: open /home/wwwroot/new e/files/article/txt/1/1703/.syncthing.1276133.txt.tmp: too many open filese@0xc000104000 parent not missing files/article/txt/0/435/225621.txt-TIHB2AO e/files/article/txt/0/435/.syncthing.225621.txt.tmp: too many open filese" (Zhaoe), file “files/article/txt/0/435/225621.txt”): finisher: dst create: open /home/wwwroot/new e/files/article/txt/0/435/.syncthing.225621.txt.tmp: too many open filese@0xc000104000 parent not missing files/article/txt/0/891/18585436.txt-TIHB2AO e/files/article/txt/0/891/.syncthing.18585436.txt.tmp: too many open filese" (Zhaoe), file “files/article/txt/0/891/18585436.txt”): finisher: dst create: open /home/wwwroot/new e/files/article/txt/0/891/.syncthing.18585436.txt.tmp: too many open filese@0xc000104000 parent not missing files/article/txt/0/83/3828365.txt-TIHB2AO e/files/article/txt/0/83/.syncthing.3828365.txt.tmp: too many open files
This is a record of all log types!
The same, record a few seconds before the service restarts!
xkwy521
February 27, 2019, 2:31pm
9
I can guarantee that my server’s hard drive is healthy, I has been checked using “smartctl”, there is no error record, and the hard disk life is only 7000 hours!
It is dual hard drive hardware Raid 1
@AudriusButkevicius
xkwy521
February 27, 2019, 3:07pm
10
Synching has a lot of running processes, is this normal?
You are probably seeing threads not processes, as they all have exact same memory.
When you start seeing errors can you as root run ‘ls -lah /proc/[pid of syncthing]/fd’.
I suspect you need to raise the open file limit for the user running syncthing as guess ulimit -a will show that you are limited to 1024 files, which is quite low.
Also, quote your logs in triple backtick ""` to have sensible formatting.
system
March 29, 2019, 5:47pm
12
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.
