Syncthing stops accepting connections on ports 8384 and 2200

A couple of days ago all of our Windows Server machines(five of them in total) started having problems. They stop accepting connections on ports 22000 and 8384. Some times 8384 is open but unresponsive.

The affected servers are running Windows Server 2012 R2 and 2012 Essentials. All Windows servers that run Syncthing have this problem.

Our Linux servers are unaffected.

Restarting Syncthing fixes it, for a while. Then few hours later they stop responding.

They were running 1.11.1 when this began. I then updated them all to 1.12.0. no change.

They’ve been running for years without problems.

Explicitly setting Windows Firewall to permit ports 8484 and 22000 does nothing.

Anybody else have this problem? I was thinking a recent Windows update broke something. Nothing has changed on the servers otherwise.

Here is a piece of the log showing A startup that soon failed:

[start] 11:16:13 INFO: syncthing v1.12.0 "Fermium Flea" (go1.15.5 windows-amd64) teamcity@build.syncthing.net 2020-11-27 13:33:58 UTC
[LINRD] 11:16:14 INFO: My ID: XXXXXXX-YYYYYYYY-QQ4HUM5-E55QQAL-TEET6LP-YVMFBLH-JLVDSUJ-RBMNSQE
[LINRD] 11:16:15 INFO: Single thread SHA256 performance is 212 MB/s using minio/sha256-simd (150 MB/s using crypto/sha256).
[LINRD] 11:16:15 INFO: Hashing performance is 166.55 MB/s
[LINRD] 11:16:15 INFO: Detected upgrade from v1.11.1 to v1.12.0
[LINRD] 11:16:15 INFO: Overall send rate limit is 500 KiB/s, receive rate limit is 500 KiB/s
[LINRD] 11:16:15 INFO: Rate limits do not apply to LAN connections
[LINRD] 11:16:15 INFO: Using discovery mechanism: IPv4 local broadcast discovery on port 21027
[LINRD] 11:16:15 INFO: Using discovery mechanism: IPv6 local multicast discovery on address [ff12::8384]:21027
[LINRD] 11:16:15 INFO: Ready to synchronize db-finance-amsql (sendonly)
[LINRD] 11:16:15 INFO: Ready to synchronize "Sage-Live" (lifoj-t9ntj) (sendonly)
[LINRD] 11:16:15 INFO: TCP listener ([::]:22000) starting
[LINRD] 11:16:15 INFO: QUIC listener ([::]:22000) starting
[LINRD] 11:16:15 INFO: GUI and API listening on [::]:8384
[LINRD] 11:16:15 INFO: Access the GUI via the following URL: https://127.0.0.1:8384/
[LINRD] 11:16:15 INFO: My name is "DB-FINANCE"
[LINRD] 11:16:15 INFO: Device XXXXXXX-YYYYYYYY-YXBHKWQ-SKKANXP-EQ4OS7R-LW2D27J-DZETLGS-ET7G4QN is "backup-ns" at [tcp://backup-ns:22000]
[LINRD] 11:16:15 INFO: Device XXXXXXX-YYYYYYYY-2TABPLD-5T7YXWL-KTSXG2X-KNORWJY-RLVX7QV-YXQLCQY is "backup-ws" at [tcp://backup-ws:22000]
[LINRD] 11:16:15 INFO: Device XXXXXXX-YYYYYYYY-5PVDKQL-DQGWGGK-JLCDGBD-KFEORUI-HJCHRT6-HB732QT is "backup-co" at [tcp://10.4.1.29:22000]
[LINRD] 11:16:15 INFO: Established secure connection to GAUBTQT-3HJVQHQ-5PVDKQL-DQGWGGK-JLCDGBD-KFEORUI-HJCHRT6-HB732QT at 10.4.1.25:22000-10.4.1.29:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256
[LINRD] 11:16:15 INFO: Device XXXXXXX-YYYYYYYY-5PVDKQL-DQGWGGK-JLCDGBD-KFEORUI-HJCHRT6-HB732QT client is "syncthing v1.11.1" named "backup-co" at 10.4.1.25:22000-10.4.1.29:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256
[LINRD] 11:16:15 INFO: Established secure connection to XXXXXXX-YYYYYYYY-YXBHKWQ-SKKANXP-EQ4OS7R-LW2D27J-DZETLGS-ET7G4QN at 10.4.1.25:22000-10.5.1.29:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256
[LINRD] 11:16:15 INFO: Device XXXXXXX-YYYYYYYY-YXBHKWQ-SKKANXP-EQ4OS7R-LW2D27J-DZETLGS-ET7G4QN client is "syncthing v1.11.1" named "backup-ns" at 10.4.1.25:22000-10.5.1.29:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256
[LINRD] 11:16:15 INFO: Established secure connection to XXXXXXX-YYYYYYYY-2TABPLD-5T7YXWL-KTSXG2X-KNORWJY-RLVX7QV-YXQLCQY at 10.4.1.25:22000-10.3.1.29:22000/tcp-client/TLS1.3-TLS_CHACHA20_POLY1305_SHA256
[LINRD] 11:16:15 INFO: Device XXXXXXX-YYYYYYYY-2TABPLD-5T7YXWL-KTSXG2X-KNORWJY-RLVX7QV-YXQLCQY client is "syncthing v1.11.1" named "backup-ws" at 10.4.1.25:22000-10.3.1.29:22000/tcp-client/TLS1.3-TLS_CHACHA20_POLY1305_SHA256
[LINRD] 11:16:15 INFO: Completed initial scan of sendonly folder db-finance-amsql
[LINRD] 11:16:17 INFO: Connected to already connected device XXXXXXX-YYYYYYYY-5PVDKQL-DQGWGGK-JLCDGBD-KFEORUI-HJCHRT6-HB732QT (existing: 10.4.1.25:22000-10.4.1.29:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256 new: 10.4.1.25:22000-10.4.1.29:37604/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256)
[LINRD] 11:17:16 INFO: Completed initial scan of sendonly folder "Sage-Live" (lifoj-t9ntj)
[LINRD] 11:20:08 INFO: Listen (BEP/tcp): TLS handshake: EOF
[LINRD] 11:25:08 INFO: Listen (BEP/tcp): TLS handshake: EOF
[LINRD] 11:30:08 INFO: Listen (BEP/tcp): TLS handshake: EOF
[LINRD] 11:35:08 INFO: Listen (BEP/tcp): TLS handshake: EOF
[LINRD] 11:40:08 INFO: Listen (BEP/tcp): TLS handshake: EOF
[LINRD] 11:45:08 INFO: Listen (BEP/tcp): TLS handshake: EOF
[LINRD] 11:46:18 INFO: Sent usage report (version 3)
[LINRD] 11:50:08 INFO: Listen (BEP/tcp): TLS handshake: EOF

The TLS handshake error is my Nagios monitoring of the TCP ports. Nothing is logged to indicate a problem.

How are you noticing there’s problems with connections? The following line says that Syncthing connects just fine:

[LINRD] 11:16:15 INFO: Established secure connection to XXXXXXX-YYYYYYYY-2TABPLD-5T7YXWL-KTSXG2X-KNORWJY-RLVX7QV-YXQLCQY at 10.4.1.25:22000-10.3.1.29:22000/tcp-client/TLS1.3-TLS_CHACHA20_POLY1305_SHA256

The TCP ports stop responding: they are tested every 5 minutes. And syncing to our working Linux Syncthing servers is interrupted. Those Linux servers log things like this:

[PQYW3] 21:05:36 INFO: Puller (folder smshost-stagedsql, item "model.dump"): syncing: no connected device has the required version of this file
[PQYW3] 21:05:36 INFO: Puller (folder smshost-stagedsql, item "STORESQL.dump"): syncing: no connected device has the required version of this file
[PQYW3] 21:05:36 INFO: Puller (folder smshost-stagedsql, item "it_test.dump"): syncing: no connected device has the required version of this file
[PQYW3] 21:05:36 INFO: Puller (folder smshost-stagedsql, item "msdb.dump"): syncing: no connected device has the required version of this file
[PQYW3] 21:05:36 INFO: Puller (folder smshost-stagedsql, item "master.dump"): syncing: no connected device has the required version of this file
[PQYW3] 21:05:36 INFO: smshost-stagedsql: Failed to sync 5 items
[PQYW3] 21:05:36 INFO: Folder smshost-stagedsql isn't making sync progress - retrying in 1h4m0s.

The remote server that is not responsive above is one of the Windows servers.

Usual thing for mysterious error on Windows: have you tried disabling antivirus?

1 Like

I’ll take a look. Our AV software, TrendMicro Apex One, has not logged anything for these servers so I would be surprised if it were the cause. But I’ve been wrong before. :slight_smile: I’ll disable AV on a couple of the servers. I should know by today if things have improved.

Jakob, that was it. I’ve never had out AV software silently mess with applications before. I should have thought of that. Thanks. And thanks to you and all the other developers for the great software.

4 Likes

The solution was to add c:\Program Files\Syncthing\syncthing.exe to the “Trusted Program List” in TrendMicro Apex One.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.