Connected, Bytes Not Moving

I have two Syncthing nodes which are connected, with files needing sync, but bytes aren’t moving. Pointers on how to get things moving would be appreciated.

  • Node 1 = Windows 10, running Syncthing 1.9.0
  • Node 2 = Ubuntu 18.04.3, running Syncthing 1.6.1

Node 1 says “Established secure connection to (node 2)”. Node 2 says

2020-09-18 17:22:52 Established secure connection to YY4ECWC-SEM7A2I-5RZ26GT-ZQTKARQ-TJ6WLGK-E4ZVWDT-RXJXVE3-UUWMNAA at 192.168.8.10:22000-192.168.8.203:49561/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256
2020-09-18 17:22:52 Device YY4ECWC-SEM7A2I-5RZ26GT-ZQTKARQ-TJ6WLGK-E4ZVWDT-RXJXVE3-UUWMNAA client is "syncthing v1.9.0" named "GACDTL01KK4056" at 192.168.8.10:22000-192.168.8.203:49561/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256

Node 1 says there are 15 items to sync. Node 2 says there are 344 items to sync. Both say the upload and download rate is 0 B/s.

I’d prefer not to upgrade ST on the Ubuntu node unless we know it would fix the issue. I installed it with apt, and I’d have to convert it to a manual install or mess with my source repository settings.

If I watch it for about 5 minutes, it will briefly switch to an upload and/or a download rate of 6 bps. With 4 gig to transfer, that’s going to take an ‘undesirable’ amount of time. :wink:

You can enable model debugging to see what it’s doing. It’s likely still exchanging the indexes.

Is this running on some silly underpowered devices?

And yes, you should update, I don’t think we’d be going deep into understanding of why this is happening just because of how much of an outdated version you are using.

Thank you.

I do sync some underpowered devices, but these aren’t them. These devices are on the same LAN and ping with 10K frames at 2 ms, so network shouldn’t be the problem.

I suspect the Windows box of being the problem because the Linux box successfully syncs with 5 different machines with no problem. The Windows box also took a pushed software update recently, where the Linux box has been static.

Here’s 5 minutes of model debugging on the Windows box. If you don’t see anything, I might try rebuilding the index on the Windows box.

[YY4EC] 2020/09/18 18:24:05.279304 api.go:649: INFO: Enabled debug data for "model"
[YY4EC] 2020/09/18 18:25:35.094322 progressemitter.go:296: DEBUG: progress emitter: bytes completed for mylaptop-c-users-kevink: 0
[YY4EC] 2020/09/18 18:25:35.095250 model.go:829: DEBUG: model@0xc00020dc80 Completion(BJHPEG7, "mylaptop-c-users-kevink"): map[completion:96.05683694037768 globalBytes:41499344974 globalItems:88134 needBytes:1636386841 needDeletes:0 needItems:13]
[YY4EC] 2020/09/18 18:25:35.096279 progressemitter.go:296: DEBUG: progress emitter: bytes completed for mylaptop-flash-drive: 0
[YY4EC] 2020/09/18 18:25:35.097924 model.go:829: DEBUG: model@0xc00020dc80 Completion(BJHPEG7, "mylaptop-flash-drive"): map[completion:100 globalBytes:1298244181 globalItems:6000 needBytes:0 needDeletes:0 needItems:0]
[YY4EC] 2020/09/18 18:25:35.099247 progressemitter.go:296: DEBUG: progress emitter: bytes completed for mylaptop-network-folder: 0
[YY4EC] 2020/09/18 18:25:35.100253 model.go:829: DEBUG: model@0xc00020dc80 Completion(BJHPEG7, "mylaptop-network-folder"): map[completion:99.80853915913379 globalBytes:944745668 globalItems:11274 needBytes:1808818 needDeletes:2 needItems:14]
[YY4EC] 2020/09/18 18:25:35.101250 progressemitter.go:296: DEBUG: progress emitter: bytes completed for keepass-kevin: 0
[YY4EC] 2020/09/18 18:25:35.103250 model.go:829: DEBUG: model@0xc00020dc80 Completion(6P5C52J, "keepass-kevin"): map[completion:100 globalBytes:344210 globalItems:4 needBytes:0 needDeletes:0 needItems:0]
[YY4EC] 2020/09/18 18:29:08.831236 folder.go:743: INFO: Completed initial scan of sendonly folder "mylaptop-c-users-kevink" (mylaptop-c-users-kevink)
[YY4EC] 2020/09/18 18:29:08.831236 folder.go:268: DEBUG: sendonly/mylaptop-c-users-kevink@0xc000282a80 next rescan in 26h51m25.024766392s
[YY4EC] 2020/09/18 18:29:08.831236 folder.go:195: DEBUG: sendonly/mylaptop-c-users-kevink@0xc000282a80 Scan due to watcher
[YY4EC] 2020/09/18 18:29:08.861314 model.go:1980: DEBUG: indexSender@0xc003014000 for mylaptop-c-users-kevink to  at 192.168.8.203:63969-192.168.8.10:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256: Sending 10 files (<1662 bytes)
[YY4EC] 2020/09/18 18:29:08.867195 model.go:937: DEBUG: Index update (in): BJHPEG7 / "mylaptop-c-users-kevink": 10 files
[YY4EC] 2020/09/18 18:29:08.871227 folder.go:195: DEBUG: sendonly/mylaptop-c-users-kevink@0xc000282a80 Scan due to watcher
[YY4EC] 2020/09/18 18:29:09.948708 progressemitter.go:296: DEBUG: progress emitter: bytes completed for mylaptop-c-users-kevink: 0
[YY4EC] 2020/09/18 18:29:09.948708 model.go:829: DEBUG: model@0xc00020dc80 Completion(BJHPEG7, "mylaptop-c-users-kevink"): map[completion:96.05683694037768 globalBytes:41499344974 globalItems:88134 needBytes:1636386841 needDeletes:0 needItems:13]
[YY4EC] 2020/09/18 18:29:50.905760 folder.go:195: DEBUG: sendonly/mylaptop-c-users-kevink@0xc000282a80 Scan due to watcher
[YY4EC] 2020/09/18 18:29:50.926968 model.go:1980: DEBUG: indexSender@0xc003014000 for mylaptop-c-users-kevink to  at 192.168.8.203:63969-192.168.8.10:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256: Sending 2 files (<246 bytes)
[YY4EC] 2020/09/18 18:29:50.970969 model.go:937: DEBUG: Index update (in): BJHPEG7 / "mylaptop-c-users-kevink": 2 files
[YY4EC] 2020/09/18 18:29:52.085137 progressemitter.go:296: DEBUG: progress emitter: bytes completed for mylaptop-c-users-kevink: 0
[YY4EC] 2020/09/18 18:29:52.087774 model.go:829: DEBUG: model@0xc00020dc80 Completion(BJHPEG7, "mylaptop-c-users-kevink"): map[completion:96.05683696470216 globalBytes:41499345230 globalItems:88136 needBytes:1636386841 needDeletes:0 needItems:13]

Oh yeah! I decided I didn’t have anything to lose by rebuilding the index, so I did. Now data is moving at 2 Mb/sec, which is much better than 6 B/s.

In fact, it finished transferring everything while I typed that last paragraph.

I think you should simply upgrade, there have been tons of bugs fixed with missing index entries since 1.6.0.

Is there a step-by-step on migrating an apt-installed Syncthing to a manual install?

That’s the sticking point for me to upgrade. It isn’t a philosophical objection to an upgrade. If I saw a simple upgrade option, I’d take it.

My apt-installed Syncthing is capped at its current version because that’s the version that goes with Ubuntu 18, and that’s what I’m running on the server. (And this server has enough funky hardware that upgrading to Ubuntu 20 is a month-long effort.)

Oops. Wrong! My Syncthing is a downloaded Docker container. I’ll have to investigate rolling my own container.

Well, that was easier than expected. Edit the Dockerfile. Replace 1.6.1 with 1.9. Stop the container. Rebuild the container. Start the container.

Ouch! I found out why sync stopped. The database was corrupted. Then I rebuilt the index and sync resumed. Then it stopped again. The database was corrupt again.

Then I found out why my DB kept getting corrupted – the underlying disk has a problem. Call it a hardware problem.

2 Likes

Yeah, leveldb is pretty good at finding bad hardware.

3 Likes