Large Files not Synching

We have recently setup Syncthing. We have 2 DELL servers (24 core/64GB Ram/640GB disk/1GB eth/plenty of disk space/Ubuntu 18.04/Syncthing 1.34 default install) in two data centers about 50mi apart. Synchronization is mostly successful.

The problem seems to be with very large files. It seems to be completely stuck when attempting to synchronize anything over about 300MiB. Of the 4 files it is stuck on, 2 are in the 400MiB range, and the other two are in the 5GiB range.

These are Virtual Machine images. Knowing they couldn’t be synchronized while running, we shut them down for a while (2 days). All other files in the folder synchronize, but those files are all much smaller. Even after 2 days of the files not changing, they still won’t sync.

We did notice a symptom. When it hits those files, it begins to rapidly go through a disconnect, reconnect cycle, which seems to exacerbate the problem. I really think it is related, but I do not know how to prove it.

If I remove this folder, then add a folder containing 90K files with a combined size of over 5GiB, it synchronizes without issue, but there are no files in the bunch much larger than a few MiB. Most are a few KiB.

We had 4 others that wouldn’t transfer, but they were very old ZIP backups we simply decided we could delete. We don’t have the same option on these, and we want to add 4 more approximately the same size.

I’m sure it can do it, but not sure what to do to make it succeed.

I did find a post suggesting we increase the scan time to a longer time. Right now it is set at the default of 1h. Any suggestions on what length of time would be required?

Also, I see some debugging options, but little idea on which ones to turn on to get meaningful output. I’ll gladly try anything with proper direction.

You say this, and I assume you mean 1.3.4, but the symptoms sound like an older version.

Logs, please.

That means 0.14.43 -> no need for logs,Provide logs for certainty, or just upgrade. That’s too old (and has a known limitation with new versions that results in exactly the behaviour you describe).

Should we add a check to hello messages to drop connection with a useful error message for <0.14.46 (not entirely sure on the message). Given old ubuntuses (and debians, but less so) will be around “forever”, it might still be worthwhile…

Well, there is a claimed issue with a Syncthing release that doesn’t exist. Logs will show conclusively what goes on and whether “1.34” means “0.14.43” (the keys are, like, right next to each other?) or “1.3.4”.

1 Like

Yes. It says 1.3.4 Linux 64 bit (I missed a dot). I don’t see the other number (0.14.43) anywhere in the about screen, though I’m sure I don’t know where to look for it.

I saw posts on this about having an older version and some dissenting remarks about versions being out there forever. Yet, we followed the exact instructions posted on the Syncthing web site for the stable latest version. So, if we are to be admonished for having an outdated version, admonishments belong to the mirror. Please hold off of the dissenting remarks for new users when those users followed the exact instructions you post for “stable” and “latest” versions. Update your web site to avoid such mistakes from new users.

We’ll be happy to zap it and upgrade to the newest solution once we are pointed to the latest stable version instructions. That is, we generally don’t have the time to fight with candidates or test versions.

Logs, please, so we can see what’s going on?

1 Like

3 posts were split to a new topic: Files stuck downloading (1.4.0; macOS)

Seems like we just need to upgrade to the latest version.

Is it as simple as downloading the Linux 1.4 tar and extracting somewhere?

Is 1.4 the latest version?

There’s no difference in syncing large files between 1.3.4 and 1.4.0 that I’m aware of. But, you can upgrade the same way you installed. Since you’re on Ubuntu, https://apt.syncthing.net/ should be the best way.

I’m not participating here further without logs posted.

1 Like

So we managed to upgrade to 1.4, but interestingly, one server upgraded to 1.4.0, the other 1.4.43. Here are the logs:

2020-03-23 12:56:06 My ID: ******************************************************************************
2020-03-23 12:56:07 Single thread SHA256 performance is 203 MB/s using minio/sha256-simd (160 MB/s using crypto/sha256).
2020-03-23 12:56:07 Hashing performance is 175.65 MB/s
2020-03-23 12:56:07 Migrating database to schema version 9...
2020-03-23 12:56:09 Compacting database after migration...
2020-03-23 12:56:10 Detected upgrade from v1.3.4 to v1.4.0
2020-03-23 12:56:10 Overall send rate is unlimited, receive rate is unlimited
2020-03-23 12:56:10 Using discovery server https://discovery.syncthing.net/v2/?noannounce&id=******************************************************************************
2020-03-23 12:56:10 Using discovery server https://discovery-v4.syncthing.net/v2/?nolookup&id=******************************************************************************
2020-03-23 12:56:10 Using discovery server https://discovery-v6.syncthing.net/v2/?nolookup&id=******************************************************************************
2020-03-23 12:56:10 ...
2020-03-23 12:56:10 Ready to synchronize "Default Folder" (default) (sendreceive)
2020-03-23 12:56:10 Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
2020-03-23 12:56:10 TCP listener ([::]:22000) starting
2020-03-23 12:56:10 QUIC listener ([::]:22000) starting
2020-03-23 12:56:10 Ready to synchronize "Mantis MySQL" (mzazw-uy9ef) (sendreceive)
2020-03-23 12:56:10 Ready to synchronize "Development" (sqz2f-9hkcp) (sendreceive)
2020-03-23 12:56:10 Ready to synchronize "Datapower Appliances" (yyhvg-zk5c5) (sendreceive)
2020-03-23 12:56:10 Completed initial scan of sendreceive folder "Default Folder" (default)
2020-03-23 12:56:10 Completed initial scan of sendreceive folder "Mantis MySQL" (mzazw-uy9ef)
2020-03-23 12:56:12 GUI and API listening on ###.###.###.###:8384
2020-03-23 12:56:12 Access the GUI via the following URL: https://###.###.###.###:8384/
2020-03-23 12:56:12 My name is "DellR710-1"
2020-03-23 12:56:12 Device****************************************************************************** is "Sharley's Server" at [dynamic]
2020-03-23 12:56:12 Established secure connection to****************************************************************************** at ###.###.###.###:44544-###.###.###.###:9320/tcp-client/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
2020-03-23 12:56:12 Device****************************************************************************** client is "syncthing v0.14.43-ds1" named "Dell-R710" at ###.###.###.###:44544-###.###.###.###:9320/tcp-client/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
2020-03-23 12:56:12 Device****************************************************************************** folder "Default Folder" (default) has a new index ID (0x98BED7AFBA364BDE)
2020-03-23 12:56:12 Device****************************************************************************** folder "Default Folder" (default) has mismatching index ID for us (0xEDF7CBA4BDC18879 != 0x21D5F159C6011DDE)
2020-03-23 12:56:12 Ignoring folder "Mantis MYSQL Files" (msc4o-qj6qt) from device****************************************************************************** since we are configured to
2020-03-23 12:56:12 Device****************************************************************************** folder "Mantis MySQL" (mzazw-uy9ef) has a new index ID (0x37684E4EC31F52B7)
2020-03-23 12:56:12 Device****************************************************************************** folder "Mantis MySQL" (mzazw-uy9ef) has mismatching index ID for us (0x702B8D638F1F4D34 != 0x7BAA437B080B93E3)
2020-03-23 12:56:12 Device****************************************************************************** folder "Development" (sqz2f-9hkcp) has a new index ID (0xB361473530671DE5)
2020-03-23 12:56:18 Device****************************************************************************** folder "Development" (sqz2f-9hkcp) has mismatching index ID for us (0x8A8FC9BF5F6D9213 != 0x946CEB00DF80CE19)
2020-03-23 12:56:18 Ignoring folder "Mantis MYSQL" (stqee-xnsef) from device****************************************************************************** since we are configured to
2020-03-23 12:56:18 Device****************************************************************************** folder "Datapower Appliances" (yyhvg-zk5c5) has a new index ID (0x767F69C2492263C3)
2020-03-23 12:56:18 Device****************************************************************************** folder "Datapower Appliances" (yyhvg-zk5c5) has mismatching index ID for us (0x66C885F705453BE4 != 0x9032ACBE067D243A)
2020-03-23 12:56:25 Completed initial scan of sendreceive folder "Development" (sqz2f-9hkcp)
2020-03-23 12:56:25 Detected 1 NAT service
2020-03-23 12:56:30 quic://0.0.0.0:22000 detected NAT type: Port restricted NAT
2020-03-23 12:56:30 quic://0.0.0.0:22000 resolved external address quic://70.134.33.199:22000 (via stun.syncthing.net:3478)
2020-03-23 12:56:47 Connection to****************************************************************************** at ###.###.###.###:44544-###.###.###.###:9320/tcp-client/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 closed: reading length: read tcp ###.###.###.###:44544->###.###.###.###:9320: read: connection reset by peer
2020-03-23 12:56:49 Joined relay relay://168.235.72.156:22067
2020-03-23 12:57:01 Completed initial scan of sendreceive folder "Datapower Appliances" (yyhvg-zk5c5)
2020-03-23 12:57:13 Established secure connection to****************************************************************************** at ###.###.###.###:33630-168.235.72.156:22067/relay-server/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
2020-03-23 12:57:13 Device****************************************************************************** client is "syncthing v0.14.43-ds1" named "Dell-R710" at ###.###.###.###:33630-168.235.72.156:22067/relay-server/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
2020-03-23 12:57:13 Ignoring folder "Mantis MYSQL Files" (msc4o-qj6qt) from device****************************************************************************** since we are configured to
2020-03-23 12:57:13 Ignoring folder "Mantis MYSQL" (stqee-xnsef) from device****************************************************************************** since we are configured to
2020-03-23 12:57:15 Established secure connection to****************************************************************************** at ###.###.###.###:45198-###.###.###.###:9320/tcp-client/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
2020-03-23 12:57:15 Replacing old connection ###.###.###.###:33630-168.235.72.156:22067/relay-server/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 with ###.###.###.###:45198-###.###.###.###:9320/tcp-client/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 for******************************************************************************
2020-03-23 12:57:15 Connection to****************************************************************************** at ###.###.###.###:33630-168.235.72.156:22067/relay-server/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 closed: reading length: read tcp ###.###.###.###:33630->168.235.72.156:22067: use of closed network connection
2020-03-23 12:57:15 Device****************************************************************************** client is "syncthing v0.14.43-ds1" named "Dell-R710" at ###.###.###.###:45198-###.###.###.###:9320/tcp-client/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
2020-03-23 12:57:15 Ignoring folder "Mantis MYSQL Files" (msc4o-qj6qt) from device****************************************************************************** since we are configured to
2020-03-23 12:57:15 Ignoring folder "Mantis MYSQL" (stqee-xnsef) from device****************************************************************************** since we are configured to
2020-03-23 12:57:16 Connection to****************************************************************************** at ###.###.###.###:45198-###.###.###.###:9320/tcp-client/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 closed: reading length: read tcp ###.###.###.###:45198->###.###.###.###:9320: read: connection reset by peer
2020-03-23 12:57:31 Established secure connection to****************************************************************************** at ###.###.###.###:33640-168.235.72.156:22067/relay-server/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
2020-03-23 12:57:31 Device****************************************************************************** client is "syncthing v0.14.43-ds1" named "Dell-R710" at ###.###.###.###:33640-168.235.72.156:22067/relay-server/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
2020-03-23 12:57:31 Ignoring folder "Mantis MYSQL Files" (msc4o-qj6qt) from device****************************************************************************** since we are configured to
2020-03-23 12:57:31 Ignoring folder "Mantis MYSQL" (stqee-xnsef) from device****************************************************************************** since we are configured to
2020-03-23 12:57:42 Connection to****************************************************************************** at ###.###.###.###:33640-168.235.72.156:22067/relay-server/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 closed: reading length: read tcp ###.###.###.###:33640->168.235.72.156:22067: read: connection reset by peer
2020-03-23 12:57:48 Established secure connection to****************************************************************************** at ###.###.###.###:33646-168.235.72.156:22067/relay-server/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
2020-03-23 12:57:48 Device****************************************************************************** client is "syncthing v0.14.43-ds1" named "Dell-R710" at ###.###.###.###:33646-168.235.72.156:22067/relay-server/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
2020-03-23 12:57:48 Ignoring folder "Mantis MYSQL Files" (msc4o-qj6qt) from device****************************************************************************** since we are configured to
2020-03-23 12:57:48 Ignoring folder "Mantis MYSQL" (stqee-xnsef) from device****************************************************************************** since we are configured toèsý$†VÊ?͗!7

You should post a screenshot from both sides, as it’s not syncing.

0.14.43, not 1… you need to upgrade that client

1 Like

Yeah… it looks like both of then say that. How do I upgrade just the client?

I did find something that said to force the upgrade of syncthing via:

 syncthing -upgrade

But on mine, I get a warning:

 WARNING:  Upgrade:  upgrade unsupported

Otherwise, I’m not finding a way.

Do it the same way you did on the other one, or start actually reading my posts which include relevant links.

1 Like

I haven’t managed to get too far with either. We updated once today and got to 1.4.0 on one and 1.4.43 on the other… why I’m getting two different versions is strange to me. But, as Simon noted, the client version is lagging.

I would have suspected when we upgraded and got the 1.4.x versions, the client versions would have upgraded as well, but that doesn’t seem to be the case, as both machines are reading a client version of v0.14.43-ds1.

I was hoping I could force the upgrade of just the client.

P.S. Mr. Borg. You are likely an excellent developer. Considering all the technologies and libraries your team uses, its pretty impressive. However, you should not be in customer support. Your demeanor and derision is uncalled for.

If you had actually been reading my posts, I clearly noted we did the upgrade on both machines and ended up with two different versions. How that happens, I have no idea.

What makes you think I’m not reading your posts? On the other hand, by the way you address others as if we’re stupid, why would anyone want to read your posts?

I don’t have the energy.

2 Likes

Given OP can’t see it.

If that doesn’t help, I’m sorry, but you are on your own for being causing controversy.

Perhaps searching for -ds1 in the forums will lead you on the right track.

1 Like

What am I not seeing here:

josephmmorgan@DellR710-1:~$ cat /etc/apt/sources.list.d/syncthing.list deb https://apt.syncthing.net/ syncthing stable

josephmmorgan@DellR710-1:~$ sudo apt-get update … Hit:6 https://apt.syncthing.net syncthing InRelease
Get:7 https://apt.syncthing.net syncthing/stable amd64 Packages [1,798 B] Get:8 https://apt.syncthing.net syncthing/stable i386 Packages [1,801 B] …

josephmmorgan@DellR710-1:~$ sudo apt-get install syncthing Reading package lists… Done Building dependency tree
Reading state information… Done syncthing is already the newest version (1.4.0). 0 upgraded, 0 newly installed, 0 to remove and 18 not upgraded.

This, is what I should expect as I did this earlier today. On the other server, the version reads 1.4.43, but otherwise the exact same information.

I found only one apparently meaningful post regarding “-ds1”, which simply says to upgrade using the https://apt.syncthing.net/ link. I’m now lost on what do do.

P.S. I’d also love to know how I am causing controversy. I’ve not been anything but courteous and provided all the insight I can provide. I have followed every direction given (except one of providing logs from both sides since it became clear my client version was out of date).

Sorry buddy, but you’re on your own here.

We do this for free, and last thing we want is some internet knight teaching us how to do customer service. You can always ask for your money back if you are unhappy with customer service.

And you never answered my question. And you wonder why you need to be taught customer service. NVM. I’ll find another tool.