Using Syncthing with about 20TB and more

Ah sorry, my bad, mixed something up - that’s indeed a failed item error message. Unless you have a disconnected device that error should be transitional, i.e. go away by itself.
Are any of these failures related to the file that doesn’t sync?

Sorry for late response – On Saturday last week everything was in sync again, but I didn’t do anything like searching for the conflicts or out-of-sync items. On Monday morning the sync was ‘stopped’, and I just restart the service on one side, and it is working until right now.

So I am searching for a workaround or use a simple script to see if there’s a problem or not. Problem is that with the amount of data it is hard to see if there’s a problem with a file or not, since it could be OK some hours later.

I still don’t understand what stopped means.

Neither do I. You really need to figure out what the problem is instead of just restarting. Check/save logs, try scanning and/or pausing/restarting the folder (and saving the logs) and describe what happens (aka take screenshots when something unexpected happens).

That’s “good”, in the sense that I now have an idea of the general problem. Before I saw out-of-sync and failed items, which in a way is less problematic, as Syncthing is telling you what’s up.

So the screenshot shows the “receiving side” ® and when you look at actual data, it is missing a new file or change that is in place on the “sending side” (S), right? If in this situation you hit rescan on the “sending side”, does the file get synced? Also from screenshots I assume all clients are at least on v1.0.1 (if there were any older ones, there’s a bug that could cause this behaviour).

Yes, all clients are on 1.0.1, unfortunately one server running on Ubuntu 16.04.2 and the other one on 18.04.2. Both ‘sides’ of this folder are in Send & Receive mode.

I’ll enable some other logging options and have a look in the next days what is going on. Great would be to see exactly a timeframe when it stops or waits for something.

You need to show screenshots from the other side, as the other side is refusing or failing to download files.

It is happening! This a screenshot of the other side, currently 42 items out of sync, I’ve started a rescan now, but nothing else so far:

So there are 42 items out of sync shown, but the current situation is I don’t know maybe since yesterday (I don’t know a timeframe unfortunately) it is ‘offline’ – there would be lots of items needed to be synced when I would restart the service (what I didn’t do now).

As I checked various files from at around 15:00 are missing on the other side, but that can’t be only that 42 files.

Hitting Pause and Resume on the other other side (so not the one on the screenshot I posted before) resynced the folder, but it looks like stoped again afterwards – so there is still a problem with out-of-sync items preventing the sync?

:sleepy: “Body is too similar to what you recently posted”

Out of sync items don’t prevent sync of anything other than themselves, so to speak. Look on the destination to see the reason they can’t sync.

OK, I looked in the log on the destination side and the last entry is from 2019-04-10 12:45:52 – so it seems to be stopped? That’s when my restarting of the service comes into action :sunglasses: But I didn’t do that now.

Can you post the last 20 lines from the log?

Apr 10 12:13:46 rb-fs-04 syncthing[14257]: [JDMEP] INFO: Puller (folder "pool-data" (qaime-mxfuw), file "Jobs/XXX/.DS_Store"): finisher: file modified but not rescanned; will try again later
Apr 10 12:45:00 rb-fs-04 syncthing[14257]: [JDMEP] INFO: Puller (folder "pool-data" (qaime-mxfuw), file "Jobs/XXX.zip"): finisher: pull: peers who had this file went away, or the file has changed while syncing. will retry later
Apr 10 12:45:00 rb-fs-04 syncthing[14257]: [JDMEP] INFO: Connection to XYZ at 10.132.85.221:22000-10.170.0.21:44920/tcp-server closed: reading message: read tcp 10.132.85.221:22000->10.170.0.21:44920: read: connection reset by peer
Apr 10 12:45:00 rb-fs-04 syncthing[14257]: [JDMEP] INFO: Puller (folder "pool-data" (qaime-mxfuw), file "Jobs/XXX/.info"): finisher: pull: peers who had this file went away, or the file has changed while syncing. will retry later
Apr 10 12:45:00 rb-fs-04 syncthing[14257]: [JDMEP] INFO: Puller (folder "pool-data" (qaime-mxfuw), file "Jobs/XXX/.info"): finisher: pull: peers who had this file went away, or the file has changed while syncing. will retry later
Apr 10 12:45:00 rb-fs-04 syncthing[14257]: [JDMEP] INFO: Puller (folder "pool-data" (qaime-mxfuw), file "Jobs/XXX/.info"): finisher: pull: peers who had this file went away, or the file has changed while syncing. will retry later
Apr 10 12:45:00 rb-fs-04 syncthing[14257]: [JDMEP] INFO: Puller (folder "pool-data" (qaime-mxfuw), file "Jobs/XXX/.DS_Store"): pull: no connected device has the required version of this file
Apr 10 12:45:00 rb-fs-04 syncthing[14257]: [JDMEP] INFO: Puller (folder "pool-data" (qaime-mxfuw), file "Jobs/XXX/.info"): pull: no connected device has the required version of this file
Apr 10 12:45:00 rb-fs-04 syncthing[14257]: [JDMEP] INFO: Puller (folder "pool-data" (qaime-mxfuw), file "Jobs/XXX/.info"): pull: no connected device has the required version of this file
Apr 10 12:45:00 rb-fs-04 syncthing[14257]: [JDMEP] INFO: Puller (folder "pool-data" (qaime-mxfuw), file "Diverses/XXX/.DS_Store"): pull: no connected device has the required version of this file
Apr 10 12:45:00 rb-fs-04 syncthing[14257]: [JDMEP] INFO: Puller (folder "pool-data" (qaime-mxfuw), file "Jobs/XXX.zip"): pull: no connected device has the required version of this file
Apr 10 12:45:02 rb-fs-04 syncthing[14257]: [JDMEP] INFO: Puller (folder "pool-data" (qaime-mxfuw), file "Jobs/XXX.info"): pull: no connected device has the required version of this file
Apr 10 12:45:02 rb-fs-04 syncthing[14257]: [JDMEP] INFO: Puller (folder "pool-data" (qaime-mxfuw), file "Jobs/XXX.zip"): pull: no connected device has the required version of this file
Apr 10 12:45:02 rb-fs-04 syncthing[14257]: [JDMEP] INFO: Puller (folder "pool-data" (qaime-mxfuw), file "Jobs/XXX.info"): pull: no connected device has the required version of this file
Apr 10 12:45:02 rb-fs-04 syncthing[14257]: [JDMEP] INFO: Puller (folder "pool-data" (qaime-mxfuw), file "Jobs/XXX/.DS_Store"): pull: no connected device has the required version of this file
Apr 10 12:45:52 rb-fs-04 syncthing[14257]: [JDMEP] INFO: Established secure connection to XYZ at 10.132.85.221:22000-10.170.0.21:44936/tcp-server (TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384)
Apr 10 12:45:52 rb-fs-04 syncthing[14257]: [JDMEP] INFO: Device XYZ client is "syncthing v1.0.1" named "rb-fs-03" at 10.132.85.221:22000-10.170.0.21:44936/tcp-server```

I think I’ll enable some more logging options also here on this side.

Again a report back: Now since Apr. 11th 2019 without a restart or problem.

Horay! But I really do not know what I did or didn’t do.

3 Likes