Proper way for "Sync complete" notification and actual idle detection

Status quo

As you likely know I’m developing Syncthing Tray. Who has already tested it has likely noticed, that the following features do not work very well so far:

  • Showing notifications when a directory has been synchronized, eg. “Synchronization of folder x complete”
  • Detect when Syncthing really does nothing, eg. to shutdown after all local changes have been uploaded:
    syncthingctl wait-for-idle --timeout 1800000 --at-least 5000 && systemctl poweroff

Why it doesn’t work

The idle detection simply checks whether the directory status is idle. To implement the notifications, I simply check for the status transition syncidle and emit the notification in that case.

Sounds like a simple and reasonable approach but there are two problems:

  1. A directory only has the status sync when it is pulling changes, but not when pushing. So whether files are (still) being uploaded can not really be determined.
  2. Users of Syncthing Tray might have noticed that after uploading local changes a “Sync complete” notification is emitted nevertheless. However, I believe this is only the case because some index update is pulled back which causes the directory to be in status sync only for a short moment.

The behaviour described in 2. can even happen quite often during upload. This is very annoying when those notifications are enabled and likely the reason everybody turned them off in the settings. Eg. uploading an album produces a notification for each uploaded track.

Any false statements or misunderstandings on my side so far?

How to improve those features?

I think those features would be useful so I’d like to implement them properly. However, I’m not sure how to do it. The ideas and questions I’ve got so far:

  • Since the state sync only means downloading, is there some way to determine whether a directory is uploading? That would be quite useful. The events ItemStarted and ItemFinished are not useful here because they also only cover downloading if I understand correctly.

  • Or maybe is there a way to find out whether each file/chunk/unit has at least been uploaded once?
    That would actually be the required criteria because it would mean I can finally turn off the computer who “introduced” the new versions and the rest of the cluster can still become completely up-to-date on its own. So the “sync is complete” from the perspective of the particular node (which introduced the new versions).

Any other ideas how to improve the two features?

There is no concept of uploading in syncthing. You can’t gauge someones activity externally, as someone might ask you for random parts of random files, and they might even discard it once they got it, as it doesn’t match a hash, so it means nothing.

There are various shortcuts which might make the syncing state very short, but you can work around this by not displaying transition periods that are very short, as it could imply that this only renamed a file and no transfers were done, and potentially only display it once after you had no more state transitions.

If you are using inotify and if your transfers are fast, it’s possible that we are producing an event for every file.

Shutting down once sync completes should be done by checking completion of other devices, not much related to events, as we don’t emit anything useful for the state of other devices.

Yes. Technically it uploads, but this does not seem to be propagated via the REST/Event-API.

That is also a possibility of course.

I could query GET /rest/db/completion for all directories and for all connected devices and check whether those still need bytes. For those directory/device combinations which still require bytes, I could then wait for the FolderCompletion event. If waiting for the event isn’t possible, I could also just poll GET /rest/db/completion.

However, I suppose this way of querying completion does not allow to distinguish whether the needed chunks can only be downloaded from the current instance.


I also noticed that there’s a Remote Download Progress event. The documentation says:

The files in questions are currently being downloaded on the remote device and belong to folder.

This sounds ambiguous to me. It could either mean

The files in questions are currently being downloaded by the remote device and belong to folder.

or

The files in questions are currently being downloaded from the remote device and belong to folder.

@imsodin What did you mean? I suppose the from version. Unfortunately this means that the event is not very useful here. Either way, I’d create a PR for making the documentation less ambiguous.


But thinking further about the wait-for-idle use case, I have found a solution which likely does not even require to change anything in my code :slight_smile: . One can just invoke syncthingctl for the remote instance where the files are expected to be downloaded:

syncthingctl wait-for-idle --timeout 1800000 --at-least 5000 --dir concerning-dir --url http://remote.instance --api-key foo-bar && systemctl poweroff

Still, it would be cool if one node could just ask the others whether they could get up-to-date “without me”. But likely this is not worth the implementation effort.

Because nothing that we know is meaningful. X asked for 128kb at location A on file B, it doesn’t say how far X is within the file, as the download order is random, nor does this mean that X will ask us about this file ever again, as it might be getting the file from other peers.

FolderCompletion is local folder, so it has no significance of what you are trying to achieve.

You’d have to wait for RemoteIndexUpdate, as that’s the only significance that remote side has received something, and then check completion for each folder/device. GUI code already does something like that, so that’s the best place to check.

No, there is no easy way to check if you are the only available device with the files, but in a decentralized system you can’t reliably tell that anyway. You are A, who is syncing with B and C. B is in sync, C is yet not, yet you don’t necessarily know if B and C are interconnected or not. Even if they are not, there might be a device D which you don’t even know that connects B and C.

In practise (or let’s say what I have as use-case in mind) I only care about the subset of the network which is visible to me. I know that B is ‘the server’ so the sync can considered completed when B has everything. It is as simple as that. Of course it unfeasible to take connections between other nodes into account.

Yes, that sounds like way to do it. So I’ve just implemented this in Syncthing Tray and it seems to work in general.

Only one thing is still annoying. When uploading files, the FolderSummary-event is often emitted. Sometimes the needed statistics are non-zero and sometimes zero. Since I use transition of non-zero to zero, the sync complete for the local folder is often emitted during uploads. In fact, I would have expected the needed variables to be always zero during upload. But maybe this is just a mistake in my code.

FolderSummary is emitted in both cases, as someone sending an index update to is might mean that we need to start downloading or that they finished downloading. You can infer which one it is from the values, and only check completion if you believe someone else has finished a download.

It seems that this event is also emitted for remote devices. If it is emitted for a remote device, the event data is the same as what you get via GET /rest/db/completion. In fact, that this event is also about remote folders is even stated in the documentation.

So I now just consume the FolderCompleted event and distinguish between local and remote changes by the device ID. I have already done a few tests and it seems to work nicely.