Know when files have fully synced across multiple nodes

Support
1

Is there a way to know when a file has been successfully pushed out to all nodes connected?

We have a main server that we use to read/write files to with 3 mounts(folders), each mount is then connected to 2 servers each.

We track changes using the “Recent Changes” feature on the main server UI. They show up here when the main server has picked up a change but not necessarily pushed it out to the other connected servers.

Is there a way to know from the main server that the connected servers have successfully received the files?

All servers are running v1.2.2, Linux (64 bit).

Topology looks like

  • Main Server
    • Folder 1
      • Server 1
      • Server 2
    • Folder 2
      • Server 3
      • Server 4
    • Folder 3
      • Server 5
      • Server 6
2

I don’t think there is anything for individual remote file status.

There is a rest call for overall folder completion, and a (undocumented) rest endpoint for files missing by the remote side, but ignored files for example would not be missing.

The out of sync (files remote side does not have) is visible in the UI in the device box.

Can you explain what you are doing? It seems you want to transfer files from A to B and know once the file is there, in which case just using rsync is much more easy and gives an easily observable outcome.

3

We’re actually using this to sync our load-balanced web servers, which works extremely well for the most part.

We have some legacy websites that aren’t deployable and are edited live (awful, I know). We give the developers access to the main server UI to check for file changes so they know when they can start testing their changes.

Often they can see the change recorded in the UI but it could take another 2-5 minutes to push out to the other servers.

The idea is, if we need to scale up then we can automatically spin up new servers and use the discovery feature to automatically connect them to the main storage server to sync, and stay in sync.

4

You could call the rest api on the remote server to see what version of the file it has.

5

That looks like it could work. Is that just the /rest/db/file call?

So ideally track the main changes, see which servers it sits on, then keep checking the status of the file until the versions match and report it changed in the UI.

6

Yeah

1 Like
7

Shouldn’t the remote devices section on the right give that information already (i.e. it’s “Up to Date” once the device has the new file).

8

We’re tracking 1.7m files so it’s kind of hard to track that way.

9

Hui, big deployments - nice. I guess I am conditioned to tiny setups with weird problems on the forums :slight_smile:

/rest/db/file on any server should contain all the information: The local part tells you whether the server you’re on already has the new/global version and the availability part lists all other devices, which already have it.

10

Brilliant, thanks :slight_smile:

We have a couple of small sync issues with permissions and unsynced files but it’s prod so we just kind of live with it :smiley:

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.