Keep a big folder synced beween 2 Linux webservers

Support
1

Hi All,

I have a folder containing web stuff and I would like to keep them in sync between 2 Linux webservers. Both sides should be able to write the synced folder and I would like to see the changes on both servers as soon as possible. I use Syncthing 2.0.16 and went through the initial phases, so the 2 folders are in sync.

The folder contains 1.8 million files mostly images and other files, the total size is 540GB.

When I write 10 x 10MB file to a directory it starts scanning rather than copy over the file immediately. Only after ~5-6 mins when the scan finished it copies over the newly created files.

What does it scan at this point? Why not it just copy over the files that inotify told? Why does that scan take sooo long? I am sure that scaning and hashing 100MB does not take that much.

Do I feel correctly that Syncthing is not designed for what I thought?

2

Indeed scanning a hundred megs should be quick, so my guess would be that the bulk of the time is in checking that the other 1.8 million files are still there, in order to deduce if the new files are indeed new or just a rename. I don’t think there’s a way around that today, other than splitting things out into smaller folders.

3

Thanks for your answer.

Bud why it does check the other 1.8M files if inotify did not mention other changes?

It should strictly rely what inotify says… (or at least is should have a mode like that)

4

For one, inotify generally doesn’t tell you about renames, so we can’t. So yeah, it’s a limitation. :person_shrugging:

1 Like
5

I have just found in the inotify manpage:

rename("dir1/myfile", "dir2/myfile");

Generates an IN_MOVED_FROM event for dir1, an IN_MOVED_TO event for dir2, and an IN_MOVE_SELF event for myfile. The IN_MOVED_FROM and IN_MOVED_TO events will have the same cookie value.

Can it not be used here?

6

No, not really. That something exists and works in one scenario doesn’t mean it’s useful in all scenarios and operating systems we support. But really, I hate arguing the ”can’t” side - if you think this is a viable way forward, the proof is in the implementation.

7

You are right, sorry. I am newbie here and forgot that you guys need to support multiple OSes.

I am just a bit frustrated because I don’t find the proper tool for my usecase :frowning: Which runs on Linux, providing local files for the webservers and allow them both to write the shared folder while it continuously keep them in sync.

Syncthing is very close to that…