My UseCase - Looking For Suggestions on optimization & A few ideas of my own,


First I’d like to outline a little about my setup and use case.

I have two servers in the cloud. Server A and B. These are the primary servers with high bandwidth connectivity and small to medium storage space.

I have a bulk server at home, (far out in the woods, with a satellite internet connection). At this location there is very limited bandwidth and low data caps; but lots of cheap personal storage space.

I have temporary admin access (1-5 hrs, several times a week) to a shared family computer. This computer has limited storage space, medium internet bandwidth, and high but not non-existent data caps.

I also have access to public hotspots, and work Wi-Fi – limited bandwidth and data caps. (So I try to keep any file sync low on these networks, to avoid peeved individuals coming and hunting me down.)

Server A and B has Syncthing.

“Shared” Family Server/Computer – Has Syncthing, It can only hold a limited amount of files, is also online only in the daytime. It is also on a highspeed local network, and I have physical access.

Anyways my goal is to synchronize files from Server A & B to the bulk server, basically using sneaker net and a tablet with Syncthing. To avoid having to USB Transfer and hop files. I use the Tablet and Wifi to transfer and move the files without having to plug anything in or be physically using the Shared computer. This way others can use the shared system, while I am transferring files. I just need to be in range of the wifi.

So what I do is disable the shared computer when I am not on the local network with it. When I show up on the local network with the tablet, I enable the shared computer that has synchronized down while I was away files set by filters from Server A and B. Part of the issue I am running into is, when I enable the Shared server on the tablet, it prefers to copy down the files from Server A and B over the internet again, when there is a perfectly good copy sitting already downloaded on the local network.

I am not sure how Prioritizes sources for files, I assumed it would first prioritize local network before trying to get again from the cloud.

I think what’s happening is, and I am just guessing. If both the shared server and the tablet have a partial sync of the files from server A and B, and each of them have different parts. When they are both on the same network, they are not sharing between themselves first the differences before looking to pull in the full list from the cloud.

What I was wondering / suggesting / thinking might work, is a way to “group” devices into separate states. Devices in their group always try to come to the same state before as a group looking to bring the global state equal. This way when a tablet in a local group, entered the local network it would try to being it’s state up equal with that of the local server, before trying to get file pieces from the global state of servers A and B. Saving bandwidth as the tablet once on the network tries to get those files from the internet based servers before the local one.

There may already be a way to do this, if you know of a method to achieve this let me know.

There is nothing smart about selecting where to download files from.

It prefers to download from the “least busy” device, where least busy means “device which has least of my requests outstanding” (it has no knowledge of what requests from the other devices the chosen device has outstanding), which naturally ends up being the “fastest” device.

You can artificially control this by limiting bandwidth per device, and then toggling not to apply bandwidth limits in local networks.

You could always do something smarter and write something external that monitors device addresses and adjusts per device bandwidth limits based on that.

I think the current controls that we have are sufficient for most usecases, and if you need something more advanced, you should probably do it externally.