1 Million sync use case

General
1

My use case requires syncing a read only folder with 2,000 files, 1.5GB data to 1 million Android devices.

Is that possible if accomplished in stages of sharing and resharing?

It seems that the weak point is the device that is resharing the 2,000 files that it has itself has received by sync. How many peers can the input and output swarms be before the phone in the middle passing on the cascading sync to the next swarm begins to suffer?

2

Connections to other devices cost CPU and RAM (and hence battery, on a phone). I’d keep it on the order of the low tens on a phone. It doesn’t sound to me like Syncthing is the ideal candidate for what you want to do, probably.

1 Like
3

what kind of work are you doing to have access to so many devices? :scream:

4

I’m distributing free multi-media literacy curriculum tailored to each mother tongue.

5

So if each person just share/synched my read only directory with two others… in how many generations would I be publishing to 1 million?

6

About 20 I guess. (Assuming you mean three total; one “in” and two “out”.)

7

Also, there is no such thing as read only folders in syncthing, there is “i refuse to accept changes from others”, but that is fundamentally different

8

It is important that no one along the way in the cascading sync distribution will be ever be able to change content of the directory that I author.

9

They will, that’s not how syncthing works. There is no central authority that can enforce that, as it’s a p2p system.

10

I saw something in a past forum where someone was stuck at 63 peers. The reply was that it should be possible to go beyond that number of peers.

How does that correlate with only expect peers in the 10’s?

11

Phones tend to not be as powerful as computers.

12

Does it make any difference in how how many peers there can be if the principle device sharing is a windows computer, (2, 4, 32, 64 or 128 core) and the rest are Android devices?

13

sounds interesting, can you share a link?

as @AudriusButkevicius said you have no control over the devices that are not shared with your server directly, so I think Syncthing is not the best/easiest way to go.

idea 1:
Go with a simple app that downloads a list cointaing urls/hashes from a central server to a folder on the device. If a files is missing/changed it is downloaded again.
pros: you have control; user does not need to know anything besides installing app; you can delete/update files remotely
cons: bandwith

idea 2:
create a (weekly) torrent file containing all files, maybe split into courses for smaller download size
pros: fast downloads; bandwidth saving; no extra app from you needed; if you always name folder same existing files should be reused
cons: torrent app needs to be installed; users may exceed their bandwidth usage; you cannot delete files remotely

14

Idea 3: When setting up a folder to distribute from the originating Syncthing device have a “broadcast (BC)” checkbox.

  1. BC makes a magnet link that will auto install and auto subscribe the folder on a Syncthing installation or just auto subscribe the folder if ST is already installed.

  2. BC will have an algorithm to load balance the sync swarm size and location by sending notifications to BC folder recipients to activate/deactivate their folder share as needed.

Pros:

  1. BC can perform the ongoing sync of a folder across an infinite number of connected users.
  2. BC simplifies curriculum distribution by making the user agnostic to the to the Syncthing UI.
  3. ST is open source making it possible to add such features.

Con:

  1. Sync latency increases with increase in number of peer population. But that is ok for curriculum updates.
  2. Potential Users of the curriculum is greatly limited because ST does not yet support extSD.
  3. Coding is required.
  4. BC would only work on Androids because notifications is an Android utility?? But that’s OK since curriculum is mostly used on Androids.
15

@uok

Here is the basis of the use case…

1 Like