Syncing partial files possible? (EU-->Asia)

General
1

Hey guys,

I would Syncthing recommended a couple of times, and I’m looking into it now.

To me it’s not clear if a Synthing client can take a partial file and instantly seed it further. Does this work or do the files need to be complete?

My weird use case - maybe you have an idea

I am a Vlogger and want to send 10GB 3 times per week from Europe to Vietnam where my editor is. Thoughts so far:

  • I could use bittorrent and seed it with a couple of local clients who then all upload to Vietnam
  • the connections from VN to outside are limited, so I though having multiple streams come through would be a good idea.
  • The files could be downloaded partially to different clients in Vietnam and then synced amongst them.

–> Does this work with Syncthing as well?

Is this idea stupid? Should I go rather with Dropbox or so?

Thanks a lot
Till from Sofia
(Yes, I am googling for this topic 2 days already with little help)

2

This should work fine. You’ll probably get best performance having a cluster of several machines both in and outside of Vietnam. They might share the pieces with each other faster than over the international border, and hence cooperate.

3

Thanks, Jakob! Alright, I just ordered a VPS inside Vietnam. I’ll see how things work out.

So, in short: yes: every tiny part of the whole data gets reseeded immediately?!

4

Not every part, there are some heuristics which don’t apply (such as small files) and not immediately (the updates of stuff available is sent once every 10s or so)

5

Thanks for the clarification. Yes, that’s not an issue at all.

My concern was that IF I send an (let’say) video interview of 4GB file size, it’d continue leeching/seeding from multiple sources at the same time. This looks like the right tool now.

6

With files there is an option about the order in which their are sent/pulled (?), and the default (random) makes sure that clients get different content first. Here it is just about a single big file though - so how is block order handled?

7

As it stands now it’s random, which is not ideal in a 1 to 1 scenario.

8

Hey guys,

I ran a test overnight. I’m still blown away.

Situation:

  1. I need to upload 10GB / day to Vietnam (to my Vlog editor)
  2. I got a VPS in Vietnam and SCP’d 2.65 GB. It took 11h 40min. --> 3.8 MB/min or 4.4h for 1GB

Now with Syncthing:

  1. used my local computer as source for a 1GB file
  2. used another VPS as a third node in the US
  3. connected all 3 nodes together, dropped a 1GB file.

It looks like I more than doubled the upload speed by just switching from SCP to Syncthing and using another node. It took pretty much 51min for 1GB file.

Question: do I read the screenshot correctly? Did the file complete on the Vietnamese node at 21:20?

Is the upload speed on the right box (in the greyed area) the MEAN upload speed?

18 AM
Screen Shot 2017-08-29 at 8.33.18 AM.jpg2080×916 165 KB

If this is true, this just solved my problem. Thanks so much for making this project!!

9

It’s emwa over some small window (10s or so).

I wouldn’t trust the log so much and would actually verify when its done. Also I’d double check how much the US node (intermediate) has uploaded.