Syncthing for large pre-populated folders

Support
1

Hi !

I am new to Syncthing and would like to use it to sync a rather large (85Gb) folder between my laptop and my phone’s SD card.

The data already being mostly in sync (apart from a dozen new files on the phone), if I point the new sync to the existing folder, will all files get uploaded again or will they be recognized as already there?

It is taking around half an hour to analyze the folder for syncing, and I understand that full scans are necessary every so often. Since I only need to sync the files every once in a while and that no major modifications will be done, is Syncthing the right tool for that or is it going to use a lot of CPU power for nothing? Will I need a half hour analysis just to copy 10 new pictures every week ?

Thanks for your help and for developing this software !

[EDIT] Alright did a test on a smaller folder, and Android gives Syncthing read only permission on the SD Card :slight_smile: so I guess it’s not the right way to go… does the device have to be rooted?

2

There is no write support for sd cards unless you use root, or use unified storage (merging device storage and sd card) or set the folder to be at a very specific path (not sure it works on all roms).

3

It should, that’s a general Android thing I think. The paths are ./Android/{data,media}/com.nutomic.syncthingandroid.

2 Likes
4

Thanks !

Supposing I use the ./Android/{data,media}/com.nutomic.syncthingandroid path as a workaround, how about the rest of the question ? do all files get uploaded again?

5

If the files are the same, it will sync the metadata but not the content, but be safe and backup before hand.

7

Hi again !

After some thought, moving all pictures to the /Android/**media**/com.nutomic.syncthingandroid path seems like a good way to make it work.

However the folder doesn’t exist in the /media folder, although it exists in the /data folder. Can I just force create it?

Is there a way to make a “dry-run” of the synchronization before launching the real thing? it’s pretty scary to just let it run on a 85 GB folder…

Thanks, Have a good day

8

Yes you can, nothing “forcefully” about doing that.

What would a “dry-run” do? There’s some things you can do, depending on what is scary to you. However in the end whether or not being scared, I’d just first create a backup of the data and then let Syncthing do its thing.

9

Thanks for your answer

Just a preview of what is going to happen when the synchro starts… if I mess up some directory structure and either the phone or computer start mass deleting or copying the 85GB worth of data it’s going to be a mess to sort out, event with a backup (which I do have). Tools like Rsync have a --dry-run feature so that you know what will happen before starting to sync

10

What’s your current state - are the two sides more or less (or fully) in sync already (assuming no messing up)? Anyway for somewhat of a dry-run there is one option, but that might fail on android, as it requires the mod times to be equal on both devices. If you can manage that, then you can go on, otherwise it wont work: Set both devices to “send-only”. They will exchange metadata and if the files are the same, agree on being in sync despite both being “send-only”, thus leaving only the items that are actually different as out-of-sync. And they won’t actually sync data until you set them back to send-receive.

And yes, having to restore 85GB is annoying, but if you are on linux you could e.g. make a hardlink copy of the folder, thus restoring would be as simple as deleting the messed up data and renaming the hardlink copy.

1 Like
11

nice tip !

Thanks very much, will try it right away

12

Well, good thing I tried it with the “send only” option first… It takes more than 30 min to analyze such large folders, but the syncing starts right away…! I had to pause the sync immediately because it started transferring files that are already on both sides… Is this a known bug ? it didn’t occur while syncing smaller folders

[edit] I’m giving up ! thanks everyone for your help, but I’ll have to find a different solution, analyzing is much too long for such folders and there are too many hiccups and errors. It looks like great tool though for smaller folders not my use-case I guess.

13

Please come back! :smiley:

Of course it does, it needs to hash 85GB with a strong cipher - that takes time, especially on a phone!

What makes you think so? Possibly the remote status saying it’s out of sync and numbers changing there? That’s because it will start exchanging metadata right away, but it will not transfer actual files until everything is scanned.

After the full scan, it will even say that it is “syncing”. That means it is comparing the data both sides have and only if there are differences, and they aren’t set to “send-only”, will they exchange actual files.

TLDR:
When you add a new folder that’s sizeable, you need some patience and wait until things get settled. The first sync is rough, that’s inevitable due to the block based protocol of Syncthing.

1 Like
14

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.