Is SyncThing the solution I'm looking for my home cloud server project? I'm using NextCloud also, let me know if this idea is a good solution

Okay so here’s the deal.

This picture is about a week old so it’s not exactly what I’m doing right now but it’s a visual that might help.

The laptop is windows, the desktop is Ubuntu.

The desktop is running a NextCloud Server, and I have ~/home/work, ~/home/documents, and a Raid 1 array partition /mnt/md0/Archive mounted to it via the NextCloud external folders application.

I have my the Work and Documents folder on my Windows laptop synced by the NextCloud client to the linux desktop /home/work /home/documents folders, and I am planning on mounting the Archive partition to my laptop by WebDav as an external drive, so that with internet I can access any of my back up files etc at any time on my laptop.

I want to use NextCloud so that I can share links, send folders, collaboratively edit in the cloud etc. I thought this system I was using would work fine, but I am running into problem in the syncing of the Documents and Work folders, and according to people on the nextcloud forum, this is because I should not be directly manipulating the folders in the file manager on the desktop.

SO INSTEAD, I’m thinking about doing this:

Mount the /home/documents, /home/work, and /mnt/md0/Archive to NextCloud using external folders app so that stuff is avaiable on the web, and then sync the /hom/work to the Work folders on windows using SyncThing.

Does that seem like a good solution?

Basically here’s a picture of what I’m trying to do.

I’ve gone ahead and started trying to implement this process. I will report back with how it goes, though if anyone wanted to chime in and share their thoughts on how this solution could be different or improved, or possible problems with it, I would be much obliged.

Also here’s my original thread on the nextcloud forum in case anyone needed further context: https://help.nextcloud.com/t/how-do-i-mount-a-nextcloud-server-drive-using-sftp-as-a-virtual-drive-on-windows/34577/24

EDIT: Ugh… I just keep running into problems. I thought FINALLY with Syncthing and NextCloud working together I might have this issue solved but I’m encountering syncing issues with Syncthing too!

4 Items on my laptop (named Goffe) would not sync with my desktop (named Lovelace), and said that it was due to lack of permissions or something. So I went to my desktop and chmod 755 -R the entire folder and it’s subfolders and NOW Goffe permanently shows itself “Out of Sync” with Lovelace, and Lovelace shows me this:

EDIT2: After reading around I found a thread here that seemed to indicate what the issue was:

And so I set it to ignore and I got this after it started rescanning:

I also set the devices to “introducer” and “auto accept”. I don’t know what those mean but it sounded vaguely like it was giving the devices to more fully share the permissions on the folder etc.

I think this is a great product but perhaps there is some work that could be done on cleaning up the documentation etc. Some of the wording of the copy in the app itself is also a little confusing.

Well, syncthing has stopped saying “out of sync” on Goffe, my laptop, and instead it’s back to having 4 files that it refuses to sync for I have no clue what reason why.

What I’m most confused by is how one computer can be showing this:

while the other is showing this:

one computer shows the default folder is in sync, and the other does not. One computer shows that a certain number of files are out of sync and the other shows that same folder having a different number of folders being out of sync. It’s very confusing!

A few of the folders are scanning, so no downloads are happening for thosw. You should post a screenshot with the folder expanded for clarity.

This looks like a “normal” state during initial sync (scanning and 10s of GBs out of sync). By “sync” I do not mean, that all the data needs to be transferred, but a lot of hashing and comparing metadata needs to be done, which can take a while, you should just wait it out. After initial sync things should become smooth and fast again.

Here’s some pics from the desktop.

@imsodin I left it going all night and it still looks like no progress has been made. Should I try reinstalling or something? Wipe the process clean and start over? The initial state I gave it wasn’t optimal, the folders were already mostly partially kinda synced by NextCloud and then before that I had copied them over separately off an external HDD. So a lot of stuff was similar but also slightly different.

I’m starting to think what I should do is uninstall and reinstall on both machines, Wipe both of there folders and copy the files to an external, sync the empty folders, and then copy from the external to both the laptop and the desktop from the external in as quick succession as possible. Would that help?

So I’d check for cpu, memory, io usage on the devices. I suspect one of the devices is constrained on one of these hence why everything is slowish.

I guess adding the folders and letting them settle one by one would help.

On the other hand, you already have failed items so you should probably address those too…

Do you have any ideas of why they might be failing? I’m honestly not clear on what the issues are. I think some of it has to do with permissions but I thought syncthing was supposed to handle that stuff.

Also, how can I check on the cpu etc? I noticed looking at the web gui that it’s not using that much ram or cpu. I have 16gigs of RAM available for it to use but its only using 500mb. And I’ve got a pretty decent CPU and GPU (though I guess the GPU doesn’t matter). I don’t see anywhere in the options to crank up how much throughput the program uses though.

That was my strategy too in past cases to speed things up (probably mostly due to less seeking on spinning disks).

Figuratively no progress (as in still out of sync) or literally (as in the number of out of sync items stays constant)?

That sounds like another device or disk access is the bottleneck - probably the latter. You can check it with e.g. iotop or getting the iowait time on the cpu (e.g. with sar).

Okay here’s what I’m doing:

  • Copying laptop files to external drive.

  • copying desktop files to external drive. merging those two together and dealing with any merge conflicts they have. Calling these versions of the folders the “canon” folders. They’re the ones considered correct.

  • Emptying the folders on both the desktop and the laptop.

  • Reinstalling Synctrayzor on windows laptop and resinstalling syncthing on the Linux desktop.

  • Connect Desktop and Laptop in syncthing device settings.

  • Confirm that files sync between default folders.

  • Create new Document folder on desktop and Windows machine and sync them via the web GUI.

  • Once I can confirm that they are synced, I will then pause syncing for both devices.

  • I will then copy over the contents of the Canon Folders from the external disk -And then turn back on syncing.

  • Allow the Documents folders to sync. (10gbs)

  • Once they’ve synced I will test back and forth to make sure that changes are propogated between the two and or trouble shoot any specific issues that refuse to sync.

If all this works then repeat the process with the larger Work folder (100GBs) If that works then repeat the process again with the Resources Folder (200GBs).

Is there anything else I’m missing here that could give this process more of an edge?

The laptop is currently on wifi but I think I have another ethernet cable around here somewhere, so I’ll plug that in to give me as good a chance as possible for speed.

@imsodin Figuratively, but also maybe literally. It’s hard for me to say because the numbers seem to just jump around at random. First it will say Syncing, then it will say Syncing 17%, then it will say Syncing 71%, then it will go back to saying Syncing 17%, then it will say Out of Sync, then at another time it will say All Updated, and then half the time Device A will show it’s remote, Device B, as being disconnected while Device B will show itself and it’s remote (Device A) as disconnected. I also don’t really understand why there’s a difference between what the two devices are showing?

Like if Device A says folder Documents is 99% synced, then why does Device B show Documents as being 17% synced? If they’re synced… shouldn’t they both be the same?

I read on one of the help pages that some of these issues can be caused by the resynce operation firing before an entire sync process was done. Could that be related? I changed the resync timing from 3600 to 12000. There are some REALLY huge files in some of these projects, AND there’s also some folders with hundreds of thousands of really small files (I am for instance syncing configuration files for VScode which includes syncing it’s extensions folders, many of which are built using node modules so the number of tiny text files can balloon rather quickly).

Also, some of these files are on HDDs but both the laptop and the HDD are using SSD for their primary partitions so I don’t know how much spinning disks etc is increasing the sync speed issues.

Is syncthing maybe not well suited for this kind of job? I am putting a rather big strain on it by syncing both projects with BIG files, and also syncing projects with lots and lots of tiny files.

I’ve used GoodSync in the past to sync my windows machines and as an easy SFTP client. They have a linux version that is CLI only but maybe I should look at that? or maybe Seafile? I know you guys work on Syncthing so you’re most familar with it, but maybe I’m also trying to use the wrong tool for the job. Nextcloud was great for having a web access and larger cloud platform applications etc, but it’s syncing process just wasn’t built for my use case really, since I was running off of and using my desktop, and not leaving it alone as merely a server.

Jumping around is weird, but I wouldn’t worry much about it at this stage. Device showing as disconnected is bad though, syncing operation should not influence the connection state. Please look at the logs for any info related to terminated/established connections.

The difference in sync percentage is expected: Sync status is local compared to a global state. The global state is determined by what the local device knows about remote ones. So it is even possible at one point in time during a sync operation, that the global state is different between devices. Initial sync means lots of data changed on both sides at once, i.e. all states will differ a lot on both devices -> different sync percentages. Resolving this is a/the main part of finishing initial sync. If you are a talking about difference in between the remote percentage for device B shown on device A vs the folder percentage on device B, that’s caused by incomplete information about the remote stage - also nothing to worry about.

By resync timing do you mean the full rescan interval? In any case this shouldn’t be a relevant factor on initial sync. Later in “normal” operations it depends a bit on the data contained and how often it changes, but since watching for changes exists it’s far less relevant. An hourly (or even less often) full scan shouldn’t be a noticeable factor.

Regarding huge files: You should enable useLargeBlocks on all folders and devices. It changes the way large files are hashed: Instead of using a fixed block size, the block size is adapted to file size, reducing the size of metadata (thus less RAM and transfer size). You find the settings under actions->advanced as a checkbox for each folder.

I believe Syncthing is well suited for your job. Due to its nature with a decentralized, block based protocol it is a big effort to negotiate an initial state between devices. Changes later on will happen really fast.

Yes, that is what I meant, thank you.

Should I also set “ignore permissions” or “introducer” or “auto accept” also?

I looked into Seafile and while it’s promising it’s too centralized for my current set up.

So it looks like Syncthing is what I’ll try again, once I get done cleaning up these versions of these files on my external hdd.

I have to ask though, is it a good idea to sync the folders empty, and then add the files in? Or should I only do sync once I’ve copied versions of the files off the external hdd to both computers, that way the initial sync will be the one with the bulk of the file structure etc?

Lastly, once I start this initial sync, how long should it take? Should I avoid changing any files in either of those folders until it’s done doing it’s initial sync? (because these folders are basically all my work folders, that means that if the initial sync takes a long time, days or even a week or more, then that is a MASSIVE hit to my ability to do my work and be productive. I means basically spending that entire time with the computers syncing up and me having to do all my work on a friend’s computer or something).

Those depend on your setup. From your screenshots it looks like you have two devices, so introducer and auto accept won’t give any benefit. One automatically adds new remote devices and the other even accepts folders. Unless you have specific use case for them, keep them disabled. As for ignore permissions: If you have no need to sync them or Syncthing doesn’t have ownership (e.g. in my case on one device the syncthing user just has access via group), then I’d disable it - one less thing that could make trouble.

Both options have their advantages and disadvantages: If in the initial state one side has all the data, the other none, then all data needs to be transferred but less negotiation about the equality of files existing on both sides needs to happen. If both sides have roughly the same files, no actual data transfer has to happen, but metadata needs to be exchanged and compared, such that Syncthing can be sure the data is actually equal (that’s what you are currently doing).

How long it takes is hard to say. Did you check iotop/iowait times to check whether the disk is the bottleneck? In any case you can keep working during the initial sync, changes just aren’t picked up almost immediately as usual (as one “round” of syncing takes very long).

Here’s the output from iotop on my linux desktop:

That probably isn’t super helpful though as I have already removed sync thing from the laptop in preparation for reorganizing into an external disk and then recopying them back over to the machines for a cleaner sync process.

Which do you think would be faster?

  1. To start with 0 files on one and all the files on the other?

  2. To start with equal files across both?

I assumed at first that I should start with equal files on both because it’s A LOT of data to transfer. BUT, it does seem that with so many files, the bigger issue is comparing the metadata and stuff and perhaps it would be better to just let it copy stuff over so it isn’t getting so tied up with the metadata process.

Sure, iotop/iowait when nothing is happening wont show anything interesting :wink:

I’d say 2. is probably still faster, as even in 1. you still need to exchange metadata. You get around some “virtual” conflict resolution in 1., but I assume the transfer would take more time. I am using a lot of conditionals in here, because it is not that well known a problem. Just to reiterate: Adding one folder at a time (or pausing all but one) is definitely a good idea.

I would check the stats on the other side too.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.