manual sync doesn't work?

Hi everyone,

sorry for the hassle, but I’m having a small problem. I shared a folder on A to B (send only). It’s just about a thousand files (my collection of classical music), but most files are large: it’s about ~ 600 gb overall. To speed up sync, the destination is using an old backup of the same files (I verified by md5 that they are still identical), however it seems that B didn’t pick them up: since a few days, the progress was extremely slow. it currently says that there are 1096 files out of sync (511GB), so it’s likely re-downloading everything (the connection is quite poor).

As the list of out-of-sync files is long, it’s hard to tell and Syncthing is not very verbose about why it thinks the file differs, but it seems to list most (if not all) files.

  1. Is there any way to see what files are in sync? A command-line query to the index?

  2. Could it be a timestamp problem? Can I force B to compare files based on md5 only, instead of time/date?

In general, the only feature I’d wish is some more powerful “remote management”. Checking a box on B that says “Trust machina A” should allow me to manage B remotely from A’s web interface (and here I’m dreaming… a button “open a remote shell with ssh”…).

thanks for the help, mh

No files are in sync until they are checked.

Even timestamps would not help as the files have to be hashed and compared.

I guess if its a slow device, checking the data will probably the reason it’s slow, as that requires cpu.

Correct, it’s a slow device, but not that slow.

So you confirm that timestamp alone basically doesn’t matter? File comparison is always done by hash.

File comparison is done by hash, but you have read and hash the whole file to produce the hash which is a cpu intensive task, so I suggest you look at cpu usage.

Yes, I’m familiar with hashing. Still it seems suspicious: that machine took the md5 of all the files in a few hours. Even if you are using a slower hash, I would expect it to finish, say, in a day or so.
Instead it kept on doing network activity for ~ 48 hours (as seen from A’s Syncthing interface). Now it looks like it’s doing nothing (the sync % is stopped since this morning… I’ll have to pay a visit onsite)

That’s why I said it would be great to have (basic) remote management in Syncthing :slight_smile:

There are tons of remote management tools out there that you can use. I don’t think syncthing should bring a kitchen sink with it.

We use sha256 which is noticeably more computationally expensive than md5.

Also, if its spinning disks, we might not do then any justice reading data from a few threads causing thrashing.

So I suggest you check cpu usage and io usage

And in the ui when you open the list of out of sync items, it shows what it is doing, i.e. if it is actually downloading.

Ok, here’s an update.

The issue had nothing to do with the CPU. The machine (which is a synology NAS) was totally idle. There was a problem (that was visible only locally) about not being able to chown a file; so I double-checked permissions and discovered that they were not properly configured. I fixed the permissions, did a chown on the files… and that’s when the disaster happened :slight_smile:

Basically ST deleted 90% of the local files. Only a few were left, not totally sure why. I can tell that even some files that were identical (as contents) AND had the same filename/path were deleted. Then ST started downloading stuff so I had to pause the folder. I will retry after manually copying the folder via USB disk… Possibly the backup I used was too different (there were many renames… but this should not affect synchronisation).

Anyway, I still insist ST has a lot of room for improvement in information/monitoring/configuration of remote machines. I’m a software developer myself, so I know it’s not simple, but being able to manage everything from one place would be extremely convenient.

I don’t think syncthing should improve in that area, there are remote control solutions out there, you should use them.

You, especially being a developer, just need to sort some ssh or rdp or vnc, or even a simple reverse http proxy with auth for the remote syncthing gui. The fact that you want syncthing to provide that just seems laziness from your side to put a proper solution in place.

We somewhat follow the Unix philosophy that a single tool for a single purpose, so I don’t expect your grep/sed in a bash script to provide remote configuration and management. If we did, it would probably something half arsed anyway, that would not haved allowed you to drop in a root shell to run a bunch of chowns like you did anyway.

You are the maintainer, of course I respect that, and you know the product better than me.
You seem to forget one thing, however: if I had a remote management tool, very likely I would not need syncthing at all: rsync over ssh would probably suffice, and it would give me more control. The reason I’m using syncthing is precisely that I’m lazy, I just want to drop some files in a folder and find them somewhere else, and I don’t want to mess with opening ports on the routers, etc. (and my guess is that you user base is largely made by people who are… lazy like me :slight_smile:

Additionally, I said “information/monitoring/configuration”. At least 2 of 3 items don’t sound like a dramatic violation of the single purpose principle… On the opposite, I would consider a more serious issue not being able to precisely understand what a remote machine is doing.

anyway, thanks for listening to my rant :wink:

If you are using syncthing to copy files from a to b, then I agree, using syncing is probably not the right tool. Main purpose of syncthing is continuous bidirectional sync, not one way backup system, like a lot of people are using it for.

Not all syncthing setups are your own devices, so having remote monitoring/debugging by default would not be acceptable.

If it becomes something that people need to enable, then they might as well just enable remote ui access effectively providing what you are asking for.

I don’t think extending the protocol to send various states and statuses makes sense when all you need to do is navigate to a different page to see that.