Devices seem to not be syncing

Support
1

I have a folder that’s successfully synchronized between three devices. I now wish to add a fourth; a Raspberry Pi/10TB USB drive combo for offsite backup purposes.

The problem is that the sync file count is stuck at zero. The sync folder count matches, but no files are reported copied. The USB drive is thrashing furiously (I’m running an onsite seed before I move the setup offsite). Also, the Syncthing service on the RPi keeps crashing every few hours.

It’s been this way for four days. The service crash, however, has only been occurring for the past day or so.

This is my Dev folder hierarchy, so it contains tens of thousands of small files 1-2 kilobytes in size. I don’t know whether this has anything to do with it. But I do know that my RPi 3 was overheating at +89°C until I temporarily installed the fan from my decommissioned RPi 4. There’s no appreciable network traffic in either direction.

There do appear many temp files on the target:

image

So something is doing something. But the reported file count remains at zero nonetheless.

Here’re my stats:

Source:

image
image443×516 62.5 KB
image image

Target:

image image image

image
image886×144 31.6 KB

I figure the iNotify error will subside once everything is synchronized. Correct me, please, if I’m mistaken on that point.

What’s going on here? How do I kick this thing into gear?

2

FYI I fixed the iNotify error per the documentation.

3

Would you be willing to update Syncthing on the Raspberry Pi (see https://apt.syncthing.net)? This may not fix the problem, but it could be a good starting point. Your current version is quite outdated.

4

Would you be willing to update Syncthing on the Raspberry Pi … ?

Funny you should mention that… I just finished doing that very thing.

After reading this and this, I’ve arrived at the conclusion that we’re looking at a comedy of errors here.

First up, I believe that the Achilles’ Heel of this otherwise great tool called Syncthing is the initial seed sync, particularly with a large set of small files.

Next up, I’ve decided to take it on the chin for trying to do this on an RPi—and an RPi 3, no less! Weak CPU, low RAM, USB 2.0… well, one might say it’s got it all.

So I’ve upgraded, removed the folder from Syncthing, dumped the partition and started over from scratch. This time I seeded using rsync. That took about 10 hours or more, but at least I could glance at it and track progress every now and then.

I recreated the target folder, but I have it paused right now. I’m waiting to see what happens with another synced folder that contains only a single 728MiB file. The bytes themselves are copied over, but the file is still named .syncthing.4 Hours of Mozart.mp4.tmp and the folder status still shows a big red "Out of Sync". It’s been that way for a few hours now, so I’m not sure what to make of it. I guess it’s still hashing. FWIW, htop doesn’t show Syncthing as using up anything significant.

I’ll give it some more time, in hopes of turning on the big Dev folder once it’s done. I can hear the USB drive chugging away, and there’s nothing else going on right now, so it must be doing something.

I’ll continue my progress reports here as necessary.

5

Update: no chugging.

6

Update: I just ran a quick script to calculate a SHA256 hash on the 728MiB file. It took 23 seconds. On the other hand, the Syncthing folder has been displaying "Out of Sync" for at least five hours now. The USB drive is silent.

I’m loath to turn on the big Dev folder if I can’t even get a single file like this to sync.

What gives? What could be the holdup? Is there a way to check on it?

7

Post a screenshot of the gui on the pi side, with the folder expanded.

8
  • Is the “OFFSITE” device the Raspberry Pi 3? (which model exactly?)
  • If it is the RPi 3, is the networking wired or wireless?
  • If it is the RPi 3, are Syncthing’s database files stored on the micro SD card?
  • What type of drive is the 10TB USB drive? (make and model would be very helpful)
  • How is the 10TB USB drive formatted?
9

a comedy of errors

And what a comedy it was, indeed! I’m still laughing :unamused:

In the end, it turned out to be a permissions problem. As you’ll note here, I’ve been laboring under the requirement to sync to an NTFS partition. Well, the painful lesson in all of this is that Syncthing and Linux and NTFS don’t play well together. Syncthing was choking because it couldn’t run a proper chown where it needed to.

Sure, one might be able to take a bunch of trips 'round Robin Hood’s barn and MAYBE get it to work under NTFS, as @gadget describes in his excellent answer, but why go to the trouble when there’s an easier way (at least for my scenario)?

Bear with me…

Once I had an ext4 partition, I was able to set the directory’s owner to the user under which Syncthing is running, and then to set its permissions to allow read/write by anyone. Once that was done, I created a Syncthing folder and watched in absolute adoration as the synchronized file count started climbing immediately. Lots of nice network traffic, too.

And now I remember how it managed to work under RPi/NTFS all those many years ago (see the other thread). I was running the service under root! For shame, for shame…

So I take back all those nasty things I said about Syncthing’s heel. Apparently somebody came along after Thetis half-finished the job and turned Syncthing upright and doused his foot as well. I must have been looking the other way and missed that.

As far as disaster recovery into a Windows environment, there’s more than one way to skin a cat.

VirtualBox is a Type 2 hypervisor, which means it supports USB pass-through. In the unlikely event of a data disaster at this scale (it’d have to be a pretty serious situation to bring me to the far reaches of my plan like this), I’ll spin up an Ubuntu VM and use it to relay the files to an NTFS partition somewhere.

Either that or I’ll just turn the whole thing around and recover the files with Syncthing, simply the reverse of how they got on the drive in the first place.

Easy, peasy, Japanesey.

Thanks, everybody, for chiming in.

10

See my full reply for more info.

11

Screen shot (after switching to ext4)

image
image1420×880 226 KB

12

Yeah, trying to spot a RPi 4 in stock – and selling at MSRP – has been rarer than people spotting the Loch Ness Monster. :smirk:

The RPi 3 B has enough horsepower for Syncthing with a little work tweaking the setup.

Although slow, it might seem fast compared to the Ethernet interface because it’s 100BASE-TX on the RPi 3 (my earlier post about the RPi 3 B+ might be of interest: Too slow for school - #5 by gadget).

By default the database files are located near Syncthing’s config file. On Linux it’s usually under ~/.config/syncthing/ in a subdirectory like index-v0.14.0.d for the user running Syncthing.

It looks like your 10TB WD Elements (WDBWLG0100HBK-NESN) is CMR rather than SMR, which is good because SMR + lots of small files + lots of disk writes often doesn’t work well (especially when combined with NTFS).

Much better… :nerd_face:

A creative solution that’d work, but a few other options that might be of interest… :slightly_smiling_face:

13

I could use some of that right now! See my latest post.

Is it wise to put them somewhere else? If so, is that a how-to that’s documented somewhere? FAQ isn’t giving up anything, nor is Usage TOC. (At least nothing jumped right out at me.)

Obee kaybee… how did you manage to go about spotting that one?

:innocent:

Interesting, thanks! I’ll have a look.

14

Relocating Syncthing’s database can be beneficial depending on the type of storage media, and even filesystem that’s in use.

On an RPi, the usual default is the SD card. Because a $5-$10 SD card typically isn’t designed for extensive write cycles, the card could fail in just a few months depending on how many files and directories Syncthing is tracking.

If you don’t use the SD card, storing it on an external USB-connected SSD is a viable option (even a $20 SSD can offer a 170 TBW endurance rating).

Syncthing has a --data flag for setting the database directory and a --home flag for setting the base directory for both Syncthing’s config.xml and database. For more details, see https://docs.syncthing.net/users/config.html#description

15

Bingo. That’s what I’ll do then.

AFTER I recover from all of this.

16

The systemd service file shows single hyphens for the arguments, while the docs indicate double hyphens.

What’s up with that? Can it be either?

17

Both syntaxes are supported by Syncthing.

Programs that start out on Unix/Linux follow the convention where single character options use a single hyphen while longer options use double hyphens (e.g., -a vs. --archive in rsync).

The syntax is very useful for concatenating multiple options (e.g. -rlptgoD) without clashing with a long format option.

Unfortunately, not so orderly on Windows – single hyphen, forward slash, no case-sensitivity, etc. I rarely ever see double hyphens used for long format options.

18

Syncthing had originally supported only the single hyphen version. It was just ~2 years ago that the double hyphen variant was introduced. Not that it matters much right now, but if you’ve got to use an older Syncthing version for some reason (e.g. due to the OS incompatibility), then it’s something good to know.

19

Got it, thanks.

20

Got it, thanks.