Hi - I’ve tried my luck looking online for answers but unfortunately I don’t quite fully understand most of the troubleshooting, so I was hoping someone here could help me go through it.
I’m running a Pi as a server using SyncThing that connects to Windows machines but I’m coming across this error quite frequently:
I’m using PRETTY_NAME="Debian GNU/Linux 11 (bullseye)".
For the service start, are you using SystemD? If so, what was the command used? Was the SystemD unit file customized or are you using the stock file?
Apologies but I don’t know how to answer this one, and I don’t fully I remember how I did it then, but I think I followed a guide that copied the syncthing.service to /usr/lib/systemd/user.
While logged in as user raspi4, what’s the output from the following command?
This doesn’t seem to be working for me, here’s the output:
ls: cannot access '/media/raspi4/NVME_usb/sync_folder1/.stignore': No such file or directory
Debian (11/Bullseye), Ubuntu (Debian 11), Fedora and any Linux distributions based on those three already have Syncthing in their official package repositories. So if the install method was anything more complex than either of the following two commands, there’s a good chance you might have conflicting Syncthing versions and SystemD unit files.
Debian/Ubuntu:
apt install syncthing
Fedora:
dnf install syncthing
What’s the output from the following command?
dpkg -l | grep syncthing
Although the guide isn’t wrong, manually copying or creating a SystemD unit file isn’t necessary when using the official Syncthing Debian/Ubuntu packages because the DEB package includes a syncthing.service file that’s installed to a proper location. It’s also not necessary when using the Syncthing package from the Linux distribution.
So it’s likely that your Syncthing is being started as a service and it’s not running under user raspi4.
What’s the output from the following two commands?
systemctl status syncthing
systemctl --user status syncthing
That’s fine. I wanted to know if it would says “permission denied”, if you had an existing .stignore file, and also if your RPi was using SELinux.
It could still be loaded as syncthing@raspi4.service instead (or any other user for that matter, in that case systemctl | grep syncthing could give an indication of how it’s loaded, if it’s loaded).
I’m curious what the result of sudo ls -la /media/raspi4/NVME_usb/sync_folder1/.stignore is. At least then we know what user owns the .stignore file and what permissions were set - which could indicate a thing or two.
It means that your RPi isn’t currently running Syncthing as a service at the system level.
If you wouldn’t mind posting the output from the following command:
systemctl status
Just a quick 101 on working with SystemD…
The following two commands are equivalent, listing system-level units that are currently running:
systemctl
systemctl list-units
Display info about all running units (regardless of user):
systemctl status
Display info about all units:
systemctl status -all
Assuming that there’s a unit file named syncthing.service installed, to enable it as a system-level service:
systemctl enable syncthing.service
Because the default unit type is “service”, the .service portion isn’t required (other unit types include “mount” and “timer”). So the following command is equivalent to the previous command:
systemctl enable syncthing
Enable as a system-level service for a user named “alexa” (requires root privileges):
systemctl enable syncthing@alexa
Enable as a user-level service (no root privileges required):
systemctl --user enable syncthing
Of the three service options listed above, the systemctl --user one is generally the best for most users unless for some reason there’s a specific need to run Syncthing as root.
No problem, the other commands were just for future reference and hopefully to help with understanding SystemD a little bit more.
According to your output from systemctl status, you’ve got a user with UID 1000 running Syncthing. However, it’s not running as a service. Here’s a snippet:
Note how /usr/bin/syncthing (process ID 1056 and 1089) is running at the same level in the process tree as nm-applet (PID 1062). It’s also at the same process level as when you issued the command systemctl status. It means it’s part of your desktop session, so whenever you log off Syncthing shuts down.
And speaking of UID 1000, although it likely is, just to be certain, confirm that raspi4 is UID 1000 (-n replaces the default user and group names with their respective UID and GID):
So that definitively confirms systemctl --user hasn’t been used to set up any user services.
That’s good. The UID and GID match the user running Syncthing.
There’s nothing wrong with launching Syncthing that way. If you always stay logged onto the desktop it’s effectively the same end result as running it via SystemD, initd, or some other method. So now that we know SystemD isn’t involved, we can ignore it.
Since we’ve confirmed that Syncthing is running under user raspi4, and that the same user is also the owner of /media/raspi4/NVME_usb/sync_folder1, there should be no issue creating a .stignore file. So try the following command which will create an empty file:
If the command above fails, there’s a good chance that the filesystem on your SSD might be corrupted. Is your USB device a NVMe stick in a USB adapter?
bash: /media/raspi4/NVME_usb/sync_folder1/.stignore: No such file or directory
touch: missing file operand
Try 'touch --help' for more information.
Yes, it’s an NVMe on a USB enclosure. However, outside of this the drive seems to be fine and working properly? I can access all its data, and checking the filesystem using Disks doesn’t result in any issues or errors either. I had also formatted it to Ext4 filesystem from the get go in hopes this would be more compatible and less error prone in the future.
I keep seeing this on boot on the yellow Notice modal:
2023-05-03 20:54:31: Error on folder "sync_folder1" (sync_folder1): folder path missing
2023-05-03 20:54:31: Error on folder "sync_folder2" (sync_folder2): folder path missing
2023-05-03 20:54:31: Error on folder "sync_folder3" (sync_folder3): folder path missing
2023-05-03 20:54:31: Error on folder "sync_folder4" (sync_folder4): folder path missing
oops… I just noticed that I’d unintentionally included back ticks in the example command, resulting in the “missing file operand” error. Let’s try that again:
Too soon to tell if it’s certainly a filesystem problem. Depends on the result of the touch command above.
As long as there aren’t millions of files and directories, ext4 is a good choice.
For SSDs, btrfs is also well worth considering because it has some flash media friendly and extra data integrity features.
You’ve got a bit of a chicken-and-egg problem.
On Debian-based Linux distros, if there’s a desktop environment, removable media is auto-mounted under /media/<user>/<volume_label> (if there’s no volume label, the manufacturer name is often used instead).
Because the auto-mounter depends on you logging onto the desktop first, the path /media/raspi4/NVME_usb/ doesn’t exist yet – even if the USB drive is already plugged in.
And because /home/raspi4/.config/autostart/syncthing-start.desktop is part of your list of auto-started apps at login, Syncthing could potentially launch before `/media/raspi4/NVME_usb/sync_folder1" is actually available.
I do have multiple Cryptomator vaults which tends to break things into a million pieces so I wonder if it could be related? At least I have some out of sync items failing with the following error:
One to consider if I ever decide to nuke everything and start from scratch.
Open to suggestions! I’ve always had a Windows <-> Windows and a Windows <-> Mac setup and I’ve never had any issues, but now I’ve tried to move away from that by setting up a Raspberry Pi running Linux as a server to which both Windows and Mac sync to.
Even through it’s not ideal, I’m starting to consider re-imaging the system and start from scratch at this point, but I would need to be sure to set everything properly this time around to avoid coming across this or other future issues. Having that said, I would still obviously rather being able to simply fix my currents issues.
This is one of those “no news is good news” things. touch is normally silent except when there’s an error.
Because the command was successful, it means that there’s no permissions issue as long as syncthing is running under the same user you were logged on as for touch (which I’m assuming is “raspi4”).
It sounds like a bad time value was returned when Syncthing tried to read a file’s metadata. Having run across bit rot more than once before, I’d be concerned about filesystem corruption.
The backup program I use also dices up files into small chunks so in the past I’ve exhausted the number of inodes on a storage volume.
If you don’t already know which device node your USB NVMe is on:
df -h
Since RPi users usually boot from a SD card, your USB NVMe is probably at /dev/sda. And chances are there’s only one partition, so to get details about an ext2/3/4 volume:
tune2fs -l /dev/sda1
Check the “Inode count” and “Free inodes” values. Generally speaking, the total expected file count to be stored in the volume must be less than the inode count.
Ext2/3/4 pre-allocates a fixed number of inodes during formatting while btrfs creates them on-demand (limit is 264 inodes per subvolume – a disk partition can contain multiple subvolumes).
btrfs checksums every block written, so to start, cancel or status check a checksum verification:
On Windows, its StartUp folder is roughly equivalent to ~/.config/autostart/ on Linux.
When plugging a USB drive into Windows, it’s assigned a drive letter independent of any user desktop sessions. So by the time you log onto Windows and Syncthing is started via the StartUp folder, the USB drive is already available – to every user on the computer.
In contrast, desktop environments on Linux normally auto-mount a USB drive only after a user has logged on. It provides privacy and security on a multi-user system because the USB drive won’t immediately be accessible by every user on the computer.
(Last time I checked, macOS has similar behavior to Windows with regards to removable media.)
Without knowing what your workflow is like on your RPi – e.g., how often is your USB NVMe unplugged? – recommending a particular solution isn’t easy.
One suggestion is to simply not auto-start Syncthing. Make it part of your desktop routine instead:
Log on.
Plug in your USB NVMe.
Launch Syncthing (tip: launch in a terminal window so Syncthing’s activity log is visible. Just minimize the window when not needed).
…
Shut down Syncthing (or just close the terminal window).
Log off.
Unplug the USB NVMe.
(It’s actually what I do on my laptop.)
For Linux there are often multiple solutions to just about anything one can think of.
On my home NAS, I have Syncthing started by SystemD, but on a 15-minute time delay. Whenever the NAS is rebooted after OS updates there’s time for Linux to mount and check various storage volumes first.
Because you’re using your RPi as a server for your Windows and Mac devices, and if your USB NVMe isn’t frequently being unplugged, I’d use an /etc/fstab entry so that the USB NVMe is mounted at boot time instead of at user desktop login time.
It looks like your ext4 volume has the label “NVME_usb”, so first create the mount point…
LABEL=NVME_usb tells Linux to find a device volume with the specified label. Although a device node (e.g. /dev/sda1) also works instead of LABEL=NVME_usb, the latter is more flexible in case more drives are added (trying to keep track of which removable drive is on which device node gets messier the more drives there are).
/mnt/NVME_usb is a suitable mount point. Although you technically could use /media/raspi4/NVME_usb, the auto-mounter manages /media/raspi4 so there could be conflicts and it’s just cleaner to use `/mnt’.
ext4 specifies the filesystem type.
defaults … yup, accepting the defaults, then overriding some of them.
noatime disables recording the access time everytime a file is touched. Most users don’t need the info, it reduces wear on flash media, and it improves read times.
nodev says no character and/or block special devices (optional).
nosuid says no setuid files for better security (optional).
discard enables TRIM for use with SSD drives.
uid=1000 temporarily makes /mnt/NVME_usb owned by the user with UID 1000 (“raspi4” on your RPi).
gid=1000 temporarily makes /mnt/NVME_usb owned by the group with GID 1000 (“raspi4” on your RPi).
0 is don’t dump at mount time.
0 is no fsck at mount time (ext4 is a journaling filesystem, so unless journaling is disabled, better to do a fsck offline as needed).
Now you can leave Syncthing to auto-start when the user desktop loads because your USB NVMe will already be available early during the boot process.
