So, update time! I’ve setup things again with an HDD on an enclosure and it was all smooth sailing during the transfer of +100GB of data until I restarted the RPi. I started getting different errors so I’ve now ordered a USB 3.2x1 hub with a power adapter, this one to be precise, so hopefully this one will do the job.
Since I will now have a powered USB hub, I’ll probably also go back to setting things up again with the NVMe enclosure since the scan times on the HDD were really slow, which didn’t help with the transfer speeds either.
I have come across External Storage Configuration, which I’m thinking about following for this upcoming setup, but I would still be interested in knowing your thoughts or recommended steps, to once and for all set things up properly from the get go in a way that will hopefully not break anymore and it will just work going forward.
That USB 3.x hub from Sabrent looks nice. The bundled power supply is rated for 5W, so it’s something to keep an eye on because Samsung says its PM981 NVMe SSD pulls up to 5.9W/5.7W during active reads/writes, although typical power consumption is likely less during normal usage.
Yup, a HDD is definitely not ideal, especially when the USB bridge in the enclosure doesn’t support UAS.
That particular Raspberry Pi documentation page is a good resource and walks thru steps that are very similar to what we covered earlier.
/etc/fstab
In the example provided, the suggested /etc/fstab entry was written as…
The “defaults” option is an alias for at least “rw, suid, dev, exec, auto, nouser, async”, so specifying “auto” and “rw” doesn’t hurt, but is redundant.
For added flexibility, instead of setting a specific filesystem type (e.g., “ext4”), the option “auto” tells Linux to determine how the volume is formatted:
UUID=5C24-1453 /mnt/mydisk auto defaults,users,nofail 0 0
On a single user system or server, the “user” option probably isn’t necessary:
UUID=5C24-1453 /mnt/mydisk auto defaults,nofail 0 0
And if you’ll be using the NVMe SSD:
UUID=5C24-1453 /mnt/mydisk auto defaults,noatime,discard,nofail 0 0
UUID vs. LABEL…
The Raspberry Pi documentation suggests using a UUID for the device identifier, which is perfectly fine.
A new UUID is auto-generated whenever a volume is reformatted. So each reformat requires updating /etc/fstab to match unless extra steps are taken to manually assign a UUID (not recommended because two volumes could inadvertently be assigned the same UUID).
Alternatively, the volume LABEL can be used as the identifier. It’s usually more readily recognizable (e.g., “NVME_USB”) compared to something more cryptic (e.g., “5C24-1453”) and also often more convenient to update as needed.
Raspberry Pi OS
Because your RPi is acting as a server, I’d recommend switching to “Raspberry Pi OS Lite” instead of " Raspberry Pi OS with desktop" for reasons including:
As lean as LXDE is, it’s still consuming system resources. By default the swap partition is located on the microSD card which is going to be a lot slower than even a HDD (is your RPi 4 the 1, 2, 4 or 8 GB model?)
A desktop environment typically auto-mounts storage volumes, complicating setting up mounts in /etc/fstab so that server applications such as Syncthing have reliable directory paths.
One of your original issues was that Syncthing was being auto-started by the desktop environment while you were also trying to start it as a SystemD service.
It seems more difficult at first, but there are often a lot more buttons and menus to tweak in a GUI to get to the same results as a single CLI command.
Would you recommend something with a higher rate? I’m really trying to avoid something like a NAS case which is way overkill from what I’m trying to do and definitely not as discrete, but if you are not confident about this option please let me know so I can look into other options.
I shall follow the suggested steps then
So, just to confirm, when doing the fstab step I can just replace UUID=5C24-1453 with LABEL=NVME_USB?
My RPi is the 4GB model, and I have actually started using Syncthing more via SSH because the browser GUI is indeed so much faster and easier to use on my main machine than through Chromium. I don’t mind having a go at this, although I’m just a bit hesitant getting stuck at some point and not having the visual support to try to troubleshoot.
However, if this helps on the power saving, which could help the USB run a bit better I’m up for giving it a go because at this stage I just want things for finally work. I wouldn’t also mind looking into smaller things like disabling Bluetooth, HDMI and the audio device since I wouldn’t have a need for them.
I took a look at Sabrent’s official product page for the HB-UMP3 and it said it’s bundled with a 5V/2.5A power supply, which works out to 12.5W, so that’s more than enough – unless later on you scale up with 3 more USB drives.
Yes, as long as the volume name actually is “NVME_USB”, otherwise adjust accordingly.
The following example /etc/fstab entries are interchangeable:
/dev/sda1 /mnt/NVME_USB auto defaults,noatime,nofail
/dev/disk/by-label/NVME_USB /mnt/NVME_USB auto defaults,noatime,nofail
LABEL=NVME_USB /mnt/NVME_USB auto defaults,noatime,nofail
/dev/disk/by-uuid/5C24-1453 /mnt/NVME_USB auto defaults,noatime,nofail
UUID=5C24-1453 /mnt/NVME_USB auto defaults,noatime,nofail
Comments:
/dev/sda1 is shorthand for “first SCSI disk, first partition”. The “second SCSI disk, first partition” is /dev/sdb1, and so on. (Linux groups SCSI, SAS, SATA and USB devices together because they all share common elements with SCSI.)
In the above examples the 5th and 6th fields weren’t specified – they default to 0 when not present.
Earlier when you posted the output from sudo tune2fs -l /dev/sda1, it included the volume name and UUID at that time:
Filesystem volume name: NVME_usb
Last mounted on: /media/datapi/NVME_usb
Filesystem UUID: 3a1b4f5a-597c-485e-a3e2-61fde66c1538
This means the proper /etc/fstab entries would have been either of these:
LABEL=NVME_usb /mnt/NVME_USB auto defaults,noatime,nofail
UUID=3a1b4f5a-597c-485e-a3e2-61fde66c1538 /mnt/NVME_USB auto defaults,noatime,nofail
While there are advantages to using a UUID in a data center with hundreds of drives, volume names are often shorter and easier to remember.
The mount point doesn’t have to match the volume label. It can be any valid directory name, so /mnt/SSD is just as good as /mnt/NVME_USB.
It definitely can be comforting to have a GUI, but it’s a lot easier to find solutions online by copying and pasting text from a CLI compared to a screenshot of a GUI.
4GB of RAM would require system tuning to run a server with a desktop GUI, so switching to a CLI-only is a major advantage in terms of system resource usage.
(Chromium forks a new process for every open window or tab, so it can easily consume several gigabytes of RAM during a browsing session.)
RasPiOS has a command-line tool for toggling Bluetooth on/off, video settings, overclocking the CPU plus a variety of other features:
sudo raspi-config
So in a nutshell…
Image the microSD card with Raspberry Pi OS Lite.
Boot the RPi.
Create a user account (e.g., “datapi”).
Add a suitable /etc/fstab entry for your USB drive.
Just done all of these steps now with the aforementioned powered USB Hub. I was able to set things up and sync a small folder. After that, I shared a big folder with the RPi and shortly after only downloading 1.11GiB of data the sync stopped with the error:
Error on folder "syncpi_backup" (syncpi_backup): folder marker missing (this indicates potential data loss, search docs/forum to get information about how to proceed)
I shutdown the Syncthing service and rebooted the RPi. Upon login, the error remains, so I ran ls /mnt/NVME_USB to check on the drive and I got the following:
This was previously /dev/sda1 and not /dev/sdb1. As I was not sure if this was a problem, I updated the /etc/fstab file from using LABEL=NVME_USB to UUID=66C68DC7C68D97C7.
After shutting down Syncthing and rebooting once again, the error remained, even though this time around ls /mnt/NVME_USB and sudo blkid turned out ok.
I seem to be able to cd to it but when I try to sudo umount /mnt/NVME_USB I just get:
umount: /mnt/NVME_USB: not mounted.
I’ve also tried to lsusb -v > /tmp/lsusb.log to only get:
Couldn't open device, some information will be missing
Since nothing was mounted, I plugged the NVME_USB to my PC instead to make sure the folders are still there with the .stfolder and sure was. I plugged it back to the RPi and told it to reboot, to see the syncing resuming and making a reappearance when running df -h.
The big data folder is currently re-syncing but what else can I do to figure this out? I’m not comfortable with a system that connects/disconnects whenever it wants to since I’m at the risk of losing data to corruption like it has already happened with a few files, but luckily they weren’t essential.
The error above most often coincides with reset errors in the kernel messages (viewable via the dmesg command).
Since the powered HUB removes the risk of insufficient power for the drive, it’s more likely an issue with the drive itself than the RPi.
The main benefit of using LABEL=NVME_USB or UUID=66C68DC7C68D97C7 is that it avoids the needs to keep track of which device path (e.g., /dev/sdxx) a particular drive is currently on, so it wasn’t necessary to switch from LABEL to UUID.
There are several reasons why the USB drive would have shifted from /dev/sda1 to /dev/sdb1, but it would unfortunately take too long to explain in a meaningful way.
That’s a normal message. For your particular setup, it means that the volume wasn’t mounted at boot time so there’s nothing to unmount.
Did the ls /mnt/NVME_USB return a list of files/directories?
Keep in mind that /mnt/NVME_USB is a directory. Files and subdirectories can be created there without the USB drive connected. It means that ls /mnt/NVME_USB will succeed regardless of if the USB drive is connected. When the USB drive is mounted, it temporarily occupies /mnt/NVME_USB, effectively covering up any content that already exists there. As soon as the USB drive is unmounted, the underlying directory contents are visible again.
It’s normal because lsusb tries to probe all USB devices, but things like USB hubs are “dumb” devices that don’t usually return any info.
stdout vs. stdin vs. stderr
stdout = standard output
stdin = standard input
stderr = standard error
lsusb -v > /tmp/lsusb.log redirects the “standard output” from lsusb -v into the file /tmp/lsusb.log. But because error messages are often sent to “standard error”, they appear on the console instead of being added to the output file.
Try this:
This command…
ls -l filedoesnotexist.txt
… should return the following error (unless for some coincidence you actually have a file with that name ):
ls: cannot access 'filedoesnotexist.txt': No such file or directory
Now let’s redirect stdout to /dev/null (basically a system black hole)…
ls -l filedoesnotexist.txt > /dev/null
… which makes no difference:
ls: cannot access 'filedoesnotexist.txt': No such file or directory
But if we now redirect both stdout and stderr…
ls -l filedoesnotexist.txt > /dev/null 2>&1
The power of Unix/Linux. There’s also “standard input” (aka., “stdin”), which as the name implies, has to do with input into a command, but I’ll leave that for a Google search.
Is your PC also running Linux? Or did the filesystem change on the USB NVMe? Or did you use a tool for accessing ext4 volumes?
When the USB drive stopped responding on the USB port, Linux wouldn’t have been able to access its contents. Then Syncthing in turn wouldn’t have been able to detect /mnt/NVME_USB/.stfolder.
So far the sequence of events all make sense. It’s easy to get derailed when diagnosing issues like this because it’s a combination of hardware and software that’s in play.
I still think that the stability of the USB NVMe drive itself hasn’t been verified.
Do you have another NVMe enclosure or a PC with a compatible slot for running Samsung’s diagnostic software?
I had previously formatted the drive to NTFS on a Windows PC to subsequently use it on the RPi since I had file length issues when previously using Ext4.
If anything, I’m glad to hear it’s hasn’t been has bad as it sounds.
This is the only enclosure I have and no other available slot other than the one on this USB.
I’m guessing Samsung Magician Software is the diagnostic tool you are referring to? It doesn’t seem to be compatible with my drive according to the Supported Models section. Otherwise, I could still run it as long as this would be ok to be on the enclosure.
That’s very odd because the max file length on ext4 is 16 TiB. Since your NVMe SSD is 512GB, there shouldn’t have been an issue. This adds to my earlier suspicion that there was a filesystem corruption issue.
So now that the drive volume is formatted as NTFS, that’s something that needs to be accounted for on the RPi when used with Syncthing (ext4’s time stamp resolution is 100x more granular than NTFS).
The other wrinkle is prior to Linux kernel 5.15, NTFS support was via the ntfs-3g package (a FUSE module). The extra layer adds a performance hit when dealing with lots of small files, though that might or might not be an issue depending on your particular needs. With 5.15 and onward, there’s a kernel module option that sits alongside all the other supported filesystem types (RasPiOS, based on Debian 11/Bullseye, uses Linux kernel 5.10).
Nope, it’s all perfectly normal. It’s simply a matter of methodically ruling out each possible culprit.
Yup, Samsung Magician Software.
One wrinkle is that some enclosures have USB bridges that are opaque. To test for it, try the following command on the RPi assuming that your USB drive has been assigned /dev/sda:
Sorry, I meant I had issues with the filename length where I guess the combination of filename and path length was a bit much and was resulting in errors. I wouldn’t mind using Ext4 instead but at least I haven’t had this error again since moving to NTFS.
I had to install the tool which output the following:
sudo smartctl -a /dev/sda
smartctl 7.2 2020-12-30 r5155 [aarch64-linux-6.1.21-v8+] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Number: SAMSUNG MZVLB512HAJQ-000H1
Serial Number: S3WTNX0KA18357
Firmware Version: EXA72H1Q
PCI Vendor/Subsystem ID: 0x144d
IEEE OUI Identifier: 0x002538
Total NVM Capacity: 512,110,190,592 [512 GB]
Unallocated NVM Capacity: 0
Controller ID: 4
NVMe Version: 1.2
Number of Namespaces: 1
Namespace 1 Size/Capacity: 512,110,190,592 [512 GB]
Namespace 1 Utilization: 109,647,650,816 [109 GB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: 002538 8a81b791dd
Local Time is: Fri May 12 23:07:34 2023 BST
Firmware Updates (0x16): 3 Slots, no Reset required
Optional Admin Commands (0x0017): Security Format Frmw_DL Self_Test
Optional NVM Commands (0x001f): Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Log Page Attributes (0x03): S/H_per_NS Cmd_Eff_Lg
Maximum Data Transfer Size: 512 Pages
Warning Comp. Temp. Threshold: 81 Celsius
Critical Comp. Temp. Threshold: 82 Celsius
Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 7.02W - - 0 0 0 0 0 0
1 + 6.30W - - 1 1 1 1 0 0
2 + 3.50W - - 2 2 2 2 0 0
3 - 0.0760W - - 3 3 3 3 210 1200
4 - 0.0050W - - 4 4 4 4 2000 8000
Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 0
1 - 4096 0 0
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 29 Celsius
Available Spare: 100%
Available Spare Threshold: 5%
Percentage Used: 0%
Data Units Read: 6,306,410 [3.22 TB]
Data Units Written: 5,823,019 [2.98 TB]
Host Read Commands: 91,764,099
Host Write Commands: 127,216,369
Controller Busy Time: 620
Power Cycles: 1,136
Power On Hours: 533
Unsafe Shutdowns: 840
Media and Data Integrity Errors: 0
Error Information Log Entries: 198
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 29 Celsius
Temperature Sensor 2: 31 Celsius
Warning: NVMe Get Log truncated to 0x200 bytes, 0x200 bytes zero filled
Error Information (NVMe Log 0x01, 16 of 256 entries)
No Errors Logged
Big sync just finished with 9 files failing with input/output error so still not feeling 100% confident about the fidelity of this setup.
Ext4 has a 255-byte limit on a filename, but that’s just for the filename. There’s no defined limit on a pathname, although software can impose their own limits.
The Linux kernel by default caps it at 4,096 characters. From /usr/include/linux/limits.h:
#define PATH_MAX 4096 /* # chars in a path name including nul */
On Windows 7-11, the API limits pathnames to 260-characters (there’s Unicode support that extends it, but with caveats).
That’s great. I’ve unfortunately had to work with USB bridges that didn’t allow direct access to an attached drive.
Try initiating a long self-test:
smartctl -t long /dev/sda
The same command earlier can be used to check the status:
smartctl -a /dev/sda
Check the output from dmesg to as soon as the input/output error occurs to see if the drive resets on the USB port.
You’re welcome. I’ve been a fan of the Raspberry Pi and Linux for years, so always happy to help fellow users whenever I can.
I was hoping that the S.M.A.R.T self-test results were more detailed, but at least it didn’t fail. The 840 unsafe shutdowns across 536 power on hours isn’t exactly reassuring for a drive that’s going to be used on a server.
Reviewing the earlier and most recent dmesg output files, it appears that the problem is the active state power management feature of the USB bridge used in the Inateck NVMe enclosure.
The RTL9210-CG is advertised by Realtek as being one of the most power efficient bridges available. There’s a 10-minute preset where it powers down the link – the USB drive disconnects – after no traffic appears.
I found several NVMe enclosures from Orico, Anker and others using the same RTL9210 chip. At first I thought it was possibily a compatibility issue with the RPi’s USB chipset, but posts from users on Windows, macOS and Linux made it very unlikely:
The last one above is the most detailed. Some users have flashed the firmware, while others were able to override the 10-minute idle disconnect.
Flashing the firmware is definitely worth trying, but not to be taken lightly because you could end up with a paperweight. Ideally the firmware update would come from Inateck but none have been publicly released that I’m aware of.
But before doing a deep dive into hacking the RTL9210, a simple test/workaround would be to use a cronjob on the RPi to touch the USB drive every 5 minutes or so to ensure that it doesn’t go to sleep.
Save the following block of text to a file on the RPi:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# Min Hour DoM Month DoW Command
*/5 * * * * ls /mnt/NVME_USB > /dev/null 2>&1
Only the last line above is important. Lines starting with # are just there to help make sense of the fields.
The command above simply does a directory listing of /mnt/NVME_USB every 5 minutes 24x7x365 and tosses out the results. Its only purpose is to push/pull some data across the USB connection to keep it active.
If the configuration file is named “cronjobs.txt”, to load it…
crontab cronjobs.txt
To list the current configuration…
crontab -l
To remove the current configuration…
crontab -r
(Because crontab can be used by any user on a Linux system, there’s no need for “sudo”. Just use it as user “datapi”.)
This NVMe sticks comes from my IT department since they were going to get rid of it but because it still worked it was donated to me instead. Other than this, I have a 128GB stick that was originally on my laptop but since it was too small I swapped it. If push comes to shove, I can always look into swapping these but so far it looks like the issue comes from the enclosure controller itself?
I guess this on paper makes sense, especially when the RPi can only output so much power, but it looks like it might not be fit for this purpose?
Done this. However, I still keep getting the error below, and now with 2289 failed items as input/output error.
Error on folder "syncpi_backup" (syncpi_backup): folder marker missing (this indicates potential data loss, search docs/forum to get information about how to proceed)
Even after rebooting the RPi failed items problem persists.
Afterthought… would it be more reliable to setup a network drive in the RPi for my local backups (syncpi_backup) and limit Syncthing to folders that are intended to be shared with other devices? Not that other folders don’t hitch as well, but the backup folder does seems to be the biggest offender.
The S.M.A.R.T. results said that there has been less than 3TBW, and Samsung drives can easily handle more than 10x that, so it does appear that the USB bridge in the enclosure is the problem.
Due to lack of firmware updates from Inateck, one option is to get a different enclosure for use with your RPi server. Other manufacturers who also use Realtek’s RTL9210B chip but provide firmware updates and configuration tools:
Yup, and laptops too. The enclosure likely wasn’t intended to be used as a semi-permanent storage device on a server.
Too bad the cronjob kludge didn’t work. When the drive disappeared Syncthing couldn’t find its /mnt/NVME_USB/.stfolder.
This requires a three-part answer…
First, I hope the plan wasn’t to use Syncthing for backups.
Second, splitting up local backups from Syncthing unfortunately wouldn’t resolve the NVMe enclosure issue. If both the network share and Syncthing’s folders are still on the USB drive, you’d end up with two network services going offline when the USB drive disappears.
Third, at the same time, separating local backups from Syncthing’s sync folders is a great idea. Use a dedicated backup utility such as Duplicacy, Restic, etc.
I’m all for using/reusing electronics for as long as possible, but the USB NVMe enclosure’s top transfer speed is much faster than the RPi can keep up with.
And since it doesn’t sound like you’re going to have hundreds of Syncthing devices syncing to your RPi, a $25 512GB 2.5" SSD in a $15 SATA-to-USB enclosure might be a better deal than purchasing another NVMe enclosure (the bonus is that a SATA SSD requires much less power than a NVMe stick because that extra speed comes at a price).
Then use the Inateck enclosure, 128GB NVMe and 512GB NVMe for offline storage as part of a complete 3-2-1 backup plan.
On my own server, I use a combination of Samba + Syncthing + Duplicacy:
Samba for file sharing when there’s an always-on network connection.
Syncthing for replicating shared files between various desktop and mobile devices when a network file share isn’t practical.
In this scenario, I already have two 2.5" SATA Enclosures: a Sabrent EC-UASP and a Inateck FEU3NS-1E (sorry, that’s the closest to a product page I could find). The Sabrent enclosure seems to at least provide firmware, so in this scenario I guess I would probably be better off with using it over the Inateck.
Could this SATA enclosure suffer from the same disconnection issues the NVMe has been having, or would this simply be expected to work?
Not historically, but I do confess I was somewhat considering it for this setup
I normally run robocopy to a backup external drive I plug in from time to time but a more robust solution might not be a bad idea.
Whether either the Sabrent or Inateck SATA-to-USB enclosures would just work out-of-the-box, the combination of SSD/HDD + enclosure USB bridge chip + host USB chip + firmware always adds some uncertainty.
Inateck doesn’t even seem to acknowledge that its FEU3NS-1E exists. I wasn’t able to find any details about the USB bridge chip, but various user benchmarks showed poor random write speeds even with a SSD. As far as aesthetics, the curved/rounded top and bottom shell is a poor design because a HDD would cause the case to move as the drive platters spin up.
According to Sabrent’s user forum, the newer versions of the EC-UASP use a JMicron USB bridge chip. Users have reported needing a firmware update to fix stability issues in some setups, but otherwise it looks like a good enclosure.
So between Sabrent and Inateck, the Sabrent seems to have better manufacturer support and design for your intended use case.
Over the years I’ve built up quite a collection of so-so USB storage devices. For storage that doesn’t need to be portable, I tend to favor USB docks instead of full enclosures. The Archgon MH-3507-U3A with a “ASMT 2105” USB bridge chip from ASMedia Technologies as been rock solid for me. I think it was released back in 2016 so it’s not widely available now, but is still available online.
You’re not alone. Quite a few posts on this user forum about that very topic.
Backup isn’t Syncthing’s intended function, but it can be twisted to work somewhat like one by using its send only, ignore delete, and versioning features.
Any backup is better than no backup.
Robocopy is useful as part of a backup plan, but lacks robust error checking, backup verification and newer features including encryption, deduplication, and file compression (Robocopy only supports data transfer compression across a network).
.
):
