Files lost

Support
1

Recently I switched from Linux OS distro and I start losing files. They just disappeared. 70% of the files in a folder. I needed a recovery tool to recover them. They did not appear in the trashbin or the SyncThing restore function or elsewhere.

At one device I noticed that the Veracrypt drive I mount shows an empty folder at times, and I need to re-mount it. However it has the strange behavior it shows empty folder structure but no files, so this could likely be the cause.

I saw a similar post of a ‘dropped’ USB connection from a user also having lost files: Syncthing deleted 67000 files, how do I restore?

Or maybe it is an issue on an Android device. Causing it to delete the files, however then I should have a copy in the syncthing restore, which was not the case.

I’d like to write a python script to (email) alert me each time SyncThing detects 50+ files where deleted.

What do I turn on in the debug logger to see and log when files are being deleted? Currently my log file does not seems to provide me with any clue or information on the issue of the lost files.

Is there anyone who created a similar feature/cript in the past, to be alerted if 50+ files are deleted within a 24 hour period (or at once)?

Any ideas or suggestions?

2

(It’s not at all clear which device lost the files, so if you could clarify as distinctly as possible it would help a lot.)

  • As in you switched away from Linux or you switched from one Linux distro to another? Which Linux distro(s)?

In Syncthing, you have to enable one of the versioning modes for there to be recovery files in the trashbin or .stversions subdirectory.

Without more details, it’d difficult to diagnost, but right now it sounds like there’s a hardware issue.

This is most likely unrelated to the missing “70% of the files in a folder”, unless you’re referring to your VeraCrypt volume.

Note that encrypted volumes – such as those for VeraCrypt – can be thought of as a disk image. To an OS, to Dropbox, to Syncthing and to other applications, the encrypted volume is just a file. Then, as you know, when that encrypted volume is mounted, the mount point / drive letter allows real-time access to its contents.

However, it’s generally not guaranteed that the file can be successfully copied while the encrypted volume it contains is in use (it’s like writing to a text file while something else is copying it).

Syncthing’s marker folder feature is designed to mitigate that type of potential problem whether it’s a disconnected USB drive, network share, or the file contents of an encrypted volume.

In order to help, will need a lot more context because the description jumped from Linux to VeraCrypt to Android in just three short paragraphs. :smirk:

3

I switched my Linux Distro from QubesOS to Pop OS.

Versioning in SyncThing is enabled.

What I believe that happened is that the veracrypt drive that was mounted was lost after the laptop went into sleep mode, while in Nautilus (file explorer) I could still see some of the folder structure. I assume that the .stversions or Syncthing Folder marker where no longer visible and may have caused the deletion, however still I cannot find an explanation for many things, how could I have lost the data and needed a data recovery tool to restore the files.

I mentioned Android as one of the synced devices is an Android device and I read that Android sometimes cause issues.

I wonder what clues I can find in the SyncThing log file? I looked at it but so far did not find any info leading to a clue about the data loss.

I’d like to write a python script to (email) alert me each time SyncThing detects 50+ files where deleted, so I am made aware of the problem and can investigate further, including in the SyncThing log files.

4

So the root of your Syncthing folder is the mount point of the VeraCrypt container?

Is the Android device the other Syncthing peer or are there other devices?

You’re going to have to share console output and/or screenshots because it’s still not clear how your Syncrhing is configured, how it relates to VeraCrypt, or even where the lost files were being synced to/from.

5

One of the shares in SyncThing is a folder of the VeraCrypt container.

The Android device is the other SyncThing peer which is set to ‘Send only’.

This would mean the files where lost or removed by the Android device, however if this was the case, on the Linux OS device, these files should be moved to the stversions folder right, for 180 days as I have set it, however this is not the case.

Therefor the issue is highly strange. As I count on those fallback mechanisms to work, which they normally do.

Looking at other backup solutions I could locate other Linux users using other backup solutions also suffering from similar problems.

I found a FAQ from Filen about missing / corrupted files and it states:

“Since in 99% of cases we are only informed about these problems by Linux users, please also make sure that your distribution is compatible.”

What I really need to cover this issue, is an ‘Unusual file deletion’ Alert, or Unusual Activity Alert so I am made aware of this unusual file deletion right there and then, and not much later on when I seek for a file that is no longer there. I understand in some cases a massive file deletion is desired, but an alert would then not hurt, as this is not a daily occurrence in most situations.

6

Note that Syncthing wasn’t designed and/or intended as a backup tool/solution (but it can be part of one).

But with regards to backup solutions, almost 70% of publicly accessible websites are running Linux, and it’s likely that most of them are using some kind of backup solution. If there was widespread file corruption I’m pretty certain companies would’ve switched platforms by now.

I’d like to see the link to that FAQ because the quote above is missing a whole lot of context…

One could accurately say that “99% of people who look up at the sky during the daytime say that it’s blue” – not that the remaining 1% think it’s a different color – they simply haven’t looked up at the sky yet.

So it’s also accurate to say that Linux users tend to be more tech-savvy, have the interest, and are comfortable enough with reporting any issues that occur.

It’s similar to how there tends to be more negative product reviews on shopping sites than there are positive reviews because customers who have a poor experience tend to be more vocal about it.

In any case, I think you’re misinterpreting that FAQ entry. The part about “please also make sure that your distribution is compatible” is important.

7

That’s a tricky setup to maintain…

The VeraCrypt mount point absolutely must be available at all times in order for Syncthing to properly sync the contents.

If you’ve got auto-dismount enabled after a preset idle time has elapsed, or another number of other things that result in the mount point disappearing, it’s going to cause problems.

I don’t see any reason why Syncthing cannot be used in combination with VeraCrypt if it’s done properly. I’ve got a collection of encrypted vaults (using a variety of tools) being synced by Syncthing and Rclone, and backed up by two different backup tools without having lost a single file in those vaults.

But up to this point it’s only been a general overview of your setup without enough details to diagnose much of anything.

8

I made a change to avoid losing the mount after sleep mode:

It worked for years without any problems on QubesOS but on POP OS linux distro problems started.

About Filen go to: Filen – Support and enter Corrupted / Missing files.

I was looking at Ente and they also mention not to use mounted drives as this gives a problematic user experience.

So now I consider to purchase a new SSD large enough to eliminate the need to mount an additional drive or drive image.

What could I look for in the log files to obtain more info. The log file goes back two month’s so maybe I can find some clue there?

9

For full context, this is what the “Corrupted / Missing Files” support post says:

If you notice with the desktop client that files are either corrupted or missing, make sure you have sufficient storage access rights, no network problems or the linked files are currently not used by other programs (Individual files can only be uploaded or downloaded in one go and cannot be continued if a file that has been started is paused. These skipped files are tried again at the end of the indexed list). Since in 99% of cases we are only informed about these problems by Linux users, please also make sure that your distribution is compatible. Currently the client runs verified on vanilla Ubuntu/Debian (GNOME). Other distributions may work, but problems with other modified, trimmed or otherwise handicapped distributions may occur, as indicated on the download page. If file types of less common programs cause problems, please contact us and we will include it in the upcoming update for a complete overhaul of the desktop client.

It’s a generic warning that corrupted and/or missing files can be caused by a number of issues including:

  • Not enough permissions to write the files, resulting in missing files.
  • Network issues, resulting in corrupted and/or missing files.
  • Files that are currently in use, resulting in corrupted and/or missing files.

All of the bullet items above are OS-agnostic, and has nothing to do with Linux in particular.

The second half of the paragraph is regarding software compatibility…

Filen’s desktop app for Linux is packaged in AppImage format. Generally speaking, an AppImage includes all of the library dependencies the app requires. However, the app ultimately also needs specific OS features that may or may not be available.

It’s like trying to run an app built for Windows 11 on a Windows XP system; or running an app built for Android 14 on Android 1.0 – there would be no guarantee that it works at all or functions perfectly if it does.

Although Linux is mentioned in the paragraph above, it’s not a statement about cause and effect that 99% of Linux users are seeing corrupted / missing files. It’s just a recommendation about Filen’s desktop app system requirements and what to look for when diagnosing potential issues when running Filen’s desktop app on a distro and/or heavily customized Linux distro.

That troubleshooting referenced above is about the lack of support for filesystem notifications in most network filesystems. It’s not about discouraging the use of mounted drives – in Linux, all drives are mounted (there’s no way to boot a Linux system without mounting some kind of storage, even if it’s virtual or a ramdisk).

Without filesystem notifications, Ente’s client app would have to rely on brute-force scanning for changes, causing more system load as the number of files increases.

So neither Filen’s or Ente’s support notes are actually applicable to the lost files issue with your Syncthing setup. The core problem is that Syncing files to/from a mount point for an encrypted container requires extra care during setup and use. It doesn’t matter if it’s Linux, macOS, Windows, Android, iOS, FreeBSD, etc., etc. because the same rules apply to all of them.

10

I assume an ‘Unusual file deletion’ Alert is not possible then?

Neither finding more info in the SyncThing log file?

11

Event and REST API documenation is a good place to start.

I’m not sure that watching for unusual file deletion patterns really solves the problem.

There are advanced debugging options in the log settings, but again, it’s focusing too much on the symptoms rather than the cause.

12

For the Veracrypt mounted drive I had made the mount directory path immutable using

sudo chattr +i /media/veracript1

Like mentioned before, I once noticed the mount was no longer available, but the folder structure and some files where still visible, only when you tried to open them they where not found, leaving me to conclude that this may have caused the data loss where SyncThing did no longer saw the files, but did see the placeholder folder and deleted them on a next instance.

For the ‘Unusual file deletion’ Alert feature, what log entries I could look for using the debugging options in the log file?

13

What you saw were remnants of the disk cache after the VeraCrypt mount went offlline. Generally the only processes that can still view the cached directory entries already have the directory open. New processes would only see the contents of the directory underneath the mount point.

Not unless /media/veracript1 also contained Syncthing’s .stfolder marker folder in addition to the .stfolder that would have been inside the VeraCrypt container (the paths would have to be identical).

Then, and only then, if the VeraCrypt volume was unmounted and Syncthing was allowed to rescan /media/veracript1, would the files have been deleted because Syncthing doesn’t know if you deliberately deleted the files or the mounted volume disappeared because the physical storage is abstracted by the operating system (as designed, and as it should be).

Isn’t researching/investigating an essential part of software development? :smirk:

If I was going to do what you’re looking to do, I’d be using Syncthing’s Event and REST APIs, but if you really want to parse Syncthing’s runtime log instead, I’d consider the filesystem event watcher.

14

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.