File loss, how to debug

Twice recently I’ve created a new folder, added some files, and then when I check back the folder is empty. I need some help figuring out what happened.

The setup:

  • ~25 machines. a mix of windows, mac, linux, and 1-2 android devices
  • running for something like 4 years, adding machines occasionally
  • ~20GB spread over 30 or so different shares
  • mostly Syncthing is up to date, but certainly a few machines are running older versions. the issues involved machines I know were up to date
  • some files never change, other are added or change daily
  • there is one “central” server that’s always on. it’s a VPS
  • our internet quality is, to put it mildly, terrible. most of the swarm is behind the Chinese GFW and there’s not much we can do to improve this
  • no file versioning from syncthing (there is a separate backup system that runs on the central VPS)

1st issue:
created a new folder in an existing share, added a few files to it, shared that with a relatively new machine, and (20 minutes?) later the files were gone. Luckily I had copies elsewhere so I added them again and everything seems OK.

2nd issue:
created a new share, put a couple different folders inside it, shared it with our central server as well as an existing machine in the swarm. now checking back ~10 hours later the share is completely empty except for a lone .stfolder (nothing in it). lost about an hour and a half of work


  • I don’t think there were any android devices involved in these particular folders.
  • The central machine has backups made every 4 hours. The new share folder from the second issue is empty in all available snapshots.

I would appreciate some guidance on how to get to the bottom of this. I’ve looked through logs previously for other issues and cumulatively probably spent most of a day on it without ever really learning anything other than an appreciation for the verbosity of the logging system. At this point I don’t think I will be able to solve it myself. I do have access to config files and (probably) logs on all the machines involved. I don’t remember exactly when the first issue happened, but could track down exactly when the second issue happened in the logs, if they are available that far back.

I suggest you first narrow down the surface if error. With 20 machines it’s hard to understand what did what.

Suggest you check the recent changes log to see who caused the deletes, and if its always the same machine every time, investigate that.

Sorry, I would be doing that, I simply have no idea how to narrow the logs down to what I need to see.

There isn’t an option for “recent changes” in the Debugging Facilities tab on the Logs dialog in the UI. Can you go into a little more detail about how I could, for example, see file events for a particular machine? The ‘fs’ checkbox seems to be the only thing close, and that’s not saying anything related to file creations or deletions from what I can see.

There is a recent changes button in the ui, or -audit command line flag which then logs most file operations.

The information I need has scrolled off the end of the list in the UI. Looks like I’ll need to restart with -audit to see file changes.

I do have some raw logs… is it perhaps possible to re-construct this info from them? Ideally I’d like to avoid needing to lose data again before being able to debug this.

I don’t think there would be anything useful in the logs by default.

You could probably query the api to see who deleted some file, if you know the folder/path and it has not been restored since.

A few hours ago I re-created the files and added them to the folder again, so your suggested scenario isn’t available.

I can query the API. Is it possible to find any deletions in a particular folder? There should have been 0 deletions since it was created so learning that would be informative. I’m currently only seeing ways to list what’s there right now.

No, you can only query by file name, recent deletions etc would he in the recent changes dialog, for some definition of recent (and assuming you have not restarted the process).

I’m having a similar issue again. Can you point me to the logic that uses the API to generate the “Recent Changes” screen in the UI? I want to log that same data over a longer time frame. Searching for “recent changes” only finds the language translation files.

For anyone searching later, this seems to give you the events:

curl -H X-API-Key:YOUR_API_KEY "http://localhost:8384/rest/events/disk?limit=25"

And turning that into something a human would enjoy reading is done in globalChangesModalView.html

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.