Missing Syncthing directory, missing data on devices

I was a happy Syncthing user until the moment I encountered the mysterious disappearance of my data. Now I am trying to figure out what happened.

This is my setup:

  • A - Android phone, Galaxy Note 8, running its standard firmware, with Syncthing v1.2.0
  • B - Linux Mint 19.2 with Syncthing daemon 0.14.43-ds1
  • C - Raspberry Pi

I used Syncthing to synchronize the data produced by several apps used on the phone:

  • A diary made of plaintext files kept in /storage/emulated/0/Diary, configured to send and receive.
  • Note-taking app that keeps the notes in plaintext files in /storage/emulated/0/Documents/markor, configured to send and receive.
  • Some voice recordings made with the phone, saved to /storage/0b10-7cc5/Voice recorder on the external MicroSD card, configured to send only.

On the desktop the stuff was in ~/syncthing containing the following subdirectories:

  • Diary
  • Notes
  • Voice recordings

Here is the sequence of events, to the best of my knowledge:

  1. I clicked Syncthing’s icon on the desktop, where I use the GTK front-end, and see the red marker and the error “folder path missing”
  2. I check ~/syncthing and it was really gone. I figured that maybe I accidentally deleted it, or I accidentally dragged and dropped it onto another directory. I didn’t find it anywhere else on my disk, so I concluded that I must have deleted it without realizing it.
  3. Without checking the data on the phone, I tell myself “no problem, I’ll just recreate it and let the Syncthing get it off the phone”. I recreate ~/syncthing and the subdirectories in it.
  4. I restart the Syncthing daemon

After waiting for a while, I see that only 2 of my notes from /storage/emulated/0/Documents/markor on the phone are there, there are no voice recordings and no diary records.

I then checked the phone, and saw that what was on the desktop is really what’s on the phone; in other words - the directories were practically empty.

Fortunately the RaspberryPi contains some of the diary entries, so not everything was lost. However, I am still facing a big problem - I don’t understand how this could have happened.

  • I am the only person using my devices
  • I might accept the possibility that there was an accidental deletion of ~/syncthing on the desktop, but it is unlikely that I also accidentally deleted data on the phone
  • It is also not likely that the phone apps somehow screwed up, because they come from different vendors, and they used both, internal and external storage - everything is gone

My trust in Syncthing is now undermined, so I would like to ask the community for help in understanding what could have lead to such an outcome.

You should always back up.

Whatever process deleted/moved ~/syncthing, could have been a recursive delete/move, meaning it deletes file by file at the deepest directories before actually getting up to ~/syncthing and deleting/moving that directory.

Given you are running on Linux, we have filesystem notifications enabled, so we probably noticed those deletions/moves (which are just deletes if they leave the managed tree) and told other devices about them, so other devices followed the instructions and deleted the same items locally.

This process probably took some time, but there was probably not enough time gap between ~/syncthing directory disappearing with it’s markers and the last few files that are left (that you’ve mentioned), so we’ve assumed the folder is unhealthy and left the last few files around.

Recreating that directory is not enough either, as you technically need to create the market for it to work. If you created the marker by hand, you essentially told syncthing “this folder is ok, the market is here, but there is no content, hence delete everything everywhere”.

From what I understand Syncthing is not managing ~/syncthing directory (it’s managing it’s subdirectories based on your description), so I am not sure how syncthing can be at fault here, and why your trust can be undermined. It’s simply following instructions. You should understand what happed to ~/syncthing, and distrust whatever caused it to disappear.

Thank you for the explanation and the quick answer. If I understand you correctly, this is a hypothetical timeline of what probably happened:

  1. something deleted ~/syncthing on the computer, starting with the “leaf” entries of the subdirectories
  2. the Syncthing daemon noticed that the “deep” files started disappearing (e.g. /~syncthing/Diary/2019/May/01.txt) and began notifying other clients that these files should be removed
  3. Syncthing on the Android faithfully removed the files
  4. At some point the parent directory itself, ~/syncthing was gone, so the Syncthing daemon figured out something is odd and stopped notifying other devices about file removal operations

Thus, whatever files remained on the Android device were files that were still there when Syncthing on the computer reached step#4.

It makes sense, but it also is a bit puzzling, because the amounts involved were rather small:

  • the diary had around 50 files
  • there were around 10 files in the Notes/ directory
  • and all of this is plaintext data that was under 200 KB
  • the voice recordings were at 150MB, there were around 15 of them

In other words, I would expect this to happen in a blink of an eye, i.e. the time-span between steps #3 and #4 would be just a few seconds. Is it still a realistic explanation?

Another thing that doesn’t fit into the picture is the disappearance of the voice recordings on the Android, because the directory is marked as “send only” on the phone (and “send and receive” on the computer). If my understanding of the model is right, then this means that the deletion must have been initiated by the Android. Is this correct?

Correction: it turns out that the voice recordings are on the Android. The UI showed an “out of sync” label and an “override changes” button for this particular directory. The files were on the SD card, I presume the client notified me that it was out of sync with the other devices, so clicking “override” made my local version the canonical one.

So it boils down to this: how fast will Syncthing on Linux propagate the news about deleting ~60 files of a total size of < 200KB?

One minute after detecting them with filesystem notifications, by default.

Regardless, ~/syncthing is gone, thats the most likely cause, so I suggest you understand what happened there.

If the devices haven’t been restarted since it happened, there should be a button in the web ui showing the last 100 or so changes and who caused them and what time.

Also, I am still puzzled how you made it work by creating directories but not creating the marker manually.

I haven’t read anything else, but these two versions are not compatible, crashes (panics) will happen.

1 Like

For reference:

I thank everyone for taking their time to read my story and provide insight into what happened. As a software engineer, I understand the contributing factors. However, when I put my usability hat on, I am reminded of the joke about the suit [1].

I believe several things could be discussed to potentially improve the user experience of Syncthing.

  • A nice interface is forgiving, i.e. it allows one to recover from an error. Syncthing’s default is unforgiving. The folder creation wizard does bring it up, however the option is called File versioning - there is no immediate link between this option and possible loss of data, no link with the “trash can” metaphor that users are accustomed to. I only connected the dots now, after taking some damage, I wish I did it earlier.
  • On Android and in the Web-UI File versioning doesn’t have a one-line explanation of what this option is for. For example, Folder type is Send and receive by default and there is a label under it that says “The folder will both send changes to and receive changes from remote devices”. There is no analog string about the file versioning option, it could say something like “Define what happens when a file is deleted locally or on a remote device”. Instead of “None”, the current default, it could say “No versioning, delete immediately”. This might attract attention and prompt people to RTFM. The status quo is that only users who click around to tinker with the software will stumble upon it; some do this, but not everybody.
  • In the Web-UI it is possible to create a folder without ever going through the File version tab. As soon as I type the name, I can press Save and get going. This is potentially dangerous, because this important use case is overlooked. The status quo attitude is “users are expected to know what they’re doing”, we can assist them in this, by nudging them into the right direction (e.g., change the default, ensure they go through the other tabs, etc).
  • A good approach is to place the burden of deciding on the shoulders of those who are best equipped to handle the decision. The status quo is that the consequences of the defaults fall onto the shoulders of inexperienced users (because the experienced one enable versioning). If the default were different, the burden would be on the shoulders of the experienced users (an extra click for them to turn it off), while the clueless newbies would enjoy the extra protection without even being aware of it.
  • The remark about version compatibility/stability issues is somewhat worrying. I only became aware of it on this forum; Syncthing was operating in a “set it and forget it” mode on my systems, I was satisfied with the result and wasn’t following the news. Perhaps when such backwards compatibility situations occur, the newer version could warn the user through the UI about it? My Syncthing client on the Android didn’t say anything about it, although it is always visible in the notification area and there is always a “Syncthing is running” entry in the pull down menu. In other words, the software is displayed in a prominent place and it is getting its share of attention - perhaps this place can also be used to inform users about important actions such as this one?

A final note is about the handling of cases like the one I described. I understand the mechanics of why it happened, and although I miss my data, I don’t feel any anger (-:

In the field of human-computer interaction, several types of errors are distinguished:

  • mistake - when the user had the wrong plan, they carried it out correctly, and they got a bad outcome;
  • slip - when the user had the right plan, but they carried it out incorrectly (i.e. accidentally pressed the adjacent button, mistyped, etc.).

What happened to me was not a mistake, because I think I have a fairly accurate mental model of how Syncthing works. It must have been a slip, perhaps when I was selecting a bunch of directories to delete, my finger moved a bit too far on the touchpad, taking my Syncthing directory with it.

The software can’t figure out what it is that the user really wanted, and there’s no way for it to anticipate that the “file deleted” notifications it receives should be ignored, because they are actually a part of a recursive delete that will take down everything, including the .stfolder marker [2]. However, the software can be more forgiving.

[1] The suit joke, taken from the Internet and adapted to the UI context:

 A man goes to a tailor to try on a new custom-made suit.
 The first thing he notices is that the arms are too long.

"No problem" says the tailor. "Just bend them at the
elbow and hold them out in front of you. See, now it's fine."

"But the collar is up around my ears!"

"It's nothing. Just hunch your back up a little... No,
a little more... That's it."

"But I'm stepping on my cuffs!" the man cries in desperation.

"Bend your knees a little to take up the slack. There you go.
Look in the mirror--the suit fits perfectly."

So, twisted like a pretzel, the man lurches out onto the street.
A passer-by asks him "hey, is that a new suit?", "yes it is",
our guy replies.

"Wow, it must have been made by the most skilled tailor in the
world!".

"How so?", our guy asks.

"Well, if he could make a suit for a cripple like you, I can
 only imagine what he can do for me!"

[2] If it is possible, maybe it should be done that way.

2 Likes

If it happened that way, it’s a bug: When the .stfolder marker disappears, scanning and syncing should stop immediately and no data loss occur. What could have theoretically happened, but seems unrealistic, is that the deletion was really slow and happened depth first, so Syncthing noticed deletions faster (by default 1 min delay) than they occurred and especially, before the .stfolder in the top-level was deleted.

  • About forgiving defaults and UI:
    At least in my view it is the idea of Syncthing that defaults work fine, i.e. if the user just configures what they must (folder ID, path and who to share with) they are fine.
    With regards to versioning it might be debatable that some form of versioning by default is better than none, as it adds some data safety (after all, that’s Syncthing’s goal 1). However it is detrimental to goal 4: Being automatic. When I delete something, I really mean to delete it. That’s also the notion why I personally do not agree that not having versioning is a form of data loss: Any deletions occurs due to user action on another system. If there is no other mechanism (filesystem trashcan or backups) that leads to the removal of the data regardless of whether Syncthing is there or not. So in my mental model this kind of data safety must happen outside of Syncthing, and Syncthing’s versioning would just be an additional layer of safety (thus optional). Again, that’s my personal opinion, I might be an outlier here.

As to missing strings and generally UI could be nicer: I don’t think anyone will debate you on that. Syncthing’s UI has grown organically and definitely does have issues. The problem is there’s no-one with the motivation and time (latter is huge) to overhaul and improve it. Small changes like a proposed additional text hint can obviously easily be added (either propose a PR yourself or write up a very clear explanation of what change you’d like and where it goes etc., and hopefully someone (maybe me) will take care of it :wink: ).

Agreed. In hindsight knowing there’s quite a few users on debian (and derivatives) that will stay on ancient versions for what feels like an eternity should have been a deciding factor to at the very least implement some warning in the UI, or go with what Audrius always proposed: Make a clean cut with an incompatible version bump. Well in hindsight we are always smarter (and by the way I personally would a priori be a bit worried if I’d entrust my data to a program where instances run on 0.14.xy and 1.2.0, doesn’t that reek of potential problems?).

I tried to replicate the problem by creating a shared directory and syncing it with the Android device. The folder structure was similar to that of the Diary directory. After making sure the phone has it, I deleted it on the computer (using the nemo file manager of Linux Mint).

  • The local Syncthing client complained about the missing directory
  • Everything was intact on the phone

I tried replicating this with midnight commander, perhaps it does a depth-first delete, and I use this tool every now and then, so it could have happened. However - the outcome was the same.

One caveat is that in this experiment I used Syncthing 1.2.1, while it is possible that the problem existed with the older version. I cannot find that binary any longer, is there an repository of old versions’ DEB files, so I could try it with 0.14.43-ds1?

This is puzzling, because if the problem was caused by user error, then I would have deleted it with nemo or mc - but neither of them lead to the previous effect in this [not identical] test.

I’d be worried too if that happened right away, but this is an issue that evolved naturally. I installed Syncthing on my devices on the same day and tinkered with them. At some point there were updates on Android, which I installed - but nothing like it was happening on the Linux desktop. So how did the program versions diverge? One step at a time :slight_smile:

As for the strings, I found the json file with the texts, but it seems that it is not a matter of a missing string, but rather a matter of a UI element that is absent in principle. I will look at the code to understand what is going on and how UI elements are shown there, I’ll get back when I have questions about it.

First of all, thank for trying to reproduce, I am happy you didn’t succeed though (I am reasonably sure that should be understandable as an amicable statement :slight_smile: ).

I thought it was more benign than what happened to you, but I do remember some problems with disregarded folder error states. So it might be possible, I don’t remember specifics though.

It’s debian you’re talking about, of course there’s a place for old stuff: http://snapshot.debian.org/ :wink:

Very nice, don’t hesitate with any questions!

2 Likes

Rolling back to the old version still failed to replicate the results. Good news on one hand, bad news on the other - I still don’t know what happened to the files.

I issued a pull request https://github.com/syncthing/syncthing/pull/5988, my questions are:

  1. What is the best place to discuss the wording of the string? I wrote what I thought is a good one, but this is certainly something that could benefit from second opinions.
  2. Is the Android version using the same data as input for the strings automagically, or should I find a similar part in that code?
  3. I added the strings right into the JSON, then joined the Transifex team - the strings aren’t there, is there something else I need to do to have them show up and become available for others?
  4. I didn’t test the update because I am clueless about how to start up the web-ui locally and test my changes. I didn’t find a section about it in the docs. There’s one about building the binary itself with Go, but I just need to debug the web-ui. Is it possible to start up the web interface without going through the step of building everything else?
  1. The PR - that’s discussion specific to the actual change in code.

  2. No. Except for the “Web UI”, which is the exact same thing, the android wrapper is completely decoupled. If you want to improve that, you need to dig into https://github.com/syncthing/syncthing-android

  3. You are a bit ahead there: Transifex only gets strings later (after committing to master). You only need to add the string to source, not to jsons. Then you run go run build.go translate to add it to the appropriate json that will then later get pushed to transifex (where you can add the Russian translation). You wouldn’t even need to run the translate command, we do that occasionally too, but it’s good practice.

  4. You can copy the gui folder that you modified to your Syncthing config directory. (It’s in the FAQ, but if I hadn’t known that it existed, I wouldn’t have found it either: https://docs.syncthing.net/users/faq.html#i-don-t-like-the-gui-or-the-theme-can-it-be-changed).

You can also set the environment variable STGUIASSETS=/path/to/your/gui before launching Syncthing and it will serve the GUI files from there, just change and reload in the browser as normal.

2 Likes