# Understanding data-corruption and recovering from \`.stversions/\`
**URL:** https://forum.syncthing.net/t/understanding-data-corruption-and-recovering-from-stversions/24191
**Category:** Support
**Created:** [April 16, 2025, 5:13pm UTC](https://forum.syncthing.net/t/understanding-data-corruption-and-recovering-from-stversions/24191 "2025-04-16T17:13:09Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@foo](https://forum.syncthing.net/u/foo)
#### Post date: [April 16, 2025, 5:13pm UTC](https://forum.syncthing.net/t/understanding-data-corruption-and-recovering-from-stversions/24191/1 "2025-04-16T17:13:09Z")
A couple of days ago I discovered corrupted files within syncthing replicated folders. Some plain text files had their contents replaced with seemingly completely random gibberish binary data. Replication was of course disabled immediately.
My setup should basically be understood as having a central fileserver as hub with spokes going out to all “clients”. I do understand that is not exactly how things work, but this mental model has value. The central server has mirrored disks with enough terabytes to be configured keeping plenty of copies of older file versions, while the other devices have no `.stversions/` folders at all. It is an environment with mixed operating systems. Generally running the latest packaged version of syncthing, which likely was not the cause of files getting thrashed.
Running `find . | grep '.stversions/.*~20250414' | wc -l` tells me that there should likely be less than 86 corrupted files. A manageable amount to resolve, but prior to starting I ask; Would anyone have seen or sit on any scripts or tools around which helps to recover a bunch of backed up files from `.stversions/`? Maybe that question is pretty much answered in my next section.
Searching the forums I find _[Code share : Syncthing-Restore for .stversions folder](https://forum.syncthing.net/t/code-share-syncthing-restore-for-stversions-folder/11060)_ which looks helpful and _[restore deletion for a specific date from stversions](https://forum.syncthing.net/t/restore-deletion-for-a-specific-date-from-stversions/22454)_. There is also the post _[syncthing restore .stversions scripts](https://forum.syncthing.net/t/syncthing-restore-stversions-scripts/10188)_ which seemingly has attempted to research which stuff already existed, linking to _[Syncthing deleted 67000 files, how do I restore?](https://forum.syncthing.net/t/syncthing-deleted-67000-files-how-do-i-restore/9896)_ and _[Script for restore version files](https://forum.syncthing.net/t/script-for-restore-version-files/1842)_.
I have a strong theory of why corrupted files has happened to me, but am not fully sure. Thus it would be interesting, at least theoretically so, to have a way to determine the source of the corrupted files. The _[Data corruption](https://forum.syncthing.net/t/data-corruption/13818/2)_ thread claims it should be possible to discover the origin of data corruption using some API. Should this be understood as the `modifiedBy` property returned by calling _GET_ on the [_/rest/db/file_-endpoint](https://docs.syncthing.net/rest/db-file-get.html)?
As the database does contain the origin of files, would it be feasible and/or desirable to add the first seven characters of the device identifier to their filenames under `.stversions/` similar to how conflicting files get named? If not as core functionality, then at least through using a custom versioning script which I believe currently is not able to do so?
---
### Author:  [@tomasz86](https://forum.syncthing.net/u/tomasz86)
#### Post date: [April 16, 2025, 5:23pm UTC](https://forum.syncthing.net/t/understanding-data-corruption-and-recovering-from-stversions/24191/2 "2025-04-16T17:23:59Z")
Have you tried using the GUI? You can filter versions by date, and there is also a search box to look for specific filenames. You can restore files in bulk too.
---
### Author:  [@foo](https://forum.syncthing.net/u/foo)
#### Post date: [April 16, 2025, 10:04pm UTC](https://forum.syncthing.net/t/understanding-data-corruption-and-recovering-from-stversions/24191/3 "2025-04-16T22:04:34Z")
> [@tomasz86](#):
>
> Have you tried using the GUI?
Thanks for your answer and suggestion. But fiddling with files using something as opaque as a browser gui seems like a guarantee for data-loss in a situations like this. I prefer understand what is going on.
I ended up with this little script:
```sh
#!/bin/sh -eu
#
# Takes a syncthing folder id as its first argument, and then a list of backup
# files from under `.stversions/`. Queries syncthing for information of the
# origin of the current file version and asks the user whether to restore that
# file or not.
#
# Assumes syncthing is configured to expose its api on a unix domain socket,
# among other things. Do read and understand the code before even considering
# running it. It's short after all. Less than 50 lines!
_folder="$1"; shift
_gui="$( grep 'gui enabled' -A 2 "$HOME/.config/syncthing/config.xml" )"
_sock="$( echo "$_gui" | sed -n 's/.*address>\(.*\)<.address>/\1/p' )"
_api_key="$( echo "$_gui" | sed -n 's/.*apikey>\(.*\)<.apikey>/\1/p' )"
unset _gui
n=1
for _backup in "$@"; do
clear
_file="$( echo "$_backup" | sed 's/~[0-9]\{8\}-[0-9]\{6\}//' )"
_output="$( curl --silent -X GET -H "X-API-Key: $_api_key" \
--unix-socket "$_sock" "http://_/rest/db/file?folder=$_folder" \
--url-query "file=$_file" |
sed -n 's/.*modifiedBy.*"\(.*\)",/\1 /p'; file "$_file" )"
_answer=''
(
echo "$_output" | tr -d '\n';
echo '';
cat ".stversions/$_backup"
) | less
[-e ".stversions/$_backup"] || continue
printf '[%d/%d] Restore %s from %s (y/N) ? ' "$n" "$#" "$_file" "$_backup"
read -r answer
case "${answer:-}" in
'y' | 'Y')
echo "Restoring $_file"
mv ".stversions/$_backup" "$_file"
# The next line did not exist when I ran this, but must be.
# touch "$_file"
;;
*)
echo "Okay, keeping it"
;;
esac
unset _answer
n=$(( n + 1 ))
done
unset _backup _file _folder _output
```
When running it, I got the impression that everything worked. However for some reason syncthing isn’t picking up the changes. Not even after fully killing, restarting and waiting for a couple of hours after rescanning completed. It seems I need to explicitly touch every restored file. The moment I run touch, syncthing picks it up and replicates it, but until that happens that api endpoint keeps returning the outdated timestamp for the `modified` property rather than the correct value.
This sure seems like a bug to me. While I could understand if `rename()` calls for some reason are tricky to detect, but the full scan after a restart surely should detect the change. Right?
---
### Author:  [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [April 17, 2025, 12:19am UTC](https://forum.syncthing.net/t/understanding-data-corruption-and-recovering-from-stversions/24191/4 "2025-04-17T00:19:42Z")
If there’s a visible change to the file metadata (size, modification time, permissions), yes. Otherwise no.
---
### Author:  [@foo](https://forum.syncthing.net/u/foo)
#### Post date: [April 17, 2025, 6:19am UTC](https://forum.syncthing.net/t/understanding-data-corruption-and-recovering-from-stversions/24191/5 "2025-04-17T06:19:14Z")
This morning I have realized that there is a _/rest/folder/versions_ endpoint which can be used to return a huge list of file when called with [GET](https://docs.syncthing.net/rest/folder-versions-get.html) (it does not document taking any selection parameters) or be used to restore file(s) with [POST](https://docs.syncthing.net/rest/folder-versions-post.html). Would using this API have advantages over simply doing `mv …~YYYYMMDD-HHMMSS …`? Should that force trigger detection of the file change even if the scanner, or what you call it, would fail to do so?
> [@calmh](#):
>
> If there’s a visible change to the file metadata (size, modification time, permissions), yes. Otherwise no.
Thanks for clarifying how [scanning](https://docs.syncthing.net/users/syncing.html#scanning) works. I was indeed having the misconception that all files were rehashed with a frequency of _rescanIntervalS_. I [read](https://forum.syncthing.net/t/does-a-rescan-actually-rehash-everything/20978/2) that _“The only way to force a rehash of everything would to be reset the database”_ which definitely does not seem like something I would wish to do.
My corrupted files came from [fsck](https://en.wikipedia.org/wiki/fsck), which means the mentioned metadata properties appears to have remained constant. Still the corrupted files were detected as new and got replicated. However after restoring they do not get picked up. I fail to understand why. It can’t be explained by inotify, as syncthing was not running at the time of doing fsck. If wildly speculating, would perhaps changed inodes be a possible factor?
Both the laptop which corrupted the files and the server which fails to detect the restored ones run syncthing 1.19.2~ds1-1+b4, which admittedly is quite old even if being the “latest packaged” in the Debian world.
Understanding that this is not considered a bug which can be helped by me debugging the current state, I’ll update _mtime_ of all restored files in a few hours.
---
### Author:  [@foo](https://forum.syncthing.net/u/foo)
#### Post date: [April 18, 2025, 9:43am UTC](https://forum.syncthing.net/t/understanding-data-corruption-and-recovering-from-stversions/24191/6 "2025-04-18T09:43:27Z")
I would say a key issue with restoration of fsck-corrupted files really boils down to:
> [@foo](#):
>
> I have realized that there is a _/rest/folder/versions_ endpoint which […] be used to restore file(s) with [POST](https://docs.syncthing.net/rest/folder-versions-post.html). Would using this API […] force trigger detection of the file change even if the scanner […] would fail to do so?
My level of understanding is far from sound, but the feeling I get from reading the code is that even files restored using the API would fail to replicate, right?
From what I can see, [walkRegular()](https://github.com/syncthing/syncthing/blob/1fbd396ffa3fedf1217947963df8a4634c721b4c/lib/scanner/walk.go#L431) returns nil without sending any hash request is case the file is considered unchanged. Most relevant here is a call to [mustRescan()](https://github.com/syncthing/syncthing/blob/1fbd396ffa3fedf1217947963df8a4634c721b4c/lib/protocol/bep_fileinfo.go#L281). This is in the code path of the scanner.
When restoring a file, the exact code path traversed depends on the versioning strategy used. It essentially runs down to [restoreFile()](https://github.com/syncthing/syncthing/blob/1fbd396ffa3fedf1217947963df8a4634c721b4c/lib/versioner/util.go#L193) being called. This function seemingly merely restores the contents and metadata, relying on the scanner to pick up those changes. Something which will not happen if fsck merely corrupted the contents and not the metadata. Isn’t there some code missing in [RestoreFolderVersions()](https://github.com/syncthing/syncthing/blob/1fbd396ffa3fedf1217947963df8a4634c721b4c/lib/model/model.go#L2836) if syncthing are fully serious about being [Safe From Data Loss](https://github.com/syncthing/syncthing/blob/1fbd396ffa3fedf1217947963df8a4634c721b4c/README.md?plain=1#L19)? Some kind of side-channel message to set `FlagLocalMustRescan` high for the restored file? Possibly just an added `runner.ScheduleForceRescan(file)` inside of the loop?
Unless I’m mistaken, the current behaviour would suffice for cases where users, like me, don’t trust unknown software and do restoration themselves. However since an API and a web user-interface is provided, the functionality should preferably be rock-solid.
Am I onto something or completely wrong? Would my understanding be close enough that a patch as lined out in this post would be appreciated, or would another way of fixing the bug be preferred?
Could this thread be moved into development by a forum administrator? I don’t really know what is possible with discourse, but that seems like a better category in light of what has been discovered.
---
### Author:  [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [April 18, 2025, 11:09am UTC](https://forum.syncthing.net/t/understanding-data-corruption-and-recovering-from-stversions/24191/7 "2025-04-18T11:09:09Z")
For there to be a versioned file to begin with Syncthing must have noticed a difference and synced a file. Hence I think the versioned file will by definition differ somehow from the base file.
---
### Author:  [@foo](https://forum.syncthing.net/u/foo)
#### Post date: [April 18, 2025, 11:46am UTC](https://forum.syncthing.net/t/understanding-data-corruption-and-recovering-from-stversions/24191/8 "2025-04-18T11:46:26Z")
> [@calmh](#):
>
> For there to be a versioned file to begin with Syncthing must have noticed a difference and synced a file.
Yes, correct. Please see the first sentence of my initial post in this thread.
> [@foo](#):
>
> A couple of days ago I discovered corrupted files within syncthing replicated folders. Some plain text files had their contents replaced with seemingly completely random gibberish binary data.
However, for some reason that does not lead to the following being true:
> [@calmh](#):
>
> Hence I think the versioned file will by definition differ somehow from the base file.
Please see how my restoration has still not been detected after I ran `mv ".stversions/$_backup" "$_file"` which to the best of my understanding would be equivalent to the operation Syncthing does.
This thread is about what has actually happened to me, not about a hypothetical case.
Why the change got detected in the first place is the biggest mystery to me. Maybe it’s an intricate difference between how the filesystems treat something? The laptop runs ext2 (which obviously ought to have been ext3 or ext4, but that’s beside the point), the server uses btrfs. I would very much like to find the answer to why the data-corruption started to spread. Another imaginable theory is that the laptop were multiple versions behind and thus skipped a few when updating, but no. Out of the 86 changes of that day, 45 were corrupted files. Some of those had not otherwise seen changes for over a year.
Regardless of whether you might think it is required to add an explicit call to ScheduleForceRescan(), I’d say my experience proves that it likely is in some poorly understood but real corner case.
An interesting question to ask is; Would defensive coding hurt here? The main difference is that during the next scan an early boolean comparison would cause a rehash scheduling rather than waiting for comparing the struct, right? With that said, my understanding is that only tests forcefully trigger files for rescan right now. So maybe that functionality just isn’t considered production grade?
---
### Author:  [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [April 18, 2025, 12:00pm UTC](https://forum.syncthing.net/t/understanding-data-corruption-and-recovering-from-stversions/24191/9 "2025-04-18T12:00:41Z")
> [@foo](#):
>
> Please see how my restoration has still not been detected after I ran `mv ".stversions/$_backup" "$_file"` which to the best of my understanding would be equivalent to the operation Syncthing does.
Not necessarily. Syncthing also resets the mtime back to the original, depending on the versioning method.
Forcing a rescan won’t make any difference. What could make a difference is setting the force rescan bit on the file in the database.
---
### Author:  [@foo](https://forum.syncthing.net/u/foo)
#### Post date: [April 18, 2025, 12:54pm UTC](https://forum.syncthing.net/t/understanding-data-corruption-and-recovering-from-stversions/24191/10 "2025-04-18T12:54:11Z")
> [@calmh](#):
>
> Not necessarily. Syncthing also resets the mtime back to the original, depending on the versioning method.
So syncthing restores mtime back to the value of the backup, while mv preserves the same value. Or to be more precise, restoreFile() uses a thin wrapper to call [os.Chtimes()](https://pkg.go.dev/os#Chtimes) with the same sourceMtime value as the original file had. A value corresponding to [struct timespec st\_mtim](https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/sys_stat.h.html) documented to not be updated by [rename()](https://pubs.opengroup.org/onlinepubs/9799919799/functions/rename.html).
That sure seems like functionally identical to me.
We can probably disregard _thrashcan_ and _external_ for this thread, as they work very differently and would never be bitten by this bug.
> [@calmh](#):
>
> Forcing a rescan won’t make any difference. What could make a difference is setting the force rescan bit on the file in the database.
That bit corresponds to the `FlagLocalMustRescan` bool, right? Did I misunderstand more than runner.ScheduleForceRescan() being the way to do that?
---
### Author:  [@AudriusButkevicius](https://forum.syncthing.net/u/AudriusButkevicius)
#### Post date: [April 19, 2025, 10:17am UTC](https://forum.syncthing.net/t/understanding-data-corruption-and-recovering-from-stversions/24191/11 "2025-04-19T10:17:19Z")
Rescanning singular files will probably be grim. Best to set the bit and let the standard scan scoop it up in one go.
---
### Author:  [@system](https://forum.syncthing.net/u/system)
#### Post date: [May 19, 2025, 10:21am UTC](https://forum.syncthing.net/t/understanding-data-corruption-and-recovering-from-stversions/24191/12 "2025-05-19T10:21:54Z")
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.