Files deleted after update to v0.14.45-rc.2

That was the entire log from the panic. Apparently, there are several other logs in that directory. I can post them someplace in a zip if you think they’d be useful.

Yeah, drop it here as an attachment should work.

Any damage should be instantaneous on first scan after upgrade. We’ll put out a less dangerous rc3 tomorrowish.

@calmh - can’t upload as new user… file is here:

https://home.lunixlol.org/~scain/zomg-panics.zip

Thanks. I un-newed you in anticipation of that, but I’m sure the forum software can throw up other blocks instead. :confused:


I think I see the deadlock. Been in there since .42 I think. Root cause unknown, possibly index corruption - try running at least once with STRECHECKDBEVERY=0 to rebuild some of the index stuff… Not related to the other .45 misfortune.

1 Like

No new panics after that but of course that could mean zilch.

I have a setup with many machines some of which are Windows, but I saw this happen in one of the folders that aren’t shared with any Windows machine (all Fedora Linux).

Interestingly, it was between a machine using the release candidate and a stable one - and the syncthing stable machine had significant deleted directories, however they remained on the release candidate machine itself. I was able to revert to stable, clear the database and have it all resync, and it seems like no data was fully lost.

Just as a note, some people like me don’t have all machines running at all times, and turning them on might auto-upgrade them and cause further data loss on the last machine that might have had the full data for those people. So I really think it would be better to make a fix upgrade as soon as possible.

After upgrade to .45-rc2, I had to recover from backup, downgrade to .44 and -reset-databases on two nodes (another two nodes had not been affected because they were not online and not updated). GUI on affected nodes showing large difference between local and global file count (both below actual) although also reporting folders were up-to-date. Edit: Forgot to mention, devices are RaspberryPI3 (Mate 16.04) and Linux laptop on Ubuntu 17.10. All up-to-date now on .44.

I think I hit this same bug.

Debian install is using ext4. Alpine is using a tmpfs for testing. I do not have anything in the ignore list.

I had been running v0.44.14 for a day or two, both folders fully synced before I compiled from source.

Both the 32bit and 64bit versions were build on my 32bit debian machine.

Debian:

$ uname -a
Linux linux 4.14.18 #22 SMP Thu Feb 8 15:50:19 EST 2018 i686 GNU/Linux
./syncthing --version
syncthing v0.14.44+38-g2bbd2d6e "Dysprosium Dragonfly" (go1.9.3 linux-386) user@linux 2018-02-12 14:27:27 UTC
$ cat /etc/fstab
UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX / ext4 noatime,errors=remount-ro 0 0

Alpine:

$ uname -a 
Linux localhost 4.14.18-0-vanilla #1-Alpine SMP Fri Feb 9 18:57:14 UTC 2018 x86_64 GNU/Linux
$ ./syncthing -version
syncthing v0.14.44+38-g2bbd2d6e "Dysprosium Dragonfly" (go1.9.3 linux-amd64) user@linux 2018-02-12 14:27:27 UTC
$ cat /etc/fstab
...
tmpfs /tmp/tmpfs tmpfs size=5G,user,defaults,noatime,rw,mode=0555 0 0

A bunch of files got deleted, I have no idea which ones, but it did created a bunch of conflicts, https://gist.github.com/RealKindOne/389ec76986cb49a9e2bef3405d505060

125,732 files, 16,091 folders, ~3.08 GiB

Yes, I also see it on a folder that is only shared between three linux devices, all filesystems are ext4.

1 Like

For me, files were deleted on a Linux laptop just after it upgraded to v0.14.45-rc.2. The only other device on the network at the time was my Windows desktop.

I can restore files from a third device - but how to ensure they’re not deleted (assuming a fix is available)? Do I need to delete a database file or something?

Anybody got v0.14.45-rc.2 from apt.syncthing.net ? Mine seems stuck at v0.14.45-rc.1

There is an rc.3 now, which does not include the change responsible for the deletes.

It will not assist in undeleting the files marked for deletion though. :confused:

1 Like

The question was mainly to now if anybody on rc1 met the same deletions issue.

rc.1 contains the same flaw as rc.2, plus an additional panic. But as far as we understand you are safe if you do not have any case insensitive clients involved.

So, rc3 seems to mostly fix stuff but I have one way sync folders set up that show differences between folder/file counts in global vs local state. Despite that, the source folders do not present a big red “override changes” button to force a resync.

Any suggestion on how I fix that?

I never noticed any deletions/issues on rc1.

You can run the affected device once with STDBCHECKEVERY=0 environment variable set to recalculate those stats.

What is the “affected” device though? The counts are wrong on all of them.

All of them then.

Doesn’t seem to have helped. Also, I don’t see any indication in the logs that setting that environment variable actually caused any behavior change in syncthing. Is there a log entry that I can use to verify that it did indeed do something?