I have lots of cases where sync-conflicts files appear all over my documents, despite of using Symple File Versioning. They should be placed under .stversions, so that I can see them.
I just synced the documents on my MacBook Pro 2018 with my desktop PC and from there I synced only my source code folders to an Ubuntu virtual machine. For some reason it decided to throw away a new Makefile, replace it with an old version and then propagate that to my PC and then to my MacBook. How do I tell Syncthing to always keep the new version and stop doing stupid things?
Very ofter I see two devices not having their databases in sync, so device A thinks B is Up to date, while B thinks A is syncing. Sometimes -reset-deltas helps, but sometimes I have to -reset-database which has to rescan over 300 GB of data and is not fun. Especially when things break that often.
It’s very disappointing to see that despite of the great potential of Syncthing, it has such obvious bugs making me loose hours every time I need to sync my devices. And I always have to worry if the new changes are propagated or reverted because of a bug.
It’s bad practice to just start sharing a folder between two devices that is in a different state, this will lead to conflicts. You can’t tell syncthing to do anything, it does what it does and it decides what is new and what is old not on the timestamp but the order of observation. You can’t sync based on clocks as they drift.
Database issues could happen because of database inconsistencies from previous versions whic we since have fixwd, as well as from abrupt shutdowns/crashes. If you have a specific scenario where this happens, we can suggest ways of debugging it to understand why it happens.
You are one of few who has this experience. Yes, there are bugs and edge cases, as in every software, but majority of users seem to be happy otherwise there would be more angry people on the forum.
Perhaps you are misusing syncthing which triggers these bugs/edge cases.
Hello Audrius!
I had actually started by copying one of more folders to each device, then making all devices know each other and adding them as introducers. Finally I share the folders. Thank you for clarifying the difference between version and conflict, I thought it’s was same and that Syncthing is acting randomly.
Regarding the order of observation if I make changed on device A while Syncthing isn’t running, does it matter if I start Syncthing on device A first or B?
I never observed any crashes, and I always use Shutdown from the web page and only when everything in Up to Date.
Well, it’s possible that I’m putting Syncthing to it’s limits with my workspace. I have the following devices:
A B1 C1: Documents
B2 C2 D E F: multiple C++ project folders inside Documents
A: MacBook Pro 2018 macOS
B: MacBook Pro 2018 Windows 10
C: Desktop PC Windows 10
D E F: Ubuntu Linux virtual machines
A and B are the same laptop, so changes propagate through C1.
Here is how I manage my shares in hope to avoid conflicts, unfortunately I often get hundreds of conflicts and it’s completely unmanageable:
I start Syncthing on the device where I am working.
Let say I made some changes to files on device A.
Once ready I start Syncthing on device C1 which should accept the changes. Both are configured as Send and Receive. For the most part it works, but sometimes I get a few conflicts.
A and C share the Documents folder. Inside that there are many C++ projects. A few of the projects have to be synced to a virtual machine D running Linux. C2 is the second Syncthing instance which shares these C++ projects between C2 and D. Note that I never run C1 and C2 at the same time, to prevent conflicts, and according to the documentation I should not include these subfolders to C1 which shares the parent folder, so I created the separate instance C2.
Usually I get many conflicts here.
This might also be because of the rare usage of the virtual machines. Syncthing usually gets updated if I had not used it for a month. And like you said I noticed sync problems and database corruption after each update.
Also if the database or sync state gets corrupted, this usually causes conflicts, but I cannot just copy the folders from one device to another, because the .vs subfolder of C++ projects stores the state of open files in Visual Studio 2019 and it is using absolute paths, so this folder is unique for every device. It is added to .stignore.
Since I use many different compilers on multiple operating systems and targets, they tend to create a huge variety of temporary files and folder which eventually get synced everywhere. I have created scripts to delete such files. And I add them to .stignore when I notice them, but this means I already left some to sync. The script deletes them, but Syncthing had already indexed them and now .stignore tells they should be ignored so the database and sync-state between devices get corrupted. I’m sure this can be improved! Although I try to keep .stversions the same among devices, this is a very common cause if sync problems and conflicts. I have to reset the database to get the devices in sync, but they usually make a lot of conflicts meanwhile.
I need a way to force Syncthing to keep .stversions in sync between devices. Otherwise I have to update many files on many devices manually, and it takes a lot of time. For a single device I can edit a file and use a script to replicate it. But with 6 devices running different operating systems, the manual steps take too much energy.
Thank you for your time, and please let me know what I should do to debug the cause? Perhaps the Syncthing team may also consider my use case and improve this nice project!
Your explanation is too complicated for me to understand but there are a few key takeaways:
You switch syncthing on and off across the estate, which is most likely the reason you get conflicts. Ideally syncthing should run all the time everywhere, as this is most likely the reason you have conflicts.
You are syncing code repos when version control systems are made specifically for that purpose. You have to understand that version control system metadata is not a single file, but it’s a collection of files. Syncthing deals with individual files, so suddely some metadata file that was removed on one machine but got modified on another causes a conflict, the non-deleted file wins, and everyone gets that file potentially corrupting the metadata required for the version control system to operate (by providong an exta unexpected file).
.stversions is just a directory which you can also add and sync.
Having unignored temporary files and devices going up and down is the worst possible scenario for consistency. Your devices announce temporary files and go offline, then everyone is out of sync not having where to get the temporary file from. Ideally, you should work out your ignore patterns ahead of time and stop starting and stopping syncthing.
haven’t read all the following post, just wanted to comment on this: If I remember correctly in case on conflicts, e.g. the specific case of initial sync of non-matching data, the decision which file is the conflict, i.e. gets renamed, is based on mod time
I see what you mean, unfortunately since there are virtual machines which only run during short tests and the laptop is dual-boot and travelling, it is not possible to keep Syncthing running on all devices all the time. It would be nice if Syncthing gets improved to support such a use case. The solution would be to temporary switch one device as master and replicate its state to the rest.
By default each folder can be configured individually as Send & Receive, Send Only, or Receive Only. Please add a Global Folder Type setting on the main screen which while active overrides the Folder Type for all shared folders. Example:
Default mode: the Folder Type setting for each shared folder is respected
Send mode: Send & Receive folders become Send Only
Receive mode: Send & Receive folders become Receive Only
Force Send: all folders become Send Only
Force Receive: all folders become Receive Only
Then before syncing, the user can set the source device as Force Send, which will prevent it from receiving any changes from other devices. Next Syncthing is started on the target device and switched to Force Receive. This will perform a clean sync without any conflicts. After the sync, both devices can be reverted to Default mode. I am sure this feature would be very helpful to many users, since it is tedious to reconfigure several folders on multiple devices.
Regarding the unignored temporary files, I believe it will settle with time. In my case I started with source code of several projects designed for one operating system, then added support for many others, and since each compiler has it’s own set of temporary files… It just takes time to organise the .stignore patterns. What can be improved here is adding the feature to synchronise .stignore among all devices.
You are trying to use syncthing like rsync, deliberately put it in conditions where it’s unlikely to work sensibly and then come around on the forums talking how disappointed you are.
Perhaps you should re-evaluate the tool you are using for the job.
Hey Audrius,
I’m sorry, I did not want to offend you in any way!
For the most part I am using Syncthing the way it is designed to be used: I work on two computers, switching from one to the other, and everything syncs fine in the background. I don’t have to worry about what I change, and what direction I have to sync. Still there are certain cases where the user may have a good reason to propagate changes in one direction. Currently he can change the settings for several folders. The feature I suggested makes that one click. You’re welcome to ignore me or say I should go use another tool. Or you may consider my suggestion would be useful to others and discuss it with your colleagues.
Please let me know if you think any of the features I offered will have a negative impact on the usage! I’m open for discussion.
When choosing a program to sync my files I spent a few days learning about the available tools, their features and many details. Syncthing was carefully selected as the best tool for me and I’m happy with it. Although I read the entire documentation and looked into all settings, there are always things under the hood to be learned. I’m here to discuss my experience share ideas what can be improved.
If you want to send files one way, just use rsync.
We can add a million different features to syncthing, local sync, one way sync, shut down computer, cron, documents viewer, syncing contacts, wash dishes, the list is limitless.
But you have to draw the line somewhere and decide what is the software specialised for. I’d rather do one thing well, than do 100s of things poorly (or add 100s of things which compromise on the ability to do that one thing well).
I think at least part of your needs would be met by https://github.com/syncthing/syncthing/issues/2710, which I think would be a very sensible extension of the current send-/receive-only functionality. However that’s pretty complicated to do and as Audrius explains, outside of “Syncthing’s core business”, so needs to be done well and unintrusively to be useful.
I like the idea of keeping Syncthing simple and reliable. I love it for its simplicity and ease of use. In a real world, people are not perfect and software usually has a few things to iron out too. In a perfect world Syncthing runs all the time on all devices and everything works great. However there are mobile devices and nodes are not always online. In a rare case a node may stay powered off for a few weeks. Obviously there will be problems once it tries to sync. The user is aware that a particular device is behind and has no changes and can give a temporary hint by setting all folders Send Only or Receive Only. But Syncthing is not aware. I’ve seen it replicating the old state, and it can make quite a mess. What works best at my experience is to set all folders Send only on a node which is up to date, then force the device which is behind to catch up. Hence my idea for a quick switch to get all folders in sync faster and without any wrong actions. I am describing an environment where the user has access to configure Syncthing on both devices. The functionality is already there: we have Send Only and Receive Only folders.
Audrius, I agree that each software has a purpose, hence the question is if a feature is relevant and important to make it serve its purpose better and reduce unwanted behaviour. I strongly believe in that.
If that’s the case, it is a bug, that’s not by design. Even if you would give that hint, it wouldn’t change what Syncthing does (it just wouldn’t propagate). Syncthing will never replicate the old state no matter how long devices are unconnected (unless there is a bug). It can create conflict copies, but that’s not replicating the old state. That’s detecting that there are concurrent changes, and replicating both versions to let the user decide which is the right one. It might rename the file that you consider the canonical one as a conflict if its modification time is older (for real or if clocks are out of sync), nevertheless it’s preserved.
Essentially what I am saying is if you see Syncthing overwriting files with old version when a device connects, that has long been disconnected, something went wrong. Try to reproduce and give steps how you did it if you succeed, or provide screenshots and logs - thus we can try and identify the problem and fix any bugs.
I am pretty sure the reason you are seeing this issue is because whoever is offline actually touches the files while offline, causing a conflict, which then deterministically gets resolved not the way you expect it, because it’s all relative to when that offline device modified the file.
For all that I know, this device comes up might be running a startup script that does a recursive touch on all files, or some source code indexer fiddling with it (which is done by most IDEs), or some dodgy filesystem with low precision timestamps.
You should fix the cause of the conflicts, and not try to hack syncthing to work around the consequences of your machines being offline while they are modifying files concurrently with others.
As stated above, that’s actually a case Syncthing should handle. What I want to note is that that’s not the case you originally described – or if it was, you were extremely unclear:
This implies that you are selectively starting and stopping Syncthing in order to propagate specific changes on demand. That’s specifically not an intended use case; there are existing tools such as rsync, unison and rclone that are specifically designed for this.
There is definitely a bug. Sure my suggestions won’t fix it, but for me they make the difference between spending a day to recover from a bad sync and taking precautions and getting the job done. Completely risk free.
I use two devices, clocks difference is less than a second:
A1: Desktop PC, Windows 10 always on: Music, Documents, Mailbird, few others…
B1: MacBook Pro 2018 macOS Catalina: all the time, sleeps if inactive: Music, Documents.
B2: MacBook Pro 2018 (same laptop) Windows 10: once every few weeks: Music, Documents, Mailbird, few others.
2020-01-14 I installed updates on B2. I don’t remember if I ran Syncthing.
(14-23) I don’t remember if I used B2 or not. Are there any logs I can check?
2020-01-23 I created a backup copy of Mailbird on A1.
2020-01-25 I synced B2 to A1. There were a few source files in Documents that were marked as sync-conflict, where the old version was preserved and the new was renamed. There were tens of other conflicts where the new was preserved and the old was renamed. I wasted my entire day because of this bad sync, and resorted to compare in hash 50000 files (21 GB) against the last backup to see what I had to preserve.
2020-01-28 I noticed that Mailbird on A1 has an old state of e-mail folders and arrangement of messages from 2020-01-21.
2020-01-28 I restored the backup from 2020-01-23 to A1 and deleted the old version of Mailbird’s folder, but B2 probably still has it.
I apologise for not being more detailed, and not keeping conflict files. I ran out of schedule fighting the bad sync. In the end I just wanted to clean up the mess, so I deleted all sync-conflict files and .stversion folders, then used backups to recover.
Are there any logs we can check? I recently enabled anonymous usage reporting on all devices.
Another recipe for disaster is to add an indexed file to .stignore, optionally replicate .stignore to all devices, and finally delete the ignored file from one device. The only way to get a proper sync after this is -reset-database. Makes me think it might be another recipe for disaster, yet the only think that works to remove ignored files from the index. Are logs preserved after that command?
I read somewhere a good designed interface is easy to use properly and hard to produce unwanted results. Start adding items to the ignored list, and you’ll have something that is impossible to use correctly and very easy to make a huge mess. Audrius advice to think ahead is good, yet that just happens in real life.
A post-mortem type of write up does no good in helping us understand this, as it does not rule out you fat-fingering something which then leads to this state.
I suggest you sit down, try to reproduce your issue in isolation in a brand new folder with no prior state with one or two files, write down the steps and post it here if successful, then we can talk about bugs.
Now we can only talk about things you think you did, but you yourself stated that half of which you don’t remember, so there isn’t really a straw for us to grab on as there are no concrete steps, just faded memories.
By default the logs are not verbose enough to follow the trail (people sync millions of files, we’d be producing more logs than content we manage in the folder if we logged every operation on every file) to understand how this happened, and sadly I don’t have a crystal ball to go in the past and figure out what you did, how you did it, and what bug (if any) it triggered.
The indexed file to .stignores sounds interesting, but I actually think it works as expected, that when you unignore a file which is now deleted, it gets deleted everywhere. I do agree that’s not intuitive, even if it’s the right thing todo, but I suspect there is already a ticket for it.
Hi Audrius, I understand your concern and that Syncthing can also be used to share files between different people. However let me assure you that I am using it between my own devices and no one else has access to them. When I say that Windows bootcamp is offline, it means I’m running macOS on my laptop and the Windows NTFS partition is read-only there - no write support in macOS. If I want to sync Documents to it, first I make sure that macOS and the desktop PC are in sync, then I start bootcamp and sync it to the desktop PC. The Mailbird folder is only synced between the desktop PC and bootcamp, since it is a Windows program. And I never run it on the device which is behind.
Before making changes to any device, first I make sure it is in sync with my desktop PC. There are no startup scripts. When I change a source or header in one project if other projects are using a copy of it, there is a script to create a text file with SHA1 checksums and another script to update all local instances of this file name matching the old checksum. Then Syncthing propagates the changes correctly to the other device. When I’m done working, a cleanup script is used to delete all temporary files created by various compilers and IDEs. And finally I check that all devices are sync. All this works correctly.
IDEs - I have not noticed any of these modifying files without me saving the file explicitly. I’m only concerned about source and project files here. Intelligence and other temporary files are ignored and will be deleted when I finish working. And especially when the IDE is not started, e.g. before syncing a device that was offline. I always sync before doing anything else.
Visual Studio 2019 on Windows 10
Xcode on macOS Catalina
Compilers
Visual Studio 2019 on Windows 10
Xcode on macOS Catalina
gcc on macOS Catalina, Ubuntu Linux 18.04 LTS
OpenWRT buildroot on Ubuntu and macOS with 3 different target architectures
Each creates its own temporary files and folders which eventually get added to the cleanup script, .stignore and .gitignore.
Filesystems - please let me know if any of these has low precision timestamps!
NTFS on Windows 10
APFS case insensitive, encrypted
ext4 on Ubuntu Linux 18.04 LTS
There are NO RAID volumes
If I knew the cause of the problem I would definitely try to mitigate it myself instead of being a burden to others. So far it only happens when I try to sync with a device that has been offline for some time. Possible reasons might be modifying .stignore or using -reset-deltas or -reset-database if two devices only do a partial sync and indicate that one device still has indexed files which are ignored, and there is no way to remove them from the database. This can be optimised: remove ignored files from the index. In any case this should have synced the rest of the files and if there are any conflicts for ignored files they have to be deleted anyway. But I have conflicts in regular files. And I have conflicts in the Mailbird folder where I don’t make changes to .stignore. I’m just guessing here and can’t give a clear cause. But I can tell it’s easy to get things wrong and hard to do right when syncing a device that has been offline for a week or two.
Fair enough, I’ll provide more clear steps. It might take a couple of weeks. Meanwhile here are a couple of clear steps which I know caused problems:
Why are indexed files kept in the database after being added to the ignore list? E.g. delete on all devices, edit .stignore in notepad and Rescan all. My observations show that A considers B up to date, while B shows A as syncing e.g. 95%.
If for any reason two devices end up in the state above, stop Syncthing, run it with -reset-deltas and click Rescan all on all nodes. If that doesn’t help, do the same with -reset-database. Now both devices show each other as Up to date.
A device which was offline for a couple of weeks comes online and tries to sync, but then updates and restart Syncthing in the middle of the sync.
Because you need to tell others that the file you previously advertised is no longer available. Absence of things from the database does not allow you to know what you sent before and what you did not.
Anyways, none of the above seem like detailed set of steps one could follow to observe the issue you are seeing.