Solved: What to do if a file has vanished on one side and nodes keep requesting the file from each other

From time to time Syncthing starts choking on a file. I have a very hard time fixing this - which makes me think the error is most likely on my side. This post should verify what I keep overlooking…

This weekend it was a file with root-permissions (*.pyc) while Syncthing was running with users permissions. I’ve ssh’ed to the machine and deleted the offending file but Syncthing did not recover. Nor did it after restarting the Syncthing itself.
I’ve added the mask *.pyc to the ignore-List (on all peers) but that didn’t help either. I’ve double and triple checked on all peers, but the file is not there anymore. Syncthing steadfastly refuses to ignore the vanished file…
As a last resort I’ve shut down Syncthing and removed it’s .config/syncthing/index-v0.11.0.db/-directory (read that somewhere here in the forums). To no avail. Syncthing was restarted, rebuilt its index and keeps insisting that there is a file it cannot sync. (I wonder where it got that information from…)

Setup:
Syncthing is always installed in the most current version (12.8 as of writing) - as are all peers. The problem has almost certainly nothing to do with different versions, although the sync is done between different systems (Unix, Linux, Embedded Linux, Windows, Mac).
I’m diligent in using as much 7-bit-ASCII as possible. The filenames should not neither be the source of the problem.

Question:
How should I proceed when this happens? Is there some other file which I could delete to force Syncthing to have a fresh look at the source files/directories? Is there a F.A.Q. for this type of problem that somebody could point me to, I’d be most grateful.

What’s the actual error you’re getting, as reported in the log?

Ah. I’m sorry I forgot the most obvious…

That’s the full log after a fresh start with synching -logfile="-" -verbose

DE71364-SV-U5:~ $ syncthing -logfile="-" -verbose
[monitor] 19:41:22 INFO: Starting syncthing
[QFMBY] 19:41:22 INFO: syncthing v0.12.8 “Beryllium Bedbug” (go1.5.2 linux-arm) unknown-user@build2.syncthing.net 2015-12-13 09:39:09 UTC
[QFMBY] 19:41:22 INFO: My ID: …
[QFMBY] 19:41:22 INFO: Single thread hash performance is ~5.2 MB/s
[QFMBY] 19:41:22 VERBOSE: Starting up (/home/…/.config/syncthing)
[QFMBY] 19:41:25 OK: Ready to synchronize Scripts (read-write)
[QFMBY] 19:41:25 VERBOSE: Device … sent an index update for “Scripts” with 0 items
[QFMBY] 19:41:25 INFO: Device … is “DE71364-SV-U5.lvdw.local” at [dynamic]
[QFMBY] 19:41:25 INFO: Device … is “DE71364-CL-M3.lvdw.local” at [dynamic]
[QFMBY] 19:41:25 VERBOSE: Startup complete
[QFMBY] 19:41:25 VERBOSE: Folder “Scripts” is now scanning
[QFMBY] 19:41:25 INFO: API listening on 0.0.0.0:8090
[QFMBY] 19:41:25 INFO: GUI URL is https://127.0.0.1:8090/
[QFMBY] 19:41:26 VERBOSE: Discovered device … at [tcp://192.168.X.A:22000]
[QFMBY] 19:41:31 VERBOSE: Discovered device … at [tcp://192.168.X.B:22000]
[QFMBY] 19:41:34 VERBOSE: Login successful for username …
[QFMBY] 19:41:38 VERBOSE: Discovered device … at [tcp://192.168.X.C:22000]
[QFMBY] 19:41:40 VERBOSE: Discovered device … at [tcp://192.168.X.D:22000]
[QFMBY] 19:41:45 VERBOSE: Discovered device … at [tcp://192.168.X.E:22000]
[QFMBY] 19:41:49 INFO: Established secure connection to … at 192.168.X.D:58520-192.168.X.F:22000 (direct-dialx)
[QFMBY] 19:41:49 INFO: Device … client is “syncthing v0.12.8” named “DE71364-CL-M3.lvdw.local”
[QFMBY] 19:41:49 VERBOSE: Connected to device … at 192.168.X.F:22000 (type direct-dial)
[QFMBY] 19:42:01 VERBOSE: Scanning folder “Scripts”, 0% done (0.0 MB/s)
[QFMBY] 19:42:04 VERBOSE: Device … sent an index update for “Scripts” with 902 items
[QFMBY] 19:42:07 INFO: Completed initial scan (rw) of folder Scripts
[QFMBY] 19:42:07 VERBOSE: Device … sent an index update for “Scripts” with 667 items
[QFMBY] 19:42:07 VERBOSE: Folder “Scripts” is now idle
[QFMBY] 19:42:07 VERBOSE: Folder “Scripts” is now syncing
[QFMBY] 19:42:07 VERBOSE: Summary for folder “Scripts” is map[inSyncBytes:43236836 globalDeleted:98 localFiles:1421 localDeleted:0 globalFiles:1422 localBytes:43224050 needFiles:0 needBytes:0 globalBytes:43236836 inSyncFiles:1422 state:scanning version:181970]
[QFMBY] 19:42:08 VERBOSE: Completion for folder “Scripts” on device … is 100%
[QFMBY] 19:42:08 VERBOSE: Folder “Scripts” is now idle
[QFMBY] 19:42:08 VERBOSE: Summary for folder “Scripts” is map[state:idle needFiles:1 inSyncBytes:43242866 needBytes:11088 localFiles:1421 localBytes:43224050 inSyncFiles:1421 version:181970 globalDeleted:147 localDeleted:0 globalBytes:43253954 globalFiles:1422]
[QFMBY] 19:42:08 VERBOSE: Completion for folder “Scripts” on device … is 100%

On this machine there is only this one directory “$HOME/Scripts”. All peers have been deleted except for one to simplify troubleshooting (for me).

I don’t see anything to indicate something is stuck?

Not in the logs. The GUI does. I’m not sure screenshots would help - except make me a little more believable…
Would it help if I deleted the whole .config/syncthing-directory? Or is there maybe a file I could delete without having to setup Syncthing from scratch?

Stop deleting things. :wink: When a file can’t be synchronized correctly, the reason is printed in the log (and in the “errored items” thing in the GUI).

:blush: You are right, of course. But after some hours of head scratching I get a little desperate… I’ve just checked all the other machines - they are fine. So it’s just this one client - and it’s a test client anyway. I’ll just give it the night and see if it recovers. Sync is not always as fast as my patience is…

Thanks for your support!

It sounds to me like once the file is deleted you need to stop all of the nodes at the same time otherwise it stays in their index and they continue to request it. Even after blowing away the db the other nodes request the file from the original one.

I am guessing because it sounds a lot like what can happen when adding a file to the ignore list once the other nodes know it exists.

1 Like

kluppy that did it!!

I’ve waited for a day to give all machines (esp. those two) the chance to catch up, but that did not work. Then I’ve done what kluppy suggested for all seven nodes that are syncing this directory. After that everything resynced and all nodes agreed on this file not being there anymore. All nodes are green again and working. Marvelous!

Thank you to everybody involved! I’ve learned to a) keep my fingers off the index dbs, b) trust syncthing and c) take all nodes that have a problem with each other offline so as to give then the chance of a fresh restart.

I would change the topic to “Solved: What to do if a file has vanished on one side and nodes keep requesting the file from each other” if somebody can help me how to do this…

done.

1 Like

Thanks! :smile:

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.