Syncthing keeps syncing existing files due to duplicate entries in index

I recently upgraded to version 50.

Several things run a lot smoother now with the “receive only” folders.

But there is one thing i noticed:

It seems to me that the following is happening:

  • There are some files (about 400 in my case) which are in the index twice:
  1. DOWNLO~1/someFile.txt
  2. Downloads/someFile.txt
  • Now syncthing is trying to sync them.
  • While syncing: syncthing shows them in the detail view as “reused”, but the number of remaining files does not go down.
  • I guess there is still some bug in there which does not recognize that these files (#1 & #2) are the same in deed and it keeps syncing and syncing and syncing, never able to reach a synced state.

Addition: The one device where these files (and duplicate entries) originated from is not yet updated right now, but also is offline. (Maybe this info is useful. Maybe the device would auto-correct these entries if it were updated and online?)

But maybe not, since i had another similar case on another folder with just 5 files. In this case both devices were updated to 50.

I tried to remove and re-add these files on the device where they were shown as missing/out of sync, but it did not help.

Then i deleted an re-added them on another device, which solved the problem.

Actually when i removed the 5 files it showed about 10 or even more (not sure now) files being removed (including the ones with the “~N” suffix).

The re-add did only add the correct filenames.

A) Any ideas how to solve this with the current version?

B) Will you look into this bug, to solve it?

Greetings Fred;

Afaik, the obvious causes for this were fixed (https://github.com/syncthing/syncthing/issues/3800). The odd index data might remain, though.

I figured it out. I had another linux device, where the DOWNLO~2 dir actually existed as real directory.

Now i deleted it and i think it will be fine.

There is still trouble syncing this.

I think the problem is as follows:

The linux device can differentiate between “Downloads” and “DOWNLO~2”. The Windows devices can not.

Now that the linux device deleted the “DOWNLO~2” directory. This change is synced to the windows devices.

These (try to) delete the dir but then notice that the “Downloads” dir is missing (they are set to “receive only”).

And somwhere in this they get messed up …

EDIT: I believe it fixed itself now (after pressing “revert local changes” repeatedly). Finally it actually deleted the Downloads dir and is now re-downloading it. This is still pretty stupid … but at least finally the state is ok again.

Please look into this again. Windows devices should detect this case (identical files in the index involving short-names) and handle it gracefully.

Addition:

On the second Windows device i don’t get the “revert local changes” button.

And syncthing is stuck with this message in the log:

2018-09-15 11:15:49 Puller (folder "Downloads" (downloads), file "DOWNLO~1\\Art\\Tiere"): delete dir: directory contains unexpected files, scheduling scan
2018-09-15 11:15:49 Puller (folder "Downloads" (downloads), file "DOWNLO~1\\Art"): delete dir: directory contains unexpected files, scheduling scan
2018-09-15 11:15:49 Puller (folder "Downloads" (downloads), file "DOWNLO~1"): delete dir: directory contains unexpected files, scheduling scan

Ideally you’d reset the databases on all devices, as it will be close to impossible to get rid of these entries.

You could rename Downloads temporarily, which will allow DOWNLO~1 to be deleted, and then rename it back I guess, but the fact that the entries are still around could cause issues in the future.

Some of my devices are not very fast and it would take days to reset the DB.

Ist there a safe way to reset just one folder?

Maybe by changing its ID (on all devices)?

Just removing and readding the folder will reset the db for just that folder.

How do i need to do this in regards to multiple devices?

Or do i still need to change the folder-ID?

With the same folder-id and when not removing it on all devices simultaneously i assume i still have the old index on some device left which will be re-synced when it connects to the other devices?! Rigth?

Right, so changing the id will probably be the best course to get rid of the index for just that folder entirely.

The remote devices will still send the garbage index entries back tho, so this is not good enough.

Why? If the garbage files on disk are removed everywhere and you change the folder id, a totally new db prefix will be used.

Sure but this is still resetting the index (even if only for one folder).

I think that’s the goal: Reset the index for just one folder, because resetting it for everything would be very costly (time/resource wise).

What will happen with the index of the old folder/id?

Will it be automatically deleted? Or will it stick around and fill my drives? If so, is there a way to delete it?

If a folder is removed, any information related to it in the db is deleted - so no, it won’t stick around.

Id love to have a per-folder database and a new button to manually reset the database for that particular folder.

Resetting the local database is easy, but the point is that remote devices back-populate all the entries that were there.

Per folder database is essentially removing the folder on all devices and readding it back.

That doesn’t require per-folder dbs and would be pretty easy, but useless due to:

Alternatively as discussed above: Also change the folder ID in the process of readding, thus not needing to do it at the same time on all devices, because noone will propagate any old data back.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.