Syncthing ignores/doesn't pick up almost all files

Support
1

Created a new dir yesterday that I wanted to sync (50 000 files and lots of git repos and dot files), but today I noticed that syncthing just pics up 66 files instead of the 50 000.

I tried to reset deltas and the db but it changed nothing.

I have some dirs with 600 000+ files that sync just fine.

Today I double checked with all my dirs, I noticed several where syncthing doesn’t pick up everything and a couple thousand files are missing in the sync…

What can I do from here ? ( I mostly checked with ‘tree’) I have no ignore rules that should prevent the synchronization of those files.

I am using btrfs on the system where the files don’t get picked up. (root and home dir get snapped) Also i am syncing ‘$HOME/Sync/lots-of-dirs…’ where ‘Sync’ is a symlink to a different drive. And the daughter dirs contain the different syncthing folders

No errors or warnings in the webui. Tried to start syncthing via systemd and the provided .desktop files.

2

I am quite confused.

example dir most stuffed doens’t get picked up: Sync/pc1/config/50 000 files

When I moved the 50 000 files directly into pc1 most stuff got picked up… And when I moved them to a new subdir e.g. config2 they vanish again

I doubt this has anything todo with path / file lenght

3

My guess is When socket files are in a folder that needs to be synced on linux, some files were not synced until the socket files were removed · Issue #9872 · syncthing/syncthing · GitHub

Could be some other kind than sockets, or it could be some other error that takes the same path and currently doesn’t get surfaced. If you can try the latest nightly there is a fix in there…

4

Yeah this might be it. This branch ? GitHub - syncthing/syncthing at release-nightly nah the commit is not in there…

Ah NVM Versions & Releases — Syncthing documentation found it

5

Actually it wasn’t, I thought the build had run but it hadn’t. It is running now. :confused:

You want 1.29.0-rc.1.dev.12 (or newer) to have any luck.

6

Okay I have run v1.29.0-rc.1.dev.12. most files get picked up now(after running once the nightly build) (even by the stable release)

‘tree’ still gives me a diff of 10 000 files ‘find dir -type f | wc -l’ does not. But at very least I am not missing anything important anymore.

I think syncthing still has trouble to pick up everything ‘find dir -type f | wc -l’ still gives me a diff of multiple 10 000 in some dirs… IDK why this might be (most folders seem to be fine now…)

7

If you enable scanner debugging in the log view, and run a scan (with the dev.12 nightly), do you get Skipping non-regular file lines? Would be interesting to see…

8

There were a couple with

STTRACE=scanner ./syncthing  | tee log.txt

I removed them (9) now there is nothing. But I still have diffs with find dir -type f | wc -l

do you have a more reliable way to check the file count for syncthing? I hope this is an issue with the way those tools interact with BTRFS.

rsync -an --stats /path/to/btrfs /empty/dir

Also gives me a diff of ~30 000 regular files

Well I copied the dir to a ext4 usb but still a diff of 30 000

9

Are there other things / logs I should check?

10

If you have both sides you can diff the find output from them. I was interested in what those non-regular file log entries were, though.

11

Note that find dir -type f | wc -l along with rsync -an --stats /path/to/btrfs /empty/dir would include any versioned files under Syncthing’s .stversions directory while Syncthing’s web UI reports only the files counted in the sync.

(In Linux, user-space tools that interact with a filesystem pass thru Linux’s VFS so they’re insulated from the fact that it’s btrfs or some other filesystem type.)

1 Like
12

I don’t use syncthings file versioning.

mmmh I just remembered that ’ du ’ and similar tools apparently don’t report accurately on btrfs (e.g better use btrfs du etc…) .

I guess that is not the case for other things . thanks!

13

Well the result was rather anticlimatic

The socket issue was definitely one cause.

The rest was solved by this …

find -iname "*syncthing*tmp*" -delete

The tmp files was weren’t cleaned up ( + 200 000)

1 Like
14

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.