bug? fresh setup, single send-only folder, remote side misses 48files, global state all the same, files are really missing on remote

Support
1

bug? fresh setup, single send-only folder, remote side misses 48files, global state all the same, files are really missing on remote

I am having bad feelings about syncthing and its robustness and dependability when I think about my current situation I tried to use it for.

fresh win-x64 syncthing 1.5.0, from syncthing.net webpage, started from scratch yesterday. very simple setup. two machines involved only. both windows10 x64, ntfs everywhere.

A simple transfer of a folder, with 3331 objects and 49folders. Both machines, windows10, both machines never had syncthing running before. Never shared this folder before. both sides NTFS, both places have plenty of disk space.

created sendonly folder/object on the one side, let it scan through the whole stuff first, 66gb, only images actually inside subfolders.

the remote side then added the other machine ID, and received the folder.

on both sides the global and local states match perfectly, 3331 files and 49 folders.

only the send-only side shows, on the remotes machine status (lower right side of sycnthing webpage), and there are some hundred megabytes missing in 48files/objects.

I remote accessed the remote machine (think Teamviewer etc…) and there are really a few hundred megabytes missing, and really in select directory, three different directories, some files missing. that stuff is legit that the sending machine complains about that the remote hasnt finished syncing yet…

so what now? how is this possible? what logs and debug stuff can I provide to sort this stuff out.

thanks for helping to fix this bug if it is any. thank you.

2

restarted both sides today obviously multiple times, and I added a single more testfolder inside, with a single test-txt file on the sending side, to make sure syncthing instances would still connect and talk to each other, and they did, and now the global state is at 3332 and 50 folders. both sides. and the test dir and test-txt with its content safely arrived on the remote side as well.

those 48files are still being complained about on the senders side, and those didnt show up still. :frowning:

3

No filtes, exclusions or anything the like either. Fresh simple send-only folder created and wanted it to simply transfer. How is it even possible that such a situation fails? :frowning:

4

Can you please download the stindex tool from https://build.syncthing.net/buildConfiguration/Syncthing_Tools/66572?buildTab=artifacts and run it like stindex -mode idxck /path/to/your/index-v0.14.0.db on the side that does not miss the files.
And if it complains about incorrect sequences, can you try reproducing the problem with the walkfs and scanner debug facilities enabled (STTRACE=walkfs,scanner).
Anything special about the missing files?

5

first, thanks for your reply, I really appreciate this, as I am really having stomach aches about such things as data-loss or inconsistency of data.

that build-site the link your provided asks for an account, do you have that tool available elsewhere for my simple windows x64 systems? I can then run that tool on the source send-only side of things where the data is complete, where I actually wanted it to be transferred from to the one, single, remote side.

6

There should be a small link saying something like “proceed as guest”. Anyway here is the windows x64 binary: stindex.exe (12.3 MB)

7

Okay thanks lots, the -mode idxck results in several lines, as far as I can tell, all complaining about…

refers to wrong name… some .jpg file refers to another completely different (folder/filename)… object… it claims… several times…

some lines it confuses e.g. a .tif file with a .jpg file still within the same folder, on other lines it confuses some other graphis file formats with other graphic files but on a completely different folder… very weird…

8

Please post the stindex output, I can’t make any sense of this.

9

as I am not a developer and not too deep in such things, where do I put those debug settings exactly?

STTRACE=walkfs,scanner

some environment variable? or some startup on the command line infront of the syncthing.exe?

10

I fired up the normal 1.5.0 sycnthing.exe on the sending side and inside the web-gui in the log area, I activated those debug facilities:

[3DMZE] 2020/05/28 15:03:34.504605 api.go:644: INFO: Enabled debug data for “walkfs” [3DMZE] 2020/05/28 15:03:38.430801 api.go:644: INFO: Enabled debug data for “scanner”

and then rescanned that one folder, but it didnt output much?

[3DMZE] 2020/05/28 15:04:54.934371 walk.go:104: DEBUG: Walk [] Matcher/[]@0xc0000e2240 [3DMZE] 2020/05/28 15:04:54.934371 walk.go:104: DEBUG: Walk [] Matcher/[]@0xc0000e2120 [3DMZE] 2020/05/28 15:04:54.938373 walk.go:240: DEBUG: ignored (internal): .stfolder [3DMZE] 2020/05/28 15:04:54.939337 walk.go:240: DEBUG: ignored (internal): .stfolder [3DMZE] 2020/05/28 15:04:54.939337 walk.go:171: DEBUG: Walk progress done default [] Matcher/[]@0xc0000e2120 [3DMZE] 2020/05/28 15:04:56.098502 walk.go:171: DEBUG: Walk progress done kcnkg-icxp7 [] Matcher/[]@0xc0000e2240

11

I could not follow what you wrote about the stindex output. Could you please first post the output here so I can see what it says (that has implications on helping you).

Then when I say “reproduce the problem” I mean start from scratch, i.e. remove the folder and readd it, hoping you get the same files (or any files) to be missing again while recording these logs. Thus it’s hopefully visible where something is going wrong.

12

I sent you the log output via privatemessage due to filenames.

13

experimented some more. shut down syncthing on the senders side. restarted it with syncthing.exe -reset-deltas.

it seemed to refresh the progress bar on the lower right side on the senders side in the gui showing the remote machine, but again stopping at 46nonsynced files and 900megabytes worth of data. this didnt help.

no datatransfers beyond this level of sync.

14

Thanks, the logs you sent me show the “expected” kind of error. Not saying it’s ok for these to be there, but it is the problem I thought you’d have (sequences). There’s an improvement in v1.6.0 that prevents one way how this error can occur. However it would still be interesting to know how it happened for you.

That -reset-deltas did nothing is consistent with the problem.

If you want to help, i.e. get to the bottom of this, I need these logs with walkfs,scanner.

  1. Remove the folder in the web UI on both sides.
  2. Enable walkfs,scanner debug logging (e.g. in actions->logs in the UI) on the side that has currently no missing files (where you ran stindex).
  3. Do the same setup of folders that you did initially.

If you then see a the same problem (missing files) again, please share the logs.

15

Okay I have debug enabled on those two settings. I removed the folders on both sides. Im adding the folder on the sending side now as send-only again.

Hopefully(?) I can re-add the folder on the receivers side to the same path that already exists there and it will only tranfer those missing few hundreds of megabytes? and not all the 67gigabytes?

how do I share the logs later on as it seems like and endless line of entries rushing by on the command line.

Does it save them or can I export them as whole into a file in some way?

Also, it is still scanning the 67gigabytes on the sending side, rather a lot of data, some usb disk drive. I have not yet accepted on the receivers side, this new folder ID to be saved. Do I preferabbly wait on the receivers side until the whole scan and state on the senders side is completely 100% scanned and all 67gigabytes indexed? Or can I add the folder already early on on the receivers side?

16

The sender side now completed the scanning of the 67gigabyte.

[3DMZE] 2020/05/28 16:01:05.064890 folder.go:619: INFO: Completed initial scan of sendonly folder pbrfm-65zv3

I have not yet accepted this new folder object on the remote receiving side. Do I need to run the index stuff already now on the senders side just to make sure the index is valid right from the start and has been created properly or something?

or how can an index of a nonchanging folder (senders side doesnt put new stuff into it at all) change over time at all, especially it being a send-only file.

My whole setup here is actually only tring to transfer some emergency backup of images from one place to another. Wouldnt have thought that syncthing was failing so badly at this simple thing.

Maybe I should have gone for like rsync or something? :frowning:

17

More oddities that I notice… As I have not yet accpeted the new folder on the receivers side of things, and there is nothing else on the receivers side, but the default folder, which is unused. And I removed the old failing object on the receivers side… How is it possible, that the receivers side being shown on the remote syncthing, does not show the same amount of files missing/unsynced that totally exist.

on the syncthing on the senders side:

Global State 3.378 50 ~68 GiB
Local State 3.378 50 ~68 GiB
Folder Type Send Only

on the senders side inside syncthing gui, on the lower right side, there is the remote side machine object. it currently has this state:

Out of Sync Items 3.428 items, ~68 GiB

how? why not those 3378 objects? I think I fail to understand basic syncthing concepts :frowning:

18

Ups, from the first I didn’t expect this to be so big…

I am not hoping for this to work, I am hoping for it to fail again so we see why. As I wrote before, this is only necessary if you want to put in the time to help and get to the botom of this. If you just want to fix the problem, update to the v1.6.0 RC or run Syncthing with STRECHECKDBEVERY=1s once.

So essentially tell me now please if you just want to get things running, then I can just tell you that and stop investing time, or if you are willing to get to the bottom, then I need you to stick exactly to the plan.

19

Well obviously, I want things to work as well, otherwise I woldnt be scared by such “bugs” or whatever the reason for this is. And I am using syncthing in other places as well, this was just an additional situation that I used it for, and I didnt expect it to fail for such a basic operation.

I accepted the folder on the remote side, and pointed it to the directory over there where the files (mostly?) already exist. It now scans on its own existing folder for a while now already and every now and then briefly bursts some traffic from/between the senders side, but only about like 22kbytes/sec and goes back to zero.

Will see how this turns out.

20

So the answer is no, you don’t want to get to the bottom of this. In that case: