need help with Out Of Sync folder, hash mismatch, failed items

[rramstad]

Hi, the source Syncthing reports all folders are in sync and that the remote system is 99% in sync with 13 items to go.

The destination Syncthing reports one folder out of sync with thirteen failed items, with the console saying hash mismatch.

I am at a loss. I have updated and upgraded both systems making sure they have the latest Syncthing. I have used the GUI to rescan the folder in question on both sides, and I have patiently waited some time now to see if the problem corrects itself.

I am new to Syncthing. The files in question were being edited via SMB and so possibly saw frequent changes over a short period of time, but they haven’t had any changes since… the various systems that did the SMB editing were totally rebooted and turned off at various points.

I’ve had this problem for a couple of days now and no improvement.

Suggestions as to what to try next?

[tomasz86]

If possible, could you post screenshots showing the errors as they appear in the Web GUI? Logfiles will be welcome too.

[rramstad]

Screenshot of the source computer (Unraid) attached

Note that when expanded, it shows that sterling-morrison needs 13 items.

Logs on that system (edited)

2022-02-12 16:57:30 Ready to synchronize “tunes_other” (tunes_other) (sendonly) 2022-02-12 16:57:30 My name is “cc67544d43af” 2022-02-12 16:57:30 Device VIJZ6NK-ARELUEM-QNZ5YQC-54HALCK-F2KWVWO-PAKXEYW-G2DPX7R-W2NGQQF is “sterling-morrison” at [dynamic] 2022-02-12 16:57:40 Established secure connection to VIJZ6NK-ARELUEM-QNZ5YQC-54HALCK-F2KWVWO-PAKXEYW-G2DPX7R-W2NGQQF at 172.17.0.2:22000-10.55.0.150:22000/tcp-client/TLS1.3-TLS_CHACHA20_POLY1305_SHA256 2022-02-12 16:57:40 Device VIJZ6NK-ARELUEM-QNZ5YQC-54HALCK-F2KWVWO-PAKXEYW-G2DPX7R-W2NGQQF client is “Syncthing v1.19.0” named “sterling-morrison” at 172.17.0.2:22000-10.55.0.150:22000/tcp-client/TLS1.3-TLS_CHACHA20_POLY1305_SHA256 2022-02-12 16:59:49 Completed initial scan of sendonly folder “tunes_other” (tunes_other)

Here’s a screenshot of the destination Windows system

It shows thirteen failed items, here’s what happens when I click on that link

The log on the destination has entries like this, they repeat periodically as the system retries

2022-02-13 12:39:35 Puller (folder “tunes_other” (tunes_other), item “fromcd\SSAR202105\They Might Be Giants - Flood (1990) [9 60907-2] [CD FLAC]\14 Whistling In The Dark.flac”): syncing: pull: hash mismatch 31b857b33924812713624ae0b58701c37bd64fc55f2d0de8bda5b80967b2975f != 3586f4f9c4fe39652e7e3bee6194ace46d4c5607f7228d2dd276387c58931215 2022-02-13 12:39:35 Puller (folder “tunes_other” (tunes_other), item “fromcd\dbAR20181117\Black Sabbath\Black Sabbath, Vol. 4\09 Black Sabbath - St. Vitus’ Dance.flac”): syncing: pull: hash mismatch c9de2e9972678a811d8c0e56a78d26fab732143f99533097a96f9fbdafee76e3 != 6813fb97de4f8d34e7011f3b05b6fd03e0bbdc63ace74b6dc849d4495d8aa8f0 2022-02-13 12:39:35 Puller (folder “tunes_other” (tunes_other), item “fromcd\dbAR20181117\Grateful Dead\1973-06-22 Vancouver BC, Disc 1\01 Grateful Dead - Bertha [Live].flac”): syncing: pull: hash mismatch 27512e9427e181b4981d0b39f42e5247487b8958347a43c941d4b5df644b0efb != a62368f7985818cfb2e85589c6df373509d77c2a8dd211c9b4a7409b11451a7a 2022-02-13 12:39:35 Puller (folder “tunes_other” (tunes_other), item “fromcd\dbAR20190209\Sir Georg Solti & Wiener Philharmoniker\Der Ring des Nibelungen [Solti], Disc 8\08 Richard Wagner - Act 2, Scene 1, Zur Neidhöhle fuhr ich bei Nacht (Wanderer).flac”): syncing: pull: hash mismatch 48a7083535d68f996227745877046f282fa3ce0d0a46f3eaa75901c9635d6aa4 != 6e96f2c6eeb634c6d10c98d8c9242be499b65620e26d2fff57a92d84b30706aa 2022-02-13 12:39:35 Puller (folder “tunes_other” (tunes_other), item “fromcd\dbAR20190108\Neil Young\Songs For Judy\18 Neil Young - Old Laughing Lady.flac”): syncing: pull: hash mismatch ecc1d1f6a66757615c40e4524d078963233692abfff110a7851dac710cadef93 != 8f64c3cb3ea7e2c870f25733d1fc3e8fd405dee2d8ee9c74fa4c4b533802bb91 2022-02-13 12:39:35 Puller (folder “tunes_other” (tunes_other), item “fromcd\dbAR202008\Various Artists\Titan; It’s All Pop!, Disc 1\01 Secrets-, The - It’s Your Heart Tonight.flac”): syncing: pull: hash mismatch 04a8a7721ecc9007f8b75bb3b40aefeeb5396c2dec79fd7df923dddd502c70f8 != 850d23f223636272b7e58125c5f2a2d39633c7465f385eb30011fcd5813a760c 2022-02-13 12:39:35 Puller (folder “tunes_other” (tunes_other), item “fromcd\dbAR20190209\Sharon Jones\I Learned the Hard Way\12 Sharon Jones & the Dap-Kings - Mama Don’t Like My Man.flac”): syncing: pull: hash mismatch 6ab65e60429b44d8ecf14eaa6bcb4087d38a1c9a304003cfe6061845e032622c != f11de9a8a41da1dae93d6bf01d512208a94ffc4b43566061f90793459e703d72 2022-02-13 12:39:35 Puller (folder “tunes_other” (tunes_other), item “fromcd\dbAR20181117\Grateful Dead\1974-05-19 Portland OR, Disc 2\08 Grateful Dead - Wharf Rat [Live].flac”): syncing: pull: hash mismatch 7a859ca232aeb00a4077402e62e0f202bc7fa71847f3740fc768149fca096b7b != 759ac75a36b782e87a9a6951b86d9f8bb3a42327acdccd382d3020df9297eba3 2022-02-13 12:39:35 Puller (folder “tunes_other” (tunes_other), item “fromcd\dbAR20181117\Black Sabbath\Technical Ecstasy\04 Black Sabbath - Gypsy.flac”): syncing: pull: hash mismatch 57ef4767dcc885de3c0e9efb44bbc760d38bee937a5422c649f235b76d89d045 != 056f48ff4099c8b626e1587b213616e8ad45a18ca8cfae5c97c6fede002503b3 2022-02-13 12:39:35 Puller (folder “tunes_other” (tunes_other), item “fromcd\dbAR202008\Various Artists\Titan; It’s All Pop!, Disc 1\05 Millionaire At Midnight - Coit Tower.flac”): syncing: pull: hash mismatch 48ceac67bf974543adb506840201f4b1038ee0792d0a6cf2244de39b1c6a8280 != 1285b2f826024cc2d95558cdf80a95f660ac0e88dffbb7523f053372c3b1d3a2 2022-02-13 12:39:35 Puller (folder “tunes_other” (tunes_other), item “fromcd\dbAR20181117\Grateful Dead\1978-07-07 Morrison CO, Disc 3\03 Grateful Dead - Space.flac”): syncing: pull: hash mismatch a03232b892b1610133b352ac39e5f106bd41e7007a58a3d18589a0dde73f8ed3 != aeb36b95b7ef1d4fb399f26cc3e0ade6a042e9b08f97dd6d9dd5f5983002afba 2022-02-13 12:39:35 Puller (folder “tunes_other” (tunes_other), item “fromcd\dbAR20190108\Queen\Live at the Rainbow '74, Disc 2\23 Queen - Jailhouse Rock.flac”): syncing: pull: hash mismatch 0ecdbae0b3cc5d225918caabc9317090b0a4114986a6a279b30cb53a6667b56c != eef51596710f01c0f3de097eb4514ebec426406fd4b4b41291399ebde5ae5774 2022-02-13 12:39:35 Puller (folder “tunes_other” (tunes_other), item “fromcd\dbAR20181117\Grateful Dead\1973-06-22 Vancouver BC, Disc 2\02 Grateful Dead - I Know You Rider.flac”): syncing: pull: hash mismatch 0b18e529372238cbf1b5b9392f89d20d96f42ba5088c613b01de28ea127d66cd != 469442378d9b6200fb0db00ba77a90b6dc902f8fc488d7861598ec0a63dd4ad9 2022-02-13 12:39:35 “tunes_other” (tunes_other): Failed to sync 13 items 2022-02-13 12:39:35 Folder “tunes_other” (tunes_other) isn’t making sync progress - retrying in 1h4m0s.

Please LMK your thoughts, thanks.

[tomasz86]

This is a weird one. I’ve personally never seen this particular error, and if you search for “hash mismatch” both on the forum and on Github, you’ll find mostly very old issues related to it. One possible clue is that the problem may happen with constantly changing files, but in your case, if I understand correctly, the errors keep popping up despite the files being static now.

I assume that restarting Syncthing, etc. hasn’t helped and the hardware isn’t faulty here. Is this correct?

[rramstad]

Yeah, the various systems have been rebooted, shut down, had the Syncthing software updated (if appropriate), manually clicked on rescan, and so forth.

The destination is my personal desktop which gets a lot of use and hasn’t been problematic in the past. The source is a relatively new server but utilizing older hardware that was a desktop system and a server for a while.

Note that there’s a good amount of data on these systems, a couple of terabytes, and Syncthing was fine with keeping it all sorted for several weeks… plus quite a bit of it is FLAC audio files which have an internal fingerprint, so if there was corruption, I’d be able to tell, most likely. (the 13 failed files on the source all pass flac testing, haven’t checked to see if the destination is also OK)

Any solutions most welcome. I have considered these options:

manually copy files from source to destination and hope that fixes it

source is a docker container, I don’t know where it keeps configuration files, but I could notate the configuration items, blow away the docker entirely, reinstall the docker container, this would get me a fresh configuration, duplicate the old folder configurations, let the system rescan, and hopefully the result would be actual updates at the destination

Maybe there’s a middle ground of some sort, I am new to Syncthing, if there are some commands I could run on either side, or some settings to fiddle with, that would be great.

I agree that my research only pointed out really old topics… and yes, these files shouldn’t be changing or locked or anything, and the source shows that the files on the destination are out of date, but the destination keeps complaining that the files don’t have the right hash.

[rramstad]

Manually copying the files didn’t work.

I then decided to try removing the folder configuration from the destination system, and re-adding it.

I ran into a few problems. First, once I removed it, I wasn’t able to re-add it directly from the destination system. I ended up having to go to the source system, tell that source system to no longer share the folder in question, wait for the source system to update and rescan, showing that the folder was no longer shared.

At that point on the source system, with the folder removed on the destination configuration, I then re-enabled sharing. I got a banner on the destination system asking me if I wanted to add the folder.

I confirmed that I wanted to add the folder and pointed the destination folder to the local storage.

The second problem was that the destination system gave me this warning:

[VIJZ6] 18:03:23 WARNING: Error on folder “tunes_other” (tunes_other): folder marker missing (this indicates potential data loss, search docs/forum to get information about how to proceed)

Given that the destination was receive only, and the material was still on the source system, and that the destination seemed to still have the data in question, after reading around, I copied an empty .stfolder from one of my other shares into that directory and started a scan.

The destination system seems to be happily scanning right now and I’m hopeful this will be a solution.

I have ran into a third minor issue – the destination system seems to be claiming that there are some locally changed items – this isn’t likely as I don’t modify any of these files, ever, it’s just a copy of the source – using a file compare tool I can see that they aren’t changed in any way from the source – so I will most likely just let the sync finish, and if the sync works, and I don’t keep getting the “pull: hash mismatch” errors, I’ll make sure all the locally changed items are in fact identical and then Revert Local Changes, and the configuration should be fine.

Going forward I’ve told both systems to not instantly see changed files – I think this error was caused by a lot of changes being made on the source, and some of the copying activity to the destination being skipped somehow – the bug may also only occur when the source is in Send Only and the destination is Receive Only.

I don’t need near real time mirroring so it’s not a big deal to turn the watch feature off and let it sync every few hours.

If my solution doesn’t work, I’ll post again.

[rramstad]

So, that attempt didn’t work. The destination chugged away and eventually said I had 13 files with local changes – 26 files out of sync – listing each twice. I then did Revert Local Changes and the system did that, after giving me a warning, and then I got the same “pull: hash mismatch” errors on the same 13 files.

I am now going to try to remove the folder at the destination GUI, then remove the folder at the source GUI, then re-add the source GUI folder, then re-add the destination GUI folder.

(for the curious, I know that some of these files are different on the source and that my theory is that the destination got very confused because the source was changing rapidly)

Fun stuff. Will repost if the newest attempt fails.

[tomasz86]

@rramstad

my theory is that the destination got very confused because the source was changing rapidly

I think there must be some other factors too. The reason is that I myself have also been operating on files with rapid changes (e.g. virtual disks that undergo constant changes until the specific VM is shut down), yet I’ve never encountered these kind of issues.

Just a guess, but maybe Unraid and SMB are playing the role here.

[rramstad]

I’m certainly willing to point the finger at SMB as the application in question that was modifying the files likes to open the files multiple times for reading with different file pointers and I do think that Syncthing may have gotten confused by that sort of access. (The application does a bunch of reading on multiple streams to the same file, then closes them, then opens one connection for writing and updates the file.)

That said, I do find it pretty frustrating that I can’t simply tell Syncthing to resolve the hash problems by flushing the destination hash cache and taking things from the source, or some other simple solution.

Right now I’ve blown away the folder configs on both sides, and am creating the folder config again on the source side, with a new folder identifier, and then will do the same on the destination side… at that point they should both have totally new hash data and shouldn’t complain.

[rramstad]

It did work to blow away folder configs on both sides, create them again with a new folder identifier, and then allow them to sync, with one caveat. The destination complained that there were many locally changed items – not sure why, don’t really care – and I did a revert local changes, which then kicked off a scan.

Once that destination scan finished, everything was in sync and I was no longer getting errors.

As I mentioned, I think the source was changing so rapidly that the destination got confused. That’s the only explanation I can come up with. The files have internal checksums and they all pass on the source, so I know it’s not data corruption.

(I suppose it might have been some sort of strange networking problem, but that seems unlikely… I guess we’ll never know.)