Failed items and "access is denied" in a Receive Encrypted folder

I have this problem in one of my Receive Encrypted folders. The folder is stuck in the following state, which doesn’t go away.

image

I have checked some of the paths, and they don’t actually exist physically at all. The logs don’t really say anything else than the above.

INFO: Scanner (folder "xxx" (m9sjf-uzd7w), item "2.syncthing-enc\\RI\\AR424G1V086TJM4E4PO4118448HUHN67KFG7S6E12JS471KD9L29MG8G8QTR4Q050LP3ADBN3FD9BNELR759Q0SEL26KOGU8TQRRE7OUK6BLLH2IAODK81BA9DS4N67RMG8QOFNAAQJKLTCQDUDA2VGJMMTCUM5AIGDRLPNL7FQIEHAM9MHHD9CGCK5FL4EQ9GEDPIFF"): scan: open \\?\C:\xxx\2.syncthing-enc\RI\AR424G1V086TJM4E4PO4118448HUHN67KFG7S6E12JS471KD9L29MG8G8QTR4Q050LP3ADBN3FD9BNELR759Q0SEL26KOGU8TQRRE7OUK6BLLH2IAODK81BA9DS4N67RMG8QOFNAAQJKLTCQDUDA2VGJMMTCUM5AIGDRLPNL7FQIEHAM9MHHD9CGCK5FL4EQ9GEDPIFF: Access is denied.

This is happening in Windows 10 v1607 (LTSB). There is nothing else that accesses these folders, and Windows Defender is disabled too. The folders themselves are located on a standard, local NTFS partition (although there is a mount point involved somewhere in the parent path, yet it links to a local storage device).

Do you have any idea what the problem may be about? Is there any debug logging that I could enable to help find more information on what is going on?

I don’t think there’s much more debug info to extract – it’s trying to open the named thing for reading, and it’s getting back a denial from the OS. Check permissions, ACLs, ownership, whatnot.

1 Like

This is not due to permissions. I’ve tried to create the path manually, and it fails due to being too long.

mkdir AR424G1V086TJM4E4PO4118448HUHN67KFG7S6E12JS471KD9L29MG8G8QTR4Q050LP3ADBN3FD9BNELR759Q0SEL26KOGU8TQRRE7OUK6BLLH2IAODK81BA9DS4N67RMG8QOFNAAQJKLTCQDUDA2VGJMMTCUM5AIGDRLPNL7FQIEHAM9MHHD9CGCK5FL4EQ9GEDPIFF
The full path of AR424G1V086TJM4E4PO4118448HUHN67KFG7S6E12JS471KD9L29MG8G8QTR4Q050LP3ADBN3FD9BNELR759Q0SEL26KOGU8TQRRE7OUK6BLLH2IAODK81BA9DS4N67RMG8QOFNAAQJKLTCQDUDA2VGJMMTCUM5AIGDRLPNL7FQIEHAM9MHHD9CGCK5FL4EQ9GEDPIFF is too long.

I think it could be the same problem as https://forum.syncthing.net/t/access-is-denied-on-long-file-paths/16097, although there are no UNC paths in my case. Syncthing is supposed to work around the long path limitation, isn’t it though? I wonder what’s failing on my system. For the record, the LongPathsEnabled registry tweak mentioned in the other topic is enabled.

In this state, I’d say this is kind of a deal breaker for me when it comes to Receive Encrypted folders (for now), because in contrary to normal folders, here I’ve got no control over path length, unless I put the folder directly in C:\ or similar.

The overall path in this case isn’t that long either. It’s basically something like C:\Users\User\Desktop\Folder1\Folder2\Folder3 and then Syncthing fails to create 2.syncthing-enc\RI\AR424G1V086TJM4E4PO4118448HUHN67KFG7S6E12JS471KD9L29MG8G8QTR4Q050LP3ADBN3FD9BNELR759Q0SEL26KOGU8TQRRE7OUK6BLLH2IAODK81BA9DS4N67RMG8QOFNAAQJKLTCQDUDA2VGJMMTCUM5AIGDRLPNL7FQIEHAM9MHHD9CGCK5FL4EQ9GEDPIFF inside it.

The error is from the scanner; it’s trying to open an existing file or directory for inspection, not create anything. Syncthing uses long filename APIs and UNC paths, your test does not – and you get a different error. But sure, it’s your system, only you can really figure out what’s going on.

This bug might be related: https://github.com/syncthing/syncthing/issues/7987

The reason why I’m sure it’s got nothing to do with permissions is because there is a dozen of other Syncthing folders located in the exact same root path, including a few other Receive Encrypted ones. All of them were created at the exact same time, however the problem occurs only in this one.

This is strange, because the folder/file called AR424G1V086TJM4E4PO4118448HUHN67KFG7S6E12JS471KD9L29MG8G8QTR4Q050LP3ADBN3FD9BNELR759Q0SEL26KOGU8TQRRE7OUK6BLLH2IAODK81BA9DS4N67RMG8QOFNAAQJKLTCQDUDA2VGJMMTCUM5AIGDRLPNL7FQIEHAM9MHHD9CGCK5FL4EQ9GEDPIFF doesn’t exist at all. That’s why I did the test trying to create it manually, ending up getting the path too long error.

Edit:

For testing purposes, I’ve created a very short junction at C:\TEST pointing at the normally longer folder path, and then pointed Syncthing at it. All the “access denied” errors have disappeared, which likely confirms that the problem is about the paths being too long (although not necessarily in Windows, but rather in Syncthing, if this is related to the issue linked above).

The linked issue above is unrelated to this. It’s more likely that the single path component exceeds some system limit. In hindsight we should have probably made that shorter, just because it doesn’t hurt us and there’s apparently a few limited filesystems out there that have problems with path components of length 200. Changing that now is likely not possible or at least cumbersome without doing a braking change.

Does this mean that nothing can be done except for using very short root paths? What is the maximum length that the encrypted paths can take?

There’s no limit on the entire path (I think it’s something like original length + 100) and a single path component has a 200 chars limit.

Hmm, I’ve just checked the logs again, and there is even something like this in there.

\\?\C:\TEST\M.syncthing-enc\VD\7M6Q7KAC4LR014NA8RSRBJ7662UUBO4G7KEEBGTGRKBB4OVGODNBAMKEHNHS9MSN5MO54VRHCNV2R395A6RDMCCASH0KDPAS92KBDPQIGBVOSHMD2U5PGNA20QRCS7VA2DBJKMKRUJCA6T15LHIOTMIVLHC7LESDAOOEFFPQORF77ELO2B9VJVQRI8K7J5MTUHD9727A: Access is denied.

The path here is only 227 characters total (excluding \\?\), so now I’m again not so sure what the problem is about. I can create the path manually if I want, without any special tricks. I’m going to do more tests later on on my main machine and see what happens in my test environment using the same folder.

The problem has come back despite using the short root path symlink, and in addition to that, there are also lots of local additions in the folder.

If I revert, the local additions do go away, but they seem to come back later regardless. Please keep in mind that this is happening with https://github.com/syncthing/syncthing/commit/708a5c2070977d6b162331a0fcbc6250e8e267ce present, so there must be something else causing the problem. I’m not sure whether the local additions and the failed items are actually related though, because the latter persist independently from the former.

I still haven’t got enough time to do proper debugging on the local computer, unfortunately, and I cannot really do anything on the remote device, since I control it through a remote desktop software only with a very slow connection, so doing anything there directly is difficult.

Edit:

I’ve set up a test instance on my local machine, shared the very same folder as RE with it, and then almost immediately after the initial sync got unexpected items in it.

My gut feeling says that this could be due to using ignore patterns like

!/folder/*
/folder

on the sending side. Still, there should be no local changes in a Receive Encrypted folder, right?

I’m attaching a log file with db,model enabled from the receiving device. Because this is a real folder, the log is huge though. Please have a look at it when you’ve got some free time.

syncthing_RE_unexpected_items.zip

Edit:

I’ve just wanted to give a quick update. The number of unexpected items keeps growing with time.

This must be caused by the ignore parent / exclude child wildcard ignore patterns, because I’ve experienced the exact same issue in normal Receive Only folders. The difference is that there I could add the same ignore patterns on the receiving device to work around them, while I cannot do the same in the Receive Encrypted folder. I’m going to try to reproduce this using a much smaller subset of data and then report the issue on GitHub with smaller logs that would be easier to analyse.

I’m not getting any “access is denied” errors on my own machine though, so I guess that something else must be going on on that specific device that’s not present here. I think the two problems (access denied and unexpected items) are likely unrelated. I’m planning to do more testing with longer root paths, and then, if there’s no difference, using a mount point in the parent (as it’s the case on the other device).

The problem isn’t ignore patterns, it’s that the modification time of the file on the receive-encrypted device gets updated. That results in an unexpected item. A subsequent change on the remote creates a conflict between that unexpected local change and the correct remote change. Syncthing then creates the conflict file when syncing. You can identify the sequence by the Length and the 0x8 local flag (locally changed).

[IDWWV] 2021/10/21 10:42:42.397749 lowlevel.go:264: DEBUG: insert (local); folde
r="rs5cy-zuem4" File{Name:"U.syncthing-enc/O0/E29A1LB5ESLLEPC8IH80HLAL07M8HO5PJN
UV5RTRH63NH8N33OLBESH2S0IMER3LPLNHOM68ADMRACGVURAETO2F45PNDSHM2FKAN", Sequence:3
7269, Permissions:0644, ModTime:2009-02-14 08:31:30 +0900 KST, Version:{[{AAAAAA
A 1634780429}]}, VersionHash:, Length:68942, Deleted:false, Invalid:false, Local
Flags:0x0, NoPermissions:false, BlockSize:131112, Blocks:[Block{0/68455/0/d468ad
f0f04e538fc0b21be70747a655616428ada18910cb1b03180dc49a2517cef42f5394884b73c62209
3dcaaccdef}], BlocksHash:8828a15e26e01655ee42ad69b663f5ba11571f623b0b17187dcd8d3
f6710f8d3}
[...]
[IDWWV] 2021/10/21 10:44:03.145378 lowlevel.go:264: DEBUG: insert (local); folder="rs5cy-zuem4" File{Name:"U.syncthing-enc/O0/E29A1LB5ESLLEPC8IH80HLAL07M8HO5PJNUV5RTRH63NH8N33OLBESH2S0IMER3LPLNHOM68ADMRACGVURAETO2F45PNDSHM2FKAN", Sequence:37303, Permissions:0644, ModTime:2021-10-21 10:42:40.5978892 +0900 KST, Version:{[{AAAAAAA 1634780429} {IDWWV34 1634780643}]}, VersionHash:, Length:68942, Deleted:false, Invalid:false, LocalFlags:0x8, NoPermissions:true, BlockSize:131112, Blocks:[], BlocksHash:}
[...]
[IDWWV] 2021/10/21 10:44:03.299408 lowlevel.go:264: DEBUG: insert (local); folder="rs5cy-zuem4" File{Name:"U.syncthing-enc/O0/E29A1LB5ESLLEPC8IH80HLAL07M8HO5PJNUV5RTRH63NH8N33OLBESH2S0IMER3LPLNHOM68ADMRACGVURAETO2F45PNDSHM2FKAN", Sequence:37304, Permissions:0644, ModTime:2009-02-14 08:31:30 +0900 KST, Version:{[{AAAAAAA 1634780638}]}, VersionHash:, Length:68956, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:false, BlockSize:131112, Blocks:[Block{0/68469/0/e6bd0d9ee00b4b3b71edb0adc734d93ae9e08da8f8b98d88decd6d7a53b8343977d7636daf6d89a8b0f2595fbbc81572}], BlocksHash:05039d85df7e1776ba1d18fc80021c9f3920fcfec007b944e590312bf6b44a36}
[...]
[IDWWV] 2021/10/21 10:44:03.730495 lowlevel.go:264: DEBUG: insert (local); folder="rs5cy-zuem4" File{Name:"U.syncthing-enc/O0/E29A1LB5ESLLEPC8IH80HLAL07M8HO5PJNUV5RTRH63NH8N33OLBESH2S0IMER3LPLNHOM68ADMRACGVURAETO2F45PNDSHM2FKAN.sync-conflict-20211021-104403-", Sequence:37305, Permissions:0644, ModTime:2021-10-21 10:42:40.5978892 +0900 KST, Version:{[{IDWWV34 1634780643}]}, VersionHash:, Length:68942, Deleted:false, Invalid:false, LocalFlags:0x8, NoPermissions:true, BlockSize:131072, Blocks:[], BlocksHash:}

This in principle is all correct, just that the conflict on an encrypted device isn’t very helpful. However we also don’t want to remove that file automatically, for the usual reasons (Syncthing should never delete any data automatically). I don’t have a good idea how to better handle this scenario right now and the current behaviour is safe and can be resolved, even though it’s admittedly not obvious what happens.

What causes the modification on the receive-encrypted device though? This is happening all automatically. Neither I nor any other software is touching anything there on purpose. I just set the sync and forget it, then the unexpected items start appearing almost immediately :upside_down_face:.

This is not happening in any other folders, only this one, for some reason, hence my suspicious regarding the ignore patterns, since I only use such patterns in this specific folder. If the culprit is something else, even better.

Just a note, the logs come from my test instance, where I try to run Syncthing with default settings, while normally I run everything with conflicts set to 0. I’m not sure what exactly is going on the real device yet, but the number of locally changed files is much smaller there.

Edit:

I’ve just remotely checked the other device, and there were several local changes in basically all Receive Encrypted folders there, so I was too quick to judge that the problem was limited to one specific folder. I can say for sure though that the access denied issue hasn’t happened in any of the other folders so far.

@tomasz86 Please see https://forum.syncthing.net/t/unexpected-items-on-remote-untrusted-rpi3

Your issue is very similar and maybe a duplicate of the issues I have been reporting for my encrypted folder(s). Essentially, my problem occurs because the modtime is not correctly changed on some modified files on the encrypted side. My experiments with a parallel unencrypted folder using rsync is evidence of the modtime difference. Reverting the changes (/rest/db/revert) ensures files are in sync and the Unexpected Items are deleted. See Simon’s bug report at https://github.com/syncthing/syncthing/issues/7991

Yeah, I’ve been following your thread closely :slightly_smiling_face:.

The thing is that I’ve added that commit to my custom build (which I mentioned in one of my previous posts, but this thread is kind of long, so it’s probably easy to miss), yet I’m still encountering these local additions, so the culprit is likely something else.

Sorry, I meant issue 7987, but Simon thinks that is not relevant, he’s asked for more logs. It’s late here, a job for tomorrow.

Actually, the changed file here is a mistake in a change I made for 1.18.4-rc.1: https://github.com/syncthing/syncthing/pull/7992.

1 Like

Thank you very much for the quick fix :relieved:. I’ve cherry-picked the commit and I’m testing everything right now. There’s been no local additions so far.

I will come back in case something changes. I’m also still going to investigate the original issue of “access is denied”, but it may take a while, since I’m not getting the errors on my home computer, which means that I’ll probably have to do the testing on the remote device.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.