Is it possible for Syncthing not to lock files in Windows when hashing them?

I’m not sure whether this belongs to Development or Support, but I thought that the former fits better, so I’m creating this topic here.

When hashing files in Windows, Syncthing locks them. This means that such a file cannot be moved/renamed/deleted. I have personally experienced a problem related to this when running some software in Syncthing folders, which from time to time threw errors due to being unable to manipulate its own files.

However, the reason for creating this topic is that today I have received a help request from someone, where a 4.5 GB-sized file was being locked by Syncthing, which prevented them from moving it to a different location. The hardware is decent, but hashing such a large file still took a few minutes, which is understandable.

My question is then, is locking files in order to hash them inevitable, or is there any workaround that could prevent Syncthing from doing so? For instance, the files are not locked when they are being pushed to other devices, i.e. we can still manipulate them on the Windows machine. It seems to be during hashing that they become locked.

It doesn’t explicitly lock them.

We simply open them for reading using the standard go apis.

1 Like

I wonder if there exists any kind of a solution or a hack to get around this then :innocent:.

For the record, Dropbox has the same problem, which is actually much worse there, because it locks files until they have been uploaded to their servers, which may take quite a while. With Syncthing, it seems to only be the hashing that locks them.

The only user-side workaround that I can think of right now is to delay file watcher by a significant amount of time, e.g. 5 minutes, so that the user has enough time to do their file operations before Syncthing accesses the files.

I assume you are seeing this on a Windows machine. The file is not actually “locked” by anyone, but it’s just how Windows works. In contrast to many other (e.g. UNIX-style) operating systems, Windows tries to forbid things going away as long as they’re in use (opened) by some piece of software. The solution is easy, use a sane operating system :wink:

IMHO this Windows behavior has always been unfortunate to put it mildly. In a 1970s world where computers were hard wired room-filling beasts, nothing could ever disappear hardware-wise. But with the introduction of USB storage devices at the latest, the assumption that stuff can be disallowed from disappearing is simply wrong. Try deleting a 12 GB video file in Linux while watching it. That will happily work and the video will still keep playing until the end, because the actual data is only freed when the last accessor closes the file. Such graceful failure mechanisms are not possible in Windows, because the one deleting the file will get an access denied error. If a file disappears because the hardware is unplugged, a read / write error will occur, and the same could be expected and handled by any program using any file at any time. There is simply no way to prevent things from disappearing, so Windows trying to ensure that by giving errors to any other offending program is a lost cause from the start.

That said, the Shadow Copy function could be used by Syncthing to work on a consistent filesystem snapshot while others can happily continue modifying and deleting files. That could even be supported on some other platforms’ file systems, but it is far from universal or portable. So there is currently no code in Syncthing to use such workarounds only on Windows systems.

1 Like

“Locked” is literally the terminology used by MS (e.g. see https://techcommunity.microsoft.com/t5/windows-blog-archive/the-case-of-the-mysterious-locked-file/ba-p/723349) :wink:. Windows itself displays a warning about the file or folder being “in use”.

image

Yeah, although as mentioned above, the behaviour is not exactly consistent. If you delete the file while Syncthing is pushing it to a remote device, Windows will not complain and happily perform the operation.

In addition to that, I have myself been replacing the Syncthing binary on Linux with a different version while still running, then press restart in the GUI, and it will happily restart using the new binary. This will never work on Windows, which will not allow to replace the binary like that in the first place.

Apparently Go’s file open sets FILE_SHARE_READ | FILE_SHARE_WRITE which means to allow concurrent reads and writes to the file by others while we have it open. It does not set FILE_SHARE_DELETE which would allow delete and rename concurrently with it being opened by us. I’m not sure if that’s intentional or just an oversight on the part of whoever wrote this Go code. Someone could open an issue on golang/go to clarify. (Generally Go strives to default to Unix-like behavior, thus disabling locking by default, so I would tend to think this is an oversight and not intentional.)

We could use syscall.CreateFile directly in our filesystem abstraction for Windows and set FILE_SHARE_DELETE there, instead of os.Open. Someone could investigate that and file an issue + PR if they cared enough. :wink:

Aaaactually… I think this is the one point where the two are kind of similar? We do the rename-and-replace (but not delete) dance with the running binary on Windows during auto updates, and Linux does in fact reject opening a running binary for writing (as opposed to renaming or deleting it):

% scp syncthing ams2:
scp: ./syncthing: Text file busy

(the error is from the remote, where scp does not do the write-to-temp-file-and-rename by default, and of course in Unix terminology “text file” means “running binary” because code lives in the TEXT segment of the executable… :roll_eyes: )

:slight_smile:

2 Likes

This is kind of important for me, but to tell the truth, I have no idea how and where to modify the Syncthing code to add this functionality. At least for testing purposes, would it be enough to change

sharemode := uint32(FILE_SHARE_READ | FILE_SHARE_WRITE)

to

sharemode := uint32(FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE)

in Go, compile local Go binaries, and then use those to compile Syncthing?

You don’t need to recompile go, as that is just the compiler, the standard library is not baked into it. I think it might be enough to change the standard library code and I think go would recompile the standard library before compiling syncthing.

That atleast was the case when I was testing some networking changes.

1 Like

I modified syscall_windows.go in C:\Program Files\Go\src, then recompiled Syncthing. Go picked the changes automatically and built new binaries. I have compared them with the previous ones, and the hash indeed differed.

Then, I did some quick testing with large files using slow storage, and it was possible to move them out of the Syncthing folder while it was still hashing them. It took a few more seconds for the Web GUI to update its state, but regardless, the solution seems to be working!

Relevant proposals, both declined:

1 Like

This is unfortunate :confused:.

Here, the user keeps complaining about their files being locked, preventing them from moving them to a different location. Because of this, tomorrow we are planning to switch from the official binary to a custom built one (with FILE_SHARE_DELETE implemented), at least temporarily, since this has been hindering their work too much.