Is it possible for Syncthing not to lock files in Windows when hashing them?

tomasz86 · July 15, 2021, 12:15pm

I’m not sure whether this belongs to Development or Support, but I thought that the former fits better, so I’m creating this topic here.

When hashing files in Windows, Syncthing locks them. This means that such a file cannot be moved/renamed/deleted. I have personally experienced a problem related to this when running some software in Syncthing folders, which from time to time threw errors due to being unable to manipulate its own files.

However, the reason for creating this topic is that today I have received a help request from someone, where a 4.5 GB-sized file was being locked by Syncthing, which prevented them from moving it to a different location. The hardware is decent, but hashing such a large file still took a few minutes, which is understandable.

My question is then, is locking files in order to hash them inevitable, or is there any workaround that could prevent Syncthing from doing so? For instance, the files are not locked when they are being pushed to other devices, i.e. we can still manipulate them on the Windows machine. It seems to be during hashing that they become locked.

AudriusButkevicius · July 15, 2021, 1:04pm

It doesn’t explicitly lock them.

We simply open them for reading using the standard go apis.

tomasz86 · July 15, 2021, 3:59pm

I wonder if there exists any kind of a solution or a hack to get around this then .

For the record, Dropbox has the same problem, which is actually much worse there, because it locks files until they have been uploaded to their servers, which may take quite a while. With Syncthing, it seems to only be the hashing that locks them.

The only user-side workaround that I can think of right now is to delay file watcher by a significant amount of time, e.g. 5 minutes, so that the user has enough time to do their file operations before Syncthing accesses the files.

acolomb · July 15, 2021, 9:06pm

I assume you are seeing this on a Windows machine. The file is not actually “locked” by anyone, but it’s just how Windows works. In contrast to many other (e.g. UNIX-style) operating systems, Windows tries to forbid things going away as long as they’re in use (opened) by some piece of software. The solution is easy, use a sane operating system

IMHO this Windows behavior has always been unfortunate to put it mildly. In a 1970s world where computers were hard wired room-filling beasts, nothing could ever disappear hardware-wise. But with the introduction of USB storage devices at the latest, the assumption that stuff can be disallowed from disappearing is simply wrong. Try deleting a 12 GB video file in Linux while watching it. That will happily work and the video will still keep playing until the end, because the actual data is only freed when the last accessor closes the file. Such graceful failure mechanisms are not possible in Windows, because the one deleting the file will get an access denied error. If a file disappears because the hardware is unplugged, a read / write error will occur, and the same could be expected and handled by any program using any file at any time. There is simply no way to prevent things from disappearing, so Windows trying to ensure that by giving errors to any other offending program is a lost cause from the start.

That said, the Shadow Copy function could be used by Syncthing to work on a consistent filesystem snapshot while others can happily continue modifying and deleting files. That could even be supported on some other platforms’ file systems, but it is far from universal or portable. So there is currently no code in Syncthing to use such workarounds only on Windows systems.

tomasz86 · July 16, 2021, 12:39am

“Locked” is literally the terminology used by MS (e.g. see https://techcommunity.microsoft.com/t5/windows-blog-archive/the-case-of-the-mysterious-locked-file/ba-p/723349) . Windows itself displays a warning about the file or folder being “in use”.

Yeah, although as mentioned above, the behaviour is not exactly consistent. If you delete the file while Syncthing is pushing it to a remote device, Windows will not complain and happily perform the operation.

In addition to that, I have myself been replacing the Syncthing binary on Linux with a different version while still running, then press restart in the GUI, and it will happily restart using the new binary. This will never work on Windows, which will not allow to replace the binary like that in the first place.

calmh · July 16, 2021, 4:52am

Apparently Go’s file open sets FILE_SHARE_READ | FILE_SHARE_WRITE which means to allow concurrent reads and writes to the file by others while we have it open. It does not set FILE_SHARE_DELETE which would allow delete and rename concurrently with it being opened by us. I’m not sure if that’s intentional or just an oversight on the part of whoever wrote this Go code. Someone could open an issue on golang/go to clarify. (Generally Go strives to default to Unix-like behavior, thus disabling locking by default, so I would tend to think this is an oversight and not intentional.)

We could use syscall.CreateFile directly in our filesystem abstraction for Windows and set FILE_SHARE_DELETE there, instead of os.Open. Someone could investigate that and file an issue + PR if they cared enough.

Aaaactually… I think this is the one point where the two are kind of similar? We do the rename-and-replace (but not delete) dance with the running binary on Windows during auto updates, and Linux does in fact reject opening a running binary for writing (as opposed to renaming or deleting it):

% scp syncthing ams2:
scp: ./syncthing: Text file busy

(the error is from the remote, where scp does not do the write-to-temp-file-and-rename by default, and of course in Unix terminology “text file” means “running binary” because code lives in the TEXT segment of the executable… )

tomasz86 · July 19, 2021, 2:02pm

calmh:

Apparently Go’s file open sets FILE_SHARE_READ | FILE_SHARE_WRITE which means to allow concurrent reads and writes to the file by others while we have it open. It does not set FILE_SHARE_DELETE which would allow delete and rename concurrently with it being opened by us. I’m not sure if that’s intentional or just an oversight on the part of whoever wrote this Go code. Someone could open an issue on golang/go to clarify. (Generally Go strives to default to Unix-like behavior, thus disabling locking by default, so I would tend to think this is an oversight and not intentional.)

We could use syscall.CreateFile directly in our filesystem abstraction for Windows and set FILE_SHARE_DELETE there, instead of os.Open. Someone could investigate that and file an issue + PR if they cared enough.

This is kind of important for me, but to tell the truth, I have no idea how and where to modify the Syncthing code to add this functionality. At least for testing purposes, would it be enough to change

sharemode := uint32(FILE_SHARE_READ | FILE_SHARE_WRITE)

to

sharemode := uint32(FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE)

in Go, compile local Go binaries, and then use those to compile Syncthing?

AudriusButkevicius · July 19, 2021, 2:10pm

You don’t need to recompile go, as that is just the compiler, the standard library is not baked into it. I think it might be enough to change the standard library code and I think go would recompile the standard library before compiling syncthing.

That atleast was the case when I was testing some networking changes.

tomasz86 · July 19, 2021, 2:58pm

I modified syscall_windows.go in C:\Program Files\Go\src, then recompiled Syncthing. Go picked the changes automatically and built new binaries. I have compared them with the previous ones, and the hash indeed differed.

Then, I did some quick testing with large files using slow storage, and it was possible to move them out of the Syncthing folder while it was still hashing them. It took a few more seconds for the Web GUI to update its state, but regardless, the solution seems to be working!

imsodin · July 22, 2021, 1:50pm

Relevant proposals, both declined:

github.com/golang/go

proposal: os: Create/Open/OpenFile() set FILE_SHARE_DELETE on windows

opened 07:17PM - 16 May 19 UTC

closed 05:09PM - 09 Oct 19 UTC

networkimprov

FrozenDueToAge OS-Windows Proposal Proposal-FinalCommentPeriod release-blocker

On Linux & MacOS we can write this; on Windows it fails with a "sharing violatio…n": ``` path := "delete-after-open" fd, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0600) if err != nil { ... } err = os.Remove(path) // or os.Rename(path, path+"2") if err != nil { ... } fd.Close() ``` If you develop on Windows and deploy to Linux etc, and your code relies on this _undocumented_ GOOS=windows behavior of os.Rename() & .Remove(), it is broken and perhaps vulnerable. Note that package "os" has _fifteen mentions_ of other Windows-specific behavior. To fix this, syscall.Open() at https://golang.org/src/syscall/syscall_windows.go#L272 needs `sharemode |= FILE_SHARE_DELETE` **Microsoft** recommends this be made the default: https://github.com/golang/go/issues/32088#issuecomment-502850674 **Rust** made it a default: https://doc.rust-lang.org/std/os/windows/fs/trait.OpenOptionsExt.html **Mingw-w64** made it a default seven years ago: https://sourceforge.net/p/mingw-w64/code/HEAD/tree/stable/v3.x/mingw-w64-headers/include/ntdef.h#l858 **Erlang** made it a default over six years ago: erlang/otp@0e02f48 **Python** couldn't adopt it due to limitations of MSVC runtime: https://bugs.python.org/issue15244 ~Therefore syscall.Open() should *use file_share_delete by default*, and syscall should provide both: a) a global switch to disable it (for any existing apps that rely on its absence), and b) a flag for use with os.OpenFile() to disable it on a specific file handle.~ Update after https://github.com/golang/go/issues/32088#issuecomment-532784947 by @rsc: a) os.Create/Open/OpenFile() should *always enable file_share_delete* on Windows, b) syscall.Open() on Windows should accept a flag which enables file_share_delete, and c) syscall on Windows should export a constant for the new flag. The docs for os.Remove() should also note that to reuse a filename after deleting an open file on Windows, one must do: `os.Rename(path, unique_path); os.Remove(unique_path)`. Win API docs https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-deletefilea https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-createfilea If there is a reason not to do this, it should be documented in os.Remove() & .Rename(). cc @alexbrainman @gopherbot add OS-Windows

github.com/golang/go

proposal: syscall: define Windows O_ALLOW_DELETE for use in os.OpenFile

opened 08:26PM - 03 Oct 19 UTC

closed 06:08PM - 15 Jan 20 UTC

rsc

FrozenDueToAge Proposal

The discussion on #32088 has so far been unable to reach consensus on setting FI…LE\_SHARE\_DELETE unconditionally in os.OpenFile. Right now it looks likely to be declined, possibly revisited in a few years. In the interim, it is more difficult on Windows than on Unix to pass an extra flag to open a file with package syscall. syscall.Open takes fake O\_\* flags, not actual Windows permission bits. It wraps syscall.CreateFile, but it does a lot of argument preparation before making that call: // package syscall func Open(path string, mode int, perm uint32) (fd Handle, err error) { if len(path) == 0 { return InvalidHandle, ERROR_FILE_NOT_FOUND } pathp, err := UTF16PtrFromString(path) if err != nil { return InvalidHandle, err } var access uint32 switch mode & (O_RDONLY | O_WRONLY | O_RDWR) { case O_RDONLY: access = GENERIC_READ case O_WRONLY: access = GENERIC_WRITE case O_RDWR: access = GENERIC_READ | GENERIC_WRITE } if mode&O_CREAT != 0 { access |= GENERIC_WRITE } if mode&O_APPEND != 0 { access &^= GENERIC_WRITE access |= FILE_APPEND_DATA } sharemode := uint32(FILE_SHARE_READ | FILE_SHARE_WRITE) var sa *SecurityAttributes if mode&O_CLOEXEC == 0 { sa = makeInheritSa() } var createmode uint32 switch { case mode&(O_CREAT|O_EXCL) == (O_CREAT | O_EXCL): createmode = CREATE_NEW case mode&(O_CREAT|O_TRUNC) == (O_CREAT | O_TRUNC): createmode = CREATE_ALWAYS case mode&O_CREAT == O_CREAT: createmode = OPEN_ALWAYS case mode&O_TRUNC == O_TRUNC: createmode = TRUNCATE_EXISTING default: createmode = OPEN_EXISTING } h, e := CreateFile(pathp, access, sharemode, sa, createmode, FILE_ATTRIBUTE_NORMAL, 0) return h, e } A direct call to syscall.CreateFile would have to duplicate all that code. A direct call to syscall.Open would still lose out on the adjustments made inside os.Open, most importantly the call to fixLongPath: // package os func openFile(name string, flag int, perm FileMode) (file *File, err error) { r, e := syscall.Open(fixLongPath(name), flag|syscall.O_CLOEXEC, syscallMode(perm)) if e != nil { return nil, e } return newFile(r, name, "file"), nil } If package syscall on Windows defined a bit O\_ALLOW\_DELETE (maybe 0x100000), then syscall.Open could convert that bit into FILE\_SHARE\_DELETE. Then it would be easy for calls to either syscall.Open or os.OpenFile to cause the underlying call to use FILE\_SHARE\_DELETE if they really needed it. This proposal assumes #32088 is declined. It should be considered withdrawn if #32088 is accepted.

tomasz86 · July 22, 2021, 2:06pm

This is unfortunate .

Here, the user keeps complaining about their files being locked, preventing them from moving them to a different location. Because of this, tomorrow we are planning to switch from the official binary to a custom built one (with FILE_SHARE_DELETE implemented), at least temporarily, since this has been hindering their work too much.

Catfriend1 · August 27, 2023, 5:40pm

@tomasz86 Where exactly did you modify syscall_windows.go?

tomasz86 · August 27, 2023, 8:16pm

You just need to replace sharemode := uint32(FILE_SHARE_READ | FILE_SHARE_WRITE) with sharemode := uint32(FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE).

I use this PowerShell script to do this automatically after every new Go installation.

$SyscallWindowsGoFile = Join-Path -Path $Env:ProgramFiles -ChildPath "go" | Join-Path -ChildPath "src" | Join-Path -ChildPath "syscall" | Join-Path -ChildPath "syscall_windows.go"

if (Test-Path -LiteralPath $SyscallWindowsGoFile -PathType Leaf) {
	(Get-Content -LiteralPath $SyscallWindowsGoFile) -replace "sharemode := uint32\(FILE_SHARE_READ \| FILE_SHARE_WRITE\)","sharemode := uint32(FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE)" | Set-Content -LiteralPath $SyscallWindowsGoFile
}

Pause

mraneri · August 28, 2023, 3:38am

For what it’s worth I hit this issue regularly. I’m regularly moving and or renaming files in watched folders which causes rehashing and problems renaming parent directories, or whatever. I have to launch the UI, pause the folder to stop the hashing and then remember to resume the folder when I am done. It would be great for this fix to make it into the production release. Syncthing should ideally work without interfering with what we are trying to do with the files.

gadget · August 28, 2023, 4:21am

Not sure it would help with rehashing since Syncthing won’t know that a moved/renamed file contains previously known blocks until it hashes it.

The exact behavior also depends on the underlying filesystem. Microsoft’s ReFS, Oracle’s ZFS/OpenZFS, XFS, btrfs plus other copy-on-write (COW) filesystems support deletes, renames, and writing (including overwriting a binary executable that’s in use).

My Syncthing folders are on a btrfs volume so renaming/overwriting/deleting files and directories hasn’t been an issue. Having the volume on a SSD also makes disk I/O from Syncthing’s rehashing low impact.

There are a few hashing-related parameters that can be tuned if the overhead is slowing down your system.

mraneri · August 28, 2023, 11:20am

Don’t care about the rehashing speed. But I would like the OS to allow me to move the file or rename the parent directory while the hashing is ongoing. If the file disappears during the hashing just move onto the next file and queue a rescan or something.

calmh · August 28, 2023, 11:35am

To be clear, this is not about importing a fix into Syncthing. It’s the Go standard library that needs the change.

tomasz86 · August 28, 2023, 11:41am

Implementing what you mentioned previously would be fine, right?

Because I’m eventually planning to give it a try sometime in the future .

calmh · August 28, 2023, 11:42am

Depends on what the PR looks like. If it ends up duplicating hundreds of lines of internal standard library code that we then need to maintain in perpetuity it may not, in fact, be fine. If the ugliness is manageable, sure. Maybe we already do all the heavy lifting required.

Catfriend1 · September 10, 2023, 10:00am

Did I unerstand that correctly that Syncthing does not lock files on Linux?

So for example if my backup tool writes to/deletes from the folder while Syncthing is scanning this same folder, the backup tool won’t be blocked to do so.

I’d like to be sure that Syncthing operation won’t affect the backup files while they are created. That’s why I let Syncthing only run by manually issuing “systemctl start Syncthing” after made sure no backup jobs were running at the same time. And after syncing, I manually stopped it. If that can be configured to always run Syncthing in the background without additional data integrity risk, I would be very glad