How to sync files that are open with read-only access?

When viewing continuous log files or collaborating on projects it’s very often that I will have files with full rw privileges open as read-only. However with default settings syncthing will not update these files while they are opened locally. I would assume this is easy to do, but where is the setting to allow overwriting files that are opened locally as read-only?

Not sure what you mean.

Syncthing always replaces files, so it downloads a new file with a temporary name, and once the download is finished, deletes old file and renames the temporary one to it’s real name.

This is done to make sure that you never see partial content as the download is happening.

There is no setting to change that, as this is incredibly important not to corrupt files as part of a crash/power outage (as you can’t tell difference between a partially downloaded file due to crash and user change)

You can’t overwrite/rename files that are locked/in use etc, there is no flag to do that, as it’s a safeguard of the operating system.

One main use for opening files with “read-only” access is specifically for cases like this, where you want some other program to be able to update a file, while it’s “in use”, as long as you open it with read access. I know windows and linux at least allow this. I would assume it’s a basic functionality of all operating systems with file systems that allow opening files with rw or r access?

On Windows it’s app-specific. The application with a file open must set a flag allowing other applications to also open the same file in parallel. There’s really nothing Syncthing can do to insure that it can always overwrite a file that’s in use.

On Linux, filesystems that support copy-on-write (COW) will allow multiple readers at the same time as a writer because a file is copied first whenever a process wants write access.

I don’t know about Windows, but on Linux, because Syncthing writes to a temp file, the temp file has a new inode, so the file rename step simply reassigns the filename to the new inode. Whatever process has the original file open is using a file handle that points to the old inode, which will be freed as soon as the file handle is closed. But ultimately it’s also not under Syncthing’s control.

Linux: does the end result not mean that files “in use” but opened as read only, can freely be modified by other processes? Including renaming (mv)? If so, syncthing would not need to care about if the file is in use or not, only that it is able to obtain rw access to it.

Windows: this sounds much trickier! I’m assuming this means that each app will have its own app specific flag? But in that case it would mean that the os does not enforce that only one program accesses a file with write privileges, but that apps are responsible for that themselves. That sounds very strange, with all kinds of potential race condition issues. Is that really how windows works?

EDIT: some quick searching seems to imply that files in windows can be opened with a OS specific “write sharing enabled” flag. And files that are opened that way can be opened with write access by other processes. If that is indeed true, it should be trivial for synthing to be able to securely write (or rename) to files like that?

Usually, but it depends on the filesystem and also the app.

For example, Firefox uses a lock file to detect when multiple instances of Firefox are trying to use the same profile so you’d get a warning from Firefox. But you could do a more ~/.mozilla/firefox/default/prefs.js and then launch Firefox without Firefox having any problems with updating its preferences file.

For renaming (mv), as long as a file remains in the same filesystem, an existing file handle is still good. You could start playing a music file, then rename the file (including moving it to another directory), and the playback will continue uninterrupted. It’s only when the music app tries to replay the track again that an error might occur because the old pathname is no longer valid.

The problem is that Syncthing doesn’t have a guarantee that every potential process that opens a file being watched is enabling the write sharing flag.

Renaming a file that’s in use on Windows is generally not possible. Try the following:

  1. Use Windows Notepad to create a text file named note.txt with more than 25 lines.
  2. In a command prompt, open the text file in read-only mode via more note.txt.
  3. Open Windows Explorer and rename note.txt.

Ok, so linux is basically easy, and syncthing could with it’s current operating type handle that. Sync over file to a temp name. Check if target file is available for write access, and if it is, rename and delete.

Windows would then apparently require some more work from syncthing. Sync over new file to temp name, if target file is set as “write share enable” read the parts from temp file needed, and write to the target file directly. Delete temp file. More potential for issues for sure, like what happens if the write process gets interrupted etc. And would need a change in how syncthing works.

For multi-user projects though, I don’t see any other solution to enable using syncthing? (many programs specifically have “collaboration” or “multi-user” modes where they can set some files to read only, and check if the file has been updated on disk to update the “view” of it)

Everyone involved in the projects switch to Linux? :wink:

In Linux it’s a very common practice to open a log file for reading while a service is continuously appending to it. You can even watch updates in near real-time (e.g., tail -0f some.log).

Out of curiosity, I ran a little test on a Linux NAS with btrfs-formatted volumes that provides SMB network shares and wasn’t able to rename a file in Windows while the file was open read-only. So Windows applies the restriction to even network filesystems.

A Windows solution will likely involve some kind of kludge. One potential option is VSS, but it might be overly cumbersome because you’d have to create a snapshot → programs open read-only from the snapshot → delete snapshot → create snapshot → read → delete → create… (rinse and repeat).

However, a hub-and-spoke setup where all of the shared files are hosted on a Linux server and provided to Windows hosts via SMB would work. Programs on Windows can open files from the network shares provided by the Linux server while Syncthing on the server is updating files as needed. Ideally, use ZFS, btrfs or another copy-on-write filesystem on the server. For extra measure, make the network shares read-only so that Windows won’t allow read/write requests from any programs.

That’s what I currently have setup. A linux system that has windows computers reading and writing to it. (unfortunately some software that I need is only available for windows :frowning: )

Moving over to syncthing would be very interesting for multiple reasons. Being able to update files open for read only is one thing that would need solving before that’s possible though.

How are you envisioning using Syncthing in your current setup?

Ideally I would like to only need to use syncthing. Would bring a lot of convenience, some performance, and future simplicity benefits. One same system for off site and on site needs. Maybe adjust the package sync delay time down for LAN links, and keep the default for laptop/offsite use :slight_smile: As far as I have been able to figure out, this is the only issue currently preventing “full use” at least in my case.

There is no work to be done by syncthing. Syncthing already does the right thing, its all other applications that don’t. I.e open files with locking or exclusive access.

Syncthing can’t fix/make other applications behave.

It sounds like you want to replace your Linux file server with Syncthing installed on every device for a peer-to-peer file share configuration – the benefits of centralized file sharing, but without a central storage hub.

If so, it’s unfortunately not viable with Windows devices in the mix. There’s nothing Syncthing can do about the file open modes requested by Windows programs and the limitations of NTFS.

Yes, I would love to replace the complicated “multi-level” file access/sharing system with just one system. Would be more reliable, better performance, much more scalable, and easier to maintain.

Technically possible for sure even on windows (updating files that programs have open for read is done all the time, every day in my use case, which is very ordinary).

But would probably be a big job in syncthing’s case to change how the files are updated. Maybe not a realistic update to do. Maybe I will one day find a single solution system to use :slight_smile:

The question here is if syncthing should update/overwrite/rename to local files that are only open with read access, when syncthing could obtain write access to that file.

Currently it does not update a file if it is “in use”, regardless of if there is write access available or not.

Syncthing attempts to do that. It doesn’t check if a file is open in any manner, it just prepares a new version and attempts to replace the old version. If that fails it’s because it was prevented by the OS.

1 Like

True, it’s technically possible on Windows, but only under limited circumstances. :smirk:

1 Like

On Windows…

  • Renaming a file already open by another process – read-only or not – isn’t allowed.
  • Overwriting a file already open by another process isn’t allowed.
  • Inserting/appending/replacing bytes in a file already open by another process is allowed, but only if the other process sets the appropriate flag.

It leaves Syncthing only one option – modifying existing files in-place. While conceptually simple, it involves way too many scenarios where things could go south.

While log files can often be appended to while already open by other programs, there are a lot of other file types where modifying the contents in-place will cause problems even if other programs only have them open in read-only mode.

Ah then I misunderstood!

So on the linux side this should already “work”. But on windows, syncthing would require functionality change to write the changes into “opened” files, since they can’t be renamed?

I wonder if windows could be workarounded with WSL somehow…

It turns out to be more difficult than I’d originally thought. In Windows on a NTFS volume, if a file is open read-only but allows shared writing by another process, data can be inserted and replaced. However, the file can only remain the same size or grow, it cannot shrink.

Over multiple tests the file got corrupted while attempting to delete data. After the file handle was released a review of the file contents didn’t match what had been shown in the application window during the write process (the disturbing part was there was no warning from Windows).

So while it’s possible to replace the contents of an open file, the final results can be indeterminate if the updated file is smaller than the file it’s replacing.

WSL 1 and WSL 2 are two very different beasts…

  • WSL 1 is a Linux compatibility layer (essentially what Wine is to Linux).
  • WSL 2 relies on Hyper-V to run a semi-custom version of Ubuntu in a virtual machine.

WSL 1 might be able to run the Linux version of Syncthing, but file I/O is subject to the same restrictions as for native Windows applications because WSL is simply mapping Linux system calls to the Windows counterparts.

WSL 2 can definitely run Syncthing for Linux, but then Samba, NFS, or some other network service must be used to provide access to the storage volumes inside the VM (ext4 filesystem by default). At that point, it’d be like maintaining a mini Linux server on each Windows workstation.

Probably be easier to just switch to Linux. :wink:

1 Like