First of all, having recently discovered Syncthing, thanks! I’m really impressed with how well it works, and great cross-platform support (*BSD support is a big deal for me).
Right now I’m looking for advice on a particular problem. I have a dual-homed workstation (Windows 10), connected to Internet via NAT router, and connected to NAS (Synology, DSM 7.0.1) via autoconf (169.254/16) network. The workstation is fairly limited in storage, accesses the NAS via SMB, and the NAS has no Internet connectivity. Ultimately I want to synchronise folders on the NAS. This is a frame store for editing/compositing.
No I’m not going to reconfigure the network. It’s not how I would set it up, but I don’t intend to volunteer as network administrator for this environment.
So I’m considering potential solutions:
Syncthing on NAS, proxied via workstation; or
Syncthing on workstation, synchronising folder from NAS via SMB
For option #1, Could Syncthing Relay Server running on a workstation provide access to discovery, syncronisation etc., and is it possible for that to be effectively private? I can’t provide a public relay through this network. Installing the SynoCommunity package, or pulling a Docker image without Internet connectivity presents its own challenges. Any advice with respect to that would be appreciated.
As for option #2, I know this is not supported. Is there a semi-sane configuration that would make it work? I could split folders into send/receive only, and sacrifice some consistency/immediacy. I’d accept some amount of babysitting would be necessary.
I’m looking at other options too, but being able to check the GUI and see the current state is helpful, and that’s one reason I’d like to keep Syncthing in the mix.
I haven’t really thought through your setup and what exactly you would like to “proxy”. Just regarding your solution #1, why do you find installing the SynoCommunity SPK hard without internet access? It’s quite easy using their Package Center to provide the SPK file you previously downloaded from the SynoCommunity package website. Just need to pick the right architecture.
If you don’t want the workstation to mirror all the data because of its limited storage capacity, then for whom are you going to use Syncthing with the NAS? Off-site backup? Other users working on the same data set, but with enough storage on their workstations?
I’d really recommend against solution #2. Syncthing works under the assumption that access to local files is cheap. Mounting over some network file system would cause lots of churn just for scanning and to determine that nothing has changed. You would likely get terrible performance.
The main question I have with solution #1 is how to proxy traffic in order to run Syncthing. I intend to synchronise folders from the isolated NAS to multiple remote users, and Syncthing discovery also requires Internet access. Is Syncthing Relay Server capable of proxying that traffic (or only to facilitate connections between chronically NATed systems), and can it be configured as a private relay?
I know solution #2 is undesirable, but I’m looking to quantify “terrible performance” so I can decide whether it’s good enough, and understand any show-stopping issues with that approach (i.e. “cheap” file access is relative, but filesystem semantics are absolute). My requirements are pretty loose, I can stand a few hours (even overnight) delay on synchronisation. So if scanning only causes rehash if metadata has changed, I can potentially bear that. I’m willing to do some testing here, but hoping someone already has.
Although the general recommendation is not to combine Syncthing with a source folder that’s on a network share, it will work under specific conditions.
You mentioned that your Windows 10 workstation is mapping a SMB share provided by a Synology NAS running DSM 7.0.1. I don’t have a Synology box to verify this, but there’s a good chance it’s close enough to version 7.1 (Synology Directory Server Technical Specifications | Synology Inc.). If so, then it’s using Samba 4.
The reason why Samba 4 on your Synology NAS is a good thing is because there’s better support for Windows 10 and higher, plus support for file/directory change event notification in the SMB protocol.
With a suitable version of Windows (I’ve tested it with Windows 7, but it might also work in Vista), Samba supports notification requests from Windows. Apps that tap into the Windows file system watcher subsystem can get event notifications sent by Samba to Windows.
This means that Syncthing doesn’t need to rely only on its polling schedule (defaults to 3600 seconds), which is expensive in terms of system load on the Synology NAS, network I/O, and system load on your Windows workstation. Syncthing will be able to use its watcher to detect most changes and use polling as a backup.
You also mentioned that the SMB share from your Synology NAS is a “frame store for editing/compositing”, which sounds like there could be lots of files. Depending on the typical file size and number of files, grouping files into smaller folders and setting up multiple folders in Syncthing will be better than bulk syncing hundreds of thousands of files under a single folder.
One caveat is that if there are a lot of files being added and/or changed, Windows might not be able to pass on every event because the default event buffer is 8KB (8192 bytes). Increasing the buffer size (must be in 4K increments) eats up RAM, and it cannot be swapped to virtual memory. If/when the event buffer is full any missed events will require Syncthing’s polling to catch any missed changes.
My current setup:
Syncthing 1.22.1 on a Windows 7 workstation.
Samba 4.5.16 on a DIY NAS providing a network share to the workstation.
Destination is Syncthing 1.22.0 on a Fedora Linux server.