How To Share Db w/ Relative Paths

dragave · June 14, 2022, 3:29am

This is a continuation of my previous thread, Constant Conflicts on Dual Boot System w/ Shared Drive, which the forum will not allow me to reply to.

Due to frequent conflicts & extra syncing on my Windows+Linux dual boot system, I’d like to setup a shared config/database, and just wanted to confirm the setup. As I understand it, the process would be:

Make sure Windows & Linux are using the same version of SyncThing, & disable auto-update (since they’ll be using shared config+db, they must always be on the same version - thus, only explicit manual updates).
First I’ll setup Linux. Launch SyncThing with the --home arg to specify a location on the shared partition, i.e. /shared/partition/configs/syncthing
Since Windows & Linux use different path formats, shared folders should be specified with relative paths. This is my main question: relative to what? To the --home location? For instance, if --home is /shared/partition/configs/syncthing, and a shared folder is /shared/partition/data/shared-folder, then the path would be “…/…/data/shared-folder”?
Reboot to Windows, & configure SyncThing to launch with --home, using the equivalent Windows path /shared/partition/configs/syncthing, i.e. D:/configs/syncthing

And that should be it - good to go?

Thanks again!

tomasz86 · June 14, 2022, 5:15am

I’d say you’ve got to share not only config and database, but all Syncthing folders themselves also need to be re-used between the two operating system. It’s similar as if you ran a portable installation of Syncthing from a USB stick or an external hard drive, really.

For example, this can be achieved the following way.

On Windows, create a new folder called D:\Syncthing.
Download Syncthing binaries for both Windows and Linux and place them in that folder, i.e.
```
D:\Syncthing\syncthing.exe
D:\Syncthing\syncthing
```
Create a new batch script called syncthing.cmd, and inside it put
```
pushd "%~dp0"
syncthing.exe -home=home -no-upgrade
```
On Linux, mount the D:\Syncthing folder e.g. to /syncthing.

Create a new bash script called syncthing.sh , and inside it put

#!/bin/bash
pushd "$(dirname "$0")"
syncthing -home=home -no-upgrade

You now can start Syncthing using syncthing.cmd in Windows, and syncthing.sh in Linux. Because Syncthing is started from that folder (thanks to changing the path with the pushd commands), all future paths become relative to it. You can then create a new folder in Syncthing with its path set to e.g. ./folders/Sync and so on. You should use forward slashes in Syncthing regardless of the OS, as they work anywhere, while backslashes are Windows-specific.

The overall folder structure should end up looking like this.

Syncthing
    folders
        Sync
    home
        index-v0.14.0.db
        cert.pem
        config.xml
        csrftokens.txt
        https-cert.pem
        https-key.pem
        key.pem
    syncthing
    syncthing.cmd
    syncthing.exe
    syncthing.sh

dragave · June 15, 2022, 12:08am

If you mean the Shared Folders are re-used, that’s indeed the case (that’s the only situation that would lead to conflicts, right? Unless I’m misunderstanding?). i.e. in my example, Linux’s /shared/partition/data/shared-folder is the same folder as D:/data/shared-folder (on a shared ntfs partition that’s mounted to both OSs).

Or did you mean that not only the database, config, & Shared Folders, but even the folder containing the SyncThing binaries must be shared (and paths are relative not to home or db or config, but to the location of the actual SyncThing binary?)

If that’s the case, might I suggest an optional command-line arg, i.e. --shared-folder-relative-base? Then we could simply specify the basedir on each OS, but SyncThing itself could remain installed in the usual way (i.e. via installer, or apt, or whatever)

For a little more context, I’m actually using SyncTrayzor on Windows & SyncThing Tray on Linux.

tomasz86 · June 15, 2022, 5:46am

Yeah, there are workarounds, but in my experience, the simplest solution is to have the binaries located in the same folder as well. However, when it comes to SyncthingTray, it really doesn’t matter where the binaries are located and executed from, because this particular program can be run separately from Syncthing proper. Not sure about SyncTrayzor though. If SyncTrayzor turns out problematic, I’d suggest to just use SyncthingTray on Windows as well.

The key point here is to set the Syncthing binaries up and run them following the steps described previously, and only then start SyncthingTray and configure it to connect to the already running instance of Syncthing.

acolomb · June 15, 2022, 7:22am

The relative paths have nothing to do with where the binary is located. It solely depends on the concept of a working directory regardless of operating system, see:

How to influence that depends on how you start Syncthing. The script solution from @tomasz1986 takes care of it. On Windows, you can specify the working directory in the proper tier of a shortcut you create to launch the binary. If run from a terminal (either OS), you need to cd into the right directory before starting the executable. There may be options in SyncTrayzor or SyncthingTray as well to set the working directory, I don’t know. But I’d assume that if not, the working directory for Syncthing will be the same as what was used for the wrapper application.

dragave · June 15, 2022, 2:59pm

If SyncTrayzor turns out problematic, I’d suggest to just use SyncthingTray on Windows as well.

For the moment I strongly prefer SyncTrazor, as SyncthingTray doesn’t support any sort of notification when conflicts are present (the author expressed a willingness to implement it, but not until Syncthing itself makes this info available through its api - Add the number of sync conflicts to the UI · Issue #8114 · syncthing/syncthing · GitHub)

Yeah, there are workarounds

From acolomb’s follow-up reply, does this specifically mean running it with a different cwd?

The key point here is to set the Syncthing binaries up and run them following the steps described previously, and only then start SyncthingTray and configure it to connect to the already running instance of Syncthing.

Yeah, I understand. Just curious though, would there be any downside to having a more direct/explicit --base-dir argument? Although you can of course trick the environment into simulating this, pushing & popping cwds, avoiding the standard installers, putting binaries in a particular location, etc, seems like that would be significantly easier/cleaner, as well as platform-agnostic. Just tack on one command-line arg to tell it where the db/config is, another command-line arg to tell it where the base path, & it’s done. Then a standard install (apt/windows installer/etc) would work in every case. No?

dragave · June 15, 2022, 3:01pm

Gotcha, it’s relative to working directory, not the binary - thanks. Pls see my reply above to tomasz86 for some additional thoughts

Thanks again!

calmh · June 15, 2022, 3:04pm

There’s one possible cheat that may or may not be more convenient – lying to Syncthing about where the user home directory is. On Windows that’s %HomeDrive%/%HomePath% or %USERPROFILE%, and on unixes it’s $HOME. Set those before starting Syncthing and you can use ~ to refer to that location in your folder locations.

Syncthing will want to put config and database in that place as well, but you can override that specifically with -home if it’s not right.

tomasz86 · June 15, 2022, 3:09pm

I don’t think this works in Windows, does it? I’ve tried setting the USERPROFILE variable to a custom path prior to starting Syncthing in the command line, but ~ still expands to the original USERPROFILE path.

Edit:

Yeah, although in Windows this is called just “current directory”. However, one big advantage on having both Windows and Linux binaries in the same folder is that it’s easier to make sure that both are the exact same version. This is important, because you absolutely do not want to mix different versions of Syncthing when running a portable installation like that.

calmh · June 15, 2022, 3:12pm

Of course it’s more complex in Windows; maybe USERPROFILE won’t suffice. I think it looks in multiple env vars and perhaps the registry to find the most correct home dir. But if %HomeDrive% and %HomePath is set that will win, because that’s a legacy override we have ourselves.

github.com

syncthing/syncthing/blob/b140b4a994c57af1ee1871a259ef98dfaae8d8cc/lib/fs/util.go#L37-L48

      
        
            func getHomeDir() (string, error) {
            	if runtime.GOOS == "windows" {
            		// Legacy -- we prioritize this for historical reasons, whereas
            		// os.UserHomeDir uses %USERPROFILE% always.
            		home := filepath.Join(os.Getenv("HomeDrive"), os.Getenv("HomePath"))
            		if home != "" {
            			return home, nil
            		}
            	}
            
            
	return os.UserHomeDir()
            }

(maybe the comment is a lie? Or may have been true at one point, some of this is from before os.UserHomeDir() existed, and it may have done something different when introduced than it does today…)

acolomb · June 15, 2022, 5:58pm

I guess wherever you will be using

syncthing --base-dir=/foo

it will be just as easy to instead enter

cd /foo; syncthing

for the same result. Plus that is standard behavior for pretty much any application running on any OS. I wouldn’t want Syncthing to bake in a special solution here.

If in doubt, put the above into a script and point your wrapper to execute that instead of the regular exe.

dragave · June 17, 2022, 3:53am

that is standard behavior for pretty much any application running on any OS. I wouldn’t want Syncthing to bake in a special solution here.

Yeah, that’s of course standard for working directory. However, just like we have an explicit toggle to tell it where the config would be, it seems like it would be a nice & much simpler convenience to also have an explicit toggle to tell it what to use as its base data dir. That would work the same regardless of platform, if it’s launched via .desktop file or windows shortcut or a wrapper application, etc. For example, both SyncTrayzor & Syncthing Tray have an explicit box where you can specify the command-line args to pass to SyncThing.

Of course, it’s possible to make it work without that, it just requires different platform-specific hacks in each place (which took quite a bit of trial and error to get right):

On Linux, I change the ~/.config/autostart .desktop file to bash -c "cd /my/shared/partition; /opt/syncthingtray/bin/syncthingtray. Then Syncthing Tray is launched with that working directory, which in turn launches SyncThing.
On Windows, I disable the option SyncTrayzor->File->Settings->SyncTrayzor->Automatically start on login. Then I manually create a shortcut, set its “Start in” to “D:\”, & put that in the start menu’s Startup folder. & Again SyncTrayzor then launches SyncThing with the same working directory.

Ultimately it’s more just a matter of clarity/convenience. To me, being able to specify args as --home=/config/path --data=/data/path is MUCH clearer & more user-friendly than above

acolomb · June 17, 2022, 5:25am

Having a command line argument in Syncthing to change the working directory still wouldn’t fully solve the problem though. Your wrapper actually needs to run with the same working directory if you want the extra “Open Folder” buttons to work, for example. So your working solution is the way to go. Adding code to handle a separate base directory to Syncthing itself and every wrapper is just a massive effort compared to the already in-place and working standard way using the working directory.

I’d suggest if you want to make the portable use case easier to set up, try to convince the wrapper developers to extend their installation procedure and optionally set the working directory in the created shortcuts / registry entries. For systemd, there might even be an option to specify the working directory for a service unit? If so, it would be a one-line override file to use it. Similar for the autostart .desktop file. Whoever creates that could include an option to bake in a specific working directory.

dragave · June 17, 2022, 5:53am

Having a command line argument in Syncthing to change the working directory

Not necessarily to change the working directory specifically - just to have some notion of a base data directory. Could be implemented via working directory, or otherwise.

Adding code to handle a separate base directory to Syncthing itself and every wrapper is just a massive effort

Yeah, valid - fair enough.

Basically, was just trying to communicate that --home=/dir is so easy, convenient, & platform-agnostic. But trying to achieve the equivalent of --data=/dir requires any one of a number of hacks that are dependent on many things: platform, install method, launch method, wrapper, etc. From a purely usability standpoint (not from an ease of implementation standpoint), it’s far, far more user-friendly to just add a command-line arg rather than to try to learn about .desktop files, systemd, Windows launch shortcuts, install locations, etc, etc, etc.

Totally understand if it’s too much of a heavy lift to actually implement. Just wanted to express that from a purely UX standpoint.

acolomb · June 17, 2022, 6:19am

A heavy lift not by itself, in the backend. But getting all the wrappers to respect it, let alone offer an option to set it, is where things start to get complicated. In that way, the working directory solution is superior because it already works for both wrappers and backend.

Again, what would really help the “portable” use case would be an option in the respective installers to hard-code the directory when creating shortcuts / startup scripts. I guess if you contact the various wrapper developers here, most wouldn’t oppose building such a portable mode if they have time.

dragave · June 17, 2022, 2:29pm

getting all the wrappers to respect it, let alone offer an option to set it

Not sure I follow - all the wrappers already have an option to set command-line args. That’s kinda the point.

But even if they don’t (due to issues w/ i.e. ‘open folder’ buttons), standard behavior always begins with the base app & isn’t instant, I get that. If Syncthing were to provide a standardized way to set its base path, surely the wrappers would eventually follow.

Again, what would really help the “portable” use case would be an option in the respective installers

Portable is not what I’m after specifically, and not everyone uses these wrappers. Again, relying on the external installers is the same issue I’m addressing. Just a platform-agnostic way to tell the underlying SyncThing a base dir, so that platform-agnostic paths could be used without needing to resort to external trickery (scripts, manipulating shortcuts, installers, etc).

Anyway, I’ve said my piece. I have it working as-is so I’m fine with it, it was just hugely not intuitive, & required external platform-specific finagling. There is a knob to give it a config dir. A knob to give it a data dir really doesn’t seem like that far of a stretch from a usability standpoint. I understand that it wouldn’t be instant, it wouldn’t be instantly supported by all external tools, there are design considerations, etc. That’s obvious. Nevertheless, setting a data dir is clearly a situation for which there is a recurring use-case, and the “standard” way to do it requires external workarounds/scripts/commands that are not nearly as easy or intuitive as --config=/dir. Take it for what you will.

dragave · June 18, 2022, 4:43am

Anyway, so I’ve got Linux+Windows successfully sharing a config (with all folders setup as relative paths, as discussed)

Everything is working great, with the exception of one new behavior:

One of my shared folders has a large number of files (i.e. ~30k). Whenever I reboot from Windows->Linux or Linux->Windows, that folder shows "Scanning (x%), which takes awhile, and as its scan percentage increases, the NAS (under “Remote Devices”) shows “Syncing” with its percentage decreasing (if I expand that device, I can see that the number of Out of Sync Items is going up). Once the folder is done scanning (& shows “Up to Date”) then the NAS’s Out of Sync Items gradually decreases to 0. The whole time, it transfers no data - in other words, it’s not actually uploading anything; the item count just slowly counts down. Because there are so many files, this whole process takes 10-15min after a reboot, before it thinks it’s done syncing.

This doesn’t occur after rebooting Linux->Linux or Windows->Windows: in that case, all folders (including the large one) pretty much just immediately show “Up to date.”

This also doesn’t occur when Windows & Linux have separate configs/dbs: it scanned for maybe ~10sec, then showed Up To Date (+the NAS never showed syncing).

Why would the shared config cause it to spend all that extra time re-scanning & counting down Out of Sync Items after a reboot between OSs?

acolomb · June 18, 2022, 6:08am

Could be related to the timestamps on the files? Linux may be storing different values not including the time zone offset for example, while Windows counts it in. I’m not really firm on the details here, just a direction you could try to investigate. Do the file modification times change on the NAS after these reboots with long scanning?

imsodin · June 18, 2022, 9:52am

Do you ignore permissions? That’s a candidate for perpetual difference on linux vs windows. Otherwise you can enable scan or db debug logging, it will be very verbose and print the difference between old and new metadata when the scan produces the differences you describe.

dragave · June 18, 2022, 7:24pm

Do you ignore permissions?

Yup, ignore permissions is enabled on all folders on all devices.

Otherwise you can enable scan or db debug logging

Under “debugging facilities” I see “db” but no “scan,” so I just did “db” for now. I’m not 100% certain what it is I’m looking for, but here’s a snippet - hopefully it can reveal the issue:

[EYXSD] 2022/06/18 07:18:40.364430 lowlevel.go:239: DEBUG: insert (local); folder="ekods-gqfir" File{Name:"WhatsApp/Media/WhatsApp Images/IMG-20130822-WA0010.jpg", Sequence:189457, Permissions:0755, ModTime:2018-05-14 15:25:36.502340243 -0700 PDT, Version:{[{EYXSDNG 1655561803} {IS4X3VD 1654367071} {KJLVN4T 1} {U6X5OV5 2} {XCG2ZGU 10} {267TD2Q 1637078273}]}, VersionHash:, Length:54315, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:true, BlockSize:131072, Blocks:[Block{0/54315/3540714598/b9d4570e37f7c4068a52f9230a498207382438b70ce995806be8a239cee0a183}], BlocksHash:641338cc5ebf1155899ef82b7a21a1009226f2f34adb782888bfd84cc213665b}
[EYXSD] 2022/06/18 07:18:40.367070 transactions.go:635: DEBUG: update global; folder="ekods-gqfir" device=7777777 file="WhatsApp/Media/WhatsApp Images/IMG-20130822-WA0010.jpg" version={[{EYXSDNG 1655561803} {IS4X3VD 1654367071} {KJLVN4T 1} {U6X5OV5 2} {XCG2ZGU 10} {267TD2Q 1637078273}]} invalid=false
[EYXSD] 2022/06/18 07:18:40.367070 transactions.go:649: DEBUG: new global for "WhatsApp/Media/WhatsApp Images/IMG-20130822-WA0010.jpg" after update: {{Version:{[{EYXSDNG 1655561803} {IS4X3VD 1654367071} {KJLVN4T 1} {U6X5OV5 2} {XCG2ZGU 10} {267TD2Q 1637078273}]}, Deleted:false, Devices:{7777777}, Invalid:{}}, {Version:{[{EYXSDNG 1655526125} {IS4X3VD 1654367071} {KJLVN4T 1} {U6X5OV5 2} {XCG2ZGU 10} {267TD2Q 1637078273}]}, Deleted:false, Devices:{IS4X3VD, U6X5OV5, XGD6WDY}, Invalid:{}}, {Version:{[{IS4X3VD 1654367071} {KJLVN4T 1} {U6X5OV5 2} {XCG2ZGU 10} {267TD2Q 1637078273}]}, Deleted:false, Devices:{XCG2ZGU}, Invalid:{}}}
[EYXSD] 2022/06/18 07:18:40.367070 lowlevel.go:260: DEBUG: adding sequence; folder="ekods-gqfir" sequence=189457 WhatsApp/Media/WhatsApp Images/IMG-20130822-WA0010.jpg
[EYXSD] 2022/06/18 07:18:40.367070 lowlevel.go:378: DEBUG: removing sequence; folder="ekods-gqfir" sequence=165357 WhatsApp/Media/WhatsApp Images/IMG-20130822-WA0011.jpg
[EYXSD] 2022/06/18 07:18:40.367070 lowlevel.go:239: DEBUG: insert (local); folder="ekods-gqfir" File{Name:"WhatsApp/Media/WhatsApp Images/IMG-20130822-WA0011.jpg", Sequence:189458, Permissions:0755, ModTime:2018-05-14 15:25:36.682340243 -0700 PDT, Version:{[{EYXSDNG 1655561803} {IS4X3VD 1654367071} {KJLVN4T 1} {U6X5OV5 2} {XCG2ZGU 10} {267TD2Q 1637078273}]}, VersionHash:, Length:61040, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:true, BlockSize:131072, Blocks:[Block{0/61040/836336904/6226473b070f46ed76e4bad9cbc8dc268555dff5677c9ef6d90db03698846484}], BlocksHash:ba75d7f9ad0f570a5543159792dc771cb3591824c1a877ba4647736158d88682}
[EYXSD] 2022/06/18 07:18:40.367070 transactions.go:635: DEBUG: update global; folder="ekods-gqfir" device=7777777 file="WhatsApp/Media/WhatsApp Images/IMG-20130822-WA0011.jpg" version={[{EYXSDNG 1655561803} {IS4X3VD 1654367071} {KJLVN4T 1} {U6X5OV5 2} {XCG2ZGU 10} {267TD2Q 1637078273}]} invalid=false
[EYXSD] 2022/06/18 07:18:40.367070 transactions.go:649: DEBUG: new global for "WhatsApp/Media/WhatsApp Images/IMG-20130822-WA0011.jpg" after update: {{Version:{[{EYXSDNG 1655561803} {IS4X3VD 1654367071} {KJLVN4T 1} {U6X5OV5 2} {XCG2ZGU 10} {267TD2Q 1637078273}]}, Deleted:false, Devices:{7777777}, Invalid:{}}, {Version:{[{EYXSDNG 1655526125} {IS4X3VD 1654367071} {KJLVN4T 1} {U6X5OV5 2} {XCG2ZGU 10} {267TD2Q 1637078273}]}, Deleted:false, Devices:{IS4X3VD, U6X5OV5, XGD6WDY}, Invalid:{}}, {Version:{[{IS4X3VD 1654367071} {KJLVN4T 1} {U6X5OV5 2} {XCG2ZGU 10} {267TD2Q 1637078273}]}, Deleted:false, Devices:{XCG2ZGU}, Invalid:{}}}
[EYXSD] 2022/06/18 07:18:40.367070 lowlevel.go:260: DEBUG: adding sequence; folder="ekods-gqfir" sequence=189458 WhatsApp/Media/WhatsApp Images/IMG-20130822-WA0011.jpg
[EYXSD] 2022/06/18 07:18:40.367070 lowlevel.go:378: DEBUG: removing sequence; folder="ekods-gqfir" sequence=165358 WhatsApp/Media/WhatsApp Images/IMG-20130822-WA0012.jpg
[EYXSD] 2022/06/18 07:18:40.367070 lowlevel.go:239: DEBUG: insert (local); folder="ekods-gqfir" File{Name:"WhatsApp/Media/WhatsApp Images/IMG-20130822-WA0012.jpg", Sequence:189459, Permissions:0755, ModTime:2018-05-14 15:25:36.886340243 -0700 PDT, Version:{[{EYXSDNG 1655561803} {IS4X3VD 1654367071} {KJLVN4T 1} {U6X5OV5 2} {XCG2ZGU 10} {267TD2Q 1637078273}]}, VersionHash:, Length:70674, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:true, BlockSize:131072, Blocks:[Block{0/70674/4215995126/f2a6fc422662179acb90eb27199d639e6292d17bdfd968d74287df2e30043b31}], BlocksHash:41af4b89a0dc3cd095d86c0d9bd59bf9444c6ea7f71479299e1ceecb73123707}
[EYXSD] 2022/06/18 07:18:40.369115 transactions.go:635: DEBUG: update global; folder="ekods-gqfir" device=7777777 file="WhatsApp/Media/WhatsApp Images/IMG-20130822-WA0012.jpg" version={[{EYXSDNG 1655561803} {IS4X3VD 1654367071} {KJLVN4T 1} {U6X5OV5 2} {XCG2ZGU 10} {267TD2Q 1637078273}]} invalid=false
[EYXSD] 2022/06/18 07:18:40.369115 transactions.go:649: DEBUG: new global for "WhatsApp/Media/WhatsApp Images/IMG-20130822-WA0012.jpg" after update: {{Version:{[{EYXSDNG 1655561803} {IS4X3VD 1654367071} {KJLVN4T 1} {U6X5OV5 2} {XCG2ZGU 10} {267TD2Q 1637078273}]}, Deleted:false, Devices:{7777777}, Invalid:{}}, {Version:{[{EYXSDNG 1655526125} {IS4X3VD 1654367071} {KJLVN4T 1} {U6X5OV5 2} {XCG2ZGU 10} {267TD2Q 1637078273}]}, Deleted:false, Devices:{IS4X3VD, U6X5OV5, XGD6WDY}, Invalid:{}}, {Version:{[{IS4X3VD 1654367071} {KJLVN4T 1} {U6X5OV5 2} {XCG2ZGU 10} {267TD2Q 1637078273}]}, Deleted:false, Devices:{XCG2ZGU}, Invalid:{}}}
[EYXSD] 2022/06/18 07:18:40.369115 lowlevel.go:260: DEBUG: adding sequence; folder="ekods-gqfir" sequence=189459 WhatsApp/Media/WhatsApp Images/IMG-20130822-WA0012.jpg

Do the file modification times change on the NAS after these reboots with long scanning?

I don’t believe so. I grabbed an FTP listing of a particular folder on the NAS (the WhatsApp Images folder above), then rebooted the PC to the other OS, waited for it to do its 15min rescan/resync, then refreshed that folder on FTP again, & I didn’t notice the file timestamps changing.