Confused about options, flags and the config.xml

Feature
1

Hi,

long story short: I’d like to use the config file to set more of the default behaviour and I don’t know if that is even possible. A short answer would be NO.

My insight into these things: I stopped every attempt at actually learning to do something like this at the hello world stage and I only started because some sort of edumacational facility forced me to. I’m just not the right starting material.

Anyways: Obviously there are some flags (or options?) that can be attached when starting Syncthing (I can use the command line, I’m not stupid)

$ syncthing -h

 Usage:
 syncthing [options]

 Options:
  -audit                  Write events to audit file
  -generate=""            Generate key and config in specified dir, then exit
  -gui-address=""         Override GUI address
  -gui-apikey=""          Override GUI API key
  -gui-authentication=""  Override GUI authentication; username:password
  -home=""                Set configuration directory
  -logfile="-"            Log file name (use "-" for stdout)
  -logflags="2"           Select information in log line prefix
  -no-browser             Do not start browser
  -no-restart             Do not restart; just exit
  -reset                  Reset the database
  -upgrade                Perform upgrade
  -upgrade-check          Check for available upgrade
  -upgrade-to=""          Force upgrade directly from specified URL
  -verbose                Print verbose log output
  -version                Show version

As usual these options are used to overwrite default behaviour that is either set or inherited somewhere. Some of these defaults are set in the in the options (dictionary?) of the config.xml or that is at least what I assume.

e.g.

  <startBrowser>true</startBrowser>

is overwritten by

syncthing -no-browser

I tried to trace the code on github. The flags are translated into to the vars in the main.go file thing. Ok, then some other listing in there looks like it sets the pecking order of these different ways of adjusting the the behaviour. That makes sense.

What I did not find is something that looked like I could somehow pass an integer for logFlags, “logflags”, logFlags and a strings for logFile, “logfile”, “”, via the config.xml.

Is this not defined? Am I looking in the wrong place? simply not possible?

(obviously you can’t set the home folder in a file that needs to read from the home folder without a handstand. I understand that there might be other probably completely different caveats)

No! It is not strictly necessary but I want the process to behave as I choose in the config file and not to be dependend on the way I start it. Because this can vary from LaunchAgent over loginItem to a droplet.

Ideally would already exist. It is definitely not working with all the terms I tried, but that could also be because the default is set to 2 by something that outranks the config.xml I simply don’t know.

cheers for reading

2

No, you cannot set logFlags and logFile in the config.

Under the ideal conditions, you don’t need to tweak these values. The general expectation is that the default logging we provide is sufficient for normal operation, the only reason we specify additional flags is for debugging, in which case it’s fine to be forced to specify a command line option.

Regarding the log file, on Windows it defaults to $SYNCTHING_HOME/syncthing.log, this is because Windows does not have all the awesome service managers which could deal with log aggregation for us, so it’s there just for reference.

On non-Windows, we simply do not encourage using the logfile option, as ideally syncthing should log to stdout, and you should either redirect it yourself wherever you want, or just let the service manager deal with it, like it does for all other services available on the system.

That said, I don’t think it’s rocket science to make these options part of the config, but I don’t really see a need for that, given these are either for debugging purposes, or not really expected to be fiddled with in general, and expected to be handled by other things (service managers).

3

There may be some overlap, ambiguousness and history here (especially with flags vs env vars, which I won’t try to defend here). However, there are also actual reasons for command line flags being command line flags:

  -audit                  Write events to audit file

It should not be trivial to disable this once enabled. There are situations where the user handing the GUI is not the same as the person who set up the service. Accessing the log requires CLI access anyway, as it’s not accessible from the web interface.

  -generate=""            Generate key and config in specified dir, then exit

This is a one shot option, doesn’t make sense in the config.

  -gui-address=""         Override GUI address
  -gui-apikey=""          Override GUI API key
  -gui-authentication=""  Override GUI authentication; username:password

Intentionally overrides the config, for cases you want to take command from the command line (or from a launcher), regardless of the configuration.

  -home=""                Set configuration directory

Sets where we load the config from, so cannot be in the config.

  -logfile="-"            Log file name (use "-" for stdout)
  -logflags="2"           Select information in log line prefix

Affects the logging. Logging is not visible in the GUI and is only relevant on the console, so is best managed from there.

  -no-browser             Do not start browser

Can be set from the config as well, but it’s nice to be able to override it on a launch by launch basis, or strictly enforce it regardless of the config. For example by the tests and by GUI wrappers.

  -no-restart             Do not restart; just exit

Affects how we handle restarts, and depends entirely on how the process is launched (i.e. from systemd or manually, etc). So you want to set it in the place you do the launch.

  -reset                  Reset the database
  -upgrade                Perform upgrade
  -upgrade-check          Check for available upgrade
  -upgrade-to=""          Force upgrade directly from specified URL
  -verbose                Print verbose log output
  -version                Show version

These are all one shot things as well.

4

Do NOT Read this. It is gotten way to long. But I had time in between the different runs of syncthing.

I assumed so, also that it is partly not sensible and partly not possible to use all the flags in the config.xml. That is why I asked for the log flags explicitly, sort of hoping that all the options had a representation anyway because all was defined at once wether it would be needed or not. In OSX there is hardly a thing that isn’t controlled in one of those files which would make it a good fit for AppleScript. I’m certainly not asking someone to spent time fixing my problems for free.

Anyways, thank you guys for your time and your work! I really like the concept of this program. Not only because I do not have to upload my data to someone I don’t trust. Most people are only concerned with the information in the data but that is only half the story. You are also forced to use their software at the version they dictate no matter how it behaves and you will have to suffer their policies which might change just with the next incoming email. Certainly not suitable for anything that is actually important.This is the best thing about your software. I do not need anything that I can’t control. Not even the announcement servers. Feature wise everything I need is sort of provided, I just need to put the pieces together and understand why it does the things that it does.

And I already realise that there will be a few pre flight checks for non default behaviour that I wouldn’t have had expected due to naivity on my side. Syncthing will actually just instantly crash without a single line if it is told to write to a log file in a folder that doesn’t exist or that it can’t reach because of permissions/ownership mismatch (where should it go, and why even. Should have been obvious). Not existent/permissive homefolders or such filled with non writeable files will be tried 4 times followed by a graceful exit.

Why would I even want to run syncthing with different Users? For synching home folders it is not only permissions but also ownership by UID and whatever is written in the extended attributes. At least on OSX that really matters. While the owner ship can be transferred OSX seems to want to bind the extended attributes not only to the user but also to Software and Devices, even when you inherit them using the Migration Assistent or tmutils you are not getting a clone of what you had. Apart from whole disc cloning the only feasible way to have identical iTunes and Photos Libraries on different machines is to make sure that the owner UIDs are kept identical.

Syncthing can do this. It can also run multiple working instances on the same machine simoultanously when started from different users. Altough this obviously only would make sense if you could get it to behave differently (Actually later I found a reason for running more then one syncthing for one user at the same time). The nice/renice commands OSX offers can keep other processes prioritised but all available idle time is given to synchthing. How about at least not putting the cpu in turbo with all cores, that would be a start. But it will crash when artificially starved as done by cpulimit, even when only attempting minor adjustments. So, yes I do want to play around with your developer options. I don’t expect it to work better, I only want it to shut up.

Yes all this amounts to a stupid hack on my part especially since what should actually be used is a modified mechanism of what Timemachine is using to merge snapshots. But instead of setting additional attributes that lock the files down attributes should be used to identify which files are shared with whom. It would be like putting a .stignore marker directly onto each file. But this is for a smarter guy then me to figure out. Having a single file for each file to be shared would also help to avoid the only thing that actually botherd me to the point I wanted to stop experimenting and simply go back to rsync sessions. Although thanks to some very simple measure I probably only have to survive the very worst once for all the machine pairings. The actual proble is this: If I start with a clean machine and a used one some of the installed files are newer to syncthing even though I prefer the config files I have edited months ago to the ones that come fresh from the install. So you put one machine into master mode. After the new machine is in synch you turn the master of and restart syncthing. This leads to the silly thing assuming it needs to download all the stuff it just uploaded. But at least it checks if it would fit.

verbose.go:42: VERBOSE: Summary for folder "dichnixan" is map[needBytes:469745069070 inSyncBytes:30721873474 localBytes:437132862703 inSyncFiles:3269 globalDeleted:0 localFiles:125504 needFiles:125910 state:scanning version:255191 globalFiles:129179 globalBytes:500466942544 localDeleted:0]
verbose.go:42:VERBOSE: Completion for folder "dichnixan" on device 000000-000000-000000-000000-000000-000000-000000-L65A7AN is 99.99999693086622%
verbose.go:42:VERBOSE: Updated index for folder "dichnixan" with 1 items
rwfolder.go:331:INFO: Completed initial scan (rw) of folder dichnixan
verbose.go:42:VERBOSE: Folder "dichnixan" is now idle
verbose.go:42:VERBOSE: Folder "dichnixan" is now syncing
rwfolder.go:516: WARNING: Folder "dichnixan": insufficient disk space available to pull 125911 files (447982.39 MiB)

So you have to force a complete rescan after removing the master status. With just being shy of 500gb that is a long time spend for being out of synch by 2 files. The same seems to apply when I change the .stignore. You restart syncthing and it carries on as if it hadn’t read the file. So you have to kill the database again.

If this it would be the standard procedure to have it rescann all its stuff just so it actually applies the set pattern you can’t really sync all the data on a mobile computer relying on whatever Internet you come accross. Because neither can you do substantial rescanning on battery nor can you upload a 2gb image file that was slightly altered. Which I is why I tried to circumvent this by using a “on the road .stignore” which in my case simply contains every file over 50MB. Actually realising this is really simple. Just start two syncthings and let them into the actual share root over a symlink. Set them to announce on different ports, turn off upnp and use little snitch on the mobile computer to pick and choose which one you can actually connect to.

Well that was sort of a recap of my journey with syncthing. It might sound stupidly complicated but it beats getting everything in a mess over time and not really knowing what is what and sorting it again with fdupes and diff just to start over.

5

You say that syncthing crashes in a number of situations, where I think it does not (unwriteable log file, renicing). If you can reproduce these crashes, please do file issues on them.

As for resource usage, that calms right down when initial scanning and sync is done. See also GOMAXPROCS in the help output.

The “not enough free space” thing is a bug, hopefully solved in the release I’m about to make today, sorry about that.