Config file format

It seems a bit awkward that the configuration format is XML, which isn’t really one. I think that toml, yaml or ini would be much nicer to work with. Was there any particular reason that XML was chosen?

Why does this matter? Mostly for use on a server where you don’t want the web UI running.

Quoting myself from https://github.com/syncthing/syncthing/issues/528:

INI has no standard way of conveying structure other than [section] and possibly dot-separated keys. Arrays are a pain in the ass.

JSON is even less human friendly than XML, although it’s nicer in almost all other ways, in that it is extremely finicky about not allowing a spurious comma on the last item in an object, doesn’t have any mechanism for comments, and things like that.

YAML requires understanding that indentation carries meaning and that - and — etc are magical, that there’s a significant difference between a line having a - in front of it or not even though it just looks like a bulleted list, etc.

TOML looks kind of nice but didn’t have a serializer and I couldn’t be bothered to write one.

XML is verbose and generally nasty, but follows a pattern most half way technical people can deduce just by looking at it, and many people are familiar enough with HTML for it not to be a shock. There are built in serializers and deserializers in like every programming language. Winner!

But I agree in principle. Suggestions for a new config format will be entertained, as long as they are accompanied by code for loading and saving the config in the new format, and the config is notably clearer when expressed in the new format.

Example: http://godoc.org/github.com/BurntSushi/toml#example-Encoder-Encode

Neat! Repo created three weeks after I selected XML. :slight_smile: Let’s have a look what the config would look like in TOML…

It would look like this:

Version = 4

[[Repositories]]
  ID = "dropbox"
  Directory = "~/Dropbox"
  ReadOnly = true
  RescanIntervalS = 300
  IgnorePerms = false

  [[Repositories.Nodes]]
    NodeID = "NFGKEKE-7Z6RTH7-I3PRZXS-DEJF3UJ-FRWJBFO-VBBTDND-4SGNGVZ-QUQHJAG"

  [[Repositories.Nodes]]
    NodeID = "P56IOI7-MZJNU2Y-IQGDREY-DM2MGTI-MGL3BXN-PQ6W5BM-TBBZ4TJ-XZWICQ2"
  [Repositories.Versioning]
    Type = ""
    [Repositories.Versioning.Params]

[[Repositories]]
  ID = "lightroom"
  Directory = "~/Lightroom"
  ReadOnly = true
  RescanIntervalS = 180
  IgnorePerms = false

  [[Repositories.Nodes]]
    NodeID = "ME6QVQK-2B4BFY3-WIANFJC-SN76Q25-GMH3NZI-SD6LAYU-ME6CSDS-CPE47Q2"

  [[Repositories.Nodes]]
    NodeID = "NFGKEKE-7Z6RTH7-I3PRZXS-DEJF3UJ-FRWJBFO-VBBTDND-4SGNGVZ-QUQHJAG"

  [[Repositories.Nodes]]
    NodeID = "P56IOI7-MZJNU2Y-IQGDREY-DM2MGTI-MGL3BXN-PQ6W5BM-TBBZ4TJ-XZWICQ2"

  [[Repositories.Nodes]]
    NodeID = "SFPK3MX-DWCMRUU-2GV2OQT-4NXCTZX-RG7VHQZ-75P4CQO-IOOWCZ4-5BXXYA5"

  [[Repositories.Nodes]]
    NodeID = "ZJCXQL7-M3NP4IC-4KQ7WFU-3NANYUX-AD74QRL-Q5LJ7BH-72KYZHK-GHTAOAK"
  [Repositories.Versioning]
    Type = "staggered"
    [Repositories.Versioning.Params]
      cleanInterval = "3600"
      maxAge = "31536000"
      versionsPath = ""

[[Repositories]]
  ID = "default"
  Directory = "~/Sync"
  ReadOnly = false
  RescanIntervalS = 60
  IgnorePerms = false

  [[Repositories.Nodes]]
    NodeID = "AIR6LPZ-7K4PTTV-UXQSMUU-CPQ5YWH-OEDFIIQ-JUG777G-2YQXXR5-YD6AWQR"

  [[Repositories.Nodes]]
    NodeID = "GYRZZQB-IRNPV4Z-T7TC52W-EQYJ3TT-FDQW6MW-DFLMU42-SSSU6EM-FBK2VAY"

  [[Repositories.Nodes]]
    NodeID = "P56IOI7-MZJNU2Y-IQGDREY-DM2MGTI-MGL3BXN-PQ6W5BM-TBBZ4TJ-XZWICQ2"
  [Repositories.Versioning]
    Type = ""
    [Repositories.Versioning.Params]

[[Nodes]]
  NodeID = "ABCDEFG-H23ABC7-DEFGH23-ABCDEFY-GH23ABC-DEFGH2V-3ABCDEF-GH23IQN"
  Name = "test"
  Addresses = ["dynamic"]
  Compression = true
  CertName = ""

[[Nodes]]
  NodeID = "AIR6LPZ-7K4PTTV-UXQSMUU-CPQ5YWH-OEDFIIQ-JUG777G-2YQXXR5-YD6AWQR"
  Name = "freebsd"
  Addresses = ["dynamic"]
  Compression = true
  CertName = ""

[[Nodes]]
  NodeID = "GYRZZQB-IRNPV4Z-T7TC52W-EQYJ3TT-FDQW6MW-DFLMU42-SSSU6EM-FBK2VAY"
  Name = "win7"
  Addresses = ["dynamic"]
  Compression = true
  CertName = ""

[[Nodes]]
  NodeID = "ME6QVQK-2B4BFY3-WIANFJC-SN76Q25-GMH3NZI-SD6LAYU-ME6CSDS-CPE47Q2"
  Name = "zlogin3"
  Addresses = ["dynamic"]
  Compression = true
  CertName = ""

[[Nodes]]
  NodeID = "NFGKEKE-7Z6RTH7-I3PRZXS-DEJF3UJ-FRWJBFO-VBBTDND-4SGNGVZ-QUQHJAG"
  Name = "udev2"
  Addresses = ["dynamic"]
  Compression = true
  CertName = ""

[[Nodes]]
  NodeID = "P56IOI7-MZJNU2Y-IQGDREY-DM2MGTI-MGL3BXN-PQ6W5BM-TBBZ4TJ-XZWICQ2"
  Name = "jborg-mbp"
  Addresses = ["dynamic"]
  Compression = true
  CertName = ""

[[Nodes]]
  NodeID = "SFPK3MX-DWCMRUU-2GV2OQT-4NXCTZX-RG7VHQZ-75P4CQO-IOOWCZ4-5BXXYA5"
  Name = "acro-syncer"
  Addresses = ["dynamic"]
  Compression = true
  CertName = ""

[[Nodes]]
  NodeID = "U7ZFRQL-NIXNJXG-FDLMBQ3-LZLIXVE-QDJTJWD-XUKZOX6-EVO4MMG-OGVZGAO"
  Name = "winbox"
  Addresses = ["dynamic"]
  Compression = true
  CertName = ""

[[Nodes]]
  NodeID = "ZJCXQL7-M3NP4IC-4KQ7WFU-3NANYUX-AD74QRL-Q5LJ7BH-72KYZHK-GHTAOAK"
  Name = "anto-syncer"
  Addresses = ["2001:470:deeb:32::13"]
  Compression = true
  CertName = ""

[GUI]
  Enabled = true
  Address = "[::1]:8080"
  User = ""
  Password = ""
  UseTLS = false
  APIKey = "pvam35gvvpqsvoj82ji8o0p183igqv"

[Options]
  ListenAddress = ["0.0.0.0:22000"]
  GlobalAnnServer = "announce.syncthing.net:22026"
  GlobalAnnEnabled = true
  LocalAnnEnabled = true
  LocalAnnPort = 21025
  LocalAnnMCAddr = "[ff32::5222]:21026"
  ParallelRequests = 16
  MaxSendKbps = 0
  ReconnectIntervalS = 60
  StartBrowser = false
  UPnPEnabled = true
  UPnPLease = 0
  UPnPRenewal = 30
  URAccepted = -1

The pluralized section names would look better as singular, which is trivial to change of course.

With the corresponding current XML file being:

<configuration version="4">
    <repository id="dropbox" directory="~/Dropbox" ro="true" rescanIntervalS="300" ignorePerms="false">
        <node id="NFGKEKE-7Z6RTH7-I3PRZXS-DEJF3UJ-FRWJBFO-VBBTDND-4SGNGVZ-QUQHJAG"></node>
        <node id="P56IOI7-MZJNU2Y-IQGDREY-DM2MGTI-MGL3BXN-PQ6W5BM-TBBZ4TJ-XZWICQ2"></node>
        <versioning></versioning>
    </repository>
    <repository id="lightroom" directory="~/Lightroom" ro="true" rescanIntervalS="180" ignorePerms="false">
        <node id="ME6QVQK-2B4BFY3-WIANFJC-SN76Q25-GMH3NZI-SD6LAYU-ME6CSDS-CPE47Q2"></node>
        <node id="NFGKEKE-7Z6RTH7-I3PRZXS-DEJF3UJ-FRWJBFO-VBBTDND-4SGNGVZ-QUQHJAG"></node>
        <node id="P56IOI7-MZJNU2Y-IQGDREY-DM2MGTI-MGL3BXN-PQ6W5BM-TBBZ4TJ-XZWICQ2"></node>
        <node id="SFPK3MX-DWCMRUU-2GV2OQT-4NXCTZX-RG7VHQZ-75P4CQO-IOOWCZ4-5BXXYA5"></node>
        <node id="ZJCXQL7-M3NP4IC-4KQ7WFU-3NANYUX-AD74QRL-Q5LJ7BH-72KYZHK-GHTAOAK"></node>
        <versioning type="staggered">
            <param key="cleanInterval" val="3600"></param>
            <param key="maxAge" val="31536000"></param>
            <param key="versionsPath" val=""></param>
        </versioning>
    </repository>
    <repository id="default" directory="~/Sync" ro="false" rescanIntervalS="60" ignorePerms="false">
        <node id="AIR6LPZ-7K4PTTV-UXQSMUU-CPQ5YWH-OEDFIIQ-JUG777G-2YQXXR5-YD6AWQR"></node>
        <node id="GYRZZQB-IRNPV4Z-T7TC52W-EQYJ3TT-FDQW6MW-DFLMU42-SSSU6EM-FBK2VAY"></node>
        <node id="P56IOI7-MZJNU2Y-IQGDREY-DM2MGTI-MGL3BXN-PQ6W5BM-TBBZ4TJ-XZWICQ2"></node>
        <versioning></versioning>
    </repository>
    <node id="SFPK3MX-DWCMRUU-2GV2OQT-4NXCTZX-RG7VHQZ-75P4CQO-IOOWCZ4-5BXXYA5" name="acro-syncer" compression="true">
        <address>dynamic</address>
    </node>
    <node id="ZJCXQL7-M3NP4IC-4KQ7WFU-3NANYUX-AD74QRL-Q5LJ7BH-72KYZHK-GHTAOAK" name="anto-syncer" compression="true">
        <address>2001:470:deeb:32::13</address>
    </node>
    <node id="AIR6LPZ-7K4PTTV-UXQSMUU-CPQ5YWH-OEDFIIQ-JUG777G-2YQXXR5-YD6AWQR" name="freebsd" compression="true">
        <address>dynamic</address>
    </node>
    <node id="P56IOI7-MZJNU2Y-IQGDREY-DM2MGTI-MGL3BXN-PQ6W5BM-TBBZ4TJ-XZWICQ2" name="jborg-mbp" compression="true">
        <address>dynamic</address>
    </node>
    <node id="ABCDEFG-H23ABC7-DEFGH23-ABCDEFY-GH23ABC-DEFGH2V-3ABCDEF-GH23IQN" name="test" compression="true">
        <address>dynamic</address>
    </node>
    <node id="NFGKEKE-7Z6RTH7-I3PRZXS-DEJF3UJ-FRWJBFO-VBBTDND-4SGNGVZ-QUQHJAG" name="udev2" compression="true">
        <address>dynamic</address>
    </node>
    <node id="GYRZZQB-IRNPV4Z-T7TC52W-EQYJ3TT-FDQW6MW-DFLMU42-SSSU6EM-FBK2VAY" name="win7" compression="true">
        <address>dynamic</address>
    </node>
    <node id="U7ZFRQL-NIXNJXG-FDLMBQ3-LZLIXVE-QDJTJWD-XUKZOX6-EVO4MMG-OGVZGAO" name="winbox" compression="true">
        <address>dynamic</address>
    </node>
    <node id="ME6QVQK-2B4BFY3-WIANFJC-SN76Q25-GMH3NZI-SD6LAYU-ME6CSDS-CPE47Q2" name="zlogin3" compression="true">
        <address>dynamic</address>
    </node>
    <gui enabled="true" tls="false">
        <address>[::1]:8080</address>
        <apikey>pvam35gvvpqsvoj82ji8o0p183igqv</apikey>
    </gui>
    <options>
        <listenAddress>0.0.0.0:22000</listenAddress>
        <globalAnnounceServer>announce.syncthing.net:22026</globalAnnounceServer>
        <globalAnnounceEnabled>true</globalAnnounceEnabled>
        <localAnnounceEnabled>true</localAnnounceEnabled>
        <localAnnouncePort>21025</localAnnouncePort>
        <localAnnounceMCAddr>[ff32::5222]:21026</localAnnounceMCAddr>
        <parallelRequests>16</parallelRequests>
        <maxSendKbps>0</maxSendKbps>
        <reconnectionIntervalS>60</reconnectionIntervalS>
        <startBrowser>false</startBrowser>
        <upnpEnabled>true</upnpEnabled>
        <upnpLeaseMinutes>0</upnpLeaseMinutes>
        <upnpRenewalMinutes>30</upnpRenewalMinutes>
        <urAccepted>-1</urAccepted>
    </options>
</configuration>

It’s not entirely obvious to me that it’s much better… Clearly, the Options section is much better in TOML. But the repository and node configs? Double [[]]? Maybe I’m just used enough to the XML for Stockholm syndrome to kick in…

Yes, I understand what [[foo]] means.

To clarify my position, the following are properties I consider important in the config format (in no particular order):

  • Human-friendliness. Exact what this means is subjective, but I see it as some combination of having a visible structure, not being unnecessarily cryptic and being somewhat tolerant to being copied and pasted through an email or similar.

  • Machine friendliness. Obviously it must be possible to read and write in Go, preferably it shouldn’t be so oddball that it’s a pain to find serializers for other languages either. If it’s a format that there already exists tools to mangle and parse in different ways, all the better.

  • Extensibility. If we suddenly discover that we need to add fields to some struct, or remove them, or make what was previously just a string into a struct of some kind, then we should be able to do so with as little pain as possible. This includes not invalidating previous config versions, as far as possible.

A new config format should be clearly better in one or more areas, and not significantly worse in the other. Changing the config format is fairly painful. The config handling needs to be rewritten, all tests need to be rewritten, code needs to be in place to handle the transition, the documentation must be updated. GUI wrappers, config templates, etc. must be rewritten for the new format and people have to re-learn it, so there are ripple effects through the whole “ecosystem”. For this to be worth it at this point, the advantages have to be pretty clear, and it’s not something we’ll do willy-nilly.

Successfully arguing for a change in format requires addressing these concerns. I estimate that we have at most one more try at getting the config format right before the pain of changing it becomes unbearable, so we better be sure. :wink:

Looking specifically at TOML I don’t see it as clearly better for humans (yes, it looks somewhat nicer than XML, but it’s indentation sensitive and still depends on brackets of various kinds), nor clearly better for machines (XML support is pretty wide spread) and an even draw on extensibility. The effort looks pretty large compared to what we gain?

1 Like