Howto unmarshall /rest/system/connections in Go

Hi,

How to unmarshall /rest/system/connections in Golang?

I mean I know how to use json.Unmarshall(), I’m not sure how to construct right struct since the json has dynamic keys. I tried everything including lots of examples from the web and can’t get it going.

This is what I have right now:

	type ConnectionInfo struct {
		Connections struct {
			Device map[string]struct {
				At        string `json:"at"`
				...
			}
		} `json:"connections"`
		Total struct {
			At string `json:"at"`
			...
		} `json:"total"`
	}

Total.At is fine but I can’t map deviceIDs in to map keys.

Do I have to resort to using map[string]interface{}, or is there a better struct that I could use?

Any help would be appreciated!

Generally deserialising with map[string]interface{} is a good generic approach. In this case especially using go, you can also just directly use Syncthing’s types:

Thanks for your answer. I looked at the ConnectionInfo type in model.go before however it doesn’t address the problem of the dynamic key.

Keep in mind that the json looks like this:

{
  "connections": {
    "XXXX-YYYY-ZZZZ": {
      "at": "0001-01-01T00:00:00Z",
      "inBytesTotal": 0,
      "outBytesTotal": 0,
      ...
    },
    "AAAA-BBBB-CCCC": {
      "at": "0001-01-01T00:00:00Z",
      "inBytesTotal": 0,
      "outBytesTotal": 0,
      ...
    },
    ....
  }
  "total": {
    "at": "2022-02-16T05:07:09-08:00",
    "inBytesTotal": 240597027,
    "outBytesTotal": 3367569691
  }
}

DeviceID is a dynamic key. I see how it’s constructed in m.ConnectionStats() but I did not find an elegant way to unmarshall this to a map of strings without resorting to interface{}.

I was hoping that someone could provide an example. Thanks

Looks like a

type response struct {
  Connections map[string]protocol.ConnectionInfo
  Total       protocol.Statistics
}

I’m not sure why the producing side is all interface{} about it.

1 Like

This does not relate to OPs question, rather than my previous comment to other maintainers of using maps in rest interfaces being a bad idea.

2 Likes

Care to elaborate or summarize your arguments? I believe you have good reasons for that assessment. Just would like to learn what practical challenges you are referring to.

I think I’ve explained my stance twice already. Last one was on a PR removing deceiving numbers from that endpoint I think, can’t remember where the other one was.

Sorry, but I’m just too tired to type it all out on the phone again after a day of skiing.

1 Like

No problem. If you ever find the time to summarize it or point to where I can find the arguments, just leave a link somewhere. Sorry the other discussions obviously didn’t make it clear enough for me, maybe missing some background or context :man_shrugging:

If not, also okay.

The point was, I think, to use lists for collections and not maps, as things are often simpler when keys are known beforehand.

1 Like

That’s the point I don’t understand. When a bunch of “things” each have a unique ID and some content / attributes, isn’t it much easier to consume that when it’s sent as an “associative array” (JSON object) than having to run through the whole list, searching for the entry you want by comparing the ID attribute?

What exactly do you mean with “known beforehand”?

Yeah I don’t think it’s particularly complicated either but then we get threads like these.

I guess it’s something people are not used to?

Json is not for people, it’s for computers. I’d say programming languages are not used to it.

I do read it with map[string]interface{} but it’s far from being elegant.

Yeah, that is the main argument to my understanding: Some more “basic languages” don’t handle “dynamic objects” (aka maps) well. However in go I don’t see why you can’t use map[string]protocol.ConnectionInfo as @calmh proposed above?

But the point of an api is to be able to call it easily … from any language.

It works in go is not a good argument, as we’d be using gob instead of json if that is all we cared about.

Standards like openapi and others discourage (and perhaps not even support) maps for specifically the interop reason.

Maps also don’t really fit into object relational model.

3 Likes

Okay the example by @calmh worked, thank you very much for it! Looks like I was close to it just had it nested struct in a struct instead of map of struct. Anyways thank you all for help! :slight_smile:

2 Likes

Suppose I want to adhere to your advice and not return “associative arrays” in an API response. But the structure of the collection is clearly as with mentioned connections: An ID string and an object of always the same structure containing attributes of the identified thing.

So I should always use a format like this?

{
  "interestingThings": [
    {
      "thingId": "abc",
      "stuffs": [1, 2, 3, 4]
    },
    {
      "thingId": "xyz",
      "stuffs": [3, 4, 5, 6]
    }
  ]
}

Or just skip the outer object and return a collection (array) directly? Example: GET /rest/events — Syncthing v1 documentation

What I previously thought would be nicer to consume (because we can do direct lookups by ID) is the following:

{
  "abc": {
    "stuffs": [1, 2, 3, 4]
  },
  "xyz": {
    "stuffs": [3, 4, 5, 6]
  }
}

Of course, if there is only the single stuffs attribute, its value could be used directly without wrapping in another object. Provided it’s clear from the endpoint name what “stuff” we are getting:

{
  "abc": [1, 2, 3, 4],
  "xyz": [3, 4, 5, 6]
}

Examples are GET /rest/cluster/pending/devices — Syncthing v1 documentation (which I added without knowing any better) and GET /rest/system/connections — Syncthing v1 documentation mentioned here (which I erroneously took as inspiration).

Implementing the first style above means to find out if the tuple ("abc", 3) is among the results, I need to walk every entry and compare both thingId and search in the stuffs attribute. Or rearrange the results into nested objects client-side after retrieving.

What I’d like to do in this case would be the equivalent to Python’s found = ("abc", 3) in results or at the least found = 3 in results["abc"]. Ideally without burning cycles needlessly on the client side when the generating Go code already used efficient maps internally.

Ping @AudriusButkevicius, am I understanding that correctly? Don’t want to design the next API endpoint with the “wrong” usage pattern.

What you showed in your first example is the right way.

The latter two are both wrong, and “what’s nicer” becomes irrelevant.

You cannot take that structure, and map it to structs and arrays in C ABI. You need hashmaps for lookups, which not all languages have.

Both of the examples from our docs are wrong based on the above constraints.

You can build your lookup after you received the data from the rest API if that’s the way you want to query.

As for “burning cycles”, it’s best to understand what things cost before trying to save the cycles.

image

Sure, this will matter at 1000s of elements, but I doubt we’ll ever get there, so in our case, array iteration is actually faster. Even if we do get there, the transformation will have close to zero cost.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.