# A Syncthing indicator

**URL:** https://forum.syncthing.net/t/a-syncthing-indicator/29
**Category:** Feature
**Created:** [April 21, 2014, 9:26am UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29 "2014-04-21T09:26:07Z")
**Posts on this page:** 20
**Page:** 1

<div class="post-metadata">

### Author: ![sil](https://forum.syncthing.net/user_avatar/forum.syncthing.net/sil/32/8_2.png) [@sil](https://forum.syncthing.net/u/sil)
#### Post date: [April 21, 2014, 9:26am UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29/1 "2014-04-21T09:26:07Z")

</div>

I’ve now got syncthing all rigged up how I want it (and thank you @calmh for all the help!). As promised, I’d like to put together a simple status menu for Syncthing to live in my Ubuntu panel so I can see whether it’s doing things without having to open up the web GUI in a tab. This should be fine using the API (which I admit isn’t documented, but it’s not hard to see how it works by looking at the GUI’s js code). However, in thinking about this, I’ve hit an issue.

Syncthing’s GUI does stuff by polling. The steady state of syncthing is that everything is synced. This means that almost all the time, when you ask for the current state of syncthing, it’ll be “everything is 100% synced”. If I, for example, drop a small new file into one of my repositories at time 0, I likely won’t notice anything in my indicator because that file will be detected by the syncthing polling interval at some point later (at time 60, say), but my indicator will poll syncthing’s API for interesting changes at time 37 and then at time 97, and at both of those times we’ll be 100% synced (we assume the file is small). So I can’t actually tell whether the file I’ve just added has been synced or not by polling the API; I don’t know whether we’re at 100% synced because it’s been noticed and synced, or because it hasn’t been noticed yet.

The obvious way to solve this would be to keep a record of the last few files that were synced; then the indicator can show that information. As far as I can tell, I can’t get this information out of the API. Is there a way to do that, or is there a better way to tell the difference between “100% synced because we haven’t noticed the new stuff” and “100% synced because we have noticed it and have synced it”? Another alternative might be an API endpoint listing a stream of “events”, or similar…?

---

<div class="post-metadata">

### Author: ![calmh](https://forum.syncthing.net/user_avatar/forum.syncthing.net/calmh/32/15311_2.png) [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [April 21, 2014, 12:08pm UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29/2 "2014-04-21T12:08:44Z")

</div>

Indeed. So, if you didn’t already find it, the best “documentation” of the current available requests is in [https://github.com/calmh/syncthing/blob/master/cmd/syncthing/gui.go#L36-L50](https://github.com/calmh/syncthing/blob/master/cmd/syncthing/gui.go#L36-L50). The current GUI calculates the percentage on the repo (indicating how in sync _this_ node is) by using the `/rest/model/<somereponame>` call and comparing `globalBytes` to `insyncBytes`. For data on _other_ nodes the `/rest/connections` call returns the in sync percentage directly.

As you say, both are likely to be at 100% at any given time if only small changes happen occasionally. I could see it being useful to have the event log you mention and I think that wouldn’t be very tricky to implement. Off hand, I think we could log events for:

- Detected change in local file _x_
- Started / completed syncing file _y_ from cluster
- Connected / disconnected to node _n_
- Node _n_ requested some blocks from file _x_
- In sync percentage for repo _r_ changed to _p_
- In sync percentage for node _n_ change to _p_

Obviously, polling (as opposed to streaming) sucks in the general case, increasing load unnecessarily etc. In this specific case we’re only likely to have at most a handful of users connected to the admin interface at any given time so simplicity wins over efficiency…

Still, it would be cool to expose this live over a streaming websocket, for example. I might look into that.

---

<div class="post-metadata">

### Author: ![sil](https://forum.syncthing.net/user_avatar/forum.syncthing.net/sil/32/8_2.png) [@sil](https://forum.syncthing.net/u/sil)
#### Post date: [April 21, 2014, 12:49pm UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29/3 "2014-04-21T12:49:53Z")

</div>

All that seems good to me, and that would give a third-party app enough knowledge to be able to know whether something has changed since the last time it looked. It would be more convenient to do that if the log were in timestamp order and each event had a timestamp, because then when I hit the log I can remember the timestamp of the most recent item and then just ignore everything before that. (Of course, this could be an increasing number rather than a timestamp if that’s more convenient.) Agreed completely that polling sucks for efficiency but we’re not serving a million users so it’s much better; the issue, as noted, is that when things start and them complete entirely between two polling intervals, you’ve got no idea that they happened at all 🙂

Streaming websocket would be nice, indeed, although reading a websocket if you’re not actually a web browser can be a bit fiddly; this is where “the API is basically for the web GUI” and “the API is for any client, where the web GUI is just one example” come slightly into conflict. A long-poll JSON-style thing or a server-sent-event thing might be better in this case?

---

<div class="post-metadata">

### Author: ![calmh](https://forum.syncthing.net/user_avatar/forum.syncthing.net/calmh/32/15311_2.png) [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [April 22, 2014, 7:56am UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29/4 "2014-04-22T07:56:00Z")

</div>

Yep. I have a prototype of this cooking. The interface is doing a GET of `/rest/events/<lastSeenId>` where `<lastSeenId>` is the ID of the last event you’ve already seen, or zero. You get back an array of events that has happened since that ID (up to a point; currently it keeps the last 100 events so if you’ve missed more than that there will be a gap). If there are no new events it blocks for up to 300 seconds waiting for an event, then returns a timeout event with ID -1 if nothing interesting happened. Every event has `id`, `type`, `timestamp` and `params` fields. The results of a typical polling loop might look something like this:

```
jb@jborg-mbp:~ $ curl -s -u st:test http://localhost:8080/rest/events/0 | jsonpp
[
  {
    "id": 1,
    "timestamp": "2014-04-22T09:03:11.274499963+02:00",
    "type": "NODE_CONNECTED",
    "params": {
      "node": "LGFPDIT7SKNNJVJZA4FC7QNCRKCE753K72BW5QD2FOZ7FRFEP57Q"
    }
  },
  {
    "id": 2,
    "timestamp": "2014-04-22T09:03:11.335947892+02:00",
    "type": "NODE_CONNECTED",
    "params": {
      "node": "ME6QVQK2B4BFYWIANFJCSN76Q2GMH3NZISD6LAYME6CSDSCPE47Q"
    }
  },
  {
    "id": 3,
    "timestamp": "2014-04-22T09:03:13.128464+02:00",
    "type": "NODE_INDEX",
    "params": {
      "hasBytes": 7229008723,
      "node": "ME6QVQK2B4BFYWIANFJCSN76Q2GMH3NZISD6LAYME6CSDSCPE47Q",
      "repo": "lightroom",
      "totalBytes": 7229008723
    }
  }
]
jb@jborg-mbp:~ $ curl -s -u st:test http://localhost:8080/rest/events/3 | jsonpp
## blocks for a while ... ##
[
  {
    "id": 4,
    "timestamp": "2014-04-22T09:04:33.220233006+02:00",
    "type": "NODE_INDEX",
    "params": {
      "hasBytes": 7229008723,
      "node": "LGFPDIT7SKNNJVJZA4FC7QNCRKCE753K72BW5QD2FOZ7FRFEP57Q",
      "repo": "lightroom",
      "totalBytes": 7229008723
    }
  }
]
jb@jborg-mbp:~ $ curl -s -u st:test http://localhost:8080/rest/events/4 | jsonpp
## blocks for five minutes ... ##
[
  {
    "id": -1,
    "timestamp": "2014-04-22T09:09:37.526111988+02:00",
    "type": "TIMEOUT",
    "params": null
  }
]

```

That seem OK? I’ll write something up on the event types and their expected params as I get along.

---

<div class="post-metadata">

### Author: ![calmh](https://forum.syncthing.net/user_avatar/forum.syncthing.net/calmh/32/15311_2.png) [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [April 22, 2014, 7:35pm UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29/5 "2014-04-22T19:35:36Z")

</div>

[https://github.com/calmh/syncthing/wiki/Event-Interface](https://github.com/calmh/syncthing/wiki/Event-Interface) (work in progress)

---

<div class="post-metadata">

### Author: ![sil](https://forum.syncthing.net/user_avatar/forum.syncthing.net/sil/32/8_2.png) [@sil](https://forum.syncthing.net/u/sil)
#### Post date: [April 23, 2014, 1:02am UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29/6 "2014-04-23T01:02:03Z")

</div>

`PULL_START: Generated when syncthing begins synchronizing a file to a newer version.`

If I’m hitting the `/rest/events/` endpoint on machine THIS, which shares a repo with machine OTHER, and I edit a file on OTHER, syncthing will then pull those changes over to THIS, and presumably fire a `PULL_START` event. If I edit a file on THIS and syncthing starts sending it over to OTHER, will that also fire PULL\_START? Or is there a “PUSH\_START” in planning? I may be looking at that page when it’s only half finished 😄

---

<div class="post-metadata">

### Author: ![calmh](https://forum.syncthing.net/user_avatar/forum.syncthing.net/calmh/32/15311_2.png) [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [April 23, 2014, 6:25am UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29/7 "2014-04-23T06:25:36Z")

</div>

Yeah, it’s half finished. The “pull\_\*” stuff is all about getting files from OTHER to THIS.

For the other direction, I’ll expose “the local repo changed: the file _f_ is new/changed/deleted” and _maybe_ “the remote node OTHER requested blocks for file _f_”. The “maybe” on the latter is just because that is slightly more involved in that we need to rate limit it (i.e. keep state so we don’t generate an event for _every_ block requested).

---

<div class="post-metadata">

### Author: ![sil](https://forum.syncthing.net/user_avatar/forum.syncthing.net/sil/32/8_2.png) [@sil](https://forum.syncthing.net/u/sil)
#### Post date: [April 23, 2014, 7:41am UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29/8 "2014-04-23T07:41:37Z")

</div>

Makes sense. One of the big end goals here, the questions that I think a user will want to ask and have answered, is: I have just edited file _f_ on my computer. Has that file _f_ been successfully synced to another node?

I don’t know whether it’s actually _possible_ to answer that question from syncthing’s PoV; syncthing on THIS may only know that OTHER has requested blocks for file _f_, and not know whether OTHER now has an up-to-date copy of file _f_. However, I think that that’s what people actually care about; if it’s not possible to answer that question, maybe a brief protocol augmentation might make it so?

---

<div class="post-metadata">

### Author: ![calmh](https://forum.syncthing.net/user_avatar/forum.syncthing.net/calmh/32/15311_2.png) [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [April 24, 2014, 7:06am UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29/9 "2014-04-24T07:06:37Z")

</div>

Well, it could be answered, as in the local syncthing node knows. What actually happens protocol wise is the following:

1. nodeA discovers a change to the file F, rescans it and gets a new set of block hashes + metadata
2. nodeA sends an “index update” to it’s peer, nodeB (and any other peers; we’ll do the exercise for one only) containing the new file data and a bumbed version number. At this point nodeA knows that nodeB _doesn’t_ have the updated file, because it hasn’t seen an index update for that file with the new version number and hashes etc. In the current GUI, nodeB’s “in sync percentage” will be dropped from 100% to something lower since it does not have all the bytes that make up the latest version of all files.
3. nodeB receives the index update and notices the local file is out of date. It is scheduled for “pulling”.
4. nodeB figures out which blocks have changed compared to the old file, requests those from it’s peers and puzzles together the new version of the file.
5. Assuming no problems happen during #4, the new version of the file is OK and nodeB sends an index update to it’s peers announcing that it has the file.
6. nodeA sees the index update from above and knows that nodeB has successfully updated the file. nodeB is shown at 100% again.

From there it’s “just” a question of what events to expose. Technically we could generate events for each file that is changed for each connected peer.

---

<div class="post-metadata">

### Author: ![sil](https://forum.syncthing.net/user_avatar/forum.syncthing.net/sil/32/8_2.png) [@sil](https://forum.syncthing.net/u/sil)
#### Post date: [April 24, 2014, 10:04am UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29/10 "2014-04-24T10:04:29Z")

</div>

> [@calmh](#):
>
> Assuming no problems happen during #4, the new version of the file is OK and nodeB sends an index update to it’s peers announcing that it has the file.

Aha. Does nodeB send an index update per file updated, or does it send one at the end of the whole process? An event for “file f up-to-date on node b” would exactly cover the use case from above, I think.

---

<div class="post-metadata">

### Author: ![calmh](https://forum.syncthing.net/user_avatar/forum.syncthing.net/calmh/32/15311_2.png) [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [April 24, 2014, 10:09am UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29/11 "2014-04-24T10:09:16Z")

</div>

The index update is sent every five seconds, if there are changes to send. The update then includes all files changed since last update. So there is a delay but not very long.

---

<div class="post-metadata">

### Author: ![sil](https://forum.syncthing.net/user_avatar/forum.syncthing.net/sil/32/8_2.png) [@sil](https://forum.syncthing.net/u/sil)
#### Post date: [April 24, 2014, 3:36pm UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29/12 "2014-04-24T15:36:39Z")

</div>

That’d be perfect, then: have a node fire an event for “files updated on node B: f1, f2, f3, f4, f5”?

---

<div class="post-metadata">

### Author: ![sil](https://forum.syncthing.net/user_avatar/forum.syncthing.net/sil/32/8_2.png) [@sil](https://forum.syncthing.net/u/sil)
#### Post date: [April 29, 2014, 7:40pm UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29/13 "2014-04-29T19:40:27Z")

</div>

Just to keep people in the loop, [https://github.com/stuartlangridge/syncthing-ubuntu-indicator](https://github.com/stuartlangridge/syncthing-ubuntu-indicator). Since the event interface isn’t actually in syncthing yet, I threw in a tiny test server which pretends to be the syncthing events API, and you can generate fake pull events and so on. So, a work in progress, and it’ll need extra once it gets hooked up to the real syncthing, but progress is made.

---

<div class="post-metadata">

### Author: ![calmh](https://forum.syncthing.net/user_avatar/forum.syncthing.net/calmh/32/15311_2.png) [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [April 30, 2014, 11:20am UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29/14 "2014-04-30T11:20:40Z")

</div>

Cool! I haven’t had time to finish the event interface, turned out to be a bunch of code in between where the events need to be fired and where they are to be reported… But I’ll get there. 🙂

---

<div class="post-metadata">

### Author: ![icaruseffect](https://forum.syncthing.net/user_avatar/forum.syncthing.net/icaruseffect/32/257_2.png) [@icaruseffect](https://forum.syncthing.net/u/icaruseffect)
#### Post date: [August 5, 2014, 10:00am UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29/15 "2014-08-05T10:00:22Z")

</div>

Hi, i have taken your code and adopted it to the new event interface (see [https://github.com/icaruseffect/syncthing-ubuntu-indicator](https://github.com/icaruseffect/syncthing-ubuntu-indicator)). Pull request on Github is out.

So i have a question: are you still working on the Project (?), so there is no work done double. Thank you for the good code basis anyway. It was much better than what I’ve been coding, before i found yours.

---

<div class="post-metadata">

### Author: ![icaruseffect](https://forum.syncthing.net/user_avatar/forum.syncthing.net/icaruseffect/32/257_2.png) [@icaruseffect](https://forum.syncthing.net/u/icaruseffect)
#### Post date: [August 5, 2014, 10:24am UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29/16 "2014-08-05T10:24:24Z")

</div>

@calmh Until now i’ve never fetched an “ItemStarted” type from the event interface. Could it be this one is missing?

---

<div class="post-metadata">

### Author: ![calmh](https://forum.syncthing.net/user_avatar/forum.syncthing.net/calmh/32/15311_2.png) [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [August 5, 2014, 10:57am UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29/17 "2014-08-05T10:57:25Z")

</div>

ItemStarted should be emitted when a file begins syncing, looks like it should work at least. 🙂

---

<div class="post-metadata">

### Author: ![icaruseffect](https://forum.syncthing.net/user_avatar/forum.syncthing.net/icaruseffect/32/257_2.png) [@icaruseffect](https://forum.syncthing.net/u/icaruseffect)
#### Post date: [August 5, 2014, 11:00am UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29/18 "2014-08-05T11:00:19Z")

</div>

Hmm, then I’ve missed something. Thank you for checking 🙂

---

<div class="post-metadata">

### Author: ![calmh](https://forum.syncthing.net/user_avatar/forum.syncthing.net/calmh/32/15311_2.png) [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [August 5, 2014, 11:03am UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29/19 "2014-08-05T11:03:40Z")

</div>

Ah, no. The event is emitted, but the `/rest/events` endpoint doesn’t actually expose ItemStarted and ItemCompleted. The reason was that the amount of events can be overwhelming and it’s not used by the current GUI. I was intending to add a separate endpoint for that for when it was actually needed.

Edit: I exposed them on the same event interface now, it’s not _that_ expensive (I think). Will be in next release.

---

<div class="post-metadata">

### Author: ![icaruseffect](https://forum.syncthing.net/user_avatar/forum.syncthing.net/icaruseffect/32/257_2.png) [@icaruseffect](https://forum.syncthing.net/u/icaruseffect)
#### Post date: [August 5, 2014, 1:19pm UTC](https://forum.syncthing.net/t/a-syncthing-indicator/29/20 "2014-08-05T13:19:27Z")

</div>

Perfect 🙂 So on the next release ItemStarted & ItemCompleted are exposed?

[Next page](https://forum.syncthing.net/t/a-syncthing-indicator/29.md?page=2)