syncthing-map: a tool to map syncthing devices and folders

I started to build a utility to map synching devices and folders, based on their configuration.xml files.

This is a very first draft version, but it works for me :slight_smile:

Thank you very much!

Testing right now, however it fails when trying to generate the graph.

.\syncthing-map.exe graph
syncthing-map: No subcommand or positional value found at position 1.
Available subcommands: read graph clean

For the record, this is under Windows 10.

I built in under Windows 10 as well. I will have a look at that tomorrow morning when I have my laptop.

Sorry for that, I was in a hurry to send something before Monday, should have tested more the executable (this is the first time I am using flaggy and the issue is there)

No problem :slightly_smiling_face: . I’m looking forward to testing the new version when it’s ready.

It is fixed. Sorry for that.

It works :slightly_smiling_face:. The readme probably needs an update due to the command change.

Howver, there’s a problem though. The resulting HTML doesn’t render with the following error.

image

There are 49 folders and 7 devices in the config.xml. Is this too much? The HTML file itself seems fine code wise.

Hmmmm. I have no idea, I never hit a limit with mermaid.js.

Let me open a question over there. If this is an insurmountable obstacle then I will look for another graphing tool (d3, …).

Yes, I sent another commit right after updating the binaries.

I’ve done more testing. The problem seems to happen when the HTML starts exceeding ~50 KB. I’ve now taken config files from 3 devices. If I add only two of them, the graph still renders, however with all three it stops rendering with the error. Also, it really seems to be just the amount of data to be the culprit, as I’ve tried to obfuscate the file (e.g. by shortening device IDs, etc.), and then it did render in the end.

(removing previous comments that are outdated now)

This is actually the right guess. In mermaid.js code there is

// Check the maximum allowed text size
  // TODO: Remove magic number
  if (text.length > (config?.maxTextSize ?? 50000)) {
    text = MAX_TEXTLENGTH_EXCEEDED_MSG;
  }

Could you please try

syncthing-map.exe (5.3 MB)

I changed the default max size of the code from 50000 to 5000000 (so 5 Mb if I am not mistaken).

It’s loading fine now! :sunglasses:

However, I’ve observed one very strange thing. I’m following the readme, i.e.

.\syncthing-map.exe add --device "d1" --file ".\config1.xml" --device "d2" --file ".\config2.xml" --device "d3" --file "config3.xml"

and the problem is that in the graph, the first device’s name is wrong. Not sure how to describe this, but I’ve seen it named either just “device” or it was as if if took a part of the other device’s name. Also, it looks like the output differs depending on whether quotes are used, e.g. the two commands

.\syncthing-map.exe add --device "d1" --file ".\config1.xml" --device "d2" --file ".\config2.xml" --device "d3" --file "config3.xml"

and

.\syncthing-map.exe add --device d1 --file .\config1.xml --device d2 --file .\config2.xml --device d3 --file config3.xml

produce different results.

Good, this means that the change in the max size was fine, thanks for testing. I will generate a new version tonight. Fixed with Release 0.3 · wsw70/syncthing-map · GitHub

As to your points: the add command is supposed to be ran repeatedly for each file, not with all files at once (to be frank I did not even envision that so I do not know how it works with the command parsing library that I just discover as I go). You should be doing

.\syncthing-map.exe add --device my-device-laptop --file config-laptop.xml
.\syncthing-map.exe add --device john-android --file config-ja.xml
(...)

This way, each time the data file (data.json) will be updated with the extra configuration.

I just tried with my files and this is what I see:

PS D:\dev-perso\syncthing-map> .\syncthing-map.exe add --device srv --file config-srv.xml
{"level":"info","time":"2023-01-09T19:46:02+01:00","message":"wrote data.json file"}
PS D:\dev-perso\syncthing-map> .\syncthing-map.exe add --device router --file config-router.xml
{"level":"info","time":"2023-01-09T19:46:16+01:00","message":"wrote data.json file"}

I do not use any quotes.

Now - as I mentioned, I am just discovering the library for arguments parsing (and I am an amateur dev :)) so there may be edge cases (which I will try to cover, such as the repeated arguments in your case) but if you go for the vanilla version as above it should be fine.

With this - please let me know if this helps.

EDIT: I’ve added your points to Better control cli arguments · Issue #3 · wsw70/syncthing-map · GitHub to keep track.

EDIT 2: ahhh, I get it now why you say that you did that following the README. I write “repeatedly” which indeed could be interpreted as “with each pair in one single command”. I will correct that. Corrected in the README.

Thank you! Now everything is clear and the separate commands seem to be working as expected :wink:. I’m going to test with more config files and come back with new observations (if any).

You’re very much welcome, powodzenia (if I got your first name as Polish right :))

A stable version was released, and will always be available at Release stable version · wsw70/syncthing-map · GitHub

The development will happen in the dev branch, executables will also be automatically built and released as for the stable one.

NEW EXPERIMENTAL FEATURE

map-syncthing can now work in server mode and build maps dynamically. This is an experimental feature that will be merged in the stable one after some testing.

Binaries are available in the development release and details are available in the README of the dev branch.

Let me know what you think. I think this is uber-cool :slight_smile:

Server mode is now stable!

After endless tests over two days (and discovery of how complicated my setup is, and finding some discrepancies) I pushed the server mode to stable.

I also swapped the Automated and Manual mode in the README because, well, the automated mode is way cooler :sunglasses:

Excited to try this out!

Readme needs some clarifications for less experienced users.

  • It appears that syncthing-map-server.yaml must be manually created first – this should be stated.
  • Automated Server mode sounds like it will connect to Synthing and sync the configs; I don’t think this is what is meant.
  • It would also help to clarify the difference between “Automated” and “Manual” before explaining the steps in each use.
  • Clarify the device name “defined in the device” – specify where this info is found.

This is a great utility.

First attempt failed with:

Syntax error in graph mermaid version 9.3.0

Tried:

  • All configs – Fail
  • Each device & path alone (just two lines in .yaml) – Fail
    • In case one of the configs had something confusing
  • Relative path to config – Fail
  • Absolute path to config – Fail
  • Quoted path to config – Go error

Example output:

2023-01-13T17:27:26-05:00 INFO starting server on port 3000
2023-01-13T17:27:42-05:00 INFO no data-server.json file
2023-01-13T17:27:42-05:00 INFO wrote data-server.json file
2023-01-13T17:27:42-05:00 INFO parsed configuration for Cesium
2023-01-13T17:27:42-05:00 INFO wrote data-server.json file
2023-01-13T17:27:42-05:00 INFO parsed configuration for argon
2023-01-13T17:27:42-05:00 INFO wrote data-server.json file
2023-01-13T17:27:42-05:00 INFO parsed configuration for atlas
2023-01-13T17:27:42-05:00 INFO wrote data-server.json file
2023-01-13T17:27:42-05:00 INFO parsed configuration for buf
2023-01-13T17:27:42-05:00 INFO wrote data-server.json file
2023-01-13T17:27:42-05:00 INFO parsed configuration for ord
2023-01-13T17:27:42-05:00 INFO wrote data-server.json file
2023-01-13T17:27:42-05:00 INFO parsed configuration for astraeus
2023-01-13T17:27:42-05:00 INFO wrote syncthing-map-server.html
2023/01/13 17:27:42 "GET http://127.0.0.1:3000/ HTTP/1.1" from 127.0.0.1:63220 - 200 30720B in 146.2295ms

No apparent error messages. No difference between the syncthing-map-server.html and the server file.

Suggestions on troubleshooting?

Cool, thanks!

No, it is generated automatically as a “by-product” of calling the web interface.

Very good points, I updated the documentation. Thanks.

I did not get that. Can you give an example of what is not clear?

Sure. Sorry for the trouble.

I assume that you are starting with the “Automated” version, that is one where in syncthing-map-server.yaml you manually created entries such as

- device: <the name of a device>
  folder: <the path to the folder containing config.xml>

I will take the case of your first device: Cesium. Do you have

- device: Cesium
  folder: /path/to/folder

Let’s stick with only one entry in syncthing-map-server.yaml for now (the map will be minimalist, but it will be there). Delete data-server.json.

When you run syncthing-map server do you get as output

2023-01-13T17:27:26-05:00 INFO starting server on port 3000
2023-01-13T17:27:42-05:00 INFO no data-server.json file
2023-01-13T17:27:42-05:00 INFO wrote data-server.json file
2023-01-13T17:27:42-05:00 INFO parsed configuration for Cesium
2023-01-13T17:27:42-05:00 INFO wrote data-server.json file

Can you share the content of syncthing-map-server.yaml, data.json and syncthing-map-server.html? (you can shorten the IDs to the first section if you want and redact whatever, but keep them consistent so that I can compare - especially if there are some strageness in the folder of device names).

Do you have any special characters in your folders/device names? (wild guess here)

Since the error is Syntax error in graph mermaid version 9.3.0 I guess that the buildup of the mermaid code (the code of the HTML file) failed with some edge case. I would be glad to debug this.

EDIT: (we will try the Manual approach to simplifythings) can you please first copy Cesium config.xml next to the syncthing-map executable and run

syncthing-map add --device Cesium --file config.xml
synching graph

and check the resulting syncthing-map.html file? (and if the issue is still there - share data.json and syncthing-map.html)

Note: you can PM me the files if you prefer