Shell script to clean (redact) log files before posting

Howto
21

Thanks!

You’re right,

I used your script on my Mac where I copied the log.txt but the config.xml on the Mac does not have that device mentioned… So my bad, sorry.

Maybe a short message at the end of the script’s execution should warn users like me, that not all was sanitized?.. :-}

22

Hmm, the thing is that it knows “what should be sanitized” based on the config…

I can add a warning for the un-cleared long ID formats (i.e. IDIDID-IDIDID-IDIDID-IDIDID…), since it has a distinct pattern, but I can’t do anything about other such stuff, like folder/device names, partial IDs, etc. since there’s no (known to me) way to accurately detect them without knowing them in advance first.

23

I see, thanks for explaining.

Well, its better than nothing!

I have another/unrelated question:

Would you mind to add an additional “modus” to your script to just redact the given config.xml ? (Or maybe a question to the user, if the config.xml should also be redacted)

Pre-posting my config, I need to search/replace it manually like:

<configuration version="27">
<folder id="my private folder id" label="my private label" path="my private path" type="readonly" rescanIntervalS="10800" fsWatcherEnabled="true" fsWatcherDelayS="60" ignorePerms="true" autoNormalize="true">

to

    <configuration version="27">
    <folder id="F1" label="L1" path="P1" ...

for the first folder.

For the second folder, I do

<folder id="F2" label="L2" path="P2" ...

etc.

For the Devices, I do similar seearch/replace:

<folder id="F1" label="L1" path="P1" type="readonly" rescanIntervalS="10800" fsWatcherEnabled="true" fsWatcherDelayS="60" ignorePerms="true" autoNormalize="true">
        <filesystemType>basic</filesystemType>
        <device id="DDDDDD-DDDDDD-DDDDDD-DDDDDD" introducedBy=""></device>

to

<folder id="F1" label="L1" path="P1" type="readonly" ...>
<filesystemType>basic</filesystemType>
<device id="D1" introducedBy=""></device>

Your script could help here to speed up the process and generate a ready2go config_redacted.xml

It might be a simple check first on how many folders/devices are present in the config and then replaceing the strings for each device/folder.

That would be a great enhancement.

Example: https://pastebin.com/zsKPFVgj (but I was too lazy with the paths…)

24

Oh, another thing.

Reading the redacted logs, I think, they’re not very well readable, because of the looong Device IDID-IDID-IDID-IDID_n’s.

Would it be possible to define just a “ID_n” instead of IDID-IDID-IDID-IDID_n" ?

And the is a small ‘l’ after the folder label. Is this correct?

2018-04-11 13:07:23 Puller (folder "FOLDER_LABEL_1l" (FOLDER_ID_1), file "###"): delete dir: directory is not empty
2018-04-11 13:07:26 Folder "FOLDER_LABEL_1l" (FOLDER_ID_1) isn't making progress. Pausing puller for 16m0s.
2018-04-11 13:23:28 Puller (folder "FOLDER_LABEL_1l" (FOLDER_ID_1), file "###"): delete dir: directory is not empty
2018-04-11 13:23:32 Folder "FOLDER_LABEL_1l" (FOLDER_ID_1) isn't making progress. Pausing puller for 32m0s.
2018-04-11 13:23:35 File "foo\información.png" path has UTF8 encoding conflict with another file; ignoring.
2018-04-11 13:39:07 Established secure connection to IDID-IDID-IDID-IDID_1 at IPv4_1:PORT_0-IPv4_0:PORT_4/FOLDER_ID_9p-server (TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305)
2018-04-11 13:39:07 Device IDID-IDID-IDID-IDID_1 client is "syncthing v0.14.46" named "DEVICE_1" at IPv4_1:PORT_0-IPv4_0:PORT_4/FOLDER_ID_9p-server
2018-04-11 13:39:18 Puller (folder "FOLDER_LABEL_1l" (FOLDER_ID_1), file "###"): delete dir: directory is not empty
2018-04-11 13:39:24 Folder "FOLDER_LABEL_1l" (FOLDER_ID_1) isn't making progress. Pausing puller for 1h4m0s.
2018-04-11 13:41:46 Connection to IDID-IDID-IDID-IDID_1 at IPv4_1:PORT_0-IPv4_0:PORT_4/FOLDER_ID_9p-server closed: writing message: write FOLDER_ID_9p IPv4_1:PORT_0->IPv4_0:PORT_4: wsasend: Eine vorhandene Verbindung wurde vom Remotehost geschlossen.
2018-04-11 13:45:52 Established secure connection to IDID-IDID-IDID-IDID_1 at IPv4_1:PORT_7-IPv4_0:PORT_0/FOLDER_ID_9p-client (TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305)
2018-04-11 13:45:52 Device IDID-IDID-IDID-IDID_1 client is "syncthing v0.14.46" named "DEVICE_1" at IPv4_1:PORT_7-IPv4_0:PORT_0/FOLDER_ID_9p-client
2018-04-11 13:45:52 Connection to IDID-IDID-IDID-IDID_1 at IPv4_1:PORT_7-IPv4_0:PORT_0/FOLDER_ID_9p-client closed: reading length: read FOLDER_ID_9p IPv4_1:PORT_7->IPv4_0:PORT_0: wsarecv: Eine vorhandene Verbindung wurde vom Remotehost geschlossen.
2018-04-11 13:46:52 Established secure connection to IDID-IDID-IDID-IDID_1 at IPv4_1:PORT_8-IPv4_0:PORT_0/FOLDER_ID_9p-client (TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305)
2018-04-11 13:46:52 Device IDID-IDID-IDID-IDID_1 client is "syncthing v0.14.46" named "DEVICE_1" at IPv4_1:PORT_8-IPv4_0:PORT_0/FOLDER_ID_9p-client

better readable, IMHO

2018-04-11 13:07:23 Puller (folder "FL_1l" (FID_1), file "###"): delete dir: directory is not empty
2018-04-11 13:07:26 Folder "FL_1l" (FID_1) isn't making progress. Pausing puller for 16m0s.
2018-04-11 13:23:28 Puller (folder "FL_1l" (FID_1), file "###"): delete dir: directory is not empty
2018-04-11 13:23:32 Folder "FL_1l" (FID_1) isn't making progress. Pausing puller for 32m0s.
2018-04-11 13:23:35 File "foo\información.png" path has UTF8 encoding conflict with another file; ignoring.
2018-04-11 13:39:07 Established secure connection to ID_1 at IPv4_1:PORT_0-IPv4_0:PORT_4/FID_9p-server (TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305)
2018-04-11 13:39:07 Device ID_1 client is "syncthing v0.14.46" named "DEVICE_1" at IPv4_1:PORT_0-IPv4_0:PORT_4/FID_9p-server
2018-04-11 13:39:18 Puller (folder "FL_1l" (FID_1), file "###"): delete dir: directory is not empty
2018-04-11 13:39:24 Folder "FL_1l" (FID_1) isn't making progress. Pausing puller for 1h4m0s.
2018-04-11 13:41:46 Connection to ID_1 at IPv4_1:PORT_0-IPv4_0:PORT_4/FID_9p-server closed: writing message: write FID_9p IPv4_1:PORT_0->IPv4_0:PORT_4: wsasend: Eine vorhandene Verbindung wurde vom Remotehost geschlossen.
2018-04-11 13:45:52 Established secure connection to ID_1 at IPv4_1:PORT_7-IPv4_0:PORT_0/FID_9p-client (TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305)
2018-04-11 13:45:52 Device ID_1 client is "syncthing v0.14.46" named "DEVICE_1" at IPv4_1:PORT_7-IPv4_0:PORT_0/FID_9p-client
2018-04-11 13:45:52 Connection to ID_1 at IPv4_1:PORT_7-IPv4_0:PORT_0/FID_9p-client closed: reading length: read FID_9p IPv4_1:PORT_7->IPv4_0:PORT_0: wsarecv: Eine vorhandene Verbindung wurde vom Remotehost geschlossen.
2018-04-11 13:46:52 Established secure connection to ID_1 at IPv4_1:PORT_8-IPv4_0:PORT_0/FID_9p-client (TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305)
2018-04-11 13:46:52 Device ID_1 client is "syncthing v0.14.46" named "DEVICE_1" at IPv4_1:PORT_8-IPv4_0:PORT_0/FID_9p-client
25

Redacting xml config is an easy thing to add. I can pass it through the same procedure as the log files.

As for redact format - i.e. FOLDER_ID_# vs FID_#, I actually prefer the longer one, as FID isn’t as obvious to what it stands for in my opinion.

The IDID-IDID-IDID-IDID can be improved however.

Side note - since there’s some “action” with this tiny tool, I put it on GitHub to track issues/requests, etc. You can chime in there if you like:

1 Like