gzip audit-logs

opensource21 · June 13, 2021, 4:46pm

I’m running syncthing with --audit. I would appreciate if old auditlogs would be gzipped and every day a new file will be used.

AudriusButkevicius · June 13, 2021, 6:14pm

Best I can Suggest is to set it up to log to stdout/stderr, and use standard rotation, or some log management systems, support setting up a pipe/named socket for an application to write to, which then then do rotation etc.

We have ways to do log rotation, which could be reused here, but I don’t think we’ll do gzipping etc.

opensource21 · June 15, 2021, 1:25pm

Sorry, but I don’t understand your suggestions. The audit-log always wrote to a file as far as I know. Furtermore the name is unpredictable (contains the start time). So tools like newsyslog and logrotate doesn’t work. I only see the possibilities that the logik is implemented in syncthing or that syncthing always write to a file audit.log. Then logrotate can do the job.

bt90 · June 15, 2021, 1:30pm

Logrotate allows wildcards AFAIK.

Alex · June 15, 2021, 1:31pm

you can specify the file it writes to:

--auditfile=PATH           Specify audit file (use "-" for stdout, "--" for stderr)

AudriusButkevicius · June 15, 2021, 1:45pm

For log rotate to work, we’d need to periodically reopen the file, which we don’t do.

I know some logging daemons expose a named pipe you can write to, which then does log rotation on stuff read from the pipe.

opensource21 · June 15, 2021, 5:38pm

Does this mean if I set the audit-file to audit.log and then newlogsystem bzip to autit.0.log. The audit-log will become broken?

What logging deamons you have in mind?

AudriusButkevicius · June 15, 2021, 5:46pm

I believe so, because we will keep writing to a file descriptor that doesn’t exist on the filesystem anymore, because you’ve archived the file and deleted it.

I can’t tell you a specific one, I just recall seeing something like that before.

Effectively what I am talking about is something like example 2 shown here:

You point syncthing to write audit logs to the named pipe (which I think should work), wand then your script that reads off of that named pipe decides what to do with the logs, when to roll a new file, archive it etc.

bt90 · June 15, 2021, 5:48pm

Would copytruncate not work?

AudriusButkevicius · June 15, 2021, 5:50pm

Good question/point, maybe it would.

opensource21 · June 15, 2021, 6:19pm

that seems in deed an option. So I can write in a fix file. Run a job, that copy it to another file and truncate the original one. The new file can then rotate with newsyslog which is the default at mac. Must think about it. Perhaps it’s easier to stop syncthing. gzip the log and restart it.

Thanks for all the input. In java this is normally only a thing of a few configurations.

opensource21 · June 15, 2021, 7:25pm

Interesting note: There seems some code in sycnthing which deletes old audit-logs

calmh · June 16, 2021, 4:47am

Yep:

github.com

syncthing/syncthing/blob/89740490acc61e052447698c553fd8239c4f3313/cmd/syncthing/main.go#L936-L948


	patterns := map[string]time.Duration{
		"panic-*.log":        7 * 24 * time.Hour,  // keep panic logs for a week
		"audit-*.log":        7 * 24 * time.Hour,  // keep audit logs for a week
		"index":              14 * 24 * time.Hour, // keep old index format for two weeks
		"index-v0.11.0.db":   14 * 24 * time.Hour, // keep old index format for two weeks
		"index-v0.13.0.db":   14 * 24 * time.Hour, // keep old index format for two weeks
		"index*.converted":   14 * 24 * time.Hour, // keep old converted indexes for two weeks
		"config.xml.v*":      30 * 24 * time.Hour, // old config versions for a month
		"*.idx.gz":           30 * 24 * time.Hour, // these should for sure no longer exist
		"backup-of-v0.8":     30 * 24 * time.Hour, // these neither
		"tmp-index-sorter.*": time.Minute,         // these should never exist on startup
		"support-bundle-*":   30 * 24 * time.Hour, // keep old support bundle zip or folder for a month
	}

(on startup)

opensource21 · July 10, 2021, 7:46pm

Is it possible to stop the syncthing-process at the mac-os-version from commandline? The idea is to regulary stop syncthing gzip the audit-log and then restart the service. I tried to do it with the REST-Api and curl, but the mac-os-version restart it immediatly. One idea is to gzip everything and then shutdown. But it’s not really clean.

calmh · July 10, 2021, 8:01pm

Probably the service manager restarts Syncthing as soon as it exits. This is fine though; restart Syncthing so it creates a new audit log, then gzip the old audit log.