Add feature to enable 'persistent' kernel logging

[mwbringmann]

Some distros or configurations do not enable/disable persistent kernel event logging
i.e. /var/log/messages content showing events of prior boot cycles. In the event that
the system crashes, the messages file prior to the crash is thus lost. It would be useful
to add an option to reset the kernel configuration to enable persistent event logging, so
that the maximum amount of information about crashes is retained.

[sourabhjains]

This check should be part of a new kernel logging plugin.

Comment 2 in #5

• Sourabh Jain

[lucianochavez]

To be more specific, we keep running into issues in SLES 15 where journal log entries do not persist for previous boots, only the current boot. This is not helpful for when the user runs into a problem but reboots and then attempts to save the log.
In order to get persistent logging, there seem to be a couple of ways. According to https://access.redhat.com/solutions/696893 for RHEL, you can do
# mkdir -p /var/log/journal
if Storage=Auto is set in /etc/systemd/journald.conf.
With SLES, they say to change Storage=Auto in /etc/systemd/journald.conf to Storage=persistent and then restart the journald service with
systemctl restart systemd-journald

[sourabhjains]

Thanks @lucianochavez for the detailed explanation. I think it's better to have a separate plugin to handle journald configuration and it should be a mandatory plugin and always run by default.

A quick note on the mandatory/optional plugins.
It's an upcoming feature in ServcieReport where workload-specific plugins, for example, HTX will be marked as optional. The optional plugins will not part of the default run and will require additional arguments to enable them.

[ananthmg]

https://documentation.suse.com/sles/15-SP1/html/SLES-all/cha-journalctl.html#sec-journalctl-persistent is the corresponding SLES documentation.

[sourabhjains]

Thanks @ananthmg for sharing the document.

I did bit of experiments and found that we don't need two different approaches for SLES and RHEL to enable persistent journald logs.
Changing the Storage=persistent and rerunning the services is good enough to enable persistent journald logs.

• Sourabh Jain

[sourabhjains]

Hello @lucianochavez @mwbringmann

Do you use SystemMaxFileSize or SystemMaxFiles journald config options to limit the journald disk consumption?

If yes then please let me know your preferred values.

• Sourabh Jain

[mwbringmann]

No, I have not used these options before. Usually, I have bypassed the configuration issue and grabbed the full log files, directly.

[sourabhjains]

This feature was dependent on the optional plugin feature in ServiceReport.

Will upstream the journalctl plugin to have persistent logging soon.

[lucianochavez]

Hi @sourabhjains

Any outlook on this enhancement? We continue to run across systems with the default in situations where the system was rebooted (due to a crash or hang or reboot test) and thus have lost error log entries from the prior boot.