Hmm, changes to the service side (for prefix) would be a bit difficult at the moment. And moreover the journal is a bit luxurious where we get the timestamp, hostname, process name and it's PID. So I would like to have that luxury in the logs.
I have a thought raised from your suggestion. Is it fine to have an extra service to each service (for that 40 services) where the new service runs "journalctl -f -u <main_service>" and lets its stdout be connected to the respective log file?. This way we can take journal logs and put it in the respective log file. But a basic question is, would running 40-50 instances of journalctl impact the load?
I will perform an experiment on that as well.
--
Arjun
On Tue, Sep 28, 2021 at 11:32 AM Mantas Mikulėnas <grawity@xxxxxxxxx> wrote:
On Tue, Sep 28, 2021, 06:07 Arjun D R <drarjun95@xxxxxxxxx> wrote:Thank you Mantas for the details.How do you currently get the logs "every few seconds"?> Actually we have a script that will be triggered every 10 seconds. That script will run "journalctl -u <service>" and redirect the output to the respective log file. We will run journalctl for around 40-50 services for every 10 seconds and redirect it to the respective log files. That may be a bad idea, but this is how we are collecting logs as of now. We need to separate the logs for every service and that's why we ended up with this implementation.So replace it with 40-50 instances of continuously running `[stdbuf -o0] journalctl -f -u <service>`.Although syslogd might be easier, since then you'd only need one process doing the monitoring. (I know both syslogds can access journal fields like _SYSTEMD_UNIT, but I don't know how, so usually I just filter by the traditional "program name" and it does the job.)You could also have a custom daemon that reads from `journalctl -f -o json` and writes to the appropriate text log...Ah, ok so StandardOutput:file:<log_file> will allow the service to open the fd and directly connect it to the service stdout.Yes, so if you want timestamps they have to be provided by your service, pretty much like how most services implement direct logging to files. (Probably the only advantage of StandardOutput is that the service doesn't need permissions to /var/log...)
