Re: logging in RAM and journald configuration issue

[Umut Tezduyar Lindskog <umut@xxxxxxxxxxxx>]

Back then, this is something we have tried and it works. Not with bind
mount though. We also have SD-card as primary storage. As soon as the
storage service mounted the SD-Card, we have created a symbolink link
under /var/log to the persistent and triggered journald flush. It
worked just fine!

On Wed, Mar 20, 2019 at 4:54 PM Colin Guthrie <gmane@xxxxxxxxxxxxxx> wrote:
>
> Lennart Poettering wrote on 20/03/2019 13:37:
> > On Mi, 20.03.19 09:57, Dave Howorth (systemd@xxxxxxxxxxxxxx) wrote:
> >
> >>> On Di, 19.03.19 20:45, Dave Howorth (systemd@xxxxxxxxxxxxxx) wrote:
> >>>
> >>>> In the case of a machine that uses an SD card as its primary backing
> >>>> store, it is desirable to reduce the number of write operations to
> >>>> the card in order to prolong its life. journald is quite
> >>>> well-behaved in this regard since it offers the choice of a
> >>>> temporary or permanent journal and limits the frequency of its
> >>>> writes, except in emergencies.
> >>>>
> >>>> However, its permanent journal is written under /var/log/journal and
> >>>> there is no way to configure the path as far as I am aware. I think
> >>>> this is a problem.
> >>>>
> >>>> The reason is that on this type of machine people sometimes map
> >>>>  /var/log to RAM using tmpfs and then perhaps persist the logs
> >>>> using a program like log2ram. When this is done, journald's
> >>>> emergency writing capability is lost and crash analysis becomes
> >>>> more difficult.
> >>>>
> >>>> Of course it is possible instead to redirect the log files of
> >>>> programs individually to temporary memory using systemd-tmpfiles
> >>>> wherever needed. But this involves reconfiguring each and every
> >>>> program that uses /var/log both initially and whenever new programs
> >>>> are installed. This is tedious not only in quantity but because
> >>>> each program has a different detailed format of configuration file.
> >>>>
> >>>> So making /var/log into a tmpfs is a more attractive option. But
> >>>> ideally the journal would be placed somewhere else in persistent
> >>>> storage so its contents are available after a crash. This does not
> >>>> seem to be possible through lack of a config option.
> >>>>
> >>>> Is my analysis correct? Are there any other ways to resolve this
> >>>> difficulty? Otherwise, is it possible to consider a log location
> >>>> config option for journald?
> >>>
> >>> Right now, when persistent mode is enabled journald will store its log
> >>> data in /var/log. When it is disabled it will store things in /run/log
> >>> instead.
> >>>
> >>> It has been requested that we add a hybrid mode that makes journald
> >>> log to both locations at the same time, but filter by log priority so
> >>> that log msgs higher than some priority go to one location and the
> >>> ones below it go to the other. A patch like that would probably be
> >>> relatively straight-forward and short. Would be happy to review/merge
> >>> a patch for that.
> >>>
> >>> I think if that's implemented the log location should really stay
> >>> unmodified: /var should be persistant and /run not, and there would be
> >>> no need to remount any of those paths in a different way.
> >>
> >> Many thanks for the quick reply. I'm not clear how that resolves the
> >> situation I explained, since it neither provides an alternative
> >> persistent log location nor provides an alternative means of
> >> arranging the logs of other programs. It doesn't seem to address my
> >> issue at all?
> >
> > I don't understand the problem then?
> >
> > The journal only collects logs on stdout/stderr, syslog and its native
> > protocol. Nothing else. It's not a tool that can collect arbitrary
> > per-app logs that don't go via these transport hence. And it really
> > shouldn't be.
> >
> > Also: /var is generally understood to be persistent, and /run to be
> > volatile. If you want to redefine that you are welcome to, but of
> > course you have to patch through all kinds of software then.
>
> As I understand it, you want /var/log to be tmpfs but /var/log/journal
> to be persistent (as journald's write behaviour is considerate enough
> for you not to worry about backing store fatigue etc)?
>
> Just as a less complex suggestion, you're presumably bootstrapping
> /var/log to be on tmpfs at some point in early boot (i.e. before
> journald is started or at least before it's being told to start writing
> to /var/log/journal). If this is the case, then why don't you just amend
> that bootstrapping to bind mount /var/log/journal to persistent storage
> rather than letting it be just a subdir on the tmpfs? That way you can
> get your mem-only syslog (or native logging) setup as you do now and get
> persistent journal logs with only very minor changes (this could all be
> done with appropriate systemd units and deps AFAICT).
>
> I can't remember off the top of my head what makes journald start
> writing to /var/log/journal, so you may have to change that slightly
> (e.g. it might notice is as soon as you mkdir it in the tmpfs but before
> you do the bind mount), but I'm sure this would be simple enough.
>
> Just a thought.
>
> Col
>
> --
>
> Colin Guthrie
> gmane(at)colin.guthr.ie
> http://colin.guthr.ie/
>
> Day Job:
>   Tribalogic Limited http://www.tribalogic.net/
> Open Source:
>   Mageia Contributor http://www.mageia.org/
>   PulseAudio Hacker http://www.pulseaudio.org/
>   Trac Hacker http://trac.edgewall.org/
> _______________________________________________
> systemd-devel mailing list
> systemd-devel@xxxxxxxxxxxxxxxxxxxxx
> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
_______________________________________________
systemd-devel mailing list
systemd-devel@xxxxxxxxxxxxxxxxxxxxx
https://lists.freedesktop.org/mailman/listinfo/systemd-devel