Antw: [EXT] Re: Journald: Re-use /var/log/journal hash entry after system upgrade

["Ulrich Windl" <Ulrich.Windl@xxxxxxxxxxxxxxxxxxxx>]

>>> Mantas Mikulenas <grawity@xxxxxxxxx> schrieb am 01.03.2022 um 16:52 in
Nachricht
<CAPWNY8XL5FtVBwrFGg80oTc7=pe-BshwD==NSV-sdiDX4gt1GQ@xxxxxxxxxxxxxx>:
> On Tue, Mar 1, 2022 at 4:39 PM Eric.Zaluzec@xxxxxxxxxx <
> Eric.Zaluzec@xxxxxxxxxx> wrote:
> 
>> #### [ System Environment ] ####
>> On an embedded x86-64 Linux system, I’m running systemd v241. I have
>> Systemd-Journald logging set to persistent with SystemMaxUse and
>> RuntimeMaxUse both set to 512MB.
>>
>> The Linux system mount loops /var/log on start-up from a var-log.ext4
>> file. The /var/log mount is given a fixed size of the disk (976MB).
Systemd
>> creates a journal entry directory given a hash name in
>> /var/log/journal/<hash1>.
>>
>>
>>
>> #### [ System Information ] ####
>>
>> Linux 4.19.0-6-2-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64
>> GNU/Linux
>>
>>
>> #### [ Problem ] ####
>>
>> When the system firmware is upgraded, Systemd creates a new journal entry
>> directory with a different hash name and no longer recognizes the previous
>> hash directory.
>>
>> The old logs from the previous journal entry can no longer be managed. The
>> old logs are never rotated and cannot be manually rotated using the
>> journaldctl cli. The disk usage calculator by Journald does not account
for
>> the previous journal entry meaning if there are two previous entries in
>> /var/log that consume 900MB of space; then the new journal entry only has
>> 76MB of space to work with. Eventually, disk space will be full. Journald
>> cannot automatically flush, vacuum, clean, rotate previous logs because it
>> does not recognize the previous journal entries.
>>
>> There must be a systemd journald check that occurs where it determines
>> these other entries are not for it to manage.
>>
> 
> This is not a 'hash' – it's the machine ID
> <https://www.freedesktop.org/software/systemd/man/machine-id.html>, which
> is directly read from /etc/machine-id. Normally the machine ID is randomly
> generated during *first *boot (it's just a random UUID), and it is supposed
> to be persistent afterwards.
> 
> It sounds like your /etc doesn't have that file and is on a read-only
> rootfs, so systemd generates a new machine-id in tmpfs every boot. The

Dumb question: Is that fact logged in the journal?

> device probably has *some* persistent storage though, so try to find a way
> to make /etc/machine-id persist as well (see the machine-id(5)
> <https://www.freedesktop.org/software/systemd/man/machine-id.html> manual
> page for a few possibilities).
> 
> The intent of using machine-id subdirectories is to allow containers'
> journal directories to be symlinked on the host, or remote system journals
> to be collected on a single system – there's actually the journalctl -m
> option to make it look at "foreign" journals, so e.g. journalctl -m -f
> could be used to watch logs of all containers.

Still I think some funcion like "remove-orphans" may seem useful (just to
prevent the user to make things worse by removing the wrong files)

> 
> (The machine ID is also used as the base for systemd-networkd's DHCPv4
> Client ID, DHCPv6 DUID, and IPv6 address generation. But other than that,
> though, I can't think of any its uses that would be visible externally –
> aside from desktop-specific things like pulseaudio – so in some cases it
> might be "good enough" to pre-define a fixed ID in the template image as a
> last resort.)


Regards,
Ulrich

> 
> -- 
> Mantas Mikulėnas