/var/log/journal full, journald is not removing journal files

[lennart@xxxxxxxxxxxxxx (Lennart Poettering)]

On Mi, 11.07.18 12:54, Chris Murphy (lists at colorremedies.com) wrote:

> systemd-238-8.git0e0aa59.fc28.x86_64
> 
> I'm really confused by what I'm seeing.
> 
> Jul 10 09:13:40 f28h.local systemd-journald[493]: System journal
> (/var/log/journal/bbe68372db9f4c589a1f67f008e70864) is 1.2G, max 1.3G,
> 90.0M free.
> [chris at f28h ~]$ du -sh /var/log/journal/bbe68372db9f4c589a1f67f008e70864/
> 1.6G    /var/log/journal/bbe68372db9f4c589a1f67f008e70864/
> [chris at f28h ~]$
> 
> [chris at f28h ~]$ df -h
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/nvme0n1p7  1.9G  1.9G   73M  97% /var/log
> 
> journald.conf is Fedora default except
> SystemMaxUse=2G
> 
> 
> 1. Somehow systemd-journald is deciding to max at 1.3G, instead of the
> specified 2G, which is fine. But I don't know how it arrives at this.
> 2. Clearly max 1.3G is being busted, the contents of var/log/journal
> are greater than the max. non-journal files total ~83M and have not
> grown at all in the last week (and multiple reboots).
> 3. If I change SystemMaxUse=1300M there is no change. No attempt by
> journald to clean up /var/log/journal, and no errors, it uses /run/log
> instead and never switches to persistent logging.
> 4. My reading of man journald.conf is that that SystemKeepFree=
> defaults to 15% of the file system space, so even with SystemMaxUse=2G
> journald should have deleted journal files before getting to 100%
> full.
> 5. If I boot with systemd.log_level=debug, there are no journald
> entries that help understand why there's no transition from volatile
> to persistent storage, i.e. hey var/log/journal is full, and also that
> I can't delete files because $reasons, or whatever.
> 
> Extra info: this is a 2G f2fs file system mounted at /var/log. Seems
> to be working well except for this little problem but I don't see it
> being the cause. journald isn't even attempting to delete its own
> journal files to free up space.

Any chance you can reproduce this on a less exotic file system?

Note that there is "journactl --vacuum-xyz=" these days, which let you
know what it's doing (in particular with debug logging turned on, by
setting the SYSTEMD_LOG_LEVEL=debug env var).

Lennart

-- 
Lennart Poettering, Red Hat