Dominique Martinet wrote on Sun, Feb 18, 2024 at 06:43:25PM +0900: > Anyway, my main question stays there same: I couldn't find any > diagnostic utility; I don't care if I just get a garbled mess but is > there a way to try to decompress what we can or whatever is needed to > get infos out of the corrupted files? Or is journalctl already doing as > much as is possible? > I've checked the files with xxd and they're not empty, I can't tell if > it's random garbage or if there's anything useful in here but would be > happy to send a couple of files to a few individuals (probably nothing > too sensible in there, but would rather not make it public as it > might contain some mail addresses or whatsnot) Hmpf. So I had copied /var/log/journal to preserve it just in case, but didn't try reading from it yesterday... Now I've tried `journalctl -D copied_dir` it's finding most of the old logs just fine up till the second the system became unresponsive, so I was able to get as much logs as I could hope for here. That just leaves a real bug of why it doesn't work directly in /var/log/journal -- I expect it'll take a few months for the bad file to rotate out if it ever does, so I guess I'll open a github issue with the same details I sent yesterday here when I have a moment... Might take a week or so, so feel free to request extra details or anything here directly if someone has an idea. For anyone who cares: running `journalctl -D . --verify` in that directory properly checks all the old files; there is also one file that is ignored from the start and one file that is corrupted upon verification but it's fairly different from the output I sent yesterday. I've attached debug output for completeness. Cheers, -- Dominique Martinet | Asmadeus
Considering root directory '.'. Root directory . added. Failed to open journal file ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b531a7b-00060f1593b6379e.journal: Bad message Failed to open journal file ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b53b7f5-00060f1fcebd0c6a.journal: Bad message Failed to open journal file ./user-1001@a10c5e2030f04d2d89a350feb75bc384-000000000b53d849-00060f2225f1f1e2.journal: Bad message Failed to open journal file ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b5448cf-00060f287b18ffe2.journal: Bad message Failed to open journal file ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b5479f9-00060f2b1a9334bf.journal: Bad message Failed to open journal file ./user-1001@a10c5e2030f04d2d89a350feb75bc384-000000000b54fd22-00060f3643c94623.journal: Bad message Failed to open journal file ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b55836c-00060f3fe3e6075d.journal: Bad message Failed to open journal file ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b55b85a-00060f43a75273db.journal: Bad message Failed to open journal file ./user-1001@a10c5e2030f04d2d89a350feb75bc384-000000000b55f6f0-00060f4a61a0aa02.journal: Bad message Failed to open journal file ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b56a9db-00060f53615ecbf6.journal: Bad message Failed to open journal file ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b577b97-00060f5ec56ad975.journal: Bad message Failed to open journal file ./user-1001@a10c5e2030f04d2d89a350feb75bc384-000000000b577852-00060f5e7f780a22.journal: Bad message Failed to open journal file ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b57ccc5-00060f61ca701cb7.journal: Bad message Failed to open journal file ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b585153-00060f680848ec19.journal: Bad message Failed to open journal file ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b58eee8-00060f7035dc3fa9.journal: Bad message Failed to open journal file ./user-1001@a10c5e2030f04d2d89a350feb75bc384-000000000b5913cd-00060f729d4f6a0f.journal: Bad message Failed to open journal file ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b5b2a72-00060f8f6dedd3f2.journal: Bad message Failed to open journal file ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b5b4abe-00060f91a0f8c147.journal: Bad message Failed to open journal file ./user-1001@a10c5e2030f04d2d89a350feb75bc384-000000000b5bd4bd-00060f9ad8fe2611.journal: Bad message Failed to open journal file ./user-1001@a10c5e2030f04d2d89a350feb75bc384-000000000b5db5d3-00060fc314acea02.journal: Bad message Journal effective settings seal=no keyed_hash=yes compress=ZSTD compress_threshold_bytes=8B File ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b61040f-00060ffd48fc1634.journal added. File ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b6105f0-00060ffd8f292ab0.journal added. File ./user-1001@a10c5e2030f04d2d89a350feb75bc384-000000000b61152b-00060fff6e333140.journal added. File ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b623145-00061011673a117b.journal added. File ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b633362-0006101f6ecf8901.journal added. File ./user-1001@a10c5e2030f04d2d89a350feb75bc384-000000000b6261e0-000610138c0a75d7.journal added. File ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b6356ec-0006102139eaa8b1.journal added. File ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b638a61-00061025af3cc686.journal added. File ./user-1001@a10c5e2030f04d2d89a350feb75bc384-000000000b63b327-00061027a9e1c9e9.journal added. File ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b64955b-00061039cfde0c35.journal added. File ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b64bc97-0006103d13d24252.journal added. File ./user-1001@a10c5e2030f04d2d89a350feb75bc384-000000000b64aec0-0006103bc7b92dd2.journal added. File ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b65c2c9-00061052ac2ab208.journal added. File ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b665ecc-0006105f99c33ba5.journal added. File ./user-1001@a10c5e2030f04d2d89a350feb75bc384-000000000b66966f-000610640367f5d9.journal added. File ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b6738e8-00061071214c5917.journal added. File ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b6949a3-0006109577b57b98.journal added. File ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b69545d-0006109640344716.journal added. File ./user-1001@a10c5e2030f04d2d89a350feb75bc384-000000000b69e309-000610a05cee0581.journal added. File ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b6ba9c6-000610bc5c617240.journal added. File ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b6e98b4-000610e8f9634f21.journal added. File ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b72d9c4-0006112a9233ddca.journal added. File ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b777769-00061170987ab1e9.journal added. File ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b789d7f-0006117d68c87a58.journal added. File ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b79546c-000611871627159f.journal added. File ./system@000611a3547cb7dc-30ac75b8bb548880.journal~ added. File ./system@000611a38b8b4850-ea9184a7bb465911.journal~ added. File ./system@564158f161f142e1a9acea4b9e126e5e-000000000b7be30e-000611a354771f2f.journal added. File ./user-1000@000611a38c9f0a95-15a0ef6c330d1d0d.journal~ added. File ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b7bd2dc-000611a29a8c1866.journal added. File ./system.journal added. File ./user-1000.journal added. File ./user-1001.journal added. Journal file ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b531a7b-00060f1593b6379e.journal corrupted, ignoring file. PASS: ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b61040f-00060ffd48fc1634.journal PASS: ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b6105f0-00060ffd8f292ab0.journal PASS: ./user-1001@a10c5e2030f04d2d89a350feb75bc384-000000000b61152b-00060fff6e333140.journal PASS: ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b623145-00061011673a117b.journal PASS: ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b633362-0006101f6ecf8901.journal PASS: ./user-1001@a10c5e2030f04d2d89a350feb75bc384-000000000b6261e0-000610138c0a75d7.journal PASS: ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b6356ec-0006102139eaa8b1.journal PASS: ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b638a61-00061025af3cc686.journal PASS: ./user-1001@a10c5e2030f04d2d89a350feb75bc384-000000000b63b327-00061027a9e1c9e9.journal PASS: ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b64955b-00061039cfde0c35.journal PASS: ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b64bc97-0006103d13d24252.journal PASS: ./user-1001@a10c5e2030f04d2d89a350feb75bc384-000000000b64aec0-0006103bc7b92dd2.journal PASS: ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b65c2c9-00061052ac2ab208.journal PASS: ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b665ecc-0006105f99c33ba5.journal PASS: ./user-1001@a10c5e2030f04d2d89a350feb75bc384-000000000b66966f-000610640367f5d9.journal PASS: ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b6738e8-00061071214c5917.journal PASS: ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b6949a3-0006109577b57b98.journal PASS: ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b69545d-0006109640344716.journal PASS: ./user-1001@a10c5e2030f04d2d89a350feb75bc384-000000000b69e309-000610a05cee0581.journal PASS: ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b6ba9c6-000610bc5c617240.journal PASS: ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b6e98b4-000610e8f9634f21.journal PASS: ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b72d9c4-0006112a9233ddca.journal PASS: ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b777769-00061170987ab1e9.journal PASS: ./system@a10c5e2030f04d2d89a350feb75bc384-000000000b789d7f-0006117d68c87a58.journal PASS: ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b79546c-000611871627159f.journal PASS: ./system@000611a3547cb7dc-30ac75b8bb548880.journal~ PASS: ./system@000611a38b8b4850-ea9184a7bb465911.journal~ PASS: ./system@564158f161f142e1a9acea4b9e126e5e-000000000b7be30e-000611a354771f2f.journal PASS: ./user-1000@000611a38c9f0a95-15a0ef6c330d1d0d.journal~ 0015f0: Invalid data hash table hash table item (102560/233016) head_hash_offset: c25c9d8e09f953fc 0015f0: Invalid object contents: Bad message File corruption detected at ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b7bd2dc-000611a29a8c1866.journal:5616 (of 8388608 bytes, 0%). FAIL: ./user-1000@a10c5e2030f04d2d89a350feb75bc384-000000000b7bd2dc-000611a29a8c1866.journal (Bad message) PASS: ./system.journal PASS: ./user-1000.journal PASS: ./user-1001.journal Root directory . removed. mmap cache statistics: 1005303406 category cache hit, 44953079 window list hit, 283 miss
