Hi, This has bitten us a couple of times now (such that we're considering re-building util-linux with the nilfs2 code commented out), so I'm wondering if anyone else has seen it [and noting the failure mode in case anyone else is confused in future] We see this with our setup of rotating media for the osd, NVMe partition for journal. What happens is that sometimes an osd refuses to start up, complaining that /var/lib/ceph/osd/ceph-XXX/journal is missing. inspecting that file will show it's a broken symlink to an entry in /dev/disk/by-partuuid: /var/lib/ceph/osd/ceph-388/journal: broken symbolic link to /dev/disk/by-partuuid/d2ace848-7e2d-4395-a195-a4428631b333 If you inspect the relevant partition, you see that it has the matching block id: blkid /dev/nvme0n1p11 /dev/nvme0n1p11: PARTLABEL="ceph journal" PARTUUID="d2ace848-7e2d-4395-a195-a4428631b333 And, if you look in syslog, you'll see this: Jan 4 09:25:29 sto-3-3 systemd-udevd[107317]: incorrect nilfs2 checksum on /dev/nvme0n1p11 The problem is that the nilfs2 checker is too promiscuous, looking for a relatively short magic number (0x3434) in 2 different places (location 0x400, and (((part_size-512)/8)*512)). So sometimes you'll be unlucky and have a ceph journal that matches, at which point the nilfs2 prober find an invalid checksum, and so systemd/udevd doesn't create the /dev/disk/by-partuuid link. You can work around this by making the symlink by hand when the failure occurs; I also understand that the nilfs2 prober in util_linux 2.29 is more robust (but that's not in any LTS distributions yet, so I've not tested it). Regards, Matthew util-linux issue: https://github.com/karelzak/util-linux/issues/361 Ubuntu bug: https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/1653936 -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |