I have a small 4-host ceph cluster. Recently, after rebooting one of the hosts, all of its daemons come back up smoothly EXCEPT the OSDs. All of the OSDs have identical journal entries, as below. 'ceph-bluestore-tool fsck' fails with: 2022-05-25T17:27:02.208+0000 7f45150a70c0 -1 bluefs _check_new_allocations invalid extent 1: 0x1a720000~10000: wasn't given but allocated for ino 1 2022-05-25T17:27:02.208+0000 7f45150a70c0 -1 bluefs mount failed to replay log: (14) Bad address 2022-05-25T17:27:02.208+0000 7f45150a70c0 -1 bluestore(/var/lib/ceph/d7511c3e-a570-11ec-9c50-3bd4b57e7e6e/osd.0) _open_bluefs failed bluefs mount: (14) Bad address fsck failed: (14) Bad address Any help/ thoughts would be very much appreciated. OSD logs below. Best, Andy Piltser-Cowan May 25 16:51:41 forth systemd[1]: Started Ceph osd.0 for d7511c3e-a570-11ec-9c50-3bd4b57e7e6e. May 25 16:51:44 forth bash[23554]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 May 25 16:51:44 forth bash[23554]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-979b08d5-747f-4103-a041-92ba46febdec/osd-block-42e57a20-3077-4208-a682-084b9dd1e634 --path /var/lib/ceph/osd/ce> May 25 16:51:44 forth bash[23554]: Running command: /usr/bin/ln -snf /dev/ceph-979b08d5-747f-4103-a041-92ba46febdec/osd-block-42e57a20-3077-4208-a682-084b9dd1e634 /var/lib/ceph/osd/ceph-0/block May 25 16:51:44 forth bash[23554]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block May 25 16:51:44 forth bash[23554]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 May 25 16:51:44 forth bash[23554]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 May 25 16:51:44 forth bash[23554]: --> ceph-volume lvm activate successful for osd ID: 0 May 25 16:51:48 forth bash[24332]: debug 2022-05-25T16:51:48.527+0000 7fee82abf080 0 set uid:gid to 167:167 (ceph:ceph) May 25 16:51:48 forth bash[24332]: debug 2022-05-25T16:51:48.527+0000 7fee82abf080 0 ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable), process ceph-osd, pid 7 May 25 16:51:48 forth bash[24332]: debug 2022-05-25T16:51:48.527+0000 7fee82abf080 0 pidfile_write: ignore empty --pid-file May 25 16:51:48 forth bash[24332]: debug 2022-05-25T16:51:48.531+0000 7fee82abf080 1 bdev(0x559b61752800 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block May 25 16:51:48 forth bash[24332]: debug 2022-05-25T16:51:48.531+0000 7fee82abf080 1 bdev(0x559b61752800 /var/lib/ceph/osd/ceph-0/block) open size 3000030461952 (0x2ba7fc00000, 2.7 TiB) block_size 4096 (4 KiB) rotational discard not sup> May 25 16:51:48 forth bash[24332]: debug 2022-05-25T16:51:48.535+0000 7fee82abf080 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 May 25 16:51:48 forth bash[24332]: debug 2022-05-25T16:51:48.535+0000 7fee82abf080 1 bdev(0x559b61752c00 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block May 25 16:51:48 forth bash[24332]: debug 2022-05-25T16:51:48.535+0000 7fee82abf080 1 bdev(0x559b61752c00 /var/lib/ceph/osd/ceph-0/block) open size 3000030461952 (0x2ba7fc00000, 2.7 TiB) block_size 4096 (4 KiB) rotational discard not sup> May 25 16:51:48 forth bash[24332]: debug 2022-05-25T16:51:48.535+0000 7fee82abf080 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 2.7 TiB May 25 16:51:48 forth bash[24332]: debug 2022-05-25T16:51:48.535+0000 7fee82abf080 1 bdev(0x559b61752c00 /var/lib/ceph/osd/ceph-0/block) close May 25 16:51:48 forth bash[24332]: debug 2022-05-25T16:51:48.827+0000 7fee82abf080 1 bdev(0x559b61752800 /var/lib/ceph/osd/ceph-0/block) close May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.111+0000 7fee82abf080 0 starting osd.0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.115+0000 7fee82abf080 -1 Falling back to public interface May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 0 load: jerasure load: lrc load: isa May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 1 bdev(0x559b62422400 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 -1 bdev(0x559b62422400 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 1 bdev(0x559b62422400 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 -1 bdev(0x559b62422400 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 0 osd.0:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 1 bdev(0x559b62422400 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 -1 bdev(0x559b62422400 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 0 osd.0:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 1 bdev(0x559b62422400 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 -1 bdev(0x559b62422400 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 0 osd.0:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 1 bdev(0x559b62422400 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 -1 bdev(0x559b62422400 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 0 osd.0:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 1 bdev(0x559b62422400 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 -1 bdev(0x559b62422400 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 0 osd.0:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (13) Permission denied May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 1 bluestore(/var/lib/ceph/osd/ceph-0) _mount path /var/lib/ceph/osd/ceph-0 May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 0 bluestore(/var/lib/ceph/osd/ceph-0) _open_db_and_around read-only:0 repair:0 May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (13) Permission denied May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 1 bdev(0x559b62422400 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 -1 bdev(0x559b62422400 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 -1 osd.0 0 OSD:init: unable to mount object store May 25 16:51:49 forth bash[24332]: debug 2022-05-25T16:51:49.147+0000 7fee82abf080 -1 ** ERROR: osd init failed: (13) Permission denied May 25 16:51:50 forth systemd[1]: ceph-d7511c3e-a570-11ec-9c50-3bd4b57e7e6e@osd.0.service: Main process exited, code=exited, status=1/FAILURE -- Andrew W. Piltser-Cowan awc34@xxxxxxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |