OK, found it. The second line in the error messages actually gives it away: stderr: 2021-02-06 13:48:27.477 7f46756b4b80 -1 bdev(0x561db199c700 /var/lib/ceph/osd/ceph-342//block) _aio_start io_setup(2) failed with EAGAIN; try increasing /proc/sys/fs/aio-max-nr On my system, the default is rather small: # sysctl fs.aio-max-nr fs.aio-max-nr = 65536 Seemingly not a problem for ceph-disk OSDs: # sysctl fs.aio-nr fs.aio-nr = 32768 However, LVM OSDs seem to be quite hungry. Increasing the value to sysctl -w fs.aio-max-nr=1048576 solved it for me. Should have read it more carefully. Best regards and a nice weekend. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@xxxxxx> Sent: 06 February 2021 14:37:15 To: ceph-users@xxxxxxx Cc: Dave Hall Subject: Re: ceph-volume bluestore _read_fsid unparsable uuid I just noticed one difference between the two servers: "Broken" server: # lvm vgs Failed to set up async io, using sync io. VG #PV #LV #SN Attr VSize VFree [listing follows] "Good" server: # lvm vgs VG #PV #LV #SN Attr VSize VFree [listing follows] Could this play a role here? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@xxxxxx> Sent: 06 February 2021 14:31:38 To: ceph-users@xxxxxxx Cc: Dave Hall Subject: ceph-volume bluestore _read_fsid unparsable uuid Hi Dave and everyone else affected, I'm responding to a thread you opened on an issue with lvm OSD creation: https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/YYH3VANVV22WGM3CNL4TN4TTL63FCEVD/ https://tracker.ceph.com/issues/43868 Most important question: is there a workaround? My observations: I'm running into the exact same issue on mimic 13.2.10. The strange thing is, that some OSDs get created and others fail. I can't see a pattern here. I have 1 host where every create worked out and another where half failed. The important lines in the log are probably: stderr: 2021-02-06 13:48:27.477 7f46756b4b80 -1 bluestore(/var/lib/ceph/osd/ceph-342/) _read_fsid unparsable uuid stderr: 2021-02-06 13:48:27.477 7f46756b4b80 -1 bdev(0x561db199c700 /var/lib/ceph/osd/ceph-342//block) _aio_start io_setup(2) failed with EAGAIN; try increasing /proc/sys/fs/aio-max-nr stderr: 2021-02-06 13:48:27.477 7f46756b4b80 -1 bluestore(/var/lib/ceph/osd/ceph-342/) mkfs failed, (11) Resource temporarily unavailable stderr: 2021-02-06 13:48:27.477 7f46756b4b80 -1 OSD::mkfs: ObjectStore::mkfs failed with error (11) Resource temporarily unavailable stderr: 2021-02-06 13:48:27.477 7f46756b4b80 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-342/: (11) Resource temporarily unavailable[0m I really need to get a decent number of disks up very soon. Any help is appreciated. I can provide more output if that helps. Best regards and good weekend! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |