Hi, I'm recently trying to enable OSD to use io_uring with our Cephadm deployment by bellow command. ceph config set osd bdev_ioring true ceph config set osd bdev_ioring_hipri true ceph config set osd bdev_ioring_sqthread_poll true However, I've ran into the issue similar to this bug. Bug #47661: Cannot allocate memory appears when using io_uring osd - bluestore - Ceph <https://tracker.ceph.com/issues/47661> I've tried setting "--ulimit memlock=-1:-1" to the docker run line in unit.run file that cephadm created for OSD service. I can confirm that the "max locked memory" is set to unlimited in the container when running ulimit -a in the container. The osd still failed to start when io_uring is enabled. Any suggestions? OSD logs: Using recent ceph image quay.io/ceph/ceph@sha256:bb6a71f7f481985f6d3b358e3b9ef64c6755b3db5aa53198e0aac38be5c8ae54 debug 2022-01-05T18:34:38.878+0000 7f06ffaee080 0 set uid:gid to 167:167 (ceph:ceph) debug 2022-01-05T18:34:38.878+0000 7f06ffaee080 0 ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable), process ceph-osd, pid 7 debug 2022-01-05T18:34:38.878+0000 7f06ffaee080 0 pidfile_write: ignore empty --pid-file debug 2022-01-05T18:34:38.878+0000 7f06ffaee080 1 bdev(0x55f113f5c800 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug 2022-01-05T18:34:38.882+0000 7f06ffaee080 -1 bdev(0x55f113f5c800 /var/lib/ceph/osd/ceph-2/block) _aio_start io_setup(2) failed: (12) Cannot allocate memory debug 2022-01-05T18:34:38.882+0000 7f06ffaee080 0 starting osd.2 osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal ulimit -a output (container is started when io_uring is disabled): core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 1030203 max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 1048576 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Regards, Gene Kuo _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |