Hello Jan, it would be nice to see how the state evolves. Is there a chance to get a log with `debug_prioritycache=20` and `debug_bluestore=20`? > How I can create osd? As a workaround you might want to poke with `osd_memory_cache_min` which, by default, is just 128 MB. CCing BlueStore folks. Regards, Radek On Fri, Aug 4, 2023 at 12:58 AM Jan Marek <jmarek@xxxxxx> wrote: > > Hello, > > I would like to add OSD to cluster, but I've got this error: > > # ceph-volume --cluster ceph lvm create --bluestore --data /dev/sda --block.wal /dev/ceph/wal.a --block.db /dev/ceph/db.a > Running command: /usr/bin/ceph-authtool --gen-print-key > Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 459cc86e-999a-4398-bd20-d03031f98a10 > Running command: vgcreate --force --yes ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb /dev/sda > stdout: Physical volume "/dev/sda" successfully created. > stdout: Volume group "ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb" successfully created > Running command: lvcreate --yes -l 1907721 -n osd-block-459cc86e-999a-4398-bd20-d03031f98a10 ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb > stdout: Logical volume "osd-block-459cc86e-999a-4398-bd20-d03031f98a10" created. > Running command: /usr/bin/ceph-authtool --gen-print-key > Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0 > --> Executable selinuxenabled not in PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin > Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb/osd-block-459cc86e-999a-4398-bd20-d03031f98a10 > Running command: /usr/bin/chown -R ceph:ceph /dev/dm-24 > Running command: /usr/bin/ln -s /dev/ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb/osd-block-459cc86e-999a-4398-bd20-d03031f98a10 /var/lib/ceph/osd/ceph-0/block > Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap > stderr: 2023-02-09T13:10:37.252+0100 7fdc2b98a700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory > 2023-02-09T13:10:37.252+0100 7fdc2b98a700 -1 AuthRegistry(0x7fdc240607d0) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx > stderr: got monmap epoch 254 > --> Creating keyring file for osd.0 > Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring > Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/ > Running command: /usr/bin/chown -h ceph:ceph /dev/ceph/wal.a > Running command: /usr/bin/chown -R ceph:ceph /dev/dm-22 > Running command: /usr/bin/chown -h ceph:ceph /dev/ceph/db.a > Running command: /usr/bin/chown -R ceph:ceph /dev/dm-23 > Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --bluestore-block-wal-path /dev/ceph/wal.a --bluestore-block-db-path /dev/ceph/db.a --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 459cc86e-999a-4398-bd20-d03031f98a10 --setuser ceph --setgroup ceph > stderr: 2023-02-09T13:10:37.728+0100 7f54531aa240 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid > stderr: ./src/common/PriorityCache.cc: In function 'void PriorityCache::Manager::balance()' thread 7f5446944700 time 2023-02-09T13:10:39.843937+0100 > stderr: ./src/common/PriorityCache.cc: 300: FAILED ceph_assert(mem_avail >= 0) > stderr: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) > stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x124) [0x563da783bf70] > stderr: 2: /usr/bin/ceph-osd(+0xc2310e) [0x563da783c10e] > stderr: 3: (PriorityCache::Manager::balance()+0x478) [0x563da8459938] > stderr: 4: (BlueStore::MempoolThread::entry()+0x82a) [0x563da7e9e27a] > stderr: 5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f5453804ea7] > stderr: 6: clone() > stderr: *** Caught signal (Aborted) ** > stderr: in thread 7f5446944700 thread_name:bstore_mempool > stderr: 2023-02-09T13:10:39.844+0100 7f5446944700 -1 ./src/common/PriorityCache.cc: In function 'void PriorityCache::Manager::balance()' thread 7f5446944700 time 2023-02-09T13:10:39.843937+0100 > stderr: ./src/common/PriorityCache.cc: 300: FAILED ceph_assert(mem_avail >= 0) > stderr: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) > stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x124) [0x563da783bf70] > stderr: 2: /usr/bin/ceph-osd(+0xc2310e) [0x563da783c10e] > stderr: 3: (PriorityCache::Manager::balance()+0x478) [0x563da8459938] > stderr: 4: (BlueStore::MempoolThread::entry()+0x82a) [0x563da7e9e27a] > stderr: 5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f5453804ea7] > stderr: 6: clone() > stderr: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) > stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7f5453810140] > stderr: 2: gsignal() > stderr: 3: abort() > stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17e) [0x563da783bfca] > stderr: 5: /usr/bin/ceph-osd(+0xc2310e) [0x563da783c10e] > stderr: 6: (PriorityCache::Manager::balance()+0x478) [0x563da8459938] > stderr: 7: (BlueStore::MempoolThread::entry()+0x82a) [0x563da7e9e27a] > stderr: 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f5453804ea7] > stderr: 9: clone() > stderr: 2023-02-09T13:10:39.852+0100 7f5446944700 -1 *** Caught signal (Aborted) ** > stderr: in thread 7f5446944700 thread_name:bstore_mempool > stderr: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) > stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7f5453810140] > stderr: 2: gsignal() > stderr: 3: abort() > stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17e) [0x563da783bfca] > stderr: 5: /usr/bin/ceph-osd(+0xc2310e) [0x563da783c10e] > stderr: 6: (PriorityCache::Manager::balance()+0x478) [0x563da8459938] > stderr: 7: (BlueStore::MempoolThread::entry()+0x82a) [0x563da7e9e27a] > stderr: 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f5453804ea7] > stderr: 9: clone() > stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > stderr: -3212> 2023-02-09T13:10:37.728+0100 7f54531aa240 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid > stderr: -1> 2023-02-09T13:10:39.844+0100 7f5446944700 -1 ./src/common/PriorityCache.cc: In function 'void PriorityCache::Manager::balance()' thread 7f5446944700 time 2023-02-09T13:10:39.843937+0100 > stderr: ./src/common/PriorityCache.cc: 300: FAILED ceph_assert(mem_avail >= 0) > stderr: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) > stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x124) [0x563da783bf70] > stderr: 2: /usr/bin/ceph-osd(+0xc2310e) [0x563da783c10e] > stderr: 3: (PriorityCache::Manager::balance()+0x478) [0x563da8459938] > stderr: 4: (BlueStore::MempoolThread::entry()+0x82a) [0x563da7e9e27a] > stderr: 5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f5453804ea7] > stderr: 6: clone() > stderr: 0> 2023-02-09T13:10:39.852+0100 7f5446944700 -1 *** Caught signal (Aborted) ** > stderr: in thread 7f5446944700 thread_name:bstore_mempool > stderr: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) > stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7f5453810140] > stderr: 2: gsignal() > stderr: 3: abort() > stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17e) [0x563da783bfca] > stderr: 5: /usr/bin/ceph-osd(+0xc2310e) [0x563da783c10e] > stderr: 6: (PriorityCache::Manager::balance()+0x478) [0x563da8459938] > stderr: 7: (BlueStore::MempoolThread::entry()+0x82a) [0x563da7e9e27a] > stderr: 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f5453804ea7] > stderr: 9: clone() > stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > --> Was unable to complete a new OSD, will rollback changes > Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it > stderr: 2023-02-09T13:10:40.072+0100 7f6963aec700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory > 2023-02-09T13:10:40.072+0100 7f6963aec700 -1 AuthRegistry(0x7f695c0607d0) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx > stderr: purged osd.0 > --> Zapping: /dev/ceph/db.a > --> Unmounting /var/lib/ceph/osd/ceph-0 > Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-0 > stderr: umount: /var/lib/ceph/osd/ceph-0 odpojen > Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph/db.a bs=1M count=10 conv=fsync > stderr: 10+0 záznamů přečteno > 10+0 záznamů zapsáno > 10485760 bajtů (10 MB, 10 MiB) zkopírováno, 0,0144712 s, 725 MB/s > --> More than 1 LV left in VG, will proceed to destroy LV only > --> Removing LV because --destroy was given: /dev/ceph/db.a > Running command: lvremove -v -f /dev/ceph/db.a > stdout: Logical volume "db.a" successfully removed > stderr: Removing ceph-db.a (254:23) > stderr: Archiving volume group "ceph" metadata (seqno 85). > stderr: Releasing logical volume "db.a" > stderr: Creating volume group backup "/etc/lvm/backup/ceph" (seqno 86). > --> Zapping: /dev/ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb/osd-block-459cc86e-999a-4398-bd20-d03031f98a10 > Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb/osd-block-459cc86e-999a-4398-bd20-d03031f98a10 bs=1M count=10 conv=fsync > stderr: 10+0 záznamů přečteno > 10+0 záznamů zapsáno > stderr: 10485760 bajtů (10 MB, 10 MiB) zkopírováno, 0,096139 s, 109 MB/s > --> Only 1 LV left in VG, will proceed to destroy volume group ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb > Running command: vgremove -v -f ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb > stderr: Removing ceph--8f15ecd8--3b3f--404f--a851--8bbba15e30bb-osd--block--459cc86e--999a--4398--bd20--d03031f98a10 (254:24) > stderr: Archiving volume group "ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb" metadata (seqno 5). > stderr: Releasing logical volume "osd-block-459cc86e-999a-4398-bd20-d03031f98a10" > stderr: Creating volume group backup "/etc/lvm/backup/ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb" (seqno 6). > stdout: Logical volume "osd-block-459cc86e-999a-4398-bd20-d03031f98a10" successfully removed > stderr: Removing physical volume "/dev/sda" from volume group "ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb" > stdout: Volume group "ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb" successfully removed > Running command: pvremove -v -f -f /dev/sda > stdout: Labels on physical volume "/dev/sda" successfully wiped. > --> Zapping: /dev/ceph/wal.a > Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph/wal.a bs=1M count=10 conv=fsync > stderr: 10+0 záznamů přečteno > 10+0 záznamů zapsáno > 10485760 bajtů (10 MB, 10 MiB) zkopírováno, 0,014648 s, 716 MB/s > --> More than 1 LV left in VG, will proceed to destroy LV only > --> Removing LV because --destroy was given: /dev/ceph/wal.a > Running command: lvremove -v -f /dev/ceph/wal.a > stdout: Logical volume "wal.a" successfully removed > stderr: Removing ceph-wal.a (254:22) > stderr: Archiving volume group "ceph" metadata (seqno 86). > stderr: Releasing logical volume "wal.a" > stderr: Creating volume group backup "/etc/lvm/backup/ceph" (seqno 87). > --> Zapping successful for OSD: 0 > --> RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --bluestore-block-wal-path /dev/ceph/wal.a --bluestore-block-db-path /dev/ceph/db.a --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 459cc86e-999a-4398-bd20-d03031f98a10 --setuser ceph --setgroup ceph > > My server have 64GB RAM. > > Version of ceph package is 17.2.5 > > ceph config dump | grep osd > > global dev osd_max_pg_log_entries 10 > global advanced osd_memory_target_autotune true > global dev osd_min_pg_log_entries 10 > global dev osd_pg_log_dups_tracked 10 > global dev osd_pg_log_trim_min 10 > global advanced osd_pool_default_min_size 2 > global advanced osd_pool_default_size 3 > osd dev bluefs_allocator bitmap * > osd advanced bluestore_allocator bitmap * > osd advanced bluestore_default_buffered_write true > osd dev bluestore_fsck_quick_fix_on_mount true > osd dev osd_memory_cache_min 134217728 > osd host:c-osd2 basic osd_memory_target 3927826739 > osd host:c-osd3 basic osd_memory_target 3927839880 > osd host:c-osd4 basic osd_memory_target 3927826739 > osd host:c-osd5 basic osd_memory_target 3927826739 > osd host:c-osd6 basic osd_memory_target 3927826739 > osd basic osd_memory_target 1073741824 > > How I can create osd? > > I've tried `ceph orch`, but its got the same error... > > Sincerely > Jan Marek > -- > Ing. Jan Marek > University of South Bohemia > Academic Computer Centre > Phone: +420389032080 > http://www.gnu.org/philosophy/no-word-attachments.cs.html > _______________________________________________ > Dev mailing list -- dev@xxxxxxx > To unsubscribe send an email to dev-leave@xxxxxxx _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
