Hello together, i need to recreate the osd´s on one ceph-node, because the nvme wal device has died. I replaced the nvme to a brand new one and i try now to recreate the osd´s on this node, but i get an error while re-creating them. Can somebody tell me why i get this error? i never saw before this is what i do: root@cd88-ceph-osdh-01:~# ceph osd out osd.2 marked out osd.2. root@cd88-ceph-osdh-01:~# systemctl stop ceph-osd@osd.2 root@cd88-ceph-osdh-01:~# ceph osd crush remove osd.2 removed item id 2 name 'osd.2' from crush map root@cd88-ceph-osdh-01:~# ceph auth del osd.2 updated root@cd88-ceph-osdh-01:~# ceph osd rm osd.2 removed osd.2 root@cd88-ceph-osdh-01:~# ceph-volume lvm zap /dev/ceph-block-0/block-0 --> Zapping: /dev/ceph-block-0/block-0 Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-block-0/block-0 bs=1M count=10 conv=fsync stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0878055 s, 119 MB/s stderr: --> Zapping successful for: <LV: /dev/ceph-block-0/block-0> root@cd88-ceph-osdh-01:~# ceph-volume lvm zap /dev/ceph-db-0/db-0 --> Zapping: /dev/ceph-db-0/db-0 Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-db-0/db-0 bs=1M count=10 conv=fsync stderr: 10+0 records in 10+0 records out stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0376431 s, 279 MB/s --> Zapping successful for: <LV: /dev/ceph-db-0/db-0> root@cd88-ceph-osdh-01:/# vi /var/lib/ceph/bootstrap-osd/ceph.keyring root@cd88-ceph-osdh-01:/# ceph-volume lvm prepare --no-systemd --bluestore --data ceph-block-0/block-0 --block.wal ceph-wal-0/waldb-0 --block.db ceph-db-0/db-0 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f2dcdce7-0b88-4a83-b9ca-2f4546c9a718 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-block-0/block-0 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-27 Running command: /usr/bin/ln -s /dev/ceph-block-0/block-0 /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-2/activate.monmap stderr: got monmap epoch 3 Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-2/keyring --create-keyring --name osd.2 --add-key AQCFXa5hn+TBJxAArSTZFWMavJqWGISwSR8Xow== stdout: creating /var/lib/ceph/osd/ceph-2/keyring added entity osd.2 auth(key=AQCFXa5hn+TBJxAArSTZFWMavJqWGISwSR8Xow==) Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/keyring Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/ Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-wal-0/waldb-0 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-40 Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-db-0/db-0 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --bluestore-block-wal-path /dev/ceph-wal-0/waldb-0 --bluestore-block-db-path /dev/ceph-db-0/db-0 --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid f2dcdce7-0b88-4a83-b9ca-2f4546c9a718 --setuser ceph --setgroup ceph stderr: 2021-12-06T18:59:19.652+0000 7f3ea7a84080 -1 bluestore(/var/lib/ceph/osd/ceph-2/) _read_fsid unparsable uuid stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/common/PriorityCache.cc: In function 'void PriorityCache::Manager::balance()' thread 7f3e835a3700 time 2021-12-06T18:59:21.531373+0000 stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/common/PriorityCache.cc: 301: FAILED ceph_assert(mem_avail >= 0) stderr: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x558fe9b69eba] stderr: 2: /usr/bin/ceph-osd(+0x56a0d4) [0x558fe9b6a0d4] stderr: 3: (PriorityCache::Manager::balance()+0x4e1) [0x558fea6f1a61] stderr: 4: (BlueStore::MempoolThread::entry()+0x889) [0x558fea17f259] stderr: 5: /lib64/libpthread.so.0(+0x814a) [0x7f3ea57db14a] stderr: 6: clone() stderr: 2021-12-06T18:59:21.528+0000 7f3e835a3700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/common/PriorityCache.cc: In function 'void PriorityCache::Manager::balance()' thread 7f3e835a3700 time 2021-12-06T18:59:21.531373+0000 stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/common/PriorityCache.cc: 301: FAILED ceph_assert(mem_avail >= 0) stderr: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x558fe9b69eba] stderr: 2: /usr/bin/ceph-osd(+0x56a0d4) [0x558fe9b6a0d4] stderr: 3: (PriorityCache::Manager::balance()+0x4e1) [0x558fea6f1a61] stderr: 4: (BlueStore::MempoolThread::entry()+0x889) [0x558fea17f259] stderr: 5: /lib64/libpthread.so.0(+0x814a) [0x7f3ea57db14a] stderr: 6: clone() stderr: *** Caught signal (Aborted) ** stderr: in thread 7f3e835a3700 thread_name:bstore_mempool stderr: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f3ea57e5b20] stderr: 2: gsignal() stderr: 3: abort() stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x558fe9b69f0b] stderr: 5: /usr/bin/ceph-osd(+0x56a0d4) [0x558fe9b6a0d4] stderr: 6: (PriorityCache::Manager::balance()+0x4e1) [0x558fea6f1a61] stderr: 7: (BlueStore::MempoolThread::entry()+0x889) [0x558fea17f259] stderr: 8: /lib64/libpthread.so.0(+0x814a) [0x7f3ea57db14a] stderr: 9: clone() stderr: 2021-12-06T18:59:21.532+0000 7f3e835a3700 -1 *** Caught signal (Aborted) ** stderr: in thread 7f3e835a3700 thread_name:bstore_mempool stderr: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f3ea57e5b20] stderr: 2: gsignal() stderr: 3: abort() stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x558fe9b69f0b] stderr: 5: /usr/bin/ceph-osd(+0x56a0d4) [0x558fe9b6a0d4] stderr: 6: (PriorityCache::Manager::balance()+0x4e1) [0x558fea6f1a61] stderr: 7: (BlueStore::MempoolThread::entry()+0x889) [0x558fea17f259] stderr: 8: /lib64/libpthread.so.0(+0x814a) [0x7f3ea57db14a] stderr: 9: clone() stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. stderr: -2953> 2021-12-06T18:59:19.652+0000 7f3ea7a84080 -1 bluestore(/var/lib/ceph/osd/ceph-2/) _read_fsid unparsable uuid stderr: -1> 2021-12-06T18:59:21.528+0000 7f3e835a3700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/common/PriorityCache.cc: In function 'void PriorityCache::Manager::balance()' thread 7f3e835a3700 time 2021-12-06T18:59:21.531373+0000 stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/common/PriorityCache.cc: 301: FAILED ceph_assert(mem_avail >= 0) stderr: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x558fe9b69eba] stderr: 2: /usr/bin/ceph-osd(+0x56a0d4) [0x558fe9b6a0d4] stderr: 3: (PriorityCache::Manager::balance()+0x4e1) [0x558fea6f1a61] stderr: 4: (BlueStore::MempoolThread::entry()+0x889) [0x558fea17f259] stderr: 5: /lib64/libpthread.so.0(+0x814a) [0x7f3ea57db14a] stderr: 6: clone() stderr: 0> 2021-12-06T18:59:21.532+0000 7f3e835a3700 -1 *** Caught signal (Aborted) ** stderr: in thread 7f3e835a3700 thread_name:bstore_mempool stderr: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f3ea57e5b20] stderr: 2: gsignal() stderr: 3: abort() stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x558fe9b69f0b] stderr: 5: /usr/bin/ceph-osd(+0x56a0d4) [0x558fe9b6a0d4] stderr: 6: (PriorityCache::Manager::balance()+0x4e1) [0x558fea6f1a61] stderr: 7: (BlueStore::MempoolThread::entry()+0x889) [0x558fea17f259] stderr: 8: /lib64/libpthread.so.0(+0x814a) [0x7f3ea57db14a] stderr: 9: clone() stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. stderr: *** Caught signal (Segmentation fault) ** stderr: in thread 7f3e835a3700 thread_name:bstore_mempool stderr: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f3ea57e5b20] stderr: 2: pthread_getname_np() stderr: 3: (ceph::logging::Log::dump_recent()+0x4b3) [0x558fea4ae233] stderr: 4: /usr/bin/ceph-osd(+0xcab9fb) [0x558fea2ab9fb] stderr: 5: /lib64/libpthread.so.0(+0x12b20) [0x7f3ea57e5b20] stderr: 6: gsignal() stderr: 7: abort() stderr: 8: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x558fe9b69f0b] stderr: 9: /usr/bin/ceph-osd(+0x56a0d4) [0x558fe9b6a0d4] stderr: 10: (PriorityCache::Manager::balance()+0x4e1) [0x558fea6f1a61] stderr: 11: (BlueStore::MempoolThread::entry()+0x889) [0x558fea17f259] stderr: 12: /lib64/libpthread.so.0(+0x814a) [0x7f3ea57db14a] stderr: 13: clone() --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.2 --yes-i-really-mean-it stderr: purged osd.2 --> RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --bluestore-block-wal-path /dev/ceph-wal-0/waldb-0 --bluestore-block-db-path /dev/ceph-db-0/db-0 --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid f2dcdce7-0b88-4a83-b9ca-2f4546c9a718 --setuser ceph --setgroup ceph root@cd88-ceph-osdh-01:/# _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |