I have tested. I install a Ceph Cluster v16.2 with cephadm.
It runs on container. And the images is form quay.io.
I add the following to /etc/ceph/ceph.conf global section.
ms_type = async+rdma
ms_async_rdma_device_name = mlx5_0
ms_async_rdma_gid_idx = 3
ms_async_rdma_local_gid = fe80:0000:0000:0000:b24a:b3d0:9469:ee67
ms_async_rdma_roce_ver = 1
when I run "/usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug" on container.
And it crashed. The crash message is much more.
------------------------
Crash message:
------------------------
/usr/include/c++/8/bits/stl_vector.h:932: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](std::vector<_Tp, _Alloc>::size_type) [with _Tp = Worker*; _Alloc = std::allocator<Worker*>; std::vector<_Tp, _Alloc>::reference = Worker*&; std::vector<_Tp, _Alloc>::size_type = long unsigned int]: Assertion '__builtin_expect(__n < this->size(), true)' failed.
*** Caught signal (Aborted) **
in thread 7f1ee55a5080 thread_name:ceph-osd
ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable)
1: /lib64/libpthread.so.0(+0x12b20) [0x7f1ee354bb20]
2: gsignal()
3: abort()
4: /usr/bin/ceph-osd(+0x6262d8) [0x55a194b822d8]
5: (RDMAStack::RDMAStack(ceph::common::CephContext*)+0x55f) [0x55a1953ff14f]
6: (NetworkStack::create(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x350) [0x55a1953f3870]
7: (AsyncMessenger::AsyncMessenger(ceph::common::CephContext*, entity_name_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long)+0xe78) [0x55a1953e6d68]
8: (Messenger::create(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, entity_name_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long)+0x10b) [0x55a1953d91bb]
9: (Messenger::create_client_messenger(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x199) [0x55a1953d9619]
10: (MonClient::get_monmap_and_config()+0x94) [0x55a195435a24]
11: (global_init(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const*, std::vector<char const*, std::allocator<char const*> >&, unsigned int, code_environment_t, int, bool)+0x7cf) [0x55a19520799f]
12: main()
13: __libc_start_main()
14: _start()
debug2021-11-17T07:47:44.731+0000 7f1ee55a5080 -1 *** Caught signal (Aborted) **
in thread 7f1ee55a5080 thread_name:ceph-osd
ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable)
1: /lib64/libpthread.so.0(+0x12b20) [0x7f1ee354bb20]
2: gsignal()
3: abort()
4: /usr/bin/ceph-osd(+0x6262d8) [0x55a194b822d8]
5: (RDMAStack::RDMAStack(ceph::common::CephContext*)+0x55f) [0x55a1953ff14f]
6: (NetworkStack::create(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x350) [0x55a1953f3870]
7: (AsyncMessenger::AsyncMessenger(ceph::common::CephContext*, entity_name_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long)+0xe78) [0x55a1953e6d68]
8: (Messenger::create(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, entity_name_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long)+0x10b) [0x55a1953d91bb]
9: (Messenger::create_client_messenger(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x199) [0x55a1953d9619]
10: (MonClient::get_monmap_and_config()+0x94) [0x55a195435a24]
11: (global_init(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const*, std::vector<char const*, std::allocator<char const*> >&, unsigned int, code_environment_t, int, bool)+0x7cf) [0x55a19520799f]
12: main()
13: __libc_start_main()
14: _start()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
debug -44> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command assert hook 0x55a19fe76540
debug -43> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command abort hook 0x55a19fe76540
debug -42> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command leak_some_memory hook 0x55a19fe76540
debug -41> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command perfcounters_dump hook 0x55a19fe76540
debug -40> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command 1 hook 0x55a19fe76540
debug -39> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command perf dump hook 0x55a19fe76540
debug -38> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command perfcounters_schema hook 0x55a19fe76540
debug -37> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command perf histogram dump hook 0x55a19fe76540
debug -36> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command 2 hook 0x55a19fe76540
debug -35> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command perf schema hook 0x55a19fe76540
debug -34> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command perf histogram schema hook 0x55a19fe76540
debug -33> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command perf reset hook 0x55a19fe76540
debug -32> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command config show hook 0x55a19fe76540
debug -31> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command config help hook 0x55a19fe76540
debug -30> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command config set hook 0x55a19fe76540
debug -29> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command config unset hook 0x55a19fe76540
debug -28> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command config get hook 0x55a19fe76540
debug -27> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command config diff hook 0x55a19fe76540
debug -26> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command config diff get hook 0x55a19fe76540
debug -25> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command injectargs hook 0x55a19fe76540
debug -24> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command log flush hook 0x55a19fe76540
debug -23> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command log dump hook 0x55a19fe76540
debug -22> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command log reopen hook 0x55a19fe76540
debug -21> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command dump_mempools hook 0x55a19ff16328
debug -20> 2021-11-17T07:47:44.706+0000 7f1ee55a5080 10 monclient: get_monmap_and_config
debug -19> 2021-11-17T07:47:44.706+0000 7f1ee55a5080 10 monclient: build_initial_monmap
debug -18> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 10 monclient: monmap:
epoch 0
fsid ea39c6f0-fb3b-11eb-9f7a-b8cef60b8e48
last_changed 2021-11-17T07:47:44.708437+0000
created 2021-11-17T07:47:44.708437+0000
min_mon_release 0 (unknown)
election_strategy: 1
0: [v2:10.10.0.2:3300/0,v1:10.10.0.2:6789/0] mon.noname-a
debug -17> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding auth protocol: cephx
debug -16> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding auth protocol: cephx
debug -15> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding auth protocol: cephx
debug -14> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding auth protocol: none
debug -13> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: secure
debug -12> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: crc
debug -11> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: secure
debug -10> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: crc
debug -9> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: secure
debug -8> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: crc
debug -7> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: crc
debug -6> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: secure
debug -5> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: crc
debug -4> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: secure
debug -3> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: crc
debug -2> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: secure
debug -1> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-0/keyring
debug 0> 2021-11-17T07:47:44.731+0000 7f1ee55a5080 -1 *** Caught signal (Aborted) **
in thread 7f1ee55a5080 thread_name:ceph-osd
ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable)
1: /lib64/libpthread.so.0(+0x12b20) [0x7f1ee354bb20]
2: gsignal()
3: abort()
4: /usr/bin/ceph-osd(+0x6262d8) [0x55a194b822d8]
5: (RDMAStack::RDMAStack(ceph::common::CephContext*)+0x55f) [0x55a1953ff14f]
6: (NetworkStack::create(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x350) [0x55a1953f3870]
7: (AsyncMessenger::AsyncMessenger(ceph::common::CephContext*, entity_name_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long)+0xe78) [0x55a1953e6d68]
8: (Messenger::create(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, entity_name_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long)+0x10b) [0x55a1953d91bb]
9: (Messenger::create_client_messenger(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x199) [0x55a1953d9619]
10: (MonClient::get_monmap_and_config()+0x94) [0x55a195435a24]
11: (global_init(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const*, std::vector<char const*, std::allocator<char const*> >&, unsigned int, code_environment_t, int, bool)+0x7cf) [0x55a19520799f]
12: main()
13: __libc_start_main()
14: _start()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 rbd_pwl
0/ 5 journaler
0/ 5 objectcacher
0/ 5 immutable_obj_cache
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 fuse
2/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
1/ 5 prioritycache
0/ 5 test
0/ 5 cephfs_mirror
0/ 5 cephsqlite
-2/-2 (syslog threshold)
99/99 (stderr threshold)
--- pthread ID / name mapping for recent threads ---
139770673647744 / ceph-osd
max_recent 10000
max_new 10000
log_file
--- end dump of recent events ---
--- begin dump of recent events ---
debug -44> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command assert hook 0x55a19fe76540
debug -43> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command abort hook 0x55a19fe76540
debug -42> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command leak_some_memory hook 0x55a19fe76540
debug -41> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command perfcounters_dump hook 0x55a19fe76540
debug -40> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command 1 hook 0x55a19fe76540
debug -39> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command perf dump hook 0x55a19fe76540
debug -38> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command perfcounters_schema hook 0x55a19fe76540
debug -37> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command perf histogram dump hook 0x55a19fe76540
debug -36> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command 2 hook 0x55a19fe76540
debug -35> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command perf schema hook 0x55a19fe76540
debug -34> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command perf histogram schema hook 0x55a19fe76540
debug -33> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command perf reset hook 0x55a19fe76540
debug -32> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command config show hook 0x55a19fe76540
debug -31> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command config help hook 0x55a19fe76540
debug -30> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command config set hook 0x55a19fe76540
debug -29> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command config unset hook 0x55a19fe76540
debug -28> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command config get hook 0x55a19fe76540
debug -27> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command config diff hook 0x55a19fe76540
debug -26> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command config diff get hook 0x55a19fe76540
debug -25> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command injectargs hook 0x55a19fe76540
debug -24> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command log flush hook 0x55a19fe76540
debug -23> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command log dump hook 0x55a19fe76540
debug -22> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command log reopen hook 0x55a19fe76540
debug -21> 2021-11-17T07:47:44.702+0000 7f1ee55a5080 5 asok(0x55a19ff28000) register_command dump_mempools hook 0x55a19ff16328
debug -20> 2021-11-17T07:47:44.706+0000 7f1ee55a5080 10 monclient: get_monmap_and_config
debug -19> 2021-11-17T07:47:44.706+0000 7f1ee55a5080 10 monclient: build_initial_monmap
debug -18> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 10 monclient: monmap:
epoch 0
fsid ea39c6f0-fb3b-11eb-9f7a-b8cef60b8e48
last_changed 2021-11-17T07:47:44.708437+0000
created 2021-11-17T07:47:44.708437+0000
min_mon_release 0 (unknown)
election_strategy: 1
0: [v2:10.10.0.2:3300/0,v1:10.10.0.2:6789/0] mon.noname-a
debug -17> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding auth protocol: cephx
debug -16> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding auth protocol: cephx
debug -15> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding auth protocol: cephx
debug -14> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding auth protocol: none
debug -13> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: secure
debug -12> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: crc
debug -11> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: secure
debug -10> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: crc
debug -9> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: secure
debug -8> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: crc
debug -7> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: crc
debug -6> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: secure
debug -5> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: crc
debug -4> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: secure
debug -3> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: crc
debug -2> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 5 AuthRegistry(0x55a1a0afca40) adding con mode: secure
debug -1> 2021-11-17T07:47:44.707+0000 7f1ee55a5080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-0/keyring
debug 0> 2021-11-17T07:47:44.731+0000 7f1ee55a5080 -1 *** Caught signal (Aborted) **
in thread 7f1ee55a5080 thread_name:ceph-osd
ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable)
1: /lib64/libpthread.so.0(+0x12b20) [0x7f1ee354bb20]
2: gsignal()
3: abort()
4: /usr/bin/ceph-osd(+0x6262d8) [0x55a194b822d8]
5: (RDMAStack::RDMAStack(ceph::common::CephContext*)+0x55f) [0x55a1953ff14f]
6: (NetworkStack::create(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x350) [0x55a1953f3870]
7: (AsyncMessenger::AsyncMessenger(ceph::common::CephContext*, entity_name_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long)+0xe78) [0x55a1953e6d68]
8: (Messenger::create(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, entity_name_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long)+0x10b) [0x55a1953d91bb]
9: (Messenger::create_client_messenger(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x199) [0x55a1953d9619]
10: (MonClient::get_monmap_and_config()+0x94) [0x55a195435a24]
11: (global_init(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const*, std::vector<char const*, std::allocator<char const*> >&, unsigned int, code_environment_t, int, bool)+0x7cf) [0x55a19520799f]
12: main()
13: __libc_start_main()
14: _start()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 rbd_pwl
0/ 5 journaler
0/ 5 objectcacher
0/ 5 immutable_obj_cache
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 fuse
2/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
1/ 5 prioritycache
0/ 5 test
0/ 5 cephfs_mirror
0/ 5 cephsqlite
-2/-2 (syslog threshold)
99/99 (stderr threshold)
--- pthread ID / name mapping for recent threads ---
139770673647744 / ceph-osd
max_recent 10000
max_new 10000
log_file /var/lib/ceph/crash/2021-11-17T07:47:44.732525Z_f498ef52-8486-4f87-9945-d78121002152/log
--- end dump of recent events ---
Aborted (core dumped)
------------------------
------------------ Original ------------------
From: "David Majchrzak, Oderland Webbhotell AB" <david@xxxxxxxxxxx>;
Date: Wed, Nov 10, 2021 06:54 PM
To: "Mason-Williams, Gabryel (RFI,RAL,-)"<gabryel.mason-williams@xxxxxxxxx>;
Cc: "GHui"<ugiwgh@xxxxxx>;"ceph-users"<ceph-users@xxxxxxx>;
Subject: Re: Re: How to enable RDMA
I think the latest docs on ceph RDMA "support" is based on Luminous.
I'd be careful using RDMA on later versions of ceph if you're running a production cluster.
Kind Regards,
David Majchrzak
CTO
Oderland Webbhotell AB
Den 10 nov. 2021, kI 11:47, "Mason-Williams, Gabryel (RFI,RAL,-)" <gabryel.mason-williams@xxxxxxxxx> skrev:
Hi GHui,
You might find this document useful: https://support.mellanox.com/s/article/bring-up-ceph-rdma---developer-s-guide
Also, I previously asked this question and there was some useful information in the thread: https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/message/5JD4ATRXKMMLIUQI5TUAUYQFGJ45Q7MJ/
Kind regards
Gabryel
From: GHui <ugiwgh@xxxxxx>
Sent: 10 November 2021 10:34
To: ceph-users <ceph-users@xxxxxxx>
Subject: How to enable RDMA
Hi,
How can I known my Ceph Cluster is enabled RDMA, and with RoCE v2?
I would very much appreciate any advice.
Best Regards,
GHui
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
