Brad Hubbard <bhubbard@xxxxxxxxxx>于2021年8月14日 周六06:11写道：
On Sat, Aug 14, 2021 at 4:06 AM Links 2004 <links2004.code@xxxxxxxxx> wrote:
>
>
> Hi,
>
> we are currently facing a strange problem on on of our ceph nodes.
> it is not possible to call `ceph -s` or start a mgr with out a 'std::runtime_error'.
>
> find below the error message and a gdb backtrace with debug symbols.
> hope this helps to understand the problem and point us in the correct direction.
>
> Thanks
>
> Markus
>
>
> OS: Debian buster
> kernel: 5.10.0-0.bpo.8-amd64 #1 SMP Debian 5.10.46-2~bpo10+1 (2021-07-22) x86_64 GNU/Linux
>
> ```
> # ceph -v
> ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
> ```
>
> ```
> # ceph -s
> terminate called after throwing an instance of 'std::runtime_error'
> what(): random_device::__x86_rdrand(void)
> Aborted
> ```
Did this issue coincide with a kernel upgrade?
Can you try and generate a lot of entropy on the system and see if the
issue goes away?
Also check the output of 'dmesg | grep random' to see if that offers any clues.
I feel the same. Looks likely that the kernel did not have enough entropy by then. Is the system not connected to a keyboard? Or it was just booted? If that’s the case, probably you could wait a while before try to launch mgr or use the ceph command line utility.
>
> ```
> # gdb --args /usr/bin/python3.7 /usr/bin/ceph -s
> GNU gdb (Debian 8.2.1-2+b3) 8.2.1
> Copyright (C) 2018 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Type "show copying" and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
>
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from /usr/bin/python3.7...Reading symbols from /usr/lib/debug/.build-id/99/21c75e6930d3e9d9fa8c942aca9dc4500bb65f.debug...done.
> done.
> (gdb) run
> Starting program: /usr/bin/python3.7 /usr/bin/ceph -s
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>
> [New Thread 0x7fffed515700 (LWP 30323)]
> [New Thread 0x7fffe7fff700 (LWP 30324)]
> [New Thread 0x7fffe77fe700 (LWP 30325)]
> [Thread 0x7fffed515700 (LWP 30323) exited]
> [New Thread 0x7fffed515700 (LWP 30326)]
> [Thread 0x7fffe7fff700 (LWP 30324) exited]
> terminate called after throwing an instance of 'std::runtime_error'
> what(): random_device::__x86_rdrand(void)
>
> Thread 4 "python3.7" received signal SIGABRT, Aborted.
> [Switching to Thread 0x7fffe77fe700 (LWP 30325)]
> __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
> 50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> (gdb)
> (gdb) bt
> #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
> #1 0x00007ffff79de535 in __GI_abort () at abort.c:79
> #2 0x00007fffeddb8983 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
> #3 0x00007fffeddbe8c6 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
> #4 0x00007fffeddbe901 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
> #5 0x00007fffeddbeb34 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
> #6 0x00007fffeddba8b7 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
> #7 0x00007fffedde6e86 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
> #8 0x00007fffedde6fd2 in std::random_device::_M_getval() () from /lib/x86_64-linux-gnu/libstdc++.so.6
> #9 0x00007fffee540ffc in std::random_device::operator() (this=0x7fffe77fba60) at /usr/include/c++/8/bits/random.h:1611
> #10 ceph::util::version_1_0_3::detail::randomize_rng<std::linear_congruential_engine<unsigned long, 16807ul, 0ul, 2147483647ul> > () at ./src/include/random.h:120
> #11 0x00007fffee54112f in ceph::util::version_1_0_3::detail::engine<std::linear_congruential_engine<unsigned long, 16807ul, 0ul, 2147483647ul> > ()
> at /usr/include/c++/8/new:169
> #12 0x00007fffee772911 in ceph::util::version_1_0_3::detail::generate_random_number<unsigned long, std::uniform_int_distribution<unsigned long>, std::linear_congruential_engine<unsigned long, 16807ul, 0ul, 2147483647ul> > (min=min@entry=0, max=max@entry=18446744073709551615) at ./src/include/random.h:170
> #13 0x00007fffee7718fe in ceph::util::version_1_0_3::generate_random_number<unsigned long, std::uniform_int_distribution<unsigned long>, std::linear_congruential_engine<unsigned long, 16807ul, 0ul, 2147483647ul> > () at ./src/include/random.h:203
> #14 Messenger::get_random_nonce () at ./src/msg/Messenger.cc:33
> #15 0x00007fffee771dd6 in Messenger::create_client_messenger (cct=0x7fffe80011c0, lname="temp_mon_client") at ./src/msg/Messenger.cc:16
> #16 0x00007fffee82c862 in MonClient::get_monmap_and_config (this=this@entry=0x7fffe77fd080) at /usr/include/c++/8/ext/new_allocator.h:79
> #17 0x00007ffff6fd9e14 in librados::v14_2_0::RadosClient::connect (this=this@entry=0x7fffe805ae80) at ./src/librados/RadosClient.cc:227
> #18 0x00007ffff6f66a0f in _rados_connect (cluster=0x7fffe805ae80) at ./src/librados/librados_c.cc:204
> #19 0x00007ffff71bf0b5 in ?? () from /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so
> #20 0x00007ffff71380ec in ?? () from /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so
> #21 0x00000000004d9850 in _PyObject_FastCallDict (kwargs={}, nargs=1, args=0x7fffe77fd880, callable=<cython_function_or_method at remote 0x7fffed6ae100>)
> at ../Objects/call.c:125
> #22 _PyObject_Call_Prepend (kwargs={}, args=<optimized out>, obj=<optimized out>, callable=<cython_function_or_method at remote 0x7fffed6ae100>) at ../Objects/call.c:904
> #23 method_call (method=<optimized out>, args=<optimized out>, kwargs=<optimized out>, method=<optimized out>, args=<optimized out>, kwargs=<optimized out>)
> at ../Objects/classobject.c:309
> #24 0x00000000005dc4f6 in PyObject_Call (callable=<method at remote 0x7ffff7634dc8>, args=<optimized out>, kwargs=<optimized out>) at ../Objects/call.c:245
> #25 0x000000000054f987 in do_call_core (kwdict={}, callargs=(), func=<method at remote 0x7ffff7634dc8>) at ../Python/ceval.c:4645
> #26 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3191
> #27 0x00000000005d91fc in PyEval_EvalFrameEx (throwflag=0,
> f=Frame 0x7ffff72c6bb8, for file /usr/lib/python3/dist-packages/ceph_argparse.py, line 1458, in run (self=<RadosThread(args=(), kwargs={}, func=<method at remote 0x7ffff7634dc8>, exception=None, _target=None, _name='Thread-3', _args=(...), _kwargs={}, _daemonic=True, _ident=140737077307136, _tstate_lock=<_thread.lock at remote 0x7fffed63af58>, _started=<Event(_cond=<Condition(_lock=<_thread.lock at remote 0x7fffed63ad00>, acquire=<built-in method acquire of _thread.lock object at remote 0x7fffed63ad00>, release=<built-in method release of _thread.lock object at remote 0x7fffed63ad00>, _waiters=<collections.deque at remote 0x7fffed6d5a70>) at remote 0x7fffed533470>, _flag=True) at remote 0x7fffed533438>, _is_stopped=False, _initialized=True, _stderr=<_io.TextIOWrapper at remote 0x7ffff7629708>) at remote 0x7fffed533358>)) at ../Python/ceval.c:547
> #28 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at ../Objects/call.c:283
> --Type <RET> for more, q to quit, c to continue without paging--c
> #29 _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:408
> #30 0x000000000054e7e0 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=0x7fffe77fdb30) at ../Python/ceval.c:4616
> #31 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3110
> #32 0x00000000005d91fc in PyEval_EvalFrameEx (throwflag=0, f=Frame 0x7fffe8000b38, for file /usr/lib/python3.7/threading.py, line 917, in _bootstrap_inner (self=<RadosThread(args=(), kwargs={}, func=<method at remote 0x7ffff7634dc8>, exception=None, _target=None, _name='Thread-3', _args=(...), _kwargs={}, _daemonic=True, _ident=140737077307136, _tstate_lock=<_thread.lock at remote 0x7fffed63af58>, _started=<Event(_cond=<Condition(_lock=<_thread.lock at remote 0x7fffed63ad00>, acquire=<built-in method acquire of _thread.lock object at remote 0x7fffed63ad00>, release=<built-in method release of _thread.lock object at remote 0x7fffed63ad00>, _waiters=<collections.deque at remote 0x7fffed6d5a70>) at remote 0x7fffed533470>, _flag=True) at remote 0x7fffed533438>, _is_stopped=False, _initialized=True, _stderr=<_io.TextIOWrapper at remote 0x7ffff7629708>) at remote 0x7fffed533358>)) at ../Python/ceval.c:547
> #33 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at ../Objects/call.c:283
> #34 _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:408
> #35 0x000000000054e7e0 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=0x7fffe77fdcc0) at ../Python/ceval.c:4616
> #36 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3110
> #37 0x00000000005da536 in PyEval_EvalFrameEx (throwflag=0, f=Frame 0x7fffed538048, for file /usr/lib/python3.7/threading.py, line 885, in _bootstrap (self=<RadosThread(args=(), kwargs={}, func=<method at remote 0x7ffff7634dc8>, exception=None, _target=None, _name='Thread-3', _args=(...), _kwargs={}, _daemonic=True, _ident=140737077307136, _tstate_lock=<_thread.lock at remote 0x7fffed63af58>, _started=<Event(_cond=<Condition(_lock=<_thread.lock at remote 0x7fffed63ad00>, acquire=<built-in method acquire of _thread.lock object at remote 0x7fffed63ad00>, release=<built-in method release of _thread.lock object at remote 0x7fffed63ad00>, _waiters=<collections.deque at remote 0x7fffed6d5a70>) at remote 0x7fffed533470>, _flag=True) at remote 0x7fffed533438>, _is_stopped=False, _initialized=True, _stderr=<_io.TextIOWrapper at remote 0x7ffff7629708>) at remote 0x7fffed533358>)) at ../Python/ceval.c:547
> #38 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at ../Objects/call.c:283
> #39 _PyFunction_FastCallDict (func=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwargs=<optimized out>) at ../Objects/call.c:322
> #40 0x00000000004d97e2 in _PyObject_FastCallDict (kwargs=0x0, nargs=1, args=0x7fffe77fde00, callable=<function at remote 0x7ffff73ead90>) at ../Objects/call.c:98
> #41 _PyObject_Call_Prepend (kwargs=0x0, args=<optimized out>, obj=<RadosThread(args=(), kwargs={}, func=<method at remote 0x7ffff7634dc8>, exception=None, _target=None, _name='Thread-3', _args=(...), _kwargs={}, _daemonic=True, _ident=140737077307136, _tstate_lock=<_thread.lock at remote 0x7fffed63af58>, _started=<Event(_cond=<Condition(_lock=<_thread.lock at remote 0x7fffed63ad00>, acquire=<built-in method acquire of _thread.lock object at remote 0x7fffed63ad00>, release=<built-in method release of _thread.lock object at remote 0x7fffed63ad00>, _waiters=<collections.deque at remote 0x7fffed6d5a70>) at remote 0x7fffed533470>, _flag=True) at remote 0x7fffed533438>, _is_stopped=False, _initialized=True, _stderr=<_io.TextIOWrapper at remote 0x7ffff7629708>) at remote 0x7fffed533358>, callable=<function at remote 0x7ffff73ead90>) at ../Objects/call.c:904
> #42 method_call (method=<optimized out>, args=<optimized out>, kwargs=<optimized out>, method=<optimized out>, args=<optimized out>, kwargs=<optimized out>) at ../Objects/classobject.c:309
> #43 0x00000000005dc4f6 in PyObject_Call (callable=<method at remote 0x7ffff7634b08>, args=<optimized out>, kwargs=<optimized out>) at ../Objects/call.c:245
> #44 0x0000000000617b63 in t_bootstrap (boot_raw=boot_raw@entry=0x7fffed63ae40) at ../Modules/_threadmodule.c:994
> #45 0x000000000062dfe4 in pythread_wrapper (arg=<optimized out>) at ../Python/thread_pthread.h:174
> #46 0x00007ffff7f6efa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
> #47 0x00007ffff7ab54cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
> ```
> _______________________________________________
> Dev mailing list -- dev@xxxxxxx
> To unsubscribe send an email to dev-leave@xxxxxxx
--
Cheers,
Brad
_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx
Regards
Kefu Chai
Kefu Chai
_______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
