downgrade to 5.10.0-0.bpo.4-amd64 where performed today, but shows the same error.
ceph -s and the mon can not start, but the rest of the system is working fin.
ceph -s and the mon can not start, but the rest of the system is working fin.
no change to the entropy_avail: 3474
Am Di., 17. Aug. 2021 um 08:23 Uhr schrieb Brad Hubbard <bhubbard@xxxxxxxxxx>:
Let us know how you go with the older kernel that is known to work.
On Tue, Aug 17, 2021 at 12:31 AM Links 2004 <links2004.code@xxxxxxxxx> wrote:
>
> Only one of the servers is failing of the 4 that have access to the cluster.
> since we where in the mid of an update the kernel and os versions are bit all over the place, but here we go:
>
> bad:
> Debian 10 (buster) - 5.10.0-0.bpo.8-amd64 #1 SMP Debian 5.10.46-2~bpo10+1 (2021-07-22) x86_64 GNU/Linux
>
> good:
> Debian 10 (buster) - 5.10.0-0.bpo.4-amd64 #1 SMP Debian 5.10.19-1~bpo10+1 (2021-03-13) x86_64 GNU/Linux
> Debian 11 (bullseye) - 5.11.22-3-pve #1 SMP PVE 5.11.22-6 (Wed, 28 Jul 2021 10:51:12 +0200) x86_64 GNU/Linux
> Debian 11 (bullseye) - 5.10.0-8-amd64 #1 SMP Debian 5.10.46-3 (2021-07-28) x86_64 GNU/Linux
>
> the proxmox one is at:
> ceph version 16.2.5 (9b9dd76e12f1907fe5dcc0c1fadadbb784022a42) pacific (stable)
>
> the rest is at:
> ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
>
> all 3 other servers are running all ceph commands fine.
> and 2 of them have mon running fine too.
> the 3 mon where on the bad one which is not starting with the same error.
>
> we stopped the bullseye update since we may kill a second server which will be bad for the cluster ;)
>
> as soon its as possible we will try going back to 5.10.0-0.bpo.4-amd64 on the bad one,
> since this is known to work on one of the other servers.
>
> the hardware of the server can not really be compared since this is a testing / dev cluster and composed of old hardware.
> And we tried the update there first to not kill the production environment.
>
>
> Am So., 15. Aug. 2021 um 00:13 Uhr schrieb Brad Hubbard <bhubbard@xxxxxxxxxx>:
>>
>> On Sat, Aug 14, 2021 at 7:13 PM Links 2004 <links2004.code@xxxxxxxxx> wrote:
>> >
>> > no kernel update in the last time, and its a server so no keyboard etc, but the entropy_avail looks good and its in the same range (-+80) as the other server.
>> > dmesg | grep random has no results.
>> > # cat /proc/sys/kernel/random/entropy_avail
>> > 3547
>> >
>> > all servers run
>> > ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
>>
>> Can you give us some more information?
>>
>> Do all of your servers fail all of the time or do some fail some of the time?
>>
>> What do the ones that fail have in common and how do they differ from
>> the ones that do not fail?
>>
>> If you have the option to try alternative kernels on the servers that
>> exhibit this behavior I'd suggest that might be a good next step.
>>
>> >
>> >
>> >
>> >
>> > Am Sa., 14. Aug. 2021 um 10:55 Uhr schrieb kefu chai <tchaikov@xxxxxxxxx>:
>> >>
>> >>
>> >>
>> >> Brad Hubbard <bhubbard@xxxxxxxxxx>于2021年8月14日 周六06:11写道：
>> >>>
>> >>> On Sat, Aug 14, 2021 at 4:06 AM Links 2004 <links2004.code@xxxxxxxxx> wrote:
>> >>> >
>> >>> >
>> >>> > Hi,
>> >>> >
>> >>> > we are currently facing a strange problem on on of our ceph nodes.
>> >>> > it is not possible to call `ceph -s` or start a mgr with out a 'std::runtime_error'.
>> >>> >
>> >>> > find below the error message and a gdb backtrace with debug symbols.
>> >>> > hope this helps to understand the problem and point us in the correct direction.
>> >>> >
>> >>> > Thanks
>> >>> >
>> >>> > Markus
>> >>> >
>> >>> >
>> >>> > OS: Debian buster
>> >>> > kernel: 5.10.0-0.bpo.8-amd64 #1 SMP Debian 5.10.46-2~bpo10+1 (2021-07-22) x86_64 GNU/Linux
>> >>> >
>> >>> > ```
>> >>> > # ceph -v
>> >>> > ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
>> >>> > ```
>> >>> >
>> >>> > ```
>> >>> > # ceph -s
>> >>> > terminate called after throwing an instance of 'std::runtime_error'
>> >>> > what(): random_device::__x86_rdrand(void)
>> >>> > Aborted
>> >>> > ```
>> >>>
>> >>> Did this issue coincide with a kernel upgrade?
>> >>>
>> >>> Can you try and generate a lot of entropy on the system and see if the
>> >>> issue goes away?
>> >>>
>> >>> Also check the output of 'dmesg | grep random' to see if that offers any clues.
>> >>
>> >>
>> >> I feel the same. Looks likely that the kernel did not have enough entropy by then. Is the system not connected to a keyboard? Or it was just booted? If that’s the case, probably you could wait a while before try to launch mgr or use the ceph command line utility.
>> >>>
>> >>>
>> >>>
>> >>> >
>> >>> > ```
>> >>> > # gdb --args /usr/bin/python3.7 /usr/bin/ceph -s
>> >>> > GNU gdb (Debian 8.2.1-2+b3) 8.2.1
>> >>> > Copyright (C) 2018 Free Software Foundation, Inc.
>> >>> > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>> >>> > This is free software: you are free to change and redistribute it.
>> >>> > There is NO WARRANTY, to the extent permitted by law.
>> >>> > Type "show copying" and "show warranty" for details.
>> >>> > This GDB was configured as "x86_64-linux-gnu".
>> >>> > Type "show configuration" for configuration details.
>> >>> > For bug reporting instructions, please see:
>> >>> > <http://www.gnu.org/software/gdb/bugs/>.
>> >>> > Find the GDB manual and other documentation resources online at:
>> >>> > <http://www.gnu.org/software/gdb/documentation/>.
>> >>> >
>> >>> > For help, type "help".
>> >>> > Type "apropos word" to search for commands related to "word"...
>> >>> > Reading symbols from /usr/bin/python3.7...Reading symbols from /usr/lib/debug/.build-id/99/21c75e6930d3e9d9fa8c942aca9dc4500bb65f.debug...done.
>> >>> > done.
>> >>> > (gdb) run
>> >>> > Starting program: /usr/bin/python3.7 /usr/bin/ceph -s
>> >>> > [Thread debugging using libthread_db enabled]
>> >>> > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> >>> >
>> >>> > [New Thread 0x7fffed515700 (LWP 30323)]
>> >>> > [New Thread 0x7fffe7fff700 (LWP 30324)]
>> >>> > [New Thread 0x7fffe77fe700 (LWP 30325)]
>> >>> > [Thread 0x7fffed515700 (LWP 30323) exited]
>> >>> > [New Thread 0x7fffed515700 (LWP 30326)]
>> >>> > [Thread 0x7fffe7fff700 (LWP 30324) exited]
>> >>> > terminate called after throwing an instance of 'std::runtime_error'
>> >>> > what(): random_device::__x86_rdrand(void)
>> >>> >
>> >>> > Thread 4 "python3.7" received signal SIGABRT, Aborted.
>> >>> > [Switching to Thread 0x7fffe77fe700 (LWP 30325)]
>> >>> > __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
>> >>> > 50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
>> >>> > (gdb)
>> >>> > (gdb) bt
>> >>> > #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
>> >>> > #1 0x00007ffff79de535 in __GI_abort () at abort.c:79
>> >>> > #2 0x00007fffeddb8983 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
>> >>> > #3 0x00007fffeddbe8c6 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
>> >>> > #4 0x00007fffeddbe901 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
>> >>> > #5 0x00007fffeddbeb34 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
>> >>> > #6 0x00007fffeddba8b7 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
>> >>> > #7 0x00007fffedde6e86 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
>> >>> > #8 0x00007fffedde6fd2 in std::random_device::_M_getval() () from /lib/x86_64-linux-gnu/libstdc++.so.6
>> >>> > #9 0x00007fffee540ffc in std::random_device::operator() (this=0x7fffe77fba60) at /usr/include/c++/8/bits/random.h:1611
>> >>> > #10 ceph::util::version_1_0_3::detail::randomize_rng<std::linear_congruential_engine<unsigned long, 16807ul, 0ul, 2147483647ul> > () at ./src/include/random.h:120
>> >>> > #11 0x00007fffee54112f in ceph::util::version_1_0_3::detail::engine<std::linear_congruential_engine<unsigned long, 16807ul, 0ul, 2147483647ul> > ()
>> >>> > at /usr/include/c++/8/new:169
>> >>> > #12 0x00007fffee772911 in ceph::util::version_1_0_3::detail::generate_random_number<unsigned long, std::uniform_int_distribution<unsigned long>, std::linear_congruential_engine<unsigned long, 16807ul, 0ul, 2147483647ul> > (min=min@entry=0, max=max@entry=18446744073709551615) at ./src/include/random.h:170
>> >>> > #13 0x00007fffee7718fe in ceph::util::version_1_0_3::generate_random_number<unsigned long, std::uniform_int_distribution<unsigned long>, std::linear_congruential_engine<unsigned long, 16807ul, 0ul, 2147483647ul> > () at ./src/include/random.h:203
>> >>> > #14 Messenger::get_random_nonce () at ./src/msg/Messenger.cc:33
>> >>> > #15 0x00007fffee771dd6 in Messenger::create_client_messenger (cct=0x7fffe80011c0, lname="temp_mon_client") at ./src/msg/Messenger.cc:16
>> >>> > #16 0x00007fffee82c862 in MonClient::get_monmap_and_config (this=this@entry=0x7fffe77fd080) at /usr/include/c++/8/ext/new_allocator.h:79
>> >>> > #17 0x00007ffff6fd9e14 in librados::v14_2_0::RadosClient::connect (this=this@entry=0x7fffe805ae80) at ./src/librados/RadosClient.cc:227
>> >>> > #18 0x00007ffff6f66a0f in _rados_connect (cluster=0x7fffe805ae80) at ./src/librados/librados_c.cc:204
>> >>> > #19 0x00007ffff71bf0b5 in ?? () from /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so
>> >>> > #20 0x00007ffff71380ec in ?? () from /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so
>> >>> > #21 0x00000000004d9850 in _PyObject_FastCallDict (kwargs={}, nargs=1, args=0x7fffe77fd880, callable=<cython_function_or_method at remote 0x7fffed6ae100>)
>> >>> > at ../Objects/call.c:125
>> >>> > #22 _PyObject_Call_Prepend (kwargs={}, args=<optimized out>, obj=<optimized out>, callable=<cython_function_or_method at remote 0x7fffed6ae100>) at ../Objects/call.c:904
>> >>> > #23 method_call (method=<optimized out>, args=<optimized out>, kwargs=<optimized out>, method=<optimized out>, args=<optimized out>, kwargs=<optimized out>)
>> >>> > at ../Objects/classobject.c:309
>> >>> > #24 0x00000000005dc4f6 in PyObject_Call (callable=<method at remote 0x7ffff7634dc8>, args=<optimized out>, kwargs=<optimized out>) at ../Objects/call.c:245
>> >>> > #25 0x000000000054f987 in do_call_core (kwdict={}, callargs=(), func=<method at remote 0x7ffff7634dc8>) at ../Python/ceval.c:4645
>> >>> > #26 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3191
>> >>> > #27 0x00000000005d91fc in PyEval_EvalFrameEx (throwflag=0,
>> >>> > f=Frame 0x7ffff72c6bb8, for file /usr/lib/python3/dist-packages/ceph_argparse.py, line 1458, in run (self=<RadosThread(args=(), kwargs={}, func=<method at remote 0x7ffff7634dc8>, exception=None, _target=None, _name='Thread-3', _args=(...), _kwargs={}, _daemonic=True, _ident=140737077307136, _tstate_lock=<_thread.lock at remote 0x7fffed63af58>, _started=<Event(_cond=<Condition(_lock=<_thread.lock at remote 0x7fffed63ad00>, acquire=<built-in method acquire of _thread.lock object at remote 0x7fffed63ad00>, release=<built-in method release of _thread.lock object at remote 0x7fffed63ad00>, _waiters=<collections.deque at remote 0x7fffed6d5a70>) at remote 0x7fffed533470>, _flag=True) at remote 0x7fffed533438>, _is_stopped=False, _initialized=True, _stderr=<_io.TextIOWrapper at remote 0x7ffff7629708>) at remote 0x7fffed533358>)) at ../Python/ceval.c:547
>> >>> > #28 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at ../Objects/call.c:283
>> >>> > --Type <RET> for more, q to quit, c to continue without paging--c
>> >>> > #29 _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:408
>> >>> > #30 0x000000000054e7e0 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=0x7fffe77fdb30) at ../Python/ceval.c:4616
>> >>> > #31 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3110
>> >>> > #32 0x00000000005d91fc in PyEval_EvalFrameEx (throwflag=0, f=Frame 0x7fffe8000b38, for file /usr/lib/python3.7/threading.py, line 917, in _bootstrap_inner (self=<RadosThread(args=(), kwargs={}, func=<method at remote 0x7ffff7634dc8>, exception=None, _target=None, _name='Thread-3', _args=(...), _kwargs={}, _daemonic=True, _ident=140737077307136, _tstate_lock=<_thread.lock at remote 0x7fffed63af58>, _started=<Event(_cond=<Condition(_lock=<_thread.lock at remote 0x7fffed63ad00>, acquire=<built-in method acquire of _thread.lock object at remote 0x7fffed63ad00>, release=<built-in method release of _thread.lock object at remote 0x7fffed63ad00>, _waiters=<collections.deque at remote 0x7fffed6d5a70>) at remote 0x7fffed533470>, _flag=True) at remote 0x7fffed533438>, _is_stopped=False, _initialized=True, _stderr=<_io.TextIOWrapper at remote 0x7ffff7629708>) at remote 0x7fffed533358>)) at ../Python/ceval.c:547
>> >>> > #33 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at ../Objects/call.c:283
>> >>> > #34 _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:408
>> >>> > #35 0x000000000054e7e0 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=0x7fffe77fdcc0) at ../Python/ceval.c:4616
>> >>> > #36 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3110
>> >>> > #37 0x00000000005da536 in PyEval_EvalFrameEx (throwflag=0, f=Frame 0x7fffed538048, for file /usr/lib/python3.7/threading.py, line 885, in _bootstrap (self=<RadosThread(args=(), kwargs={}, func=<method at remote 0x7ffff7634dc8>, exception=None, _target=None, _name='Thread-3', _args=(...), _kwargs={}, _daemonic=True, _ident=140737077307136, _tstate_lock=<_thread.lock at remote 0x7fffed63af58>, _started=<Event(_cond=<Condition(_lock=<_thread.lock at remote 0x7fffed63ad00>, acquire=<built-in method acquire of _thread.lock object at remote 0x7fffed63ad00>, release=<built-in method release of _thread.lock object at remote 0x7fffed63ad00>, _waiters=<collections.deque at remote 0x7fffed6d5a70>) at remote 0x7fffed533470>, _flag=True) at remote 0x7fffed533438>, _is_stopped=False, _initialized=True, _stderr=<_io.TextIOWrapper at remote 0x7ffff7629708>) at remote 0x7fffed533358>)) at ../Python/ceval.c:547
>> >>> > #38 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at ../Objects/call.c:283
>> >>> > #39 _PyFunction_FastCallDict (func=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwargs=<optimized out>) at ../Objects/call.c:322
>> >>> > #40 0x00000000004d97e2 in _PyObject_FastCallDict (kwargs=0x0, nargs=1, args=0x7fffe77fde00, callable=<function at remote 0x7ffff73ead90>) at ../Objects/call.c:98
>> >>> > #41 _PyObject_Call_Prepend (kwargs=0x0, args=<optimized out>, obj=<RadosThread(args=(), kwargs={}, func=<method at remote 0x7ffff7634dc8>, exception=None, _target=None, _name='Thread-3', _args=(...), _kwargs={}, _daemonic=True, _ident=140737077307136, _tstate_lock=<_thread.lock at remote 0x7fffed63af58>, _started=<Event(_cond=<Condition(_lock=<_thread.lock at remote 0x7fffed63ad00>, acquire=<built-in method acquire of _thread.lock object at remote 0x7fffed63ad00>, release=<built-in method release of _thread.lock object at remote 0x7fffed63ad00>, _waiters=<collections.deque at remote 0x7fffed6d5a70>) at remote 0x7fffed533470>, _flag=True) at remote 0x7fffed533438>, _is_stopped=False, _initialized=True, _stderr=<_io.TextIOWrapper at remote 0x7ffff7629708>) at remote 0x7fffed533358>, callable=<function at remote 0x7ffff73ead90>) at ../Objects/call.c:904
>> >>> > #42 method_call (method=<optimized out>, args=<optimized out>, kwargs=<optimized out>, method=<optimized out>, args=<optimized out>, kwargs=<optimized out>) at ../Objects/classobject.c:309
>> >>> > #43 0x00000000005dc4f6 in PyObject_Call (callable=<method at remote 0x7ffff7634b08>, args=<optimized out>, kwargs=<optimized out>) at ../Objects/call.c:245
>> >>> > #44 0x0000000000617b63 in t_bootstrap (boot_raw=boot_raw@entry=0x7fffed63ae40) at ../Modules/_threadmodule.c:994
>> >>> > #45 0x000000000062dfe4 in pythread_wrapper (arg=<optimized out>) at ../Python/thread_pthread.h:174
>> >>> > #46 0x00007ffff7f6efa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
>> >>> > #47 0x00007ffff7ab54cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
>> >>> > ```
>> >>> > _______________________________________________
>> >>> > Dev mailing list -- dev@xxxxxxx
>> >>> > To unsubscribe send an email to dev-leave@xxxxxxx
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Cheers,
>> >>> Brad
>> >>>
>> >>> _______________________________________________
>> >>> Dev mailing list -- dev@xxxxxxx
>> >>> To unsubscribe send an email to dev-leave@xxxxxxx
>> >>
>> >> --
>> >> Regards
>> >> Kefu Chai
>>
>>
>>
>> --
>> Cheers,
>> Brad
>>
--
Cheers,
Brad
_______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
