Hi,
Currently I have a cluster of 2 nodes on two raspberrypi 3 devices.
pi1 is admin/mon/mgr/osd
pi2 is osd
I am currently trying to run radosgw on pi2. I have failed multiple times
/var/lib/ceph/radosgw# /usr/bin/radosgw -f --cluster ceph --name
client.rgw.pi2 --setuser ceph --setgroup ceph
Thread::try_create(): pthread_create failed with error
11/tmp/release/Raspbian/WORKDIR/ceph-12.2.9-38-gaeeb23362d/src/common/Thread.cc:
In function 'void Thread::create(const char *, size_t)' thread
71114000 time 2020-01-22 14:58:13.793803
/tmp/release/Raspbian/WORKDIR/ceph-12.2.9-38-gaeeb23362d/src/common/Thread.cc:
152: FAILED assert(ret == 0)
ceph version 12.2.9-38-gaeeb23362d
(aeeb23362d54d07a8770c0007660d43d67909f21) luminous (stable)
1: (Thread::create(char const*, unsigned int)+0xcc) [0x726ba984]
2: (SafeTimer::init()+0x164) [0x7243ca84]
3: (RGWRealmReloader::RGWRealmReloader(RGWRados*&,
std::map<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
std::less<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > >,
std::allocator<std::pair<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const,
std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > > > >&, RGWRealmReloader::Pauser*)+0x104)
[0x28c108]
4: (main()+0x416c) [0x161bb8]
5: (__libc_start_main()+0x114) [0x71b86678]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
2020-01-22 14:58:13.796912 71114000 -1
/tmp/release/Raspbian/WORKDIR/ceph-12.2.9-38-gaeeb23362d/src/common/Thread.cc:
In function 'void Thread::create(const char *, size_t)' thread
71114000 time 2020-01-22 14:58:13.793803
/tmp/release/Raspbian/WORKDIR/ceph-12.2.9-38-gaeeb23362d/src/common/Thread.cc:
152: FAILED assert(ret == 0)
ceph version 12.2.9-38-gaeeb23362d
(aeeb23362d54d07a8770c0007660d43d67909f21) luminous (stable)
1: (Thread::create(char const*, unsigned int)+0xcc) [0x726ba984]
2: (SafeTimer::init()+0x164) [0x7243ca84]
3: (RGWRealmReloader::RGWRealmReloader(RGWRados*&,
std::map<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
std::less<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > >,
std::allocator<std::pair<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const,
std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > > > >&, RGWRealmReloader::Pauser*)+0x104)
[0x28c108]
4: (main()+0x416c) [0x161bb8]
5: (__libc_start_main()+0x114) [0x71b86678]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
0> 2020-01-22 14:58:13.796912 71114000 -1
/tmp/release/Raspbian/WORKDIR/ceph-12.2.9-38-gaeeb23362d/src/common/Thread.cc:
In function 'void Thread::create(const char *, size_t)' thread
71114000 time 2020-01-22 14:58:13.793803
/tmp/release/Raspbian/WORKDIR/ceph-12.2.9-38-gaeeb23362d/src/common/Thread.cc:
152: FAILED assert(ret == 0)
ceph version 12.2.9-38-gaeeb23362d
(aeeb23362d54d07a8770c0007660d43d67909f21) luminous (stable)
1: (Thread::create(char const*, unsigned int)+0xcc) [0x726ba984]
2: (SafeTimer::init()+0x164) [0x7243ca84]
3: (RGWRealmReloader::RGWRealmReloader(RGWRados*&,
std::map<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
std::less<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > >,
std::allocator<std::pair<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const,
std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > > > >&, RGWRealmReloader::Pauser*)+0x104)
[0x28c108]
4: (main()+0x416c) [0x161bb8]
5: (__libc_start_main()+0x114) [0x71b86678]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
*** Caught signal (Aborted) **
in thread 71114000 thread_name:radosgw
ceph version 12.2.9-38-gaeeb23362d
(aeeb23362d54d07a8770c0007660d43d67909f21) luminous (stable)
1: /usr/bin/radosgw() [0x1d3f94]
2: (__default_sa_restorer()+0) [0x71b9c6b0]
3: (gsignal()+0xa0) [0x71b9b45c]
2020-01-22 14:58:13.804425 71114000 -1 *** Caught signal (Aborted) **
in thread 71114000 thread_name:radosgw
ceph version 12.2.9-38-gaeeb23362d
(aeeb23362d54d07a8770c0007660d43d67909f21) luminous (stable)
1: /usr/bin/radosgw() [0x1d3f94]
2: (__default_sa_restorer()+0) [0x71b9c6b0]
3: (gsignal()+0xa0) [0x71b9b45c]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
0> 2020-01-22 14:58:13.804425 71114000 -1 *** Caught signal (Aborted) **
in thread 71114000 thread_name:radosgw
ceph version 12.2.9-38-gaeeb23362d
(aeeb23362d54d07a8770c0007660d43d67909f21) luminous (stable)
1: /usr/bin/radosgw() [0x1d3f94]
2: (__default_sa_restorer()+0) [0x71b9c6b0]
3: (gsignal()+0xa0) [0x71b9b45c]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
and this is from the log
-27> 2020-01-22 14:58:13.722050 668fd400 5 process_single_shard():
failed to acquire lock on obj_delete_at_hint.0000000001
-26> 2020-01-22 14:58:13.722550 668fd400 1 --
192.168.0.178:0/376421854 --> 192.168.0.130:6800/20003 --
osd_op(unknown.0.0:57 4.6
4:703a9e16:::obj_delete_at_hint.0000000002:head [call lock.lock] snapc
0=[] ondisk+write+known_if_redirected e21) v8 -- 0x67905478 con 0
-25> 2020-01-22 14:58:13.728483 6dafd400 5 --
192.168.0.178:0/376421854 >> 192.168.0.130:6800/20003 conn(0x2246100
:-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=27 cs=1 l=1). rx
osd.0 seq 16 0x6c7071a8 osd_op_reply(55 reshard.0000000000 [call] v0'0
uv78 ondisk = 0) v8
-24> 2020-01-22 14:58:13.728609 6dafd400 1 --
192.168.0.178:0/376421854 <== osd.0 192.168.0.130:6800/20003 16 ====
osd_op_reply(55 reshard.0000000000 [call] v0'0 uv78 ondisk = 0) v8
==== 162+0+11 (1215297641 0 1993775135) 0x6c7071a8 con 0x2246100
-23> 2020-01-22 14:58:13.729114 518d3400 1 --
192.168.0.178:0/376421854 --> 192.168.0.130:6800/20003 --
osd_op(unknown.0.0:58 4.7 4:e9187ab8:reshard::reshard.0000000000:head
[call lock.unlock] snapc 0=[] ondisk+write+known_if_redirected e21) v8
-- 0x67905478 con 0
-22> 2020-01-22 14:58:13.730801 6dafd400 5 --
192.168.0.178:0/376421854 >> 192.168.0.130:6800/20003 conn(0x2246100
:-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=27 cs=1 l=1). rx
osd.0 seq 17 0x6c707568 osd_op_reply(56 gc.1 [call] v0'0 uv61 ondisk =
0) v8
-21> 2020-01-22 14:58:13.730909 6dafd400 1 --
192.168.0.178:0/376421854 <== osd.0 192.168.0.130:6800/20003 17 ====
osd_op_reply(56 gc.1 [call] v0'0 uv61 ondisk = 0) v8 ==== 148+0+15
(78062353 0 682959608) 0x6c707568 con 0x2246100
-20> 2020-01-22 14:58:13.731370 670fe400 1 --
192.168.0.178:0/376421854 --> 192.168.0.130:6800/20003 --
osd_op(unknown.0.0:59 4.0 4:1e85421f:gc::gc.1:head [call lock.unlock]
snapc 0=[] ondisk+write+known_if_redirected e21) v8 -- 0x22985e8 con 0
-19> 2020-01-22 14:58:13.752731 6dafd400 5 --
192.168.0.178:0/376421854 >> 192.168.0.130:6800/20003 conn(0x2246100
:-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=27 cs=1 l=1). rx
osd.0 seq 18 0x6c707568 osd_op_reply(57 obj_delete_at_hint.0000000002
[call] v21'19 uv19 ondisk = 0) v8
-18> 2020-01-22 14:58:13.752825 6dafd400 1 --
192.168.0.178:0/376421854 <== osd.0 192.168.0.130:6800/20003 18 ====
osd_op_reply(57 obj_delete_at_hint.0000000002 [call] v21'19 uv19
ondisk = 0) v8 ==== 173+0+0 (3595957238 0 0) 0x6c707568 con 0x2246100
-17> 2020-01-22 14:58:13.753145 668fd400 1 --
192.168.0.178:0/376421854 --> 192.168.0.130:6800/20003 --
osd_op(unknown.0.0:60 4.6
4:703a9e16:::obj_delete_at_hint.0000000002:head [call timeindex.list]
snapc 0=[] ondisk+read+known_if_redirected e21) v8 -- 0x67905478 con 0
-16> 2020-01-22 14:58:13.756866 6dafd400 5 --
192.168.0.178:0/376421854 >> 192.168.0.130:6800/20003 conn(0x2246100
:-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=27 cs=1 l=1). rx
osd.0 seq 19 0x6c707568 osd_op_reply(58 reshard.0000000000 [call]
v21'80 uv80 ondisk = 0) v8
-15> 2020-01-22 14:58:13.756940 6dafd400 1 --
192.168.0.178:0/376421854 <== osd.0 192.168.0.130:6800/20003 19 ====
osd_op_reply(58 reshard.0000000000 [call] v21'80 uv80 ondisk = 0) v8
==== 162+0+0 (1541160059 0 0) 0x6c707568 con 0x2246100
-14> 2020-01-22 14:58:13.757289 518d3400 1 --
192.168.0.178:0/376421854 --> 192.168.0.130:6800/20003 --
osd_op(unknown.0.0:61 4.4 4:2c0c0c23:reshard::reshard.0000000001:head
[call lock.lock] snapc 0=[] ondisk+write+known_if_redirected e21) v8
-- 0x67905478 con 0
-13> 2020-01-22 14:58:13.768019 6dafd400 5 --
192.168.0.178:0/376421854 >> 192.168.0.130:6800/20003 conn(0x2246100
:-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=27 cs=1 l=1). rx
osd.0 seq 20 0x6c707d50 osd_op_reply(60 obj_delete_at_hint.0000000002
[call] v0'0 uv19 ondisk = 0) v8
-12> 2020-01-22 14:58:13.768100 6dafd400 1 --
192.168.0.178:0/376421854 <== osd.0 192.168.0.130:6800/20003 20 ====
osd_op_reply(60 obj_delete_at_hint.0000000002 [call] v0'0 uv19 ondisk
= 0) v8 ==== 173+0+15 (1944631971 0 2149983739) 0x6c707d50 con
0x2246100
-11> 2020-01-22 14:58:13.768530 668fd400 1 --
192.168.0.178:0/376421854 --> 192.168.0.130:6800/20003 --
osd_op(unknown.0.0:62 4.6
4:703a9e16:::obj_delete_at_hint.0000000002:head [call lock.unlock]
snapc 0=[] ondisk+write+known_if_redirected e21) v8 -- 0x67905478 con
0
-10> 2020-01-22 14:58:13.768558 6dafd400 5 --
192.168.0.178:0/376421854 >> 192.168.0.130:6800/20003 conn(0x2246100
:-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=27 cs=1 l=1). rx
osd.0 seq 21 0x6c707d50 osd_op_reply(59 gc.1 [call] v21'62 uv62 ondisk
= 0) v8
-9> 2020-01-22 14:58:13.768611 6dafd400 1 --
192.168.0.178:0/376421854 <== osd.0 192.168.0.130:6800/20003 21 ====
osd_op_reply(59 gc.1 [call] v21'62 uv62 ondisk = 0) v8 ==== 148+0+0
(3513289116 0 0) 0x6c707d50 con 0x2246100
-8> 2020-01-22 14:58:13.768849 670fe400 1 --
192.168.0.178:0/376421854 --> 192.168.0.130:6800/20003 --
osd_op(unknown.0.0:63 4.3 4:d242335b:gc::gc.2:head [call lock.lock]
snapc 0=[] ondisk+write+known_if_redirected e21) v8 -- 0x2290678 con 0
-7> 2020-01-22 14:58:13.773378 71114000 0 starting handler: civetweb
-6> 2020-01-22 14:58:13.791838 71114000 0 civetweb: 0x691678:
Cannot start worker thread 173: error 12
-5> 2020-01-22 14:58:13.793488 71114000 1 mgrc
service_daemon_register rgw.pi2 metadata
{arch=armv7l,ceph_version=ceph version 12.2.9-38-gaeeb23362d
(aeeb23362d54d07a8770c0007660d43d67909f21) luminous (stable),cpu=ARMv7
Processor rev 4 (v7l),distro=raspbian,distro_description=Raspbian
GNU/Linux 9 (stretch),distro_version=9,frontend_config#0=civetweb
port=7480,frontend_type#0=civetweb,hostname=pi2,kernel_description=#1200
SMP Tue Feb 12 20:27:48 GMT
2019,kernel_version=4.14.98-v7+,mem_swap_kb=102396,mem_total_kb=949448,num_handles=1,os=Linux,pid=19097,zone_id=e5e12905-96a2-4afd-bf15-416eed0d8cc8,zone_name=default,zonegroup_id=08fce0e1-17d5-4809-ac73-a8bd3104f967,zonegroup_name=default}
-4> 2020-01-22 14:58:13.793651 71114000 1 --
192.168.0.178:0/376421854 --> 192.168.0.130:6804/29619 --
mgropen(rgw.pi2 daemon) v2 -- 0x2312808 con 0
-3> 2020-01-22 14:58:13.795526 6dafd400 5 --
192.168.0.178:0/376421854 >> 192.168.0.130:6800/20003 conn(0x2246100
:-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=27 cs=1 l=1). rx
osd.0 seq 22 0x6c707568 osd_op_reply(61 reshard.0000000001 [call]
v21'29 uv29 ondisk = 0) v8
-2> 2020-01-22 14:58:13.795625 6dafd400 1 --
192.168.0.178:0/376421854 <== osd.0 192.168.0.130:6800/20003 22 ====
osd_op_reply(61 reshard.0000000001 [call] v21'29 uv29 ondisk = 0) v8
==== 162+0+0 (2192962868 0 0) 0x6c707568 con 0x2246100
-1> 2020-01-22 14:58:13.795994 518d3400 1 --
192.168.0.178:0/376421854 --> 192.168.0.130:6800/20003 --
osd_op(unknown.0.0:64 4.4 4:2c0c0c23:reshard::reshard.0000000001:head
[call rgw.reshard_list] snapc 0=[] ondisk+read+known_if_redirected
e21) v8 -- 0x67905478 con 0
0> 2020-01-22 14:58:13.796912 71114000 -1
/tmp/release/Raspbian/WORKDIR/ceph-12.2.9-38-gaeeb23362d/src/common/Thread.cc:
In function 'void Thread::create(const char *, size_t)' thread
71114000 time 2020-01-22 14:58:13.793803
/tmp/release/Raspbian/WORKDIR/ceph-12.2.9-38-gaeeb23362d/src/common/Thread.cc:
152: FAILED assert(ret == 0)
ceph version 12.2.9-38-gaeeb23362d
(aeeb23362d54d07a8770c0007660d43d67909f21) luminous (stable)
1: (Thread::create(char const*, unsigned int)+0xcc) [0x726ba984]
2: (SafeTimer::init()+0x164) [0x7243ca84]
3: (RGWRealmReloader::RGWRealmReloader(RGWRados*&,
std::map<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
std::less<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > >,
std::allocator<std::pair<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const,
std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > > > >&, RGWRealmReloader::Pauser*)+0x104)
[0x28c108]
4: (main()+0x416c) [0x161bb8]
5: (__libc_start_main()+0x114) [0x71b86678]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
