Re: all oas crush on start

[Vladislav Gorbunov <vadikgo@xxxxxxxxx>]

Sympthoms like on http://tracker.ceph.com/issues/4699

all OSDs the process ceph-osd crash with segfault

If i stop MONs daemons then i can start OSDs but if i start MONs back
then die all OSDs again.

more detailed log:
     0> 2013-07-15 16:42:05.001242 7ffe5a6fc700 -1 *** Caught signal
(Segmentation fault) **
 in thread 7ffe5a6fc700

 ceph version 0.61.4 (1669132fcfc27d0c0b5e5bb93ade59d147e23404)
 1: /usr/bin/ceph-osd() [0x790e5a]
 2: (()+0xfcb0) [0x7ffe6b729cb0]
 3: /usr/bin/ceph-osd() [0x893879]
 4: (crush_do_rule()+0x1e5) [0x894065]
 5: (CrushWrapper::do_rule(int, int, std::vector<int,
std::allocator<int> >&, int, std::vector<unsigned int,
std::allocator<unsigned int> > const&) const+0x7a) [0x81b2ba]
 6: (OSDMap::_pg_to_osds(pg_pool_t const&, pg_t, std::vector<int,
std::allocator<int> >&) const+0x8f) [0x80d7cf]
 7: (OSDMap::pg_to_up_acting_osds(pg_t, std::vector<int,
std::allocator<int> >&, std::vector<int, std::allocator<int> >&)
const+0xa6) [0x80d8a6]
 8: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&,
PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>,
std::less<boost::intrusive_ptr<PG> >,
std::allocator<boost::intrusive_ptr<PG> > >*)+0x190) [0x631b70]
 9: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> >
const&, ThreadPool::TPHandle&)+0x244) [0x632254]
 10: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> >
const&, ThreadPool::TPHandle&)+0x12) [0x66d0f2]
 11: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0x835a66]
 12: (ThreadPool::WorkThread::entry()+0x10) [0x837890]
 13: (()+0x7e9a) [0x7ffe6b721e9a]
 14: (clone()+0x6d) [0x7ffe69f9fccd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   0/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   0/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/ 5 hadoop
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---

2013/7/14 Vladislav Gorbunov <vadikgo@xxxxxxxxx>:
> Hello!
>
> After change the crush map all osd (ceph version 0.61.4
> (1669132fcfc27d0c0b5e5bb93ade59d147e23404)) on pool default is crushed
> with the error:
> 2013-07-14 17:26:23.755432 7f0c963ad700 -1 *** Caught signal
> (Segmentation fault) **
>  in thread 7f0c963ad700
> ...skipping...
>  10: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> >
> const&, ThreadPool::TPHandle&)+0x12) [0x66d0f2]
>  11: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0x835a66]
>  12: (ThreadPool::WorkThread::entry()+0x10) [0x837890]
>  13: (()+0x7e9a) [0x7fac4e597e9a]
>  14: (clone()+0x6d) [0x7fac4ce15ccd]
>  NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
> --- logging levels ---
>    0/ 5 none
>    0/ 1 lockdep
>    0/ 1 context
>    1/ 1 crush
>    1/ 5 mds
>    1/ 5 mds_balancer
>    1/ 5 mds_locker
>    1/ 5 mds_log
>    1/ 5 mds_log_expire
>    1/ 5 mds_migrator
>    0/ 1 buffer
>    0/ 1 timer
>    0/ 1 filer
>    0/ 1 striper
>    0/ 1 objecter
>    0/ 5 rados
>    0/ 5 rbd
>    0/ 5 journaler
>    0/ 5 objectcacher
>    0/ 5 client
>    0/ 5 osd
>    0/ 5 optracker
>    0/ 5 objclass
>    1/ 3 filestore
>    1/ 3 journal
>    0/ 5 ms
>    1/ 5 mon
>    0/10 monc
>    0/ 5 paxos
>    0/ 5 tp
>    1/ 5 auth
>    1/ 5 crypto
>    1/ 1 finisher
>    1/ 5 heartbeatmap
>    1/ 5 perfcounter
>    1/ 5 rgw
>    1/ 5 hadoop
>    1/ 5 javaclient
>    1/ 5 asok
>    1/ 1 throttle
>   -2/-2 (syslog threshold)
>   -1/-1 (stderr threshold)
>   max_recent     10000
>   max_new         1000
>   log_file /var/log/ceph/ceph-osd.2.log
> --- end dump of recent events ---
>
> ceph osd complity ignoring osd start/stop an always show this map:
> # id weight type name up/down reweight
> -2 65.52 pool iscsi
> -4 32.76 datacenter datacenter-cod
> -6 32.76 host gstore1
> 82 2.73 osd.82 up 1
> 83 2.73 osd.83 up 1
> 84 2.73 osd.84 up 1
> 85 2.73 osd.85 up 1
> 86 2.73 osd.86 up 1
> 87 2.73 osd.87 up 1
> 88 2.73 osd.88 up 1
> 89 2.73 osd.89 up 1
> 90 2.73 osd.90 up 1
> 91 2.73 osd.91 up 1
> 92 2.73 osd.92 up 1
> 93 2.73 osd.93 up 1
> -5 32.76 datacenter datacenter-rcod
> -7 32.76 host gstore2
> 94 2.73 osd.94 up 1
> 95 2.73 osd.95 up 1
> 96 2.73 osd.96 up 1
> 97 2.73 osd.97 up 1
> 98 2.73 osd.98 up 1
> 99 2.73 osd.99 up 1
> 100 2.73 osd.100 up 1
> 101 2.73 osd.101 up 1
> 102 2.73 osd.102 up 1
> 103 2.73 osd.103 up 1
> 104 2.73 osd.104 up 1
> 105 2.73 osd.105 up 1
> -1 68.96 pool default
> -3 68.96 rack unknownrack
> -9 5 host gstore5
> 2 5 osd.2 down 1
> -10 16 host cstore1
> 5 1 osd.5 down 0
> 6 1 osd.6 down 0
> 7 1 osd.7 down 0
> 8 1 osd.8 down 0
> 9 1 osd.9 down 0
> 10 1 osd.10 down 0
> 11 1 osd.11 down 0
> 12 1 osd.12 down 0
> 13 1 osd.13 down 0
> 14 1 osd.14 down 0
> 4 1 osd.4 down 0
> 47 1 osd.47 down 0
> 48 1 osd.48 down 0
> 49 1 osd.49 down 0
> 50 1 osd.50 down 0
> 51 1 osd.51 down 0
> -11 21 host cstore2
> 15 1 osd.15 down 0
> 16 1 osd.16 down 0
> 17 1 osd.17 down 0
> 18 1 osd.18 down 0
> 19 1 osd.19 down 0
> 20 1 osd.20 down 0
> 21 1 osd.21 down 0
> 22 1 osd.22 down 0
> 23 1 osd.23 down 0
> 24 1 osd.24 down 0
> 41 1 osd.41 down 0
> 42 1 osd.42 down 0
> 43 1 osd.43 down 0
> 44 1 osd.44 down 0
> 45 1 osd.45 down 0
> 46 1 osd.46 down 0
> 52 1 osd.52 down 0
> 53 1 osd.53 down 0
> 54 1 osd.54 down 0
> 55 1 osd.55 down 0
> 56 1 osd.56 up 1
> 57 0 osd.57 up 1
> -12 16 host cstore3
> 25 1 osd.25 down 0
> 26 1 osd.26 down 0
> 27 1 osd.27 down 0
> 28 1 osd.28 down 0
> 29 1 osd.29 down 0
> 30 1 osd.30 down 0
> 31 1 osd.31 down 0
> 32 1 osd.32 down 0
> 33 1 osd.33 down 0
> 34 1 osd.34 down 0
> 35 1 osd.35 down 0
> 36 1 osd.36 down 0
> 37 1 osd.37 down 0
> 38 1 osd.38 down 0
> 39 1 osd.39 down 0
> 40 1 osd.40 down 0
> -13 7.64 host cstore4
> 62 0.55 osd.62 down 0
> 63 0.55 osd.63 down 0
> 64 0.55 osd.64 down 0
> 65 0.55 osd.65 down 0
> 66 0.55 osd.66 down 0
> 67 0.55 osd.67 down 0
> 68 0.55 osd.68 down 0
> 69 0.55 osd.69 down 0
> 70 0.27 osd.70 down 0
> 71 0.27 osd.71 down 0
> 72 0.27 osd.72 down 0
> 73 0.27 osd.73 down 0
> 74 0.27 osd.74 down 0
> 75 0.27 osd.75 down 0
> 76 0.27 osd.76 down 0
> 77 0.27 osd.77 down 0
> 78 0.27 osd.78 down 0
> 79 0.27 osd.79 down 0
> 80 0.27 osd.80 down 0
> 81 0.27 osd.81 down 0
> -14 3.32 host cstore5
> 0 0.4 osd.0 down 0
> 1 0.5 osd.1 down 0
> 3 0.4 osd.3 down 0
> 58 0.4 osd.58 down 1
> 59 0.54 osd.59 down 1
> 60 0.54 osd.60 down 1
> 61 0.54 osd.61 down 1
>
> crush map is:
> # begin crush map
>
> # devices
> device 0 osd.0
> device 1 osd.1
> device 2 osd.2
> device 3 osd.3
> device 4 osd.4
> device 5 osd.5
> device 6 osd.6
> device 7 osd.7
> device 8 osd.8
> device 9 osd.9
> device 10 osd.10
> device 11 osd.11
> device 12 osd.12
> device 13 osd.13
> device 14 osd.14
> device 15 osd.15
> device 16 osd.16
> device 17 osd.17
> device 18 osd.18
> device 19 osd.19
> device 20 osd.20
> device 21 osd.21
> device 22 osd.22
> device 23 osd.23
> device 24 osd.24
> device 25 osd.25
> device 26 osd.26
> device 27 osd.27
> device 28 osd.28
> device 29 osd.29
> device 30 osd.30
> device 31 osd.31
> device 32 osd.32
> device 33 osd.33
> device 34 osd.34
> device 35 osd.35
> device 36 osd.36
> device 37 osd.37
> device 38 osd.38
> device 39 osd.39
> device 40 osd.40
> device 41 osd.41
> device 42 osd.42
> device 43 osd.43
> device 44 osd.44
> device 45 osd.45
> device 46 osd.46
> device 47 osd.47
> device 48 osd.48
> device 49 osd.49
> device 50 osd.50
> device 51 osd.51
> device 52 osd.52
> device 53 osd.53
> device 54 osd.54
> device 55 osd.55
> device 56 osd.56
> device 57 osd.57
> device 58 osd.58
> device 59 osd.59
> device 60 osd.60
> device 61 osd.61
> device 62 osd.62
> device 63 osd.63
> device 64 osd.64
> device 65 osd.65
> device 66 osd.66
> device 67 osd.67
> device 68 osd.68
> device 69 osd.69
> device 70 osd.70
> device 71 osd.71
> device 72 osd.72
> device 73 osd.73
> device 74 osd.74
> device 75 osd.75
> device 76 osd.76
> device 77 osd.77
> device 78 osd.78
> device 79 osd.79
> device 80 osd.80
> device 81 osd.81
> device 82 osd.82
> device 83 osd.83
> device 84 osd.84
> device 85 osd.85
> device 86 osd.86
> device 87 osd.87
> device 88 osd.88
> device 89 osd.89
> device 90 osd.90
> device 91 osd.91
> device 92 osd.92
> device 93 osd.93
> device 94 osd.94
> device 95 osd.95
> device 96 osd.96
> device 97 osd.97
> device 98 osd.98
> device 99 osd.99
> device 100 osd.100
> device 101 osd.101
> device 102 osd.102
> device 103 osd.103
> device 104 osd.104
> device 105 osd.105
>
> # types
> type 0 osd
> type 1 host
> type 2 rack
> type 3 row
> type 4 room
> type 5 datacenter
> type 6 pool
>
> # buckets
> host gstore5 {
> id -9 # do not change unnecessarily
> # weight 5.000
> alg straw
> hash 0 # rjenkins1
> item osd.2 weight 5.000
> }
> host cstore1 {
> id -10 # do not change unnecessarily
> # weight 16.000
> alg straw
> hash 0 # rjenkins1
> item osd.5 weight 1.000
> item osd.6 weight 1.000
> item osd.7 weight 1.000
> item osd.8 weight 1.000
> item osd.9 weight 1.000
> item osd.10 weight 1.000
> item osd.11 weight 1.000
> item osd.12 weight 1.000
> item osd.13 weight 1.000
> item osd.14 weight 1.000
> item osd.4 weight 1.000
> item osd.47 weight 1.000
> item osd.48 weight 1.000
> item osd.49 weight 1.000
> item osd.50 weight 1.000
> item osd.51 weight 1.000
> }
> host cstore2 {
> id -11 # do not change unnecessarily
> # weight 20.000
> alg straw
> hash 0 # rjenkins1
> item osd.15 weight 1.000
> item osd.16 weight 1.000
> item osd.17 weight 1.000
> item osd.18 weight 1.000
> item osd.19 weight 1.000
> item osd.20 weight 1.000
> item osd.21 weight 1.000
> item osd.22 weight 1.000
> item osd.23 weight 1.000
> item osd.24 weight 1.000
> item osd.41 weight 1.000
> item osd.42 weight 1.000
> item osd.43 weight 1.000
> item osd.44 weight 1.000
> item osd.45 weight 1.000
> item osd.46 weight 1.000
> item osd.52 weight 1.000
> item osd.53 weight 1.000
> item osd.54 weight 1.000
> item osd.55 weight 1.000
> item osd.56 weight 0.000
> item osd.57 weight 0.000
> }
> host cstore3 {
> id -12 # do not change unnecessarily
> # weight 16.000
> alg straw
> hash 0 # rjenkins1
> item osd.25 weight 1.000
> item osd.26 weight 1.000
> item osd.27 weight 1.000
> item osd.28 weight 1.000
> item osd.29 weight 1.000
> item osd.30 weight 1.000
> item osd.31 weight 1.000
> item osd.32 weight 1.000
> item osd.33 weight 1.000
> item osd.34 weight 1.000
> item osd.35 weight 1.000
> item osd.36 weight 1.000
> item osd.37 weight 1.000
> item osd.38 weight 1.000
> item osd.39 weight 1.000
> item osd.40 weight 1.000
> }
> host cstore4 {
> id -13 # do not change unnecessarily
> # weight 7.640
> alg straw
> hash 0 # rjenkins1
> item osd.62 weight 0.550
> item osd.63 weight 0.550
> item osd.64 weight 0.550
> item osd.65 weight 0.550
> item osd.66 weight 0.550
> item osd.67 weight 0.550
> item osd.68 weight 0.550
> item osd.69 weight 0.550
> item osd.70 weight 0.270
> item osd.71 weight 0.270
> item osd.72 weight 0.270
> item osd.73 weight 0.270
> item osd.74 weight 0.270
> item osd.75 weight 0.270
> item osd.76 weight 0.270
> item osd.77 weight 0.270
> item osd.78 weight 0.270
> item osd.79 weight 0.270
> item osd.80 weight 0.270
> item osd.81 weight 0.270
> }
> host cstore5 {
> id -14 # do not change unnecessarily
> # weight 3.320
> alg straw
> hash 0 # rjenkins1
> item osd.0 weight 0.400
> item osd.1 weight 0.500
> item osd.3 weight 0.400
> item osd.58 weight 0.400
> item osd.59 weight 0.540
> item osd.60 weight 0.540
> item osd.61 weight 0.540
> }
> rack unknownrack {
> id -3 # do not change unnecessarily
> # weight 67.960
> alg straw
> hash 0 # rjenkins1
> item gstore5 weight 5.000
> item cstore1 weight 16.000
> item cstore2 weight 20.000
> item cstore3 weight 16.000
> item cstore4 weight 7.640
> item cstore5 weight 3.320
> }
> pool default {
> id -1 # do not change unnecessarily
> # weight 67.960
> alg straw
> hash 0 # rjenkins1
> item unknownrack weight 67.960
> }
> host gstore1 {
> id -6 # do not change unnecessarily
> # weight 32.760
> alg straw
> hash 0 # rjenkins1
> item osd.82 weight 2.730
> item osd.83 weight 2.730
> item osd.84 weight 2.730
> item osd.85 weight 2.730
> item osd.86 weight 2.730
> item osd.87 weight 2.730
> item osd.88 weight 2.730
> item osd.89 weight 2.730
> item osd.90 weight 2.730
> item osd.91 weight 2.730
> item osd.92 weight 2.730
> item osd.93 weight 2.730
> }
> datacenter datacenter-cod {
> id -4 # do not change unnecessarily
> # weight 32.760
> alg straw
> hash 0 # rjenkins1
> item gstore1 weight 32.760
> }
> host gstore2 {
> id -7 # do not change unnecessarily
> # weight 32.760
> alg straw
> hash 0 # rjenkins1
> item osd.94 weight 2.730
> item osd.95 weight 2.730
> item osd.96 weight 2.730
> item osd.97 weight 2.730
> item osd.98 weight 2.730
> item osd.99 weight 2.730
> item osd.100 weight 2.730
> item osd.101 weight 2.730
> item osd.102 weight 2.730
> item osd.103 weight 2.730
> item osd.104 weight 2.730
> item osd.105 weight 2.730
> }
> datacenter datacenter-rcod {
> id -5 # do not change unnecessarily
> # weight 32.760
> alg straw
> hash 0 # rjenkins1
> item gstore2 weight 32.760
> }
> pool iscsi {
> id -2 # do not change unnecessarily
> # weight 65.520
> alg straw
> hash 0 # rjenkins1
> item datacenter-cod weight 32.760
> item datacenter-rcod weight 32.760
> }
>
> # rules
> rule data {
> ruleset 0
> type replicated
> min_size 1
> max_size 10
> step take default
> step chooseleaf firstn 0 type host
> step emit
> }
> rule metadata {
> ruleset 1
> type replicated
> min_size 1
> max_size 10
> step take default
> step chooseleaf firstn 0 type host
> step emit
> }
> rule rbd {
> ruleset 2
> type replicated
> min_size 1
> max_size 10
> step take default
> step chooseleaf firstn 0 type host
> step emit
> }
>
> # end crush map
>
>
> Restarting mon don't help.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com