Well, we haven't finished diagnosing #8232 yet ? I think it's actually different from #6992 ? so I can't really tell you what will fix it! Restarting the hosts might. If not, I'd love for somebody to reproduce this with "debug ms = 10" enabled so I can get a log and see what's causing it. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Wed, Apr 30, 2014 at 2:25 PM, Thanh Tran <thanhtv26 at gmail.com> wrote: > Hi Gregory, > > Sorry for replying late. The issue tracking #8232 referenced to the issue > tracking #6992 that said that this issue resolved 23 days ago. Weren't the > code for this issue updated correctly?. > > I restarted one OSD at a time, after that osds have been flapping as > described in TROUBLESHOOTING OSDS at > http://ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/. > > Does it need to reboot the hosts as Guang Yang reported in the issue > tracking #8232 to resolve this issue? > > Best regards, > Thanh Tran > > > On Wed, Apr 30, 2014 at 12:53 AM, Gregory Farnum <greg at inktank.com> wrote: >> >> Hmm, I think this might actually be another instance of >> http://tracker.ceph.com/issues/8232, which was just reported >> yesterday. >> That said, I think that if you restart one OSD at a time, you should >> be able to avoid the race condition. It was restarting all of them >> simultaneously that got you into trouble. >> -Greg >> Software Engineer #42 @ http://inktank.com | http://ceph.com >> >> >> On Tue, Apr 29, 2014 at 12:25 AM, Thanh Tran <thanhtv26 at gmail.com> wrote: >> > Hi, >> > >> > I upgraded my ceph from emperor to firefly (v0.80-rc1-16-g2708c3c), i >> > restart the whole ceph after finishing the upgrade. >> > After restarting, ceph begin to perform checking and recovering pgs, and >> > osds begin randomly to up and down constantly, some osds were crashed >> > and >> > marked as down. I tried to start the crashed osds manually, but a later >> > time >> > the other osds crashed. >> > >> > My cluster have 3 mons and 24 osds running on 3 hosts. >> > Ceph is upgraded from "deb >> > http://gitbuilder.ceph.com/ceph-deb-quantal-x86_64-basic/ref/firefly >> > quantal >> > main". >> > >> > log information of one osd crashed: >> > -1> 2014-04-28 17:42:36.751933 7f6aacb30700 2 -- >> > 10.76.0.44:6814/192022485 >> 10.76.0.42:6800/5060598 pipe(0x2af92c80 >> > sd=26 >> > :17458 s=1 pgs=0 cs=0 l=0 c=0x77636c60). got newly_acked_seq 226 vs >> > out_seq >> > 0 >> > 0> 2014-04-28 17:42:36.752674 7f6ab293a700 -1 msg/Pipe.cc: In >> > function >> > 'int Pipe::connect()' thread 7f6ab293a700 time 2014-04-28 >> > 17:42:36.750997 >> > msg/Pipe.cc: 1070: FAILED assert(m) >> > >> > ceph version 0.80-rc1-16-g2708c3c >> > (2708c3c559d99e6f3b557ee1d223efa3745f655c) >> > 1: (Pipe::connect()+0x3b61) [0xc18ab1] >> > 2: (Pipe::writer()+0x65f) [0xc193ef] >> > 3: (Pipe::Writer::entry()+0xd) [0xc240ad] >> > 4: (()+0x7e9a) [0x7f6b0a3a7e9a] >> > 5: (clone()+0x6d) [0x7f6b089523fd] >> > NOTE: a copy of the executable, or `objdump -rdS <executable>` is >> > needed to >> > interpret this. >> > >> > --- logging levels --- >> > 0/ 5 none >> > 0/ 1 lockdep >> > 0/ 1 context >> > 1/ 1 crush >> > 1/ 5 mds >> > 1/ 5 mds_balancer >> > 1/ 5 mds_locker >> > 1/ 5 mds_log >> > 1/ 5 mds_log_expire >> > 1/ 5 mds_migrator >> > 0/ 1 buffer >> > 0/ 1 timer >> > 0/ 1 filer >> > 0/ 1 striper >> > 0/ 1 objecter >> > 0/ 5 rados >> > 0/ 5 rbd >> > 0/ 5 journaler >> > 0/ 5 objectcacher >> > 0/ 5 client >> > 0/ 5 osd >> > 0/ 5 optracker >> > 0/ 5 objclass >> > 1/ 3 filestore >> > 1/ 3 keyvaluestore >> > 1/ 3 journal >> > 0/ 5 ms >> > 1/ 5 mon >> > 0/10 monc >> > 1/ 5 paxos >> > 0/ 5 tp >> > 1/ 5 auth >> > 1/ 5 crypto >> > 1/ 1 finisher >> > 1/ 5 heartbeatmap >> > 1/ 5 perfcounter >> > 1/ 5 rgw >> > 1/ 5 javaclient >> > 1/ 5 asok >> > 1/ 1 throttle >> > -2/-2 (syslog threshold) >> > -1/-1 (stderr threshold) >> > max_recent 10000 >> > max_new 1000 >> > log_file /var/log/ceph/ceph-osd.19.log >> > --- end dump of recent events --- >> > 2014-04-28 17:42:36.756085 7f6aacb30700 -1 msg/Pipe.cc: In function 'int >> > Pipe::connect()' thread 7f6aacb30700 time 2014-04-28 17:42:36.751971 >> > msg/Pipe.cc: 1070: FAILED assert(m) >> > >> > ceph version 0.80-rc1-16-g2708c3c >> > (2708c3c559d99e6f3b557ee1d223efa3745f655c) >> > 1: (Pipe::connect()+0x3b61) [0xc18ab1] >> > 2: (Pipe::writer()+0x65f) [0xc193ef] >> > 3: (Pipe::Writer::entry()+0xd) [0xc240ad] >> > 4: (()+0x7e9a) [0x7f6b0a3a7e9a] >> > 5: (clone()+0x6d) [0x7f6b089523fd] >> > NOTE: a copy of the executable, or `objdump -rdS <executable>` is >> > needed to >> > interpret this. >> > >> > 2014-04-28 17:42:36.760152 7f6ab7f54700 -1 msg/Pipe.cc: In function 'int >> > Pipe::connect()' thread 7f6ab7f54700 time 2014-04-28 17:42:36.753156 >> > msg/Pipe.cc: 1070: FAILED assert(m) >> > >> > ceph version 0.80-rc1-16-g2708c3c >> > (2708c3c559d99e6f3b557ee1d223efa3745f655c) >> > 1: (Pipe::connect()+0x3b61) [0xc18ab1] >> > 2: (Pipe::writer()+0x65f) [0xc193ef] >> > 3: (Pipe::Writer::entry()+0xd) [0xc240ad] >> > 4: (()+0x7e9a) [0x7f6b0a3a7e9a] >> > 5: (clone()+0x6d) [0x7f6b089523fd] >> > NOTE: a copy of the executable, or `objdump -rdS <executable>` is >> > needed to >> > interpret this. >> > >> > 2014-04-28 17:42:36.765117 7f6ab9792700 -1 msg/Pipe.cc: In function 'int >> > Pipe::connect()' thread 7f6ab9792700 time 2014-04-28 17:42:36.763390 >> > msg/Pipe.cc: 1070: FAILED assert(m) >> > >> > ceph version 0.80-rc1-16-g2708c3c >> > (2708c3c559d99e6f3b557ee1d223efa3745f655c) >> > 1: (Pipe::connect()+0x3b61) [0xc18ab1] >> > 2: (Pipe::writer()+0x65f) [0xc193ef] >> > 3: (Pipe::Writer::entry()+0xd) [0xc240ad] >> > 4: (()+0x7e9a) [0x7f6b0a3a7e9a] >> > 5: (clone()+0x6d) [0x7f6b089523fd] >> > NOTE: a copy of the executable, or `objdump -rdS <executable>` is >> > needed to >> > interpret this. >> > >> > --- begin dump of recent events --- >> > -4> 2014-04-28 17:42:36.753061 7f6ab7f54700 2 -- >> > 10.76.0.44:6814/192022485 >> 10.76.0.42:6819/3061991 pipe(0x771e2280 >> > sd=115 >> > :47532 s=1 pgs=0 cs=0 l=0 c=0x3ea4f760). got newly_acked_seq 475 vs >> > out_seq >> > 0 >> > -3> 2014-04-28 17:42:36.756085 7f6aacb30700 -1 msg/Pipe.cc: In >> > function >> > 'int Pipe::connect()' thread 7f6aacb30700 time 2014-04-28 >> > 17:42:36.751971 >> > msg/Pipe.cc: 1070: FAILED assert(m) >> > >> > ceph version 0.80-rc1-16-g2708c3c >> > (2708c3c559d99e6f3b557ee1d223efa3745f655c) >> > 1: (Pipe::connect()+0x3b61) [0xc18ab1] >> > 2: (Pipe::writer()+0x65f) [0xc193ef] >> > 3: (Pipe::Writer::entry()+0xd) [0xc240ad] >> > 4: (()+0x7e9a) [0x7f6b0a3a7e9a] >> > 5: (clone()+0x6d) [0x7f6b089523fd] >> > NOTE: a copy of the executable, or `objdump -rdS <executable>` is >> > needed to >> > interpret this. >> > >> > -2> 2014-04-28 17:42:36.760152 7f6ab7f54700 -1 msg/Pipe.cc: In >> > function >> > 'int Pipe::connect()' thread 7f6ab7f54700 time 2014-04-28 >> > 17:42:36.753156 >> > msg/Pipe.cc: 1070: FAILED assert(m) >> > >> > ceph version 0.80-rc1-16-g2708c3c >> > (2708c3c559d99e6f3b557ee1d223efa3745f655c) >> > 1: (Pipe::connect()+0x3b61) [0xc18ab1] >> > 2: (Pipe::writer()+0x65f) [0xc193ef] >> > 3: (Pipe::Writer::entry()+0xd) [0xc240ad] >> > 4: (()+0x7e9a) [0x7f6b0a3a7e9a] >> > 5: (clone()+0x6d) [0x7f6b089523fd] >> > NOTE: a copy of the executable, or `objdump -rdS <executable>` is >> > needed to >> > interpret this. >> > >> > -1> 2014-04-28 17:42:36.763322 7f6ab9792700 2 -- >> > 10.76.0.44:6814/192022485 >> 10.76.0.43:6808/64004052 pipe(0x43773b80 >> > sd=121 >> > :31644 s=1 pgs=0 cs=0 l=0 c=0x70e3f600). got newly_acked_seq 268 vs >> > out_seq >> > 0 >> > 0> 2014-04-28 17:42:36.765117 7f6ab9792700 -1 msg/Pipe.cc: In >> > function >> > 'int Pipe::connect()' thread 7f6ab9792700 time 2014-04-28 >> > 17:42:36.763390 >> > msg/Pipe.cc: 1070: FAILED assert(m) >> > >> > ceph version 0.80-rc1-16-g2708c3c >> > (2708c3c559d99e6f3b557ee1d223efa3745f655c) >> > 1: (Pipe::connect()+0x3b61) [0xc18ab1] >> > 2: (Pipe::writer()+0x65f) [0xc193ef] >> > 3: (Pipe::Writer::entry()+0xd) [0xc240ad] >> > 4: (()+0x7e9a) [0x7f6b0a3a7e9a] >> > 5: (clone()+0x6d) [0x7f6b089523fd] >> > NOTE: a copy of the executable, or `objdump -rdS <executable>` is >> > needed to >> > interpret this. >> > >> > --- logging levels --- >> > 0/ 5 none >> > 0/ 1 lockdep >> > 0/ 1 context >> > 1/ 1 crush >> > 1/ 5 mds >> > 1/ 5 mds_balancer >> > 1/ 5 mds_locker >> > 1/ 5 mds_log >> > 1/ 5 mds_log_expire >> > 1/ 5 mds_migrator >> > 0/ 1 buffer >> > 0/ 1 timer >> > 0/ 1 filer >> > 0/ 1 striper >> > 0/ 1 objecter >> > 0/ 5 rados >> > 0/ 5 rbd >> > 0/ 5 journaler >> > 0/ 5 objectcacher >> > 0/ 5 client >> > 0/ 5 osd >> > 0/ 5 optracker >> > 0/ 5 objclass >> > 1/ 3 filestore >> > 1/ 3 keyvaluestore >> > 1/ 3 journal >> > 0/ 5 ms >> > 1/ 5 mon >> > 0/10 monc >> > 1/ 5 paxos >> > 0/ 5 tp >> > 1/ 5 auth >> > 1/ 5 crypto >> > 1/ 1 finisher >> > 1/ 5 heartbeatmap >> > 1/ 5 perfcounter >> > 1/ 5 rgw >> > 1/ 5 javaclient >> > 1/ 5 asok >> > 1/ 1 throttle >> > -2/-2 (syslog threshold) >> > -1/-1 (stderr threshold) >> > max_recent 10000 >> > max_new 1000 >> > log_file /var/log/ceph/ceph-osd.19.log >> > --- end dump of recent events --- >> > --- begin dump of recent events --- >> > --- logging levels --- >> > 0/ 5 none >> > 0/ 1 lockdep >> > 0/ 1 context >> > 1/ 1 crush >> > 1/ 5 mds >> > 1/ 5 mds_balancer >> > 1/ 5 mds_locker >> > 1/ 5 mds_log >> > 1/ 5 mds_log_expire >> > 1/ 5 mds_migrator >> > 0/ 1 buffer >> > 0/ 1 timer >> > 0/ 1 filer >> > 0/ 1 striper >> > 0/ 1 objecter >> > 0/ 5 rados >> > 0/ 5 rbd >> > 0/ 5 journaler >> > 0/ 5 objectcacher >> > 0/ 5 client >> > 0/ 5 osd >> > 0/ 5 optracker >> > 0/ 5 objclass >> > 1/ 3 filestore >> > 1/ 3 keyvaluestore >> > 1/ 3 journal >> > 0/ 5 ms >> > 1/ 5 mon >> > 0/10 monc >> > 1/ 5 paxos >> > 0/ 5 tp >> > 1/ 5 auth >> > 1/ 5 crypto >> > 1/ 1 finisher >> > 1/ 5 heartbeatmap >> > 1/ 5 perfcounter >> > 1/ 5 rgw >> > 1/ 5 javaclient >> > 1/ 5 asok >> > 1/ 1 throttle >> > -2/-2 (syslog threshold) >> > -1/-1 (stderr threshold) >> > max_recent 10000 >> > max_new 1000 >> > log_file /var/log/ceph/ceph-osd.19.log >> > --- end dump of recent events --- >> > --- begin dump of recent events --- >> > --- logging levels --- >> > >> > >> > Log of another osd: >> > -100> 2014-04-28 19:22:13.145419 7f5a3bbdb700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.6:6805/25058567 pipe(0x5e227400 sd=55 :18619 s=4 pgs=754 cs=1 >> > l=1 >> > c=0x1fea8840).reader couldn't read tag, (0) Success >> > -99> 2014-04-28 19:22:13.145473 7f5a3bbdb700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.6:6805/25058567 pipe(0x5e227400 sd=55 :18619 s=4 pgs=754 cs=1 >> > l=1 >> > c=0x1fea8840).fault (0) Success >> > -98> 2014-04-28 19:22:13.145494 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 192.168.1.4:6829/5061602 0x1eba0f00 >> > -97> 2014-04-28 19:22:13.145526 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 192.168.1.5:6807/190001647 0x10af2000 >> > -96> 2014-04-28 19:22:13.145500 7f5a486f9700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.43:6819/192003077 pipe(0x3f309180 sd=212 :3321 s=4 pgs=5467 cs=1 >> > l=1 >> > c=0x1c62e9a0).reader couldn't read tag, (0) Success >> > -95> 2014-04-28 19:22:13.145557 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 192.168.1.5:6826/192003077 0x10af2280 >> > -94> 2014-04-28 19:22:13.145558 7f5a486f9700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.43:6819/192003077 pipe(0x3f309180 sd=212 :3321 s=4 pgs=5467 cs=1 >> > l=1 >> > c=0x1c62e9a0).fault (0) Success >> > -93> 2014-04-28 19:22:13.145540 7f5a38fa9700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.4:6829/5061602 pipe(0x1eba0f00 sd=102 :14027 s=4 pgs=4582 cs=1 >> > l=1 >> > c=0x43269760).reader couldn't read tag, (0) Success >> > -92> 2014-04-28 19:22:13.145457 7f5a62f65700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.43:6808/66054480 pipe(0x4ee41900 sd=166 :55083 s=4 pgs=2844 cs=1 >> > l=1 >> > c=0x272e3e40).reader couldn't read tag, (0) Success >> > -91> 2014-04-28 19:22:13.145590 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 10.76.0.42:6812/3051038 0x4370b400 >> > -90> 2014-04-28 19:22:13.145592 7f5a38fa9700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.4:6829/5061602 pipe(0x1eba0f00 sd=102 :14027 s=4 pgs=4582 cs=1 >> > l=1 >> > c=0x43269760).fault (0) Success >> > -89> 2014-04-28 19:22:13.145622 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 192.168.1.4:6825/3061991 0x1eb7c280 >> > -88> 2014-04-28 19:22:13.145585 7f5a3ef56700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.5:6807/190001647 pipe(0x10af2000 sd=224 :7489 s=4 pgs=5188 >> > cs=1 >> > l=1 c=0x1eae4580).reader couldn't read tag, (0) Success >> > -87> 2014-04-28 19:22:13.145596 7f5a62f65700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.43:6808/66054480 pipe(0x4ee41900 sd=166 :55083 s=4 pgs=2844 cs=1 >> > l=1 >> > c=0x272e3e40).fault (0) Success >> > -86> 2014-04-28 19:22:13.145641 7f5a3ef56700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.5:6807/190001647 pipe(0x10af2000 sd=224 :7489 s=4 pgs=5188 >> > cs=1 >> > l=1 c=0x1eae4580).fault (0) Success >> > -85> 2014-04-28 19:22:13.145606 7f5a4822a700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.5:6826/192003077 pipe(0x10af2280 sd=213 :22370 s=4 pgs=5479 >> > cs=1 >> > l=1 c=0x1c62ec60).reader couldn't read tag, (0) Success >> > -84> 2014-04-28 19:22:13.145661 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 192.168.1.4:6827/3051038 0x4370b180 >> > -83> 2014-04-28 19:22:13.145633 7f5a3bcdc700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.42:6812/3051038 pipe(0x4370b400 sd=130 :31567 s=4 pgs=4603 cs=1 >> > l=1 >> > c=0x34197600).reader couldn't read tag, (0) Success >> > -82> 2014-04-28 19:22:13.145660 7f5a6160a700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.4:6825/3061991 pipe(0x1eb7c280 sd=191 :36486 s=4 pgs=4601 cs=1 >> > l=1 >> > c=0x3f903080).reader couldn't read tag, (0) Success >> > -81> 2014-04-28 19:22:13.145695 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 192.168.1.4:6807/5060598 0x1eba0500 >> > -80> 2014-04-28 19:22:13.145705 7f5a6160a700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.4:6825/3061991 pipe(0x1eb7c280 sd=191 :36486 s=4 pgs=4601 cs=1 >> > l=1 >> > c=0x3f903080).fault (0) Success >> > -79> 2014-04-28 19:22:13.145699 7f5a66fa4700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.4:6827/3051038 pipe(0x4370b180 sd=75 :48974 s=4 pgs=4568 cs=1 >> > l=1 >> > c=0x3b891600).reader couldn't read tag, (0) Success >> > -78> 2014-04-28 19:22:13.145684 7f5a3bcdc700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.42:6812/3051038 pipe(0x4370b400 sd=130 :31567 s=4 pgs=4603 cs=1 >> > l=1 >> > c=0x34197600).fault (0) Success >> > -77> 2014-04-28 19:22:13.145730 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 192.168.1.5:6818/184055098 0x43d2d180 >> > -76> 2014-04-28 19:22:13.145767 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 192.168.1.4:6823/2060205 0x1eb7d400 >> > -75> 2014-04-28 19:22:13.145737 7f5a3e877700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.4:6807/5060598 pipe(0x1eba0500 sd=92 :9724 s=4 pgs=4362 cs=1 >> > l=1 >> > c=0x4556a9a0).reader couldn't read tag, (0) Success >> > -74> 2014-04-28 19:22:13.145734 7f5a66fa4700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.4:6827/3051038 pipe(0x4370b180 sd=75 :48974 s=4 pgs=4568 cs=1 >> > l=1 >> > c=0x3b891600).fault (0) Success >> > -73> 2014-04-28 19:22:13.145783 7f5a3e877700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.4:6807/5060598 pipe(0x1eba0500 sd=92 :9724 s=4 pgs=4362 cs=1 >> > l=1 >> > c=0x4556a9a0).fault (0) Success >> > -72> 2014-04-28 19:22:13.145664 7f5a4822a700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.5:6826/192003077 pipe(0x10af2280 sd=213 :22370 s=4 pgs=5479 >> > cs=1 >> > l=1 c=0x1c62ec60).fault (0) Success >> > -71> 2014-04-28 19:22:13.145799 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 192.168.1.4:6820/4050662 0x4370b680 >> > -70> 2014-04-28 19:22:13.145773 7f5a36228700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.5:6818/184055098 pipe(0x43d2d180 sd=245 :1882 s=4 pgs=6058 >> > cs=1 >> > l=1 c=0x78da4420).reader couldn't read tag, (0) Success >> > -69> 2014-04-28 19:22:13.145813 7f5a36228700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.5:6818/184055098 pipe(0x43d2d180 sd=245 :1882 s=4 pgs=6058 >> > cs=1 >> > l=1 c=0x78da4420).fault (0) Success >> > -68> 2014-04-28 19:22:13.145805 7f5a41fc9700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.4:6823/2060205 pipe(0x1eb7d400 sd=219 :17008 s=4 pgs=4258 cs=1 >> > l=1 >> > c=0x55b81a20).reader couldn't read tag, (0) Success >> > -67> 2014-04-28 19:22:13.145831 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 10.76.0.43:6821/184055098 0x10af3b80 >> > -66> 2014-04-28 19:22:13.145839 7f5a41fc9700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.4:6823/2060205 pipe(0x1eb7d400 sd=219 :17008 s=4 pgs=4258 cs=1 >> > l=1 >> > c=0x55b81a20).fault (0) Success >> > -65> 2014-04-28 19:22:13.145864 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 10.76.0.43:6816/75004052 0x40d48780 >> > -64> 2014-04-28 19:22:13.145860 7f5a2f5e7700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.4:6820/4050662 pipe(0x4370b680 sd=57 :2505 s=4 pgs=5020 cs=1 >> > l=1 >> > c=0x2d62edc0).reader couldn't read tag, (0) Success >> > -63> 2014-04-28 19:22:13.145892 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 192.168.1.5:6812/66054480 0x9ee16500 >> > -62> 2014-04-28 19:22:13.145872 7f5a2f6e8700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.43:6821/184055098 pipe(0x10af3b80 sd=206 :23018 s=4 pgs=6057 >> > cs=1 >> > l=1 c=0x63c0a160).reader couldn't read tag, (0) Success >> > -61> 2014-04-28 19:22:13.145897 7f5a60be1700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.43:6816/75004052 pipe(0x40d48780 sd=181 :27019 s=4 pgs=2921 cs=1 >> > l=1 >> > c=0x3f902420).reader couldn't read tag, (0) Success >> > -60> 2014-04-28 19:22:13.145950 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 10.76.0.43:6823/24030366 0x5469aa00 >> > -59> 2014-04-28 19:22:13.145936 7f5a2f6e8700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.43:6821/184055098 pipe(0x10af3b80 sd=206 :23018 s=4 pgs=6057 >> > cs=1 >> > l=1 c=0x63c0a160).fault (0) Success >> > -58> 2014-04-28 19:22:13.145959 7f5a60be1700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.43:6816/75004052 pipe(0x40d48780 sd=181 :27019 s=4 pgs=2921 cs=1 >> > l=1 >> > c=0x3f902420).fault (0) Success >> > -57> 2014-04-28 19:22:13.145951 7f5a62747700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.5:6812/66054480 pipe(0x9ee16500 sd=164 :14615 s=4 pgs=2863 >> > cs=1 >> > l=1 c=0x3f903340).reader couldn't read tag, (0) Success >> > -56> 2014-04-28 19:22:13.145979 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 192.168.1.4:6815/4056978 0x4370ac80 >> > -55> 2014-04-28 19:22:13.145981 7f5a62747700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.5:6812/66054480 pipe(0x9ee16500 sd=164 :14615 s=4 pgs=2863 >> > cs=1 >> > l=1 c=0x3f903340).fault (0) Success >> > -54> 2014-04-28 19:22:13.146029 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 192.168.1.5:6816/74002481 0x3f308000 >> > -53> 2014-04-28 19:22:13.145898 7f5a2f5e7700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.4:6820/4050662 pipe(0x4370b680 sd=57 :2505 s=4 pgs=5020 cs=1 >> > l=1 >> > c=0x2d62edc0).fault (0) Success >> > -52> 2014-04-28 19:22:13.146059 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 10.76.0.43:6814/186056525 0x4ee40c80 >> > -51> 2014-04-28 19:22:13.145991 7f5a65a5b700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.43:6823/24030366 pipe(0x5469aa00 sd=31 :56476 s=4 pgs=1294 cs=1 >> > l=1 >> > c=0x4b0de580).reader couldn't read tag, (0) Success >> > -50> 2014-04-28 19:22:13.146061 7f5a30000700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.4:6815/4056978 pipe(0x4370ac80 sd=58 :5673 s=4 pgs=4629 cs=1 >> > l=1 >> > c=0x3b8918c0).reader couldn't read tag, (0) Success >> > -49> 2014-04-28 19:22:13.146072 7f5a640c6700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.5:6816/74002481 pipe(0x3f308000 sd=208 :61372 s=4 pgs=3339 >> > cs=1 >> > l=1 c=0xb732dc0).reader couldn't read tag, (0) Success >> > -48> 2014-04-28 19:22:13.146096 7f5a30000700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.4:6815/4056978 pipe(0x4370ac80 sd=58 :5673 s=4 pgs=4629 cs=1 >> > l=1 >> > c=0x3b8918c0).fault (0) Success >> > -47> 2014-04-28 19:22:13.146081 7f5a65a5b700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.43:6823/24030366 pipe(0x5469aa00 sd=31 :56476 s=4 pgs=1294 cs=1 >> > l=1 >> > c=0x4b0de580).fault (0) Success >> > -46> 2014-04-28 19:22:13.146103 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 10.76.0.42:6806/4056978 0x4370af00 >> > -45> 2014-04-28 19:22:13.146106 7f5a640c6700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.5:6816/74002481 pipe(0x3f308000 sd=208 :61372 s=4 pgs=3339 >> > cs=1 >> > l=1 c=0xb732dc0).fault (0) Success >> > -44> 2014-04-28 19:22:13.146139 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 192.168.1.5:6815/186056525 0x4ee40780 >> > -43> 2014-04-28 19:22:13.146171 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 10.76.0.42:6817/5061602 0x1eba0a00 >> > -42> 2014-04-28 19:22:13.146202 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 10.76.0.43:6801/190001647 0x10af2f00 >> > -41> 2014-04-28 19:22:13.146186 7f5a67cd2700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.42:6806/4056978 pipe(0x4370af00 sd=82 :3622 s=4 pgs=4646 cs=1 >> > l=1 >> > c=0xd154c60).reader couldn't read tag, (0) Success >> > -40> 2014-04-28 19:22:13.146182 7f5a62646700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.5:6815/186056525 pipe(0x4ee40780 sd=176 :54924 s=4 pgs=6348 >> > cs=1 >> > l=1 c=0x42c33760).reader couldn't read tag, (0) Success >> > -39> 2014-04-28 19:22:13.146184 7f5a62b56700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.43:6814/186056525 pipe(0x4ee40c80 sd=167 :19387 s=4 pgs=6339 >> > cs=1 >> > l=1 c=0xd155080).reader couldn't read tag, (0) Success >> > -38> 2014-04-28 19:22:13.146213 7f5a67cd2700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.42:6806/4056978 pipe(0x4370af00 sd=82 :3622 s=4 pgs=4646 cs=1 >> > l=1 >> > c=0xd154c60).fault (0) Success >> > -37> 2014-04-28 19:22:13.146218 7f5a62646700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.5:6815/186056525 pipe(0x4ee40780 sd=176 :54924 s=4 pgs=6348 >> > cs=1 >> > l=1 c=0x42c33760).fault (0) Success >> > -36> 2014-04-28 19:22:13.146225 7f5a62b56700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.43:6814/186056525 pipe(0x4ee40c80 sd=167 :19387 s=4 pgs=6339 >> > cs=1 >> > l=1 c=0xd155080).fault (0) Success >> > -35> 2014-04-28 19:22:13.146231 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 10.76.0.44:6803/25058567 0x5e227900 >> > -34> 2014-04-28 19:22:13.146210 7f5a662d7700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.42:6817/5061602 pipe(0x1eba0a00 sd=101 :1604 s=4 pgs=4605 cs=1 >> > l=1 >> > c=0x62e22580).reader couldn't read tag, (0) Success >> > -33> 2014-04-28 19:22:13.146269 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 10.76.0.42:6802/5060598 0x4370a000 >> > -32> 2014-04-28 19:22:13.146241 7f5a3f360700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.43:6801/190001647 pipe(0x10af2f00 sd=223 :48654 s=4 pgs=5173 >> > cs=1 >> > l=1 c=0xd4ca000).reader couldn't read tag, (0) Success >> > -31> 2014-04-28 19:22:13.146269 7f5a662d7700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.42:6817/5061602 pipe(0x1eba0a00 sd=101 :1604 s=4 pgs=4605 cs=1 >> > l=1 >> > c=0x62e22580).fault (0) Success >> > -30> 2014-04-28 19:22:13.146297 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 192.168.1.5:6803/75004052 0x40d48c80 >> > -29> 2014-04-28 19:22:13.146285 7f5a3f360700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.43:6801/190001647 pipe(0x10af2f00 sd=223 :48654 s=4 pgs=5173 >> > cs=1 >> > l=1 c=0xd4ca000).fault (0) Success >> > -28> 2014-04-28 19:22:13.146278 7f5a31b96700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.44:6803/25058567 pipe(0x5e227900 sd=49 :52723 s=4 pgs=752 cs=1 >> > l=1 >> > c=0xa5f22c0).reader couldn't read tag, (0) Success >> > -27> 2014-04-28 19:22:13.146326 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 10.76.0.43:6805/74002481 0x4c3e9180 >> > -26> 2014-04-28 19:22:13.146327 7f5a31b96700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.44:6803/25058567 pipe(0x5e227900 sd=49 :52723 s=4 pgs=752 cs=1 >> > l=1 >> > c=0xa5f22c0).fault (0) Success >> > -25> 2014-04-28 19:22:13.146309 7f5a64bf1700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.42:6802/5060598 pipe(0x4370a000 sd=90 :37186 s=4 pgs=4370 cs=1 >> > l=1 >> > c=0x122afa20).reader couldn't read tag, (0) Success >> > -24> 2014-04-28 19:22:13.146353 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 10.76.0.42:6810/2060205 0x1eb7db80 >> > -23> 2014-04-28 19:22:13.146359 7f5a64bf1700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.42:6802/5060598 pipe(0x4370a000 sd=90 :37186 s=4 pgs=4370 cs=1 >> > l=1 >> > c=0x122afa20).fault (0) Success >> > -22> 2014-04-28 19:22:13.146381 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 10.76.0.42:6809/4050662 0x4370b900 >> > -21> 2014-04-28 19:22:13.146341 7f5a6202a700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.5:6803/75004052 pipe(0x40d48c80 sd=187 :4544 s=4 pgs=2936 cs=1 >> > l=1 >> > c=0x3f902dc0).reader couldn't read tag, (0) Success >> > -20> 2014-04-28 19:22:13.146398 7f5a6202a700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.5:6803/75004052 pipe(0x40d48c80 sd=187 :4544 s=4 pgs=2936 cs=1 >> > l=1 >> > c=0x3f902dc0).fault (0) Success >> > -19> 2014-04-28 19:22:13.146411 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 192.168.1.5:6820/24030366 0x34c8f00 >> > -18> 2014-04-28 19:22:13.146393 7f5a41bd4700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.42:6810/2060205 pipe(0x1eb7db80 sd=183 :16386 s=4 pgs=4253 cs=1 >> > l=1 >> > c=0x55b818c0).reader couldn't read tag, (0) Success >> > -17> 2014-04-28 19:22:13.146445 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 10.76.0.42:6831/3061991 0x40d49400 >> > -16> 2014-04-28 19:22:13.146421 7f5a661d6700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.43:6805/74002481 pipe(0x4c3e9180 sd=207 :25307 s=4 pgs=3343 cs=1 >> > l=1 >> > c=0x7cc3e40).reader couldn't read tag, (0) Success >> > -15> 2014-04-28 19:22:13.146419 7f5a6dc17700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.42:6809/4050662 pipe(0x4370b900 sd=132 :3860 s=4 pgs=5071 cs=1 >> > l=1 >> > c=0x3b890b00).reader couldn't read tag, (0) Success >> > -14> 2014-04-28 19:22:13.146437 7f5a41bd4700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.42:6810/2060205 pipe(0x1eb7db80 sd=183 :16386 s=4 pgs=4253 cs=1 >> > l=1 >> > c=0x55b818c0).fault (0) Success >> > -13> 2014-04-28 19:22:13.146460 7f5a6dc17700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.42:6809/4050662 pipe(0x4370b900 sd=132 :3860 s=4 pgs=5071 cs=1 >> > l=1 >> > c=0x3b890b00).fault (0) Success >> > -12> 2014-04-28 19:22:13.146451 7f5a3b03d700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.5:6820/24030366 pipe(0x34c8f00 sd=28 :2684 s=4 pgs=1300 cs=1 >> > l=1 >> > c=0x4b0df080).reader couldn't read tag, (0) Success >> > -11> 2014-04-28 19:22:13.146473 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 10.76.0.42:6815/3061091 0x1eba0280 >> > -10> 2014-04-28 19:22:13.146459 7f5a661d6700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.43:6805/74002481 pipe(0x4c3e9180 sd=207 :25307 s=4 pgs=3343 cs=1 >> > l=1 >> > c=0x7cc3e40).fault (0) Success >> > -9> 2014-04-28 19:22:13.146504 7f5a5d627700 5 -- >> > 192.168.1.6:0/21280 >> > mark_down_all 192.168.1.4:6828/3061091 0x1eba1680 >> > -8> 2014-04-28 19:22:13.146545 7f5a5b623700 1 -- >> > 192.168.1.6:0/21280 >> > mark_down 0xa5f22c0 -- pipe dne >> > -7> 2014-04-28 19:22:13.146512 7f5a387a1700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.42:6815/3061091 pipe(0x1eba0280 sd=93 :24347 s=4 pgs=4092 cs=1 >> > l=1 >> > c=0x1e12d8c0).reader couldn't read tag, (0) Success >> > -6> 2014-04-28 19:22:13.146478 7f5a64af0700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.42:6831/3061991 pipe(0x40d49400 sd=189 :57711 s=4 pgs=4609 cs=1 >> > l=1 >> > c=0x3f903600).reader couldn't read tag, (0) Success >> > -5> 2014-04-28 19:22:13.146545 7f5a3d0c2700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.4:6828/3061091 pipe(0x1eba1680 sd=95 :13902 s=4 pgs=4061 cs=1 >> > l=1 >> > c=0x7cb4160).reader couldn't read tag, (0) Success >> > -4> 2014-04-28 19:22:13.146580 7f5a64af0700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.42:6831/3061991 pipe(0x40d49400 sd=189 :57711 s=4 pgs=4609 cs=1 >> > l=1 >> > c=0x3f903600).fault (0) Success >> > -3> 2014-04-28 19:22:13.146560 7f5a387a1700 2 -- >> > 192.168.1.6:0/21280 >> >> > 10.76.0.42:6815/3061091 pipe(0x1eba0280 sd=93 :24347 s=4 pgs=4092 cs=1 >> > l=1 >> > c=0x1e12d8c0).fault (0) Success >> > -2> 2014-04-28 19:22:13.146593 7f5a3d0c2700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.4:6828/3061091 pipe(0x1eba1680 sd=95 :13902 s=4 pgs=4061 cs=1 >> > l=1 >> > c=0x7cb4160).fault (0) Success >> > -1> 2014-04-28 19:22:13.146484 7f5a3b03d700 2 -- >> > 192.168.1.6:0/21280 >> >> > 192.168.1.5:6820/24030366 pipe(0x34c8f00 sd=28 :2684 s=4 pgs=1300 cs=1 >> > l=1 >> > c=0x4b0df080).fault (0) Success >> > 0> 2014-04-28 19:22:13.245270 7f5a65cd1700 -1 *** Caught signal >> > (Aborted) ** >> > in thread 7f5a65cd1700 >> > >> > ceph version 0.80-rc1-16-g2708c3c >> > (2708c3c559d99e6f3b557ee1d223efa3745f655c) >> > 1: /usr/bin/ceph-osd() [0xa844b0] >> > 2: (()+0xfcb0) [0x7f5a7f493cb0] >> > 3: (gsignal()+0x35) [0x7f5a7d978425] >> > 4: (abort()+0x17b) [0x7f5a7d97bb8b] >> > 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f5a7e274e2d] >> > 6: (()+0x5ef26) [0x7f5a7e272f26] >> > 7: (()+0x5ef53) [0x7f5a7e272f53] >> > 8: (()+0x5f17e) [0x7f5a7e27317e] >> > 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char >> > const*)+0x43d) [0xb5e15d] >> > 10: (Pipe::connect()+0x3b61) [0xc18ab1] >> > 11: (Pipe::writer()+0x65f) [0xc193ef] >> > 12: (Pipe::Writer::entry()+0xd) [0xc240ad] >> > 13: (()+0x7e9a) [0x7f5a7f48be9a] >> > 14: (clone()+0x6d) [0x7f5a7da363fd] >> > NOTE: a copy of the executable, or `objdump -rdS <executable>` is >> > needed to >> > interpret this. >> > >> > --- logging levels --- >> > 0/ 5 none >> > 0/ 1 lockdep >> > 0/ 1 context >> > 1/ 1 crush >> > 1/ 5 mds >> > 1/ 5 mds_balancer >> > 1/ 5 mds_locker >> > 1/ 5 mds_log >> > 1/ 5 mds_log_expire >> > 1/ 5 mds_migrator >> > 0/ 1 buffer >> > 0/ 1 timer >> > 0/ 1 filer >> > 0/ 1 striper >> > 0/ 1 objecter >> > 0/ 5 rados >> > 0/ 5 rbd >> > 0/ 5 journaler >> > 0/ 5 objectcacher >> > 0/ 5 client >> > 0/ 5 osd >> > 0/ 5 optracker >> > 0/ 5 objclass >> > 1/ 3 filestore >> > 1/ 3 keyvaluestore >> > 1/ 3 journal >> > 0/ 5 ms >> > 1/ 5 mon >> > 0/10 monc >> > 1/ 5 paxos >> > 0/ 5 tp >> > 1/ 5 auth >> > 1/ 5 crypto >> > 1/ 1 finisher >> > 1/ 5 heartbeatmap >> > 1/ 5 perfcounter >> > 1/ 5 rgw >> > 1/ 5 javaclient >> > 1/ 5 asok >> > 1/ 1 throttle >> > -2/-2 (syslog threshold) >> > -1/-1 (stderr threshold) >> > max_recent 10000 >> > max_new 1000 >> > log_file /var/log/ceph/ceph-osd.18.log >> > --- end dump of recent events --- >> > >> > please help me to fix this issue and let me known if you need more >> > information. >> > >> > Best regards, >> > Thanh Tran >> > >> > _______________________________________________ >> > ceph-users mailing list >> > ceph-users at lists.ceph.com >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > >
 |