hello,
all
I can't restart most osds (version 0.80.4 or 0.80.7) in a large scale ceph cluster(nearly 1000 osd nodes), when the osds crashed.
Because there are many near full osds currently, so i add 128 new osds to the cluster, and then the osds crashed
one by one. Finally, there are only few osds alive.
Is there any one can help me?
osd restart log looks like following:
Nov 10 09:37:52 node ceph-osd: -18> 2014-11-10 09:37:52.481963 7fdad4fb47c0 10 register_cxx_method statelog.remove flags 3 0x7fdac3af5210
Nov 10 09:37:52 node ceph-osd: -17> 2014-11-10 09:37:52.481964 7fdad4fb47c0 10 register_cxx_method statelog.check_state flags 1 0x7fdac3af5a00
Nov 10 09:37:52 node ceph-osd: -16> 2014-11-10 09:37:52.481966 7fdad4fb47c0 10 _load_class statelog success
Nov 10 09:37:52 node ceph-osd: -15> 2014-11-10 09:37:52.481983 7fdad4fb47c0 20 osd.376 0 get_map 240754 - loading and decoding 0x7fdad85e3e00
Nov 10 09:37:52 node ceph-osd: -14> 2014-11-10 09:37:52.481990 7fdad4fb47c0 15 filestore(/osd/ceph/osd) read meta/c1128c24/osdmap.240754/0//-1 0~0
Nov 10 09:37:52 node ceph-osd: -13> 2014-11-10 09:37:52.482378 7fdad4fb47c0 10 filestore(/osd/ceph/osd) FileStore::read meta/c1128c24/osdmap.240754/0//-1 0~842562/842562
Nov 10 09:37:52 node ceph-osd: -12> 2014-11-10 09:37:52.482389 7fdad4fb47c0 10 osd.376 0 add_map_bl 240754 842562 bytes
Nov 10 09:37:52 node ceph-osd: -11> 2014-11-10 09:37:52.487386 7fdad4fb47c0 0 osd.376 240754 crush map has features 1074003968, adjusting msgr requires for clients
Nov 10 09:37:52 node ceph-osd: -10> 2014-11-10 09:37:52.487402 7fdad4fb47c0 0 osd.376 240754 crush map has features 1074003968, adjusting msgr requires for mons
Nov 10 09:37:52 node ceph-osd: -9> 2014-11-10 09:37:52.487405 7fdad4fb47c0 0 osd.376 240754 crush map has features 1074003968, adjusting msgr requires for osds
Nov 10 09:37:52 node ceph-osd: -8> 2014-11-10 09:37:52.487408 7fdad4fb47c0 10 osd.376 240754 create_recoverystate_perf
Nov 10 09:37:52 node ceph-osd: -7> 2014-11-10 09:37:52.487420 7fdad4fb47c0 0 osd.376 240754 load_pgs
Nov 10 09:37:52 node ceph-osd: -6> 2014-11-10 09:37:52.487423 7fdad4fb47c0 10 filestore(/osd/ceph/osd) list_collections
Nov 10 09:37:52 node ceph-osd: -5> 2014-11-10 09:37:52.487882 7fdad4fb47c0 10 osd.376 240754 load_pgs ignoring unrecognized meta
Nov 10 09:37:52 node ceph-osd: -4> 2014-11-10 09:37:52.488232 7fdad4fb47c0 10 osd.376 240754 pgid 4.33 coll 4.33_head
Nov 10 09:37:52 node ceph-osd: -3> 2014-11-10 09:37:52.488252 7fdad4fb47c0 15 filestore(/osd/ceph/osd) collection_getattr /osd/ceph/osd/current/4.33_head 'info'
Nov 10 09:37:52 node ceph-osd: -2> 2014-11-10 09:37:52.488279 7fdad4fb47c0 10 filestore(/osd/ceph/osd) collection_getattr /osd/ceph/osd/current/4.33_head 'info' = 1
Nov 10 09:37:52 node ceph-osd: -1> 2014-11-10 09:37:52.488289 7fdad4fb47c0 15 filestore(/osd/ceph/osd) omap_get_values meta/16ef7597/infos/head//-1
Nov 10 09:37:52 node ceph-osd: 0> 2014-11-10 09:37:52.489924 7fdad4fb47c0 -1 osd/PG.cc: In function 'static epoch_t PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, ceph::bufferlist*)' thread 7fdad4fb47c0 time 2014-11-10 09:37:52.488828#012osd/PG.cc: 2597: FAILED assert(values.size() == 1)#012#012 ceph version 0.80.4 (7c241cfaa6c8c068bc9da8578ca00b9f4fc7567f)#012 1: (PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, ceph::buffer::list*)+0x578) [0x7fdad5345f78]#012 2: (OSD::load_pgs()+0x1a0f) [0x7fdad525a8bf]#012 3: (OSD::init()+0x1ba1) [0x7fdad5264ca1]#012 4: (main()+0x1f12) [0x7fdad52074b2]#012 5: (__libc_start_main()+0xed) [0x7fdad2dae76d]#012 6: (()+0x23e689) [0x7fdad520b689]#012 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Nov 10 09:37:52 node ceph-osd: --- logging levels ---
Nov 10 09:37:52 node ceph-osd: 0/ 5 none
Nov 10 09:37:52 node ceph-osd: 0/ 1 lockdep
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
