Hi all,
I've just attempted to add a new node and OSD to an existing ceph cluster (it's a small one I use as a NAS at home, not like the big production ones I normally work on) and it seems to be throwing some odd errors...
Just looking for where to poke it next...
Log is below,
It's a two node cluster with 3 osd's in node A and one osd in the new node (It's going to have more eventually and node one will be retired after node three gets added)
And I've hit a weird snag.
I was running 0.80 but I ran into the 'Invalid Command' bug on the new node so I opted to jump to the latest code with the required patches already.
Please let me know what else you need..
This is the log content when attempting to start the new OSD:
2015-03-10 19:28:48.795318 7f0774108880 0 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4), process ceph-osd, pid 10810
2015-03-10 19:28:48.817803 7f0774108880 0 filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342)
2015-03-10 19:28:48.866862 7f0774108880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is supported and appears to work
2015-03-10 19:28:48.866920 7f0774108880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2015-03-10 19:28:48.905069 7f0774108880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2015-03-10 19:28:48.905467 7f0774108880 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is supported and kernel 3.18.3-1-desktop >= 3.5
2015-03-10 19:28:49.077872 7f0774108880 0 filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2015-03-10 19:28:49.078321 7f0774108880 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
2015-03-10 19:28:49.078328 7f0774108880 1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
2015-03-10 19:28:49.079721 7f0774108880 1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
2015-03-10 19:28:49.080948 7f0774108880 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello
2015-03-10 19:28:49.094194 7f0774108880 0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for clients
2015-03-10 19:28:49.094211 7f0774108880 0 osd.3 2757 crush map has features 33816576 was 8705, adjusting msgr requires for mons
2015-03-10 19:28:49.094217 7f0774108880 0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for osds
2015-03-10 19:28:49.094235 7f0774108880 0 osd.3 2757 load_pgs
2015-03-10 19:28:49.094279 7f0774108880 0 osd.3 2757 load_pgs opened 0 pgs
2015-03-10 19:28:49.095121 7f0774108880 -1 osd.3 2757 log_to_monitors {default=true}
2015-03-10 19:28:49.134104 7f0774108880 0 osd.3 2757 done with init, starting boot process
2015-03-10 19:28:49.149994 7f076384c700 -1 *** Caught signal (Aborted) **
in thread 7f076384c700
ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4)
1: /usr/bin/ceph-osd() [0xac7cea]
2: (()+0x10050) [0x7f0773013050]
3: (gsignal()+0x37) [0x7f07714e60f7]
4: (abort()+0x13a) [0x7f07714e74ca]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f0771dcbfe5]
6: (()+0x63186) [0x7f0771dca186]
7: (()+0x631b3) [0x7f0771dca1b3]
8: (()+0x633d2) [0x7f0771dca3d2]
9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7]
10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5]
11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc]
12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf]
13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7]
14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb]
15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007]
16: (DispatchQueue::entry()+0x649) [0xc6fe09]
17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d]
18: (()+0x83a4) [0x7f077300b3a4]
19: (clone()+0x6d) [0x7f0771595a4d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-135> 2015-03-10 19:28:48.790490 7f0774108880 5 asok(0x4200000) register_command perfcounters_dump hook 0x41b4030
-134> 2015-03-10 19:28:48.790565 7f0774108880 5 asok(0x4200000) register_command 1 hook 0x41b4030
-133> 2015-03-10 19:28:48.790571 7f0774108880 5 asok(0x4200000) register_command perf dump hook 0x41b4030
-132> 2015-03-10 19:28:48.790583 7f0774108880 5 asok(0x4200000) register_command perfcounters_schema hook 0x41b4030
-131> 2015-03-10 19:28:48.790599 7f0774108880 5 asok(0x4200000) register_command 2 hook 0x41b4030
-130> 2015-03-10 19:28:48.790605 7f0774108880 5 asok(0x4200000) register_command perf schema hook 0x41b4030
-129> 2015-03-10 19:28:48.790635 7f0774108880 5 asok(0x4200000) register_command perf reset hook 0x41b4030
-128> 2015-03-10 19:28:48.790646 7f0774108880 5 asok(0x4200000) register_command config show hook 0x41b4030
-127> 2015-03-10 19:28:48.790652 7f0774108880 5 asok(0x4200000) register_command config set hook 0x41b4030
-126> 2015-03-10 19:28:48.790660 7f0774108880 5 asok(0x4200000) register_command config get hook 0x41b4030
-125> 2015-03-10 19:28:48.790672 7f0774108880 5 asok(0x4200000) register_command config diff hook 0x41b4030
-124> 2015-03-10 19:28:48.790682 7f0774108880 5 asok(0x4200000) register_command log flush hook 0x41b4030
-123> 2015-03-10 19:28:48.790689 7f0774108880 5 asok(0x4200000) register_command log dump hook 0x41b4030
-122> 2015-03-10 19:28:48.790698 7f0774108880 5 asok(0x4200000) register_command log reopen hook 0x41b4030
-121> 2015-03-10 19:28:48.795318 7f0774108880 0 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4), process ceph-osd, pid 10810
-120> 2015-03-10 19:28:48.796071 7f0774108880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/10810 need_addr=1
-119> 2015-03-10 19:28:48.796123 7f0774108880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6801/10810 need_addr=1
-118> 2015-03-10 19:28:48.796144 7f0774108880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/10810 need_addr=1
-117> 2015-03-10 19:28:48.796160 7f0774108880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/10810 need_addr=1
-116> 2015-03-10 19:28:48.799414 7f0774108880 5 asok(0x4200000) init /var/run/ceph/ceph-osd.3.asok
-115> 2015-03-10 19:28:48.799446 7f0774108880 5 asok(0x4200000) bind_and_listen /var/run/ceph/ceph-osd.3.asok
-114> 2015-03-10 19:28:48.799644 7f0774108880 5 asok(0x4200000) register_command 0 hook 0x41b00c0
-113> 2015-03-10 19:28:48.799659 7f0774108880 5 asok(0x4200000) register_command version hook 0x41b00c0
-112> 2015-03-10 19:28:48.799665 7f0774108880 5 asok(0x4200000) register_command git_version hook 0x41b00c0
-111> 2015-03-10 19:28:48.799671 7f0774108880 5 asok(0x4200000) register_command help hook 0x41b4110
-110> 2015-03-10 19:28:48.799678 7f0774108880 5 asok(0x4200000) register_command get_command_descriptions hook 0x41b4120
-109> 2015-03-10 19:28:48.799760 7f0774108880 10 monclient(hunting): build_initial_monmap
-108> 2015-03-10 19:28:48.799757 7f076f4e1700 5 asok(0x4200000) entry start
-107> 2015-03-10 19:28:48.816916 7f0774108880 5 adding auth protocol: none
-106> 2015-03-10 19:28:48.816948 7f0774108880 5 adding auth protocol: none
-105> 2015-03-10 19:28:48.817130 7f0774108880 5 asok(0x4200000) register_command objecter_requests hook 0x41b41b0
-104> 2015-03-10 19:28:48.817273 7f0774108880 1 -- 0.0.0.0:6800/10810 messenger.start
-103> 2015-03-10 19:28:48.817334 7f0774108880 1 -- :/0 messenger.start
-102> 2015-03-10 19:28:48.817378 7f0774108880 1 -- 0.0.0.0:6803/10810 messenger.start
-101> 2015-03-10 19:28:48.817442 7f0774108880 1 -- 0.0.0.0:6802/10810 messenger.start
-100> 2015-03-10 19:28:48.817494 7f0774108880 1 -- 0.0.0.0:6801/10810 messenger.start
-99> 2015-03-10 19:28:48.817546 7f0774108880 1 -- :/0 messenger.start
-98> 2015-03-10 19:28:48.817666 7f0774108880 2 osd.3 0 mounting /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
-97> 2015-03-10 19:28:48.817803 7f0774108880 0 filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342)
-96> 2015-03-10 19:28:48.866862 7f0774108880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is supported and appears to work
-95> 2015-03-10 19:28:48.866920 7f0774108880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
-94> 2015-03-10 19:28:48.905069 7f0774108880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
-93> 2015-03-10 19:28:48.905467 7f0774108880 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is supported and kernel 3.18.3-1-desktop >= 3.5
-92> 2015-03-10 19:28:49.077872 7f0774108880 0 filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
-91> 2015-03-10 19:28:49.078270 7f0774108880 2 journal open /var/lib/ceph/osd/ceph-3/journal fsid 05fb481f-68b3-4442-b7de-af4c867e18d5 fs_op_seq 2
-90> 2015-03-10 19:28:49.078321 7f0774108880 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
-89> 2015-03-10 19:28:49.078328 7f0774108880 1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
-88> 2015-03-10 19:28:49.078937 7f0774108880 2 journal read_entry 8192 : seq 3 193 bytes
-87> 2015-03-10 19:28:49.079000 7f0774108880 2 journal read_entry 8192 : seq 3 193 bytes
-86> 2015-03-10 19:28:49.079022 7f0774108880 3 journal journal_replay: applying op seq 3
-85> 2015-03-10 19:28:49.079307 7f0774108880 3 journal journal_replay: r = 0, op_seq now 3
-84> 2015-03-10 19:28:49.079367 7f0774108880 2 journal read_entry 12288 : seq 4 3829 bytes
-83> 2015-03-10 19:28:49.079390 7f0774108880 3 journal journal_replay: applying op seq 4
-82> 2015-03-10 19:28:49.079667 7f0774108880 3 journal journal_replay: r = 0, op_seq now 4
-81> 2015-03-10 19:28:49.079699 7f0774108880 2 journal No further valid entries found, journal is most likely valid
-80> 2015-03-10 19:28:49.079703 7f0774108880 3 journal journal_replay: end of journal, done.
-79> 2015-03-10 19:28:49.079721 7f0774108880 1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
-78> 2015-03-10 19:28:49.080265 7f0774108880 2 osd.3 0 boot
-77> 2015-03-10 19:28:49.080948 7f0774108880 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello
-76> 2015-03-10 19:28:49.081757 7f0774108880 1 <cls> cls/log/cls_log.cc:312: Loaded log class!
-75> 2015-03-10 19:28:49.087397 7f0774108880 1 <cls> cls/refcount/cls_refcount.cc:231: Loaded refcount class!
-74> 2015-03-10 19:28:49.087735 7f0774108880 1 <cls> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class!
-73> 2015-03-10 19:28:49.093126 7f0774108880 1 <cls> cls/rgw/cls_rgw.cc:3046: Loaded rgw class!
-72> 2015-03-10 19:28:49.093484 7f0774108880 1 <cls> cls/statelog/cls_statelog.cc:306: Loaded log class!
-71> 2015-03-10 19:28:49.093723 7f0774108880 1 <cls> cls/user/cls_user.cc:367: Loaded user class!
-70> 2015-03-10 19:28:49.093924 7f0774108880 1 <cls> cls/version/cls_version.cc:227: Loaded version class!
-69> 2015-03-10 19:28:49.094194 7f0774108880 0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for clients
-68> 2015-03-10 19:28:49.094211 7f0774108880 0 osd.3 2757 crush map has features 33816576 was 8705, adjusting msgr requires for mons
-67> 2015-03-10 19:28:49.094217 7f0774108880 0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for osds
-66> 2015-03-10 19:28:49.094235 7f0774108880 0 osd.3 2757 load_pgs
-65> 2015-03-10 19:28:49.094279 7f0774108880 0 osd.3 2757 load_pgs opened 0 pgs
-64> 2015-03-10 19:28:49.094285 7f0774108880 2 osd.3 2757 superblock: i am osd.3
-63> 2015-03-10 19:28:49.094424 7f0774108880 1 accepter.accepter.start
-62> 2015-03-10 19:28:49.094573 7f0774108880 1 accepter.accepter.start
-61> 2015-03-10 19:28:49.094748 7f0774108880 1 accepter.accepter.start
-60> 2015-03-10 19:28:49.094864 7f0774108880 1 accepter.accepter.start
-59> 2015-03-10 19:28:49.094963 7f0774108880 10 monclient(hunting): init
-58> 2015-03-10 19:28:49.094980 7f0774108880 5 adding auth protocol: none
-57> 2015-03-10 19:28:49.095005 7f0774108880 10 monclient(hunting): auth_supported 1 method none
-56> 2015-03-10 19:28:49.095110 7f0774108880 10 log_channel(default) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info)
-55> 2015-03-10 19:28:49.095121 7f0774108880 -1 osd.3 2757 log_to_monitors {default=true}
-54> 2015-03-10 19:28:49.095899 7f0774108880 10 monclient(hunting): renew_subs
-53> 2015-03-10 19:28:49.095914 7f0774108880 10 monclient(hunting): _reopen_session rank -1 name
-52> 2015-03-10 19:28:49.096003 7f0774108880 10 monclient(hunting): picked mon.a con 0x43623c0 addr 192.168.0.10:6789/0
-51> 2015-03-10 19:28:49.096086 7f0774108880 10 monclient(hunting): _send_mon_message to mon.a at 192.168.0.10:6789/0
-50> 2015-03-10 19:28:49.096098 7f0774108880 1 -- 0.0.0.0:6800/10810 --> 192.168.0.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x41fcd80 con 0x43623c0
-49> 2015-03-10 19:28:49.096132 7f0774108880 10 monclient(hunting): renew_subs
-48> 2015-03-10 19:28:49.096501 7f075302b700 5 osd.3 2757 heartbeat: osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist [])
-47> 2015-03-10 19:28:49.113808 7f0774104700 1 -- 192.168.0.250:6800/10810 learned my addr 192.168.0.250:6800/10810
-46> 2015-03-10 19:28:49.131334 7f076384c700 10 monclient(hunting): renew_subs
-45> 2015-03-10 19:28:49.133513 7f076384c700 1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x41ff840 con 0x43623c0
-44> 2015-03-10 19:28:49.133572 7f076384c700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
-43> 2015-03-10 19:28:49.133587 7f076384c700 10 monclient(hunting): got monmap 1, mon.a is now rank 0
-42> 2015-03-10 19:28:49.133597 7f076384c700 10 monclient(hunting): dump:
epoch 1
fsid 3bf19e54-97e2-4a5f-b811-7c3780cd5e57
last_changed 2013-09-11 22:41:30.819687
created 2013-09-11 22:41:30.819687
0: 192.168.0.10:6789/0 mon.a
-41> 2015-03-10 19:28:49.133678 7f076384c700 1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 2 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (1541693634 0 0) 0x41ffa80 con 0x43623c0
-40> 2015-03-10 19:28:49.133716 7f076384c700 10 monclient(hunting): my global_id is 14511543
-39> 2015-03-10 19:28:49.133721 7f076384c700 1 monclient(hunting): found mon.a
-38> 2015-03-10 19:28:49.133803 7f076384c700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
-37> 2015-03-10 19:28:49.133820 7f076384c700 1 -- 192.168.0.250:6800/10810 --> 192.168.0.10:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x41e1a00 con 0x43623c0
-36> 2015-03-10 19:28:49.133906 7f076384c700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
-35> 2015-03-10 19:28:49.133925 7f076384c700 1 -- 192.168.0.250:6800/10810 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0,osdmap=2757}) v2 -- ?+0 0x41e2800 con 0x43623c0
-34> 2015-03-10 19:28:49.134016 7f076384c700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2015-03-10 19:28:19.134014)
-33> 2015-03-10 19:28:49.134061 7f0774108880 5 monclient: authenticate success, global_id 14511543
-32> 2015-03-10 19:28:49.134104 7f0774108880 0 osd.3 2757 done with init, starting boot process
-31> 2015-03-10 19:28:49.134118 7f0774108880 10 monclient: get_version osdmap req 0x4259160
-30> 2015-03-10 19:28:49.134126 7f0774108880 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
-29> 2015-03-10 19:28:49.134136 7f0774108880 1 -- 192.168.0.250:6800/10810 --> 192.168.0.10:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x42741e0 con 0x43623c0
-28> 2015-03-10 19:28:49.134362 7f0774108880 5 asok(0x4200000) register_command status hook 0x41b4290
-27> 2015-03-10 19:28:49.134388 7f0774108880 5 asok(0x4200000) register_command flush_journal hook 0x41b4290
-26> 2015-03-10 19:28:49.134405 7f0774108880 5 asok(0x4200000) register_command dump_ops_in_flight hook 0x41b4290
-25> 2015-03-10 19:28:49.134421 7f0774108880 5 asok(0x4200000) register_command ops hook 0x41b4290
-24> 2015-03-10 19:28:49.134437 7f0774108880 5 asok(0x4200000) register_command dump_historic_ops hook 0x41b4290
-23> 2015-03-10 19:28:49.134454 7f0774108880 5 asok(0x4200000) register_command dump_op_pq_state hook 0x41b4290
-22> 2015-03-10 19:28:49.134469 7f0774108880 5 asok(0x4200000) register_command dump_blacklist hook 0x41b4290
-21> 2015-03-10 19:28:49.134484 7f0774108880 5 asok(0x4200000) register_command dump_watchers hook 0x41b4290
-20> 2015-03-10 19:28:49.134500 7f0774108880 5 asok(0x4200000) register_command dump_reservations hook 0x41b4290
-19> 2015-03-10 19:28:49.134516 7f0774108880 5 asok(0x4200000) register_command get_latest_osdmap hook 0x41b4290
-18> 2015-03-10 19:28:49.134532 7f0774108880 5 asok(0x4200000) register_command setomapval hook 0x4259200
-17> 2015-03-10 19:28:49.134546 7f0774108880 5 asok(0x4200000) register_command rmomapkey hook 0x4259200
-16> 2015-03-10 19:28:49.134563 7f0774108880 5 asok(0x4200000) register_command setomapheader hook 0x4259200
-15> 2015-03-10 19:28:49.134580 7f0774108880 5 asok(0x4200000) register_command getomap hook 0x4259200
-14> 2015-03-10 19:28:49.134595 7f0774108880 5 asok(0x4200000) register_command truncobj hook 0x4259200
-13> 2015-03-10 19:28:49.134609 7f0774108880 5 asok(0x4200000) register_command injectdataerr hook 0x4259200
-12> 2015-03-10 19:28:49.134624 7f0774108880 5 asok(0x4200000) register_command injectmdataerr hook 0x4259200
-11> 2015-03-10 19:28:49.136325 7f076384c700 1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 3 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x41ffcc0 con 0x43623c0
-10> 2015-03-10 19:28:49.136622 7f076384c700 3 osd.3 2757 handle_osd_map epochs [3277,3277], i have 2757, src has [2757,3277]
-9> 2015-03-10 19:28:49.136643 7f076384c700 10 monclient: renew_subs
-8> 2015-03-10 19:28:49.136652 7f076384c700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
-7> 2015-03-10 19:28:49.136679 7f076384c700 1 -- 192.168.0.250:6800/10810 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=2758}) v2 -- ?+0 0x41e1a00 con 0x43623c0
-6> 2015-03-10 19:28:49.136943 7f076384c700 1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 4 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x41ff840 con 0x43623c0
-5> 2015-03-10 19:28:49.136994 7f076384c700 10 monclient: handle_monmap mon_map magic: 0 v1
-4> 2015-03-10 19:28:49.137010 7f076384c700 10 monclient: got monmap 1, mon.a is now rank 0
-3> 2015-03-10 19:28:49.137022 7f076384c700 10 monclient: dump:
epoch 1
fsid 3bf19e54-97e2-4a5f-b811-7c3780cd5e57
last_changed 2013-09-11 22:41:30.819687
created 2013-09-11 22:41:30.819687
0: 192.168.0.10:6789/0 mon.a
-2> 2015-03-10 19:28:49.142203 7f076384c700 1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 5 ==== osd_map(2757..2857 src has 2757..3277) v3 ==== 37648+0+0 (2648067016 0 0) 0x41ffa80 con 0x43623c0
-1> 2015-03-10 19:28:49.142316 7f076384c700 3 osd.3 2757 handle_osd_map epochs [2757,2857], i have 2757, src has [2757,3277]
0> 2015-03-10 19:28:49.149994 7f076384c700 -1 *** Caught signal (Aborted) **
in thread 7f076384c700
ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4)
1: /usr/bin/ceph-osd() [0xac7cea]
2: (()+0x10050) [0x7f0773013050]
3: (gsignal()+0x37) [0x7f07714e60f7]
4: (abort()+0x13a) [0x7f07714e74ca]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f0771dcbfe5]
6: (()+0x63186) [0x7f0771dca186]
7: (()+0x631b3) [0x7f0771dca1b3]
8: (()+0x633d2) [0x7f0771dca3d2]
9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7]
10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5]
11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc]
12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf]
13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7]
14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb]
15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007]
16: (DispatchQueue::entry()+0x649) [0xc6fe09]
17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d]
18: (()+0x83a4) [0x7f077300b3a4]
19: (clone()+0x6d) [0x7f0771595a4d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.3.log
--- end dump of recent events ---
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
