Re: cannot add new OSDs in mimic

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Fri, Jun 8, 2018 at 4:09 PM, Paul Emmerich <paul.emmerich@xxxxxxxx> wrote:
> Hi,
>
> we are also seeing this (I've also posted to the issue tracker). It only
> affects clusters upgraded from Luminous, not new ones.
> Also, it's not about re-using OSDs. Deleting any OSD seems to trigger this
> bug for all new OSDs on upgraded clusters.
>
> We are still using the pre-Luminous way to remove OSDs, i.e.:
>
> * ceph osd down/stop service
> * ceph osd crush remove
> * ceph osd auth del
> * ceph osd rm

Thanks for the easy steps to recreate, I will follow up on this one.

>
> Paul
>
>
> 2018-06-08 22:14 GMT+02:00 Michael Kuriger <mk7193@xxxxxxxxx>:
>>
>> Hi everyone,
>> I appreciate the suggestions. However, this is still an issue. I've tried
>> adding the OSD using ceph-deploy, and manually from the OSD host. I'm not
>> able to start newly added OSDs at all, even if I use a new ID. It seems the
>> OSD is added to CEPH but I cannot start it. OSDs that existed prior to the
>> upgrade to mimic are working fine. Here is a copy of an OSD log entry.
>>
>> osd.58 0 failed to load OSD map for epoch 378084, got 0 bytes
>>
>> fsid 1ce494ac-a218-4141-9d4f-295e6fa12f2a
>> last_changed 2018-06-05 15:40:50.179880
>> created 0.000000
>> 0: 10.3.71.36:6789/0 mon.ceph-mon3
>> 1: 10.3.74.109:6789/0 mon.ceph-mon2
>> 2: 10.3.74.214:6789/0 mon.ceph-mon1
>>
>>    -91> 2018-06-08 12:48:20.697 7fada058e700  1 -- 10.3.56.69:6800/1807239
>> <== mon.0 10.3.71.36:6789/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ====
>> 194+0+0 (645793352 0 0) 0x559f7a3dafc0 con 0x559f7994ec00
>>    -90> 2018-06-08 12:48:20.697 7fada058e700 10 monclient:
>> _check_auth_rotating have uptodate secrets (they expire after 2018-06-08
>> 12:47:50.699337)
>>    -89> 2018-06-08 12:48:20.698 7fadbc9d7140 10 monclient:
>> wait_auth_rotating done
>>    -88> 2018-06-08 12:48:20.698 7fadbc9d7140 10 monclient: _send_command 1
>> [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["58"]}]
>>    -87> 2018-06-08 12:48:20.698 7fadbc9d7140 10 monclient:
>> _send_mon_message to mon.ceph-mon3 at 10.3.71.36:6789/0
>>    -86> 2018-06-08 12:48:20.698 7fadbc9d7140  1 -- 10.3.56.69:6800/1807239
>> --> 10.3.71.36:6789/0 -- mon_command({"prefix": "osd crush
>> set-device-class", "class": "hdd", "ids": ["58"]} v 0) v1 -- 0x559f793e73c0
>> con 0
>>    -85> 2018-06-08 12:48:20.700 7fadabaa4700  5 -- 10.3.56.69:6800/1807239
>> >> 10.3.71.36:6789/0 conn(0x559f7994ec00 :-1
>> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25741 cs=1 l=1). rx mon.0
>> seq 8 0x559f793e73c0 mon_command_ack([{"prefix": "osd crush
>> set-device-class", "class": "hdd", "ids": ["58"]}]=0 osd.58 already set to
>> class hdd. set-device-class item id 58 name 'osd.58' device_class 'hdd': no
>> change.  v378738) v1
>>    -84> 2018-06-08 12:48:20.701 7fada058e700  1 -- 10.3.56.69:6800/1807239
>> <== mon.0 10.3.71.36:6789/0 8 ==== mon_command_ack([{"prefix": "osd crush
>> set-device-class", "class": "hdd", "ids": ["58"]}]=0 osd.58 already set to
>> class hdd. set-device-class item id 58 name 'osd.58' device_class 'hdd': no
>> change.  v378738) v1 ==== 211+0+0 (4063854475 0 0) 0x559f793e73c0 con
>> 0x559f7994ec00
>>    -83> 2018-06-08 12:48:20.701 7fada058e700 10 monclient:
>> handle_mon_command_ack 1 [{"prefix": "osd crush set-device-class", "class":
>> "hdd", "ids": ["58"]}]
>>    -82> 2018-06-08 12:48:20.701 7fada058e700 10 monclient: _finish_command
>> 1 = 0 osd.58 already set to class hdd. set-device-class item id 58 name
>> 'osd.58' device_class 'hdd': no change.
>>    -81> 2018-06-08 12:48:20.701 7fadbc9d7140 10 monclient: _send_command 2
>> [{"prefix": "osd crush create-or-move", "id": 58, "weight":0.5240, "args":
>> ["host=sacephnode12", "root=default"]}]
>>    -80> 2018-06-08 12:48:20.701 7fadbc9d7140 10 monclient:
>> _send_mon_message to mon.ceph-mon3 at 10.3.71.36:6789/0
>>    -79> 2018-06-08 12:48:20.701 7fadbc9d7140  1 -- 10.3.56.69:6800/1807239
>> --> 10.3.71.36:6789/0 -- mon_command({"prefix": "osd crush create-or-move",
>> "id": 58, "weight":0.5240, "args": ["host=sacephnode12", "root=default"]} v
>> 0) v1 -- 0x559f793e7600 con 0
>>    -78> 2018-06-08 12:48:20.703 7fadabaa4700  5 -- 10.3.56.69:6800/1807239
>> >> 10.3.71.36:6789/0 conn(0x559f7994ec00 :-1
>> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25741 cs=1 l=1). rx mon.0
>> seq 9 0x559f793e7600 mon_command_ack([{"prefix": "osd crush create-or-move",
>> "id": 58, "weight":0.5240, "args": ["host=sacephnode12", "root=default"]}]=0
>> create-or-move updated item name 'osd.58' weight 0.524 at location
>> {host=sacephnode12,root=default} to crush map v378738) v1
>>    -77> 2018-06-08 12:48:20.703 7fada058e700  1 -- 10.3.56.69:6800/1807239
>> <== mon.0 10.3.71.36:6789/0 9 ==== mon_command_ack([{"prefix": "osd crush
>> create-or-move", "id": 58, "weight":0.5240, "args": ["host=sacephnode12",
>> "root=default"]}]=0 create-or-move updated item name 'osd.58' weight 0.524
>> at location {host=sacephnode12,root=default} to crush map v378738) v1 ====
>> 258+0+0 (1998484028 0 0) 0x559f793e7600 con 0x559f7994ec00
>>    -76> 2018-06-08 12:48:20.703 7fada058e700 10 monclient:
>> handle_mon_command_ack 2 [{"prefix": "osd crush create-or-move", "id": 58,
>> "weight":0.5240, "args": ["host=sacephnode12", "root=default"]}]
>>    -75> 2018-06-08 12:48:20.703 7fada058e700 10 monclient: _finish_command
>> 2 = 0 create-or-move updated item name 'osd.58' weight 0.524 at location
>> {host=sacephnode12,root=default} to crush map
>>    -74> 2018-06-08 12:48:20.703 7fadbc9d7140  0 osd.58 0 done with init,
>> starting boot process
>>    -73> 2018-06-08 12:48:20.703 7fadbc9d7140 10 monclient: _renew_subs
>>    -72> 2018-06-08 12:48:20.703 7fadbc9d7140 10 monclient:
>> _send_mon_message to mon.ceph-mon3 at 10.3.71.36:6789/0
>>    -71> 2018-06-08 12:48:20.703 7fadbc9d7140  1 -- 10.3.56.69:6800/1807239
>> --> 10.3.71.36:6789/0 -- mon_subscribe({mgrmap=0+,osd_pg_creates=0+}) v3 --
>> 0x559f79408e00 con 0
>>    -70> 2018-06-08 12:48:20.703 7fadbc9d7140  1 osd.58 0 start_boot
>>    -69> 2018-06-08 12:48:20.703 7fadbc9d7140 10 monclient: get_version
>> osdmap req 0x559f797667a0
>>    -68> 2018-06-08 12:48:20.703 7fadbc9d7140 10 monclient:
>> _send_mon_message to mon.ceph-mon3 at 10.3.71.36:6789/0
>>    -67> 2018-06-08 12:48:20.703 7fadbc9d7140  1 -- 10.3.56.69:6800/1807239
>> --> 10.3.71.36:6789/0 -- mon_get_version(what=osdmap handle=1) v1 --
>> 0x559f79434b40 con 0
>>    -66> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command status hook 0x559f793f0700
>>    -65> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command flush_journal hook 0x559f793f0700
>>    -64> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command dump_ops_in_flight hook 0x559f793f0700
>>    -63> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command ops hook 0x559f793f0700
>>    -62> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command dump_blocked_ops hook 0x559f793f0700
>>    -61> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command dump_historic_ops hook 0x559f793f0700
>>    -60> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command dump_historic_slow_ops hook 0x559f793f0700
>>    -59> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command dump_historic_ops_by_duration hook 0x559f793f0700
>>    -58> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command dump_op_pq_state hook 0x559f793f0700
>>    -57> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command dump_blacklist hook 0x559f793f0700
>>    -56> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command dump_watchers hook 0x559f793f0700
>>    -55> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command dump_reservations hook 0x559f793f0700
>>    -54> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command get_latest_osdmap hook 0x559f793f0700
>>    -53> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command heap hook 0x559f793f0700
>>    -52> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command set_heap_property hook 0x559f793f0700
>>    -51> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command get_heap_property hook 0x559f793f0700
>>    -50> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command dump_objectstore_kv_stats hook 0x559f793f0700
>>    -49> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command dump_scrubs hook 0x559f793f0700
>>    -48> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command calc_objectstore_db_histogram hook 0x559f793f0700
>>    -47> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command flush_store_cache hook 0x559f793f0700
>>    -46> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command dump_pgstate_history hook 0x559f793f0700
>>    -45> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command compact hook 0x559f793f0700
>>    -44> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command get_mapped_pools hook 0x559f793f0700
>>    -43> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command smart hook 0x559f793f0700
>>    -42> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command list_devices hook 0x559f793f0700
>>    -41> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command setomapval hook 0x559f79767280
>>    -40> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command rmomapkey hook 0x559f79767280
>>    -39> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command setomapheader hook 0x559f79767280
>>    -38> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command getomap hook 0x559f79767280
>>    -37> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command truncobj hook 0x559f79767280
>>    -36> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command injectdataerr hook 0x559f79767280
>>    -35> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command injectmdataerr hook 0x559f79767280
>>    -34> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command set_recovery_delay hook 0x559f79767280
>>    -33> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command trigger_scrub hook 0x559f79767280
>>    -32> 2018-06-08 12:48:20.703 7fadbc9d7140  5 asok(0x559f794345a0)
>> register_command injectfull hook 0x559f79767280
>>    -31> 2018-06-08 12:48:20.704 7fadabaa4700  5 -- 10.3.56.69:6800/1807239
>> >> 10.3.71.36:6789/0 conn(0x559f7994ec00 :-1
>> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25741 cs=1 l=1). rx mon.0
>> seq 10 0x559f7958f8c0 mgrmap(e 201) v1
>>    -30> 2018-06-08 12:48:20.704 7fadabaa4700  5 -- 10.3.56.69:6800/1807239
>> >> 10.3.71.36:6789/0 conn(0x559f7994ec00 :-1
>> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25741 cs=1 l=1). rx mon.0
>> seq 11 0x559f79434b40 mon_get_version_reply(handle=1 version=378738) v2
>>    -29> 2018-06-08 12:48:20.704 7fada058e700  1 -- 10.3.56.69:6800/1807239
>> <== mon.0 10.3.71.36:6789/0 10 ==== mgrmap(e 201) v1 ==== 1776+0+0
>> (412200892 0 0) 0x559f7958f8c0 con 0x559f7994ec00
>>    -28> 2018-06-08 12:48:20.704 7fada058e700  4 mgrc handle_mgr_map Got
>> map version 201
>>    -27> 2018-06-08 12:48:20.704 7fada058e700  4 mgrc handle_mgr_map Active
>> mgr is now 10.3.74.109:6801/1015
>>    -26> 2018-06-08 12:48:20.704 7fada058e700  4 mgrc reconnect Starting
>> new session with 10.3.74.109:6801/1015
>>    -25> 2018-06-08 12:48:20.706 7fadac2a5700  2 -- 10.3.56.69:6800/1807239
>> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1
>> s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got
>> newly_acked_seq 0 vs out_seq 0
>>    -24> 2018-06-08 12:48:20.706 7fada058e700  1 -- 10.3.56.69:6800/1807239
>> --> 10.3.74.109:6801/1015 -- mgropen(unknown.58) v3 -- 0x559f79a9c000 con 0
>>    -23> 2018-06-08 12:48:20.706 7fada058e700  1 -- 10.3.56.69:6800/1807239
>> <== mon.0 10.3.71.36:6789/0 11 ==== mon_get_version_reply(handle=1
>> version=378738) v2 ==== 24+0+0 (2329122009 0 0) 0x559f79434b40 con
>> 0x559f7994ec00
>>    -22> 2018-06-08 12:48:20.706 7fada058e700 10 monclient:
>> handle_get_version_reply finishing 0x559f797667a0 version 378738
>>    -21> 2018-06-08 12:48:20.706 7fad96a13700  5 osd.58 0 heartbeat:
>> osd_stat(1.0 GiB used, 536 GiB avail, 537 GiB total, peers [] op hist [])
>>    -20> 2018-06-08 12:48:20.706 7fad96a13700 -1 osd.58 0 waiting for
>> initial osdmap
>>    -19> 2018-06-08 12:48:20.706 7fad96a13700 10 monclient: _renew_subs
>>    -18> 2018-06-08 12:48:20.706 7fad96a13700 10 monclient:
>> _send_mon_message to mon.ceph-mon3 at 10.3.71.36:6789/0
>>    -17> 2018-06-08 12:48:20.706 7fad96a13700  1 -- 10.3.56.69:6800/1807239
>> --> 10.3.71.36:6789/0 -- mon_subscribe({osdmap=378084}) v3 -- 0x559f7a3b8400
>> con 0
>>    -16> 2018-06-08 12:48:20.707 7fadac2a5700  5 -- 10.3.56.69:6800/1807239
>> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1
>> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=201245 cs=1 l=1). rx
>> mgr.44007797 seq 1 0x559f79435860 mgrconfigure(period=5, threshold=5) v2
>>    -15> 2018-06-08 12:48:20.708 7fada058e700  1 -- 10.3.56.69:6800/1807239
>> <== mgr.44007797 10.3.74.109:6801/1015 1 ==== mgrconfigure(period=5,
>> threshold=5) v2 ==== 8+0+0 (3460719617 0 0) 0x559f79435860 con
>> 0x559f79950a00
>>    -14> 2018-06-08 12:48:20.708 7fada058e700  4 mgrc handle_mgr_configure
>> stats_period=5
>>    -13> 2018-06-08 12:48:20.708 7fada058e700  4 mgrc handle_mgr_configure
>> updated stats threshold: 5
>>    -12> 2018-06-08 12:48:20.708 7fadabaa4700  5 -- 10.3.56.69:6800/1807239
>> >> 10.3.71.36:6789/0 conn(0x559f7994ec00 :-1
>> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25741 cs=1 l=1). rx mon.0
>> seq 12 0x559f79aba000 osd_map(378085..378085 src has 378085..378738
>> +gap_removed_snaps) v4
>>    -11> 2018-06-08 12:48:20.708 7fada058e700  1 -- 10.3.56.69:6800/1807239
>> --> 10.3.74.109:6801/1015 -- mgrreport(unknown.58 +54-0 packed 742) v6 --
>> 0x559f79a9c300 con 0
>>    -10> 2018-06-08 12:48:20.708 7fada058e700  1 -- 10.3.56.69:6800/1807239
>> --> 10.3.74.109:6801/1015 -- pg_stats(0 pgs tid 0 v 0) v1 -- 0x559f7958f600
>> con 0
>>     -9> 2018-06-08 12:48:20.708 7fada058e700  1 -- 10.3.56.69:6800/1807239
>> <== mon.0 10.3.71.36:6789/0 12 ==== osd_map(378085..378085 src has
>> 378085..378738 +gap_removed_snaps) v4 ==== 33348+0+0 (2799879432 0 0)
>> 0x559f79aba000 con 0x559f7994ec00
>>     -8> 2018-06-08 12:48:20.708 7fada058e700  3 osd.58 0 handle_osd_map
>> epochs [378085,378085], i have 0, src has [378085,378738]
>>     -7> 2018-06-08 12:48:20.709 7fadabaa4700  5 -- 10.3.56.69:6800/1807239
>> >> 10.3.71.36:6789/0 conn(0x559f7994ec00 :-1
>> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25741 cs=1 l=1). rx mon.0
>> seq 13 0x559f79abaa00 osd_map(378086..378125 src has 378085..378738) v4
>>     -6> 2018-06-08 12:48:20.709 7fada058e700 -1 osd.58 0 failed to load
>> OSD map for epoch 378084, got 0 bytes
>>     -5> 2018-06-08 12:48:20.710 7fadac2a5700  1 -- 10.3.56.69:6800/1807239
>> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1 s=STATE_OPEN pgs=201245
>> cs=1 l=1).read_bulk peer close file descriptor 38
>>     -4> 2018-06-08 12:48:20.710 7fadac2a5700  1 -- 10.3.56.69:6800/1807239
>> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1 s=STATE_OPEN pgs=201245
>> cs=1 l=1).read_until read failed
>>     -3> 2018-06-08 12:48:20.710 7fadac2a5700  1 -- 10.3.56.69:6800/1807239
>> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1 s=STATE_OPEN pgs=201245
>> cs=1 l=1).process read tag failed
>>     -2> 2018-06-08 12:48:20.710 7fadac2a5700  1 -- 10.3.56.69:6800/1807239
>> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1 s=STATE_OPEN pgs=201245
>> cs=1 l=1).fault on lossy channel, failing
>>     -1> 2018-06-08 12:48:20.710 7fadac2a5700  2 -- 10.3.56.69:6800/1807239
>> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1 s=STATE_OPEN pgs=201245
>> cs=1 l=1)._stop
>>      0> 2018-06-08 12:48:20.711 7fada058e700 -1
>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.0/rpm/el7/BUILD/ceph-13.2.0/src/osd/OSD.h:
>> In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7fada058e700
>> time 2018-06-08 12:48:20.710675
>>
>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.0/rpm/el7/BUILD/ceph-13.2.0/src/osd/OSD.h:
>> 828: FAILED assert(ret)
>>
>>  ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic
>> (stable)
>>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0xff) [0x7fadb3e1753f]
>>  2: (()+0x286727) [0x7fadb3e17727]
>>  3: (OSDService::get_map(unsigned int)+0x4a) [0x559f76fe4dda]
>>  4: (OSD::handle_osd_map(MOSDMap*)+0x1020) [0x559f76f921f0]
>>  5: (OSD::_dispatch(Message*)+0xa1) [0x559f76f94d21]
>>  6: (OSD::ms_dispatch(Message*)+0x56) [0x559f76f95066]
>>  7: (DispatchQueue::entry()+0xb5a) [0x7fadb3e8d74a]
>>  8: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fadb3f2df2d]
>>  9: (()+0x7e25) [0x7fadb0afde25]
>>  10: (clone()+0x6d) [0x7fadafbf134d]
>>  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
>> to interpret this.
>>
>> --- logging levels ---
>>    0/ 5 none
>>    0/ 1 lockdep
>>    0/ 1 context
>>    1/ 1 crush
>>    1/ 5 mds
>>    1/ 5 mds_balancer
>>    1/ 5 mds_locker
>>    1/ 5 mds_log
>>    1/ 5 mds_log_expire
>>    1/ 5 mds_migrator
>>    0/ 1 buffer
>>    0/ 1 timer
>>    0/ 1 filer
>>    0/ 1 striper
>>    0/ 1 objecter
>>    0/ 5 rados
>>    0/ 5 rbd
>>    0/ 5 rbd_mirror
>>    0/ 5 rbd_replay
>>    0/ 5 journaler
>>    0/ 5 objectcacher
>>    0/ 5 client
>>    1/ 5 osd
>>    0/ 5 optracker
>>    0/ 5 objclass
>>    1/ 3 filestore
>>    1/ 3 journal
>>    0/ 5 ms
>>    1/ 5 mon
>>    0/10 monc
>>    1/ 5 paxos
>>    0/ 5 tp
>>    1/ 5 auth
>>    1/ 5 crypto
>>    1/ 1 finisher
>>    1/ 1 reserver
>>    1/ 5 heartbeatmap
>>    1/ 5 perfcounter
>>    1/ 5 rgw
>>    1/ 5 rgw_sync
>>    1/10 civetweb
>>    1/ 5 javaclient
>>    1/ 5 asok
>>    1/ 1 throttle
>>    0/ 0 refs
>>    1/ 5 xio
>>    1/ 5 compressor
>>    1/ 5 bluestore
>>    1/ 5 bluefs
>>    1/ 3 bdev
>>    1/ 5 kstore
>>    4/ 5 rocksdb
>>    4/ 5 leveldb
>>    4/ 5 memdb
>>    1/ 5 kinetic
>>    1/ 5 fuse
>>    1/ 5 mgr
>>    1/ 5 mgrc
>>    1/ 5 dpdk
>>    1/ 5 eventtrace
>>   -2/-2 (syslog threshold)
>>   -1/-1 (stderr threshold)
>>   max_recent     10000
>>   max_new         1000
>>   log_file /var/log/ceph/ceph-osd.58.log
>> --- end dump of recent events ---
>> 2018-06-08 12:48:20.717 7fada058e700 -1 *** Caught signal (Aborted) **
>>  in thread 7fada058e700 thread_name:ms_dispatch
>>
>>  ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic
>> (stable)
>>  1: (()+0x8e1870) [0x559f774af870]
>>  2: (()+0xf5e0) [0x7fadb0b055e0]
>>  3: (gsignal()+0x37) [0x7fadafb2e1f7]
>>  4: (abort()+0x148) [0x7fadafb2f8e8]
>>  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x25d) [0x7fadb3e1769d]
>>  6: (()+0x286727) [0x7fadb3e17727]
>>  7: (OSDService::get_map(unsigned int)+0x4a) [0x559f76fe4dda]
>>  8: (OSD::handle_osd_map(MOSDMap*)+0x1020) [0x559f76f921f0]
>>  9: (OSD::_dispatch(Message*)+0xa1) [0x559f76f94d21]
>>  10: (OSD::ms_dispatch(Message*)+0x56) [0x559f76f95066]
>>  11: (DispatchQueue::entry()+0xb5a) [0x7fadb3e8d74a]
>>  12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fadb3f2df2d]
>>  13: (()+0x7e25) [0x7fadb0afde25]
>>  14: (clone()+0x6d) [0x7fadafbf134d]
>>  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
>> to interpret this.
>>
>> --- begin dump of recent events ---
>>      0> 2018-06-08 12:48:20.717 7fada058e700 -1 *** Caught signal
>> (Aborted) **
>>  in thread 7fada058e700 thread_name:ms_dispatch
>>
>>  ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic
>> (stable)
>>  1: (()+0x8e1870) [0x559f774af870]
>>  2: (()+0xf5e0) [0x7fadb0b055e0]
>>  3: (gsignal()+0x37) [0x7fadafb2e1f7]
>>  4: (abort()+0x148) [0x7fadafb2f8e8]
>>  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x25d) [0x7fadb3e1769d]
>>  6: (()+0x286727) [0x7fadb3e17727]
>>  7: (OSDService::get_map(unsigned int)+0x4a) [0x559f76fe4dda]
>>  8: (OSD::handle_osd_map(MOSDMap*)+0x1020) [0x559f76f921f0]
>>  9: (OSD::_dispatch(Message*)+0xa1) [0x559f76f94d21]
>>  10: (OSD::ms_dispatch(Message*)+0x56) [0x559f76f95066]
>>  11: (DispatchQueue::entry()+0xb5a) [0x7fadb3e8d74a]
>>  12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fadb3f2df2d]
>>  13: (()+0x7e25) [0x7fadb0afde25]
>>  14: (clone()+0x6d) [0x7fadafbf134d]
>>  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
>> to interpret this.
>>
>>
>> Mike Kuriger
>>
>> >> CEPH team,
>> >> Is there a solution yet for adding OSDs in mimic - specifically
>> >> re-using old IDs?  I was looking over this BUG report -
>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issues_24423&d=DwIFaQ&c=5m9CfXHY6NXqkS7nN5n23w&r=5r9bhr1JAPRaUcJcU-FfGg&m=0PCKiecm216R95S_krqboYMskCBoolGysrvgHZo8LEM&s=hfI2uudTfY0lGtBI6iIXvZWvNpme4xwBJe2SWx0_N3I&e=
>> >> and my issue is similar.  I removed a bunch of OSD's after upgrading to
>> >> mimic and I'm not able to re-add them using the new volume format.  I
>> >> haven't tried manually adding them using 'never used' IDs.  I'll try that
>> >> now but was hoping there would be a fix.
>> >>
>> >> Thanks!
>> >>
>> >> Mike Kuriger
>> >>
>> >> _______________________________________________
>> >> ceph-users mailing list
>> >> ceph-users@xxxxxxxxxxxxxx
>> >>
>> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.ceph.com_listinfo.cgi_ceph-2Dusers-2Dceph.com&d=DwIFaQ&c=5m9CfXHY6NXqkS7nN5n23w&r=5r9bhr1JAPRaUcJcU-FfGg&m=0PCKiecm216R95S_krqboYMskCBoolGysrvgHZo8LEM&s=2aoWc5hTz041_26Stz6zPtLiB5zGFw2GbX3TPjsvieI&e=
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
>
> --
> Paul Emmerich
>
> Looking for help with your Ceph cluster? Contact us at https://croit.io
>
> croit GmbH
> Freseniusstr. 31h
> 81247 München
> www.croit.io
> Tel: +49 89 1896585 90
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux