On Fri, Jun 8, 2018 at 4:09 PM, Paul Emmerich <paul.emmerich@xxxxxxxx> wrote:
> Hi,
>
> we are also seeing this (I've also posted to the issue tracker). It only
> affects clusters upgraded from Luminous, not new ones.
> Also, it's not about re-using OSDs. Deleting any OSD seems to trigger this
> bug for all new OSDs on upgraded clusters.
>
> We are still using the pre-Luminous way to remove OSDs, i.e.:
>
> * ceph osd down/stop service
> * ceph osd crush remove
> * ceph osd auth del
> * ceph osd rm
Thanks for the easy steps to recreate, I will follow up on this one.
>
> Paul
>
>
> 2018-06-08 22:14 GMT+02:00 Michael Kuriger <mk7193@xxxxxxxxx>:
>>
>> Hi everyone,
>> I appreciate the suggestions. However, this is still an issue. I've tried
>> adding the OSD using ceph-deploy, and manually from the OSD host. I'm not
>> able to start newly added OSDs at all, even if I use a new ID. It seems the
>> OSD is added to CEPH but I cannot start it. OSDs that existed prior to the
>> upgrade to mimic are working fine. Here is a copy of an OSD log entry.
>>
>> osd.58 0 failed to load OSD map for epoch 378084, got 0 bytes
>>
>> fsid 1ce494ac-a218-4141-9d4f-295e6fa12f2a
>> last_changed 2018-06-05 15:40:50.179880
>> created 0.000000
>> 0: 10.3.71.36:6789/0 mon.ceph-mon3
>> 1: 10.3.74.109:6789/0 mon.ceph-mon2
>> 2: 10.3.74.214:6789/0 mon.ceph-mon1
>>
>> -91> 2018-06-08 12:48:20.697 7fada058e700 1 -- 10.3.56.69:6800/1807239
>> <== mon.0 10.3.71.36:6789/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ====
>> 194+0+0 (645793352 0 0) 0x559f7a3dafc0 con 0x559f7994ec00
>> -90> 2018-06-08 12:48:20.697 7fada058e700 10 monclient:
>> _check_auth_rotating have uptodate secrets (they expire after 2018-06-08
>> 12:47:50.699337)
>> -89> 2018-06-08 12:48:20.698 7fadbc9d7140 10 monclient:
>> wait_auth_rotating done
>> -88> 2018-06-08 12:48:20.698 7fadbc9d7140 10 monclient: _send_command 1
>> [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["58"]}]
>> -87> 2018-06-08 12:48:20.698 7fadbc9d7140 10 monclient:
>> _send_mon_message to mon.ceph-mon3 at 10.3.71.36:6789/0
>> -86> 2018-06-08 12:48:20.698 7fadbc9d7140 1 -- 10.3.56.69:6800/1807239
>> --> 10.3.71.36:6789/0 -- mon_command({"prefix": "osd crush
>> set-device-class", "class": "hdd", "ids": ["58"]} v 0) v1 -- 0x559f793e73c0
>> con 0
>> -85> 2018-06-08 12:48:20.700 7fadabaa4700 5 -- 10.3.56.69:6800/1807239
>> >> 10.3.71.36:6789/0 conn(0x559f7994ec00 :-1
>> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25741 cs=1 l=1). rx mon.0
>> seq 8 0x559f793e73c0 mon_command_ack([{"prefix": "osd crush
>> set-device-class", "class": "hdd", "ids": ["58"]}]=0 osd.58 already set to
>> class hdd. set-device-class item id 58 name 'osd.58' device_class 'hdd': no
>> change. v378738) v1
>> -84> 2018-06-08 12:48:20.701 7fada058e700 1 -- 10.3.56.69:6800/1807239
>> <== mon.0 10.3.71.36:6789/0 8 ==== mon_command_ack([{"prefix": "osd crush
>> set-device-class", "class": "hdd", "ids": ["58"]}]=0 osd.58 already set to
>> class hdd. set-device-class item id 58 name 'osd.58' device_class 'hdd': no
>> change. v378738) v1 ==== 211+0+0 (4063854475 0 0) 0x559f793e73c0 con
>> 0x559f7994ec00
>> -83> 2018-06-08 12:48:20.701 7fada058e700 10 monclient:
>> handle_mon_command_ack 1 [{"prefix": "osd crush set-device-class", "class":
>> "hdd", "ids": ["58"]}]
>> -82> 2018-06-08 12:48:20.701 7fada058e700 10 monclient: _finish_command
>> 1 = 0 osd.58 already set to class hdd. set-device-class item id 58 name
>> 'osd.58' device_class 'hdd': no change.
>> -81> 2018-06-08 12:48:20.701 7fadbc9d7140 10 monclient: _send_command 2
>> [{"prefix": "osd crush create-or-move", "id": 58, "weight":0.5240, "args":
>> ["host=sacephnode12", "root=default"]}]
>> -80> 2018-06-08 12:48:20.701 7fadbc9d7140 10 monclient:
>> _send_mon_message to mon.ceph-mon3 at 10.3.71.36:6789/0
>> -79> 2018-06-08 12:48:20.701 7fadbc9d7140 1 -- 10.3.56.69:6800/1807239
>> --> 10.3.71.36:6789/0 -- mon_command({"prefix": "osd crush create-or-move",
>> "id": 58, "weight":0.5240, "args": ["host=sacephnode12", "root=default"]} v
>> 0) v1 -- 0x559f793e7600 con 0
>> -78> 2018-06-08 12:48:20.703 7fadabaa4700 5 -- 10.3.56.69:6800/1807239
>> >> 10.3.71.36:6789/0 conn(0x559f7994ec00 :-1
>> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25741 cs=1 l=1). rx mon.0
>> seq 9 0x559f793e7600 mon_command_ack([{"prefix": "osd crush create-or-move",
>> "id": 58, "weight":0.5240, "args": ["host=sacephnode12", "root=default"]}]=0
>> create-or-move updated item name 'osd.58' weight 0.524 at location
>> {host=sacephnode12,root=default} to crush map v378738) v1
>> -77> 2018-06-08 12:48:20.703 7fada058e700 1 -- 10.3.56.69:6800/1807239
>> <== mon.0 10.3.71.36:6789/0 9 ==== mon_command_ack([{"prefix": "osd crush
>> create-or-move", "id": 58, "weight":0.5240, "args": ["host=sacephnode12",
>> "root=default"]}]=0 create-or-move updated item name 'osd.58' weight 0.524
>> at location {host=sacephnode12,root=default} to crush map v378738) v1 ====
>> 258+0+0 (1998484028 0 0) 0x559f793e7600 con 0x559f7994ec00
>> -76> 2018-06-08 12:48:20.703 7fada058e700 10 monclient:
>> handle_mon_command_ack 2 [{"prefix": "osd crush create-or-move", "id": 58,
>> "weight":0.5240, "args": ["host=sacephnode12", "root=default"]}]
>> -75> 2018-06-08 12:48:20.703 7fada058e700 10 monclient: _finish_command
>> 2 = 0 create-or-move updated item name 'osd.58' weight 0.524 at location
>> {host=sacephnode12,root=default} to crush map
>> -74> 2018-06-08 12:48:20.703 7fadbc9d7140 0 osd.58 0 done with init,
>> starting boot process
>> -73> 2018-06-08 12:48:20.703 7fadbc9d7140 10 monclient: _renew_subs
>> -72> 2018-06-08 12:48:20.703 7fadbc9d7140 10 monclient:
>> _send_mon_message to mon.ceph-mon3 at 10.3.71.36:6789/0
>> -71> 2018-06-08 12:48:20.703 7fadbc9d7140 1 -- 10.3.56.69:6800/1807239
>> --> 10.3.71.36:6789/0 -- mon_subscribe({mgrmap=0+,osd_pg_creates=0+}) v3 --
>> 0x559f79408e00 con 0
>> -70> 2018-06-08 12:48:20.703 7fadbc9d7140 1 osd.58 0 start_boot
>> -69> 2018-06-08 12:48:20.703 7fadbc9d7140 10 monclient: get_version
>> osdmap req 0x559f797667a0
>> -68> 2018-06-08 12:48:20.703 7fadbc9d7140 10 monclient:
>> _send_mon_message to mon.ceph-mon3 at 10.3.71.36:6789/0
>> -67> 2018-06-08 12:48:20.703 7fadbc9d7140 1 -- 10.3.56.69:6800/1807239
>> --> 10.3.71.36:6789/0 -- mon_get_version(what=osdmap handle=1) v1 --
>> 0x559f79434b40 con 0
>> -66> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command status hook 0x559f793f0700
>> -65> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command flush_journal hook 0x559f793f0700
>> -64> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command dump_ops_in_flight hook 0x559f793f0700
>> -63> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command ops hook 0x559f793f0700
>> -62> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command dump_blocked_ops hook 0x559f793f0700
>> -61> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command dump_historic_ops hook 0x559f793f0700
>> -60> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command dump_historic_slow_ops hook 0x559f793f0700
>> -59> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command dump_historic_ops_by_duration hook 0x559f793f0700
>> -58> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command dump_op_pq_state hook 0x559f793f0700
>> -57> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command dump_blacklist hook 0x559f793f0700
>> -56> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command dump_watchers hook 0x559f793f0700
>> -55> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command dump_reservations hook 0x559f793f0700
>> -54> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command get_latest_osdmap hook 0x559f793f0700
>> -53> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command heap hook 0x559f793f0700
>> -52> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command set_heap_property hook 0x559f793f0700
>> -51> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command get_heap_property hook 0x559f793f0700
>> -50> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command dump_objectstore_kv_stats hook 0x559f793f0700
>> -49> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command dump_scrubs hook 0x559f793f0700
>> -48> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command calc_objectstore_db_histogram hook 0x559f793f0700
>> -47> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command flush_store_cache hook 0x559f793f0700
>> -46> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command dump_pgstate_history hook 0x559f793f0700
>> -45> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command compact hook 0x559f793f0700
>> -44> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command get_mapped_pools hook 0x559f793f0700
>> -43> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command smart hook 0x559f793f0700
>> -42> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command list_devices hook 0x559f793f0700
>> -41> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command setomapval hook 0x559f79767280
>> -40> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command rmomapkey hook 0x559f79767280
>> -39> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command setomapheader hook 0x559f79767280
>> -38> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command getomap hook 0x559f79767280
>> -37> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command truncobj hook 0x559f79767280
>> -36> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command injectdataerr hook 0x559f79767280
>> -35> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command injectmdataerr hook 0x559f79767280
>> -34> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command set_recovery_delay hook 0x559f79767280
>> -33> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command trigger_scrub hook 0x559f79767280
>> -32> 2018-06-08 12:48:20.703 7fadbc9d7140 5 asok(0x559f794345a0)
>> register_command injectfull hook 0x559f79767280
>> -31> 2018-06-08 12:48:20.704 7fadabaa4700 5 -- 10.3.56.69:6800/1807239
>> >> 10.3.71.36:6789/0 conn(0x559f7994ec00 :-1
>> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25741 cs=1 l=1). rx mon.0
>> seq 10 0x559f7958f8c0 mgrmap(e 201) v1
>> -30> 2018-06-08 12:48:20.704 7fadabaa4700 5 -- 10.3.56.69:6800/1807239
>> >> 10.3.71.36:6789/0 conn(0x559f7994ec00 :-1
>> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25741 cs=1 l=1). rx mon.0
>> seq 11 0x559f79434b40 mon_get_version_reply(handle=1 version=378738) v2
>> -29> 2018-06-08 12:48:20.704 7fada058e700 1 -- 10.3.56.69:6800/1807239
>> <== mon.0 10.3.71.36:6789/0 10 ==== mgrmap(e 201) v1 ==== 1776+0+0
>> (412200892 0 0) 0x559f7958f8c0 con 0x559f7994ec00
>> -28> 2018-06-08 12:48:20.704 7fada058e700 4 mgrc handle_mgr_map Got
>> map version 201
>> -27> 2018-06-08 12:48:20.704 7fada058e700 4 mgrc handle_mgr_map Active
>> mgr is now 10.3.74.109:6801/1015
>> -26> 2018-06-08 12:48:20.704 7fada058e700 4 mgrc reconnect Starting
>> new session with 10.3.74.109:6801/1015
>> -25> 2018-06-08 12:48:20.706 7fadac2a5700 2 -- 10.3.56.69:6800/1807239
>> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1
>> s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got
>> newly_acked_seq 0 vs out_seq 0
>> -24> 2018-06-08 12:48:20.706 7fada058e700 1 -- 10.3.56.69:6800/1807239
>> --> 10.3.74.109:6801/1015 -- mgropen(unknown.58) v3 -- 0x559f79a9c000 con 0
>> -23> 2018-06-08 12:48:20.706 7fada058e700 1 -- 10.3.56.69:6800/1807239
>> <== mon.0 10.3.71.36:6789/0 11 ==== mon_get_version_reply(handle=1
>> version=378738) v2 ==== 24+0+0 (2329122009 0 0) 0x559f79434b40 con
>> 0x559f7994ec00
>> -22> 2018-06-08 12:48:20.706 7fada058e700 10 monclient:
>> handle_get_version_reply finishing 0x559f797667a0 version 378738
>> -21> 2018-06-08 12:48:20.706 7fad96a13700 5 osd.58 0 heartbeat:
>> osd_stat(1.0 GiB used, 536 GiB avail, 537 GiB total, peers [] op hist [])
>> -20> 2018-06-08 12:48:20.706 7fad96a13700 -1 osd.58 0 waiting for
>> initial osdmap
>> -19> 2018-06-08 12:48:20.706 7fad96a13700 10 monclient: _renew_subs
>> -18> 2018-06-08 12:48:20.706 7fad96a13700 10 monclient:
>> _send_mon_message to mon.ceph-mon3 at 10.3.71.36:6789/0
>> -17> 2018-06-08 12:48:20.706 7fad96a13700 1 -- 10.3.56.69:6800/1807239
>> --> 10.3.71.36:6789/0 -- mon_subscribe({osdmap=378084}) v3 -- 0x559f7a3b8400
>> con 0
>> -16> 2018-06-08 12:48:20.707 7fadac2a5700 5 -- 10.3.56.69:6800/1807239
>> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1
>> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=201245 cs=1 l=1). rx
>> mgr.44007797 seq 1 0x559f79435860 mgrconfigure(period=5, threshold=5) v2
>> -15> 2018-06-08 12:48:20.708 7fada058e700 1 -- 10.3.56.69:6800/1807239
>> <== mgr.44007797 10.3.74.109:6801/1015 1 ==== mgrconfigure(period=5,
>> threshold=5) v2 ==== 8+0+0 (3460719617 0 0) 0x559f79435860 con
>> 0x559f79950a00
>> -14> 2018-06-08 12:48:20.708 7fada058e700 4 mgrc handle_mgr_configure
>> stats_period=5
>> -13> 2018-06-08 12:48:20.708 7fada058e700 4 mgrc handle_mgr_configure
>> updated stats threshold: 5
>> -12> 2018-06-08 12:48:20.708 7fadabaa4700 5 -- 10.3.56.69:6800/1807239
>> >> 10.3.71.36:6789/0 conn(0x559f7994ec00 :-1
>> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25741 cs=1 l=1). rx mon.0
>> seq 12 0x559f79aba000 osd_map(378085..378085 src has 378085..378738
>> +gap_removed_snaps) v4
>> -11> 2018-06-08 12:48:20.708 7fada058e700 1 -- 10.3.56.69:6800/1807239
>> --> 10.3.74.109:6801/1015 -- mgrreport(unknown.58 +54-0 packed 742) v6 --
>> 0x559f79a9c300 con 0
>> -10> 2018-06-08 12:48:20.708 7fada058e700 1 -- 10.3.56.69:6800/1807239
>> --> 10.3.74.109:6801/1015 -- pg_stats(0 pgs tid 0 v 0) v1 -- 0x559f7958f600
>> con 0
>> -9> 2018-06-08 12:48:20.708 7fada058e700 1 -- 10.3.56.69:6800/1807239
>> <== mon.0 10.3.71.36:6789/0 12 ==== osd_map(378085..378085 src has
>> 378085..378738 +gap_removed_snaps) v4 ==== 33348+0+0 (2799879432 0 0)
>> 0x559f79aba000 con 0x559f7994ec00
>> -8> 2018-06-08 12:48:20.708 7fada058e700 3 osd.58 0 handle_osd_map
>> epochs [378085,378085], i have 0, src has [378085,378738]
>> -7> 2018-06-08 12:48:20.709 7fadabaa4700 5 -- 10.3.56.69:6800/1807239
>> >> 10.3.71.36:6789/0 conn(0x559f7994ec00 :-1
>> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25741 cs=1 l=1). rx mon.0
>> seq 13 0x559f79abaa00 osd_map(378086..378125 src has 378085..378738) v4
>> -6> 2018-06-08 12:48:20.709 7fada058e700 -1 osd.58 0 failed to load
>> OSD map for epoch 378084, got 0 bytes
>> -5> 2018-06-08 12:48:20.710 7fadac2a5700 1 -- 10.3.56.69:6800/1807239
>> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1 s=STATE_OPEN pgs=201245
>> cs=1 l=1).read_bulk peer close file descriptor 38
>> -4> 2018-06-08 12:48:20.710 7fadac2a5700 1 -- 10.3.56.69:6800/1807239
>> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1 s=STATE_OPEN pgs=201245
>> cs=1 l=1).read_until read failed
>> -3> 2018-06-08 12:48:20.710 7fadac2a5700 1 -- 10.3.56.69:6800/1807239
>> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1 s=STATE_OPEN pgs=201245
>> cs=1 l=1).process read tag failed
>> -2> 2018-06-08 12:48:20.710 7fadac2a5700 1 -- 10.3.56.69:6800/1807239
>> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1 s=STATE_OPEN pgs=201245
>> cs=1 l=1).fault on lossy channel, failing
>> -1> 2018-06-08 12:48:20.710 7fadac2a5700 2 -- 10.3.56.69:6800/1807239
>> >> 10.3.74.109:6801/1015 conn(0x559f79950a00 :-1 s=STATE_OPEN pgs=201245
>> cs=1 l=1)._stop
>> 0> 2018-06-08 12:48:20.711 7fada058e700 -1
>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.0/rpm/el7/BUILD/ceph-13.2.0/src/osd/OSD.h:
>> In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7fada058e700
>> time 2018-06-08 12:48:20.710675
>>
>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.0/rpm/el7/BUILD/ceph-13.2.0/src/osd/OSD.h:
>> 828: FAILED assert(ret)
>>
>> ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic
>> (stable)
>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0xff) [0x7fadb3e1753f]
>> 2: (()+0x286727) [0x7fadb3e17727]
>> 3: (OSDService::get_map(unsigned int)+0x4a) [0x559f76fe4dda]
>> 4: (OSD::handle_osd_map(MOSDMap*)+0x1020) [0x559f76f921f0]
>> 5: (OSD::_dispatch(Message*)+0xa1) [0x559f76f94d21]
>> 6: (OSD::ms_dispatch(Message*)+0x56) [0x559f76f95066]
>> 7: (DispatchQueue::entry()+0xb5a) [0x7fadb3e8d74a]
>> 8: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fadb3f2df2d]
>> 9: (()+0x7e25) [0x7fadb0afde25]
>> 10: (clone()+0x6d) [0x7fadafbf134d]
>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
>> to interpret this.
>>
>> --- logging levels ---
>> 0/ 5 none
>> 0/ 1 lockdep
>> 0/ 1 context
>> 1/ 1 crush
>> 1/ 5 mds
>> 1/ 5 mds_balancer
>> 1/ 5 mds_locker
>> 1/ 5 mds_log
>> 1/ 5 mds_log_expire
>> 1/ 5 mds_migrator
>> 0/ 1 buffer
>> 0/ 1 timer
>> 0/ 1 filer
>> 0/ 1 striper
>> 0/ 1 objecter
>> 0/ 5 rados
>> 0/ 5 rbd
>> 0/ 5 rbd_mirror
>> 0/ 5 rbd_replay
>> 0/ 5 journaler
>> 0/ 5 objectcacher
>> 0/ 5 client
>> 1/ 5 osd
>> 0/ 5 optracker
>> 0/ 5 objclass
>> 1/ 3 filestore
>> 1/ 3 journal
>> 0/ 5 ms
>> 1/ 5 mon
>> 0/10 monc
>> 1/ 5 paxos
>> 0/ 5 tp
>> 1/ 5 auth
>> 1/ 5 crypto
>> 1/ 1 finisher
>> 1/ 1 reserver
>> 1/ 5 heartbeatmap
>> 1/ 5 perfcounter
>> 1/ 5 rgw
>> 1/ 5 rgw_sync
>> 1/10 civetweb
>> 1/ 5 javaclient
>> 1/ 5 asok
>> 1/ 1 throttle
>> 0/ 0 refs
>> 1/ 5 xio
>> 1/ 5 compressor
>> 1/ 5 bluestore
>> 1/ 5 bluefs
>> 1/ 3 bdev
>> 1/ 5 kstore
>> 4/ 5 rocksdb
>> 4/ 5 leveldb
>> 4/ 5 memdb
>> 1/ 5 kinetic
>> 1/ 5 fuse
>> 1/ 5 mgr
>> 1/ 5 mgrc
>> 1/ 5 dpdk
>> 1/ 5 eventtrace
>> -2/-2 (syslog threshold)
>> -1/-1 (stderr threshold)
>> max_recent 10000
>> max_new 1000
>> log_file /var/log/ceph/ceph-osd.58.log
>> --- end dump of recent events ---
>> 2018-06-08 12:48:20.717 7fada058e700 -1 *** Caught signal (Aborted) **
>> in thread 7fada058e700 thread_name:ms_dispatch
>>
>> ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic
>> (stable)
>> 1: (()+0x8e1870) [0x559f774af870]
>> 2: (()+0xf5e0) [0x7fadb0b055e0]
>> 3: (gsignal()+0x37) [0x7fadafb2e1f7]
>> 4: (abort()+0x148) [0x7fadafb2f8e8]
>> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x25d) [0x7fadb3e1769d]
>> 6: (()+0x286727) [0x7fadb3e17727]
>> 7: (OSDService::get_map(unsigned int)+0x4a) [0x559f76fe4dda]
>> 8: (OSD::handle_osd_map(MOSDMap*)+0x1020) [0x559f76f921f0]
>> 9: (OSD::_dispatch(Message*)+0xa1) [0x559f76f94d21]
>> 10: (OSD::ms_dispatch(Message*)+0x56) [0x559f76f95066]
>> 11: (DispatchQueue::entry()+0xb5a) [0x7fadb3e8d74a]
>> 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fadb3f2df2d]
>> 13: (()+0x7e25) [0x7fadb0afde25]
>> 14: (clone()+0x6d) [0x7fadafbf134d]
>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
>> to interpret this.
>>
>> --- begin dump of recent events ---
>> 0> 2018-06-08 12:48:20.717 7fada058e700 -1 *** Caught signal
>> (Aborted) **
>> in thread 7fada058e700 thread_name:ms_dispatch
>>
>> ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic
>> (stable)
>> 1: (()+0x8e1870) [0x559f774af870]
>> 2: (()+0xf5e0) [0x7fadb0b055e0]
>> 3: (gsignal()+0x37) [0x7fadafb2e1f7]
>> 4: (abort()+0x148) [0x7fadafb2f8e8]
>> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x25d) [0x7fadb3e1769d]
>> 6: (()+0x286727) [0x7fadb3e17727]
>> 7: (OSDService::get_map(unsigned int)+0x4a) [0x559f76fe4dda]
>> 8: (OSD::handle_osd_map(MOSDMap*)+0x1020) [0x559f76f921f0]
>> 9: (OSD::_dispatch(Message*)+0xa1) [0x559f76f94d21]
>> 10: (OSD::ms_dispatch(Message*)+0x56) [0x559f76f95066]
>> 11: (DispatchQueue::entry()+0xb5a) [0x7fadb3e8d74a]
>> 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fadb3f2df2d]
>> 13: (()+0x7e25) [0x7fadb0afde25]
>> 14: (clone()+0x6d) [0x7fadafbf134d]
>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
>> to interpret this.
>>
>>
>> Mike Kuriger
>>
>> >> CEPH team,
>> >> Is there a solution yet for adding OSDs in mimic - specifically
>> >> re-using old IDs? I was looking over this BUG report -
>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issues_24423&d=DwIFaQ&c=5m9CfXHY6NXqkS7nN5n23w&r=5r9bhr1JAPRaUcJcU-FfGg&m=0PCKiecm216R95S_krqboYMskCBoolGysrvgHZo8LEM&s=hfI2uudTfY0lGtBI6iIXvZWvNpme4xwBJe2SWx0_N3I&e=
>> >> and my issue is similar. I removed a bunch of OSD's after upgrading to
>> >> mimic and I'm not able to re-add them using the new volume format. I
>> >> haven't tried manually adding them using 'never used' IDs. I'll try that
>> >> now but was hoping there would be a fix.
>> >>
>> >> Thanks!
>> >>
>> >> Mike Kuriger
>> >>
>> >> _______________________________________________
>> >> ceph-users mailing list
>> >> ceph-users@xxxxxxxxxxxxxx
>> >>
>> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.ceph.com_listinfo.cgi_ceph-2Dusers-2Dceph.com&d=DwIFaQ&c=5m9CfXHY6NXqkS7nN5n23w&r=5r9bhr1JAPRaUcJcU-FfGg&m=0PCKiecm216R95S_krqboYMskCBoolGysrvgHZo8LEM&s=2aoWc5hTz041_26Stz6zPtLiB5zGFw2GbX3TPjsvieI&e=
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
>
> --
> Paul Emmerich
>
> Looking for help with your Ceph cluster? Contact us at https://croit.io
>
> croit GmbH
> Freseniusstr. 31h
> 81247 München
> www.croit.io
> Tel: +49 89 1896585 90
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
