On Wed, Sep 27, 2017 at 1:18 PM, Richard Hesketh
<richard.hesketh@xxxxxxxxxxxx> wrote:
> On 27/09/17 12:32, John Spray wrote:
>> On Wed, Sep 27, 2017 at 12:15 PM, Richard Hesketh
>> <richard.hesketh@xxxxxxxxxxxx> wrote:
>>> As the subject says... any ceph fs administrative command I try to run hangs forever and kills monitors in the background - sometimes they come back, on a couple of occasions I had to manually stop/restart a suffering mon. Trying to load the filesystem tab in the ceph-mgr dashboard dumps an error and can also kill a monitor. However, clients can mount the filesystem and read/write data without issue.
>>>
>>> Relevant excerpt from logs on an affected monitor, just trying to run 'ceph fs ls':
>>>
>>> 2017-09-26 13:20:50.716087 7fc85fdd9700 0 mon.vm-ds-01@0(leader) e19 handle_command mon_command({"prefix": "fs ls"} v 0) v1
>>> 2017-09-26 13:20:50.727612 7fc85fdd9700 0 log_channel(audit) log [DBG] : from='client.? 10.10.10.1:0/2771553898' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch
>>> 2017-09-26 13:20:50.950373 7fc85fdd9700 -1 /build/ceph-12.2.0/src/osd/OSDMap.h: In function 'const string& OSDMap::get_pool_name(int64_t) const' thread 7fc85fdd9700 time 2017-09-26 13:20:50.727676
>>> /build/ceph-12.2.0/src/osd/OSDMap.h: 1176: FAILED assert(i != pool_name.end())
>>>
>>> ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
>>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55a8ca0bb642]
>>> 2: (()+0x48165f) [0x55a8c9f4165f]
>>> 3: (MDSMonitor::preprocess_command(boost::intrusive_ptr<MonOpRequest>)+0x1d18) [0x55a8ca047688]
>>> 4: (MDSMonitor::preprocess_query(boost::intrusive_ptr<MonOpRequest>)+0x2a8) [0x55a8ca048008]
>>> 5: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x700) [0x55a8c9f9d1b0]
>>> 6: (Monitor::handle_command(boost::intrusive_ptr<MonOpRequest>)+0x1f93) [0x55a8c9e63193]
>>> 7: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0xa0e) [0x55a8c9e6a52e]
>>> 8: (Monitor::_ms_dispatch(Message*)+0x6db) [0x55a8c9e6b57b]
>>> 9: (Monitor::ms_dispatch(Message*)+0x23) [0x55a8c9e9a053]
>>> 10: (DispatchQueue::entry()+0xf4a) [0x55a8ca3b5f7a]
>>> 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x55a8ca16bc1d]
>>> 12: (()+0x76ba) [0x7fc86b3ac6ba]
>>> 13: (clone()+0x6d) [0x7fc869bd63dd]
>>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>>>
>>> I'm running Luminous. The cluster and FS have been in service since Hammer and have default data/metadata pool names. I discovered the issue after attempting to enable directory sharding.
>>
>> Well that's not good...
>>
>> The assertion is because your FSMap is referring to a pool that
>> apparently no longer exists in the OSDMap. This should be impossible
>> in current Ceph (we forbid removing pools if they're in use), but
>> could perhaps have been caused in an earlier version of Ceph when it
>> was possible to remove a pool even if CephFS was referring to it?
>>
>> Alternatively, perhaps something more severe is going on that's
>> causing your mons to see a wrong/inconsistent view of the world. Has
>> the cluster ever been through any traumatic disaster recovery type
>> activity involving hand-editing any of the cluster maps? What
>> intermediate versions has it passed through on the way from Hammer to
>> Luminous?
>>
>> Opened a ticket here: http://tracker.ceph.com/issues/21568
>>
>> John
>
> I've reviewed my notes (i.e. I've grepped my IRC logs); I actually inherited this cluster from a colleague who left shortly after I joined, so unfortunately there is some of its history I cannot fill in.
>
> Turns out the cluster actually predates Firefly. Looking at dates my suspicion is that it went Emperor -> Firefly -> Giant -> Hammer. I inherited it at Hammer, and took it Hammer -> Infernalis -> Jewel -> Luminous myself. I know I did make sure to do the tmap_upgrade step on cephfs but can't remember if I did it at Infernalis or Jewel.
>
> Infernalis was a tricky upgrade; the attempt was aborted once after the first set of OSDs didn't come back up after upgrade (had to remove/downgrade and readd), and setting sortbitwise as the documentation suggested after a successful second attempt caused everything to break and degrade slowly until it was unset and recovered. Never had disaster recovery involve mucking around with the pools while I was administrating it, but unfortunately I cannot speak for the cluster's pre-Hammer history. The only pools I have removed were ones I created temporarily for testing crush rules/benchmarking.
OK, so it sounds like a cluster with an interesting history and some
stories to tell :-)
> I have hand-edited the crush map (extract, decompile, modify, recompile, inject) at times because I found it more convenient for creating new crush rules than using the CLI tools, but not the OSD map.
>
> Why would the cephfs have been referring to other pools?
A filesystem can have more than one data pool, they're added at
runtime with add_data_pool/rm_data_pool commands. In old versions of
the code, someone could add a data pool, then delete the pool, and
forget to do rm_data_pool.
So, next step is to try and actually get the FSMap out of the
monitor's store to see if that's really what's happening --
unfortunately when checking how to do that I realise we missed
updating the human readable output of ceph-monstore-tool when adding
multi-filesystem support... so here's how to get it out in binary form
and then decode it separately:
ceph-monstore-tool /var/lib/ceph/<wherever...> get mdsmap > fsmap.bin
ceph-dencoder import fsmap.bin type FSMap decode dump_json
If the simple theory is correct then you'll see something referenced
in one of the pool/pools fields that doesn't really exist.
John
>
> Rich
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
