Cześć, > On Thu, 29 Dec 2016, Łukasz Chrustek wrote: >> Hi, >> >> >> >> >> >> # ceph osd tree >> >> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY >> >> -7 16.89590 root ssd-disks >> >> -11 0 host ssd1 >> >> 598798032 0 osd.598798032 DNE 0 >> >> > Yikes! >> >> Yes... indeed, I don't like this number too... >> >> >> 21940 0 osd.21940 DNE 0 >> >> 71 0 osd.71 DNE 0 >> >> >> >> My question is how to delete this osds without direct editing crushmap >> >> ? It is production system, I can't affort any service interruption :(, >> >> when I try to ceph osd crush remove then ceph-mon crushes.... >> >> >> >> I dumped crushmap, but it took 19G (!!) after decompiling (compiled >> >> file is very small). So, I cleaned this file with perl (it take very >> >> long time), and I have now small txt crushmap, which I edited. But is >> >> there any chance that ceph will still remember somewhere about this >> >> huge numbers for osds ? Is it safe to apply this cleaned crushmap to >> >> cluster ? >> >> > It sounds like the problem is the OSDMap, not CRUSH per se. Can you >> > attach the output from 'ceph osd dump -f json-pretty'? >> >> It's quite big so I put it on pastebin: >> >> http://pastebin.com/Unkk2Pa7 >> >> > Do you know how osd.598798032 got created? Or osd.21940 for that matter. >> > OSD ids should be small since they are stored internally by OSDMap as a >> > vector. This is probably why your mon is crashing. >> >> [root@cc1 /etc/ceph]# ceph osd tree >> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY >> -7 16.89590 root ssd-intel-s3700 >> -11 0 host ssd-stor1 >> 69 0 osd.69 down 0 1.00000 >> 70 0 osd.70 down 0 1.00000 >> 71 0 osd.71 down 0 1.00000 >> >> >> This the moment, when it happend: >> ]# for i in `seq 69 71`;do ceph osd crush remove osd.$i;done >> removed item id 69 name 'osd.69' from crush map >> >> >> removed item id 70 name 'osd.70' from crush map >> >> here i press ctrl+c >> >> 2016-12-28 17:38:10.055239 7f4576d7a700 0 monclient: hunting for new mon >> 2016-12-28 17:38:10.055582 7f4574233700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f456c023190 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f456c024470).fault >> 2016-12-28 17:38:30.550622 7f4574233700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.1:6789/0 pipe(0x7f45600008c0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4560001df0).fault >> 2016-12-28 17:38:54.551031 7f4574474700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f45600046c0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f45600042b0).fault > What version is this? infernalis > Can you attach the crush map too? (ceph osd crush dump -f json-pretty) I can't - ceph-mons are crushing on diffrent ceph-mon hosts: ]# ceph osd crush dump -f json-pretty 2016-12-29 21:42:47.172097 7f0c36061700 0 monclient: hunting for new mon ^CTraceback (most recent call last): File "/usr/bin/ceph", line 948, in <module> retval = main() File "/usr/bin/ceph", line 881, in main sigdict, inbuf, verbose) File "/usr/bin/ceph", line 479, in new_style_command inbuf=inbuf) File "/usr/lib/python2.7/dist-packages/ceph_argparse.py", line 1291, in json_command raise RuntimeError('"{0}": exception {1}'.format(argdict, e)) RuntimeError: "{'prefix': u'osd crush dump', 'format': 'json-pretty'}": exception "['{"prefix": "osd crush dump", "format": "json-pretty"}']": exception 'int' object is not iterable -- Pozdrowienia, Łukasz Chrustek -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
