Hi Vladimir, On Mon, 4 Jun 2012, Vladimir Bashkirtsev wrote: > Dear devs, > > While playing around with ceph with six OSDs I decided to retire two OSDs > simultaneously (I do triplication so ceph should withstand such damage) to see > how ceph will cope with it. I was doing it in different ways trying to get > ceph off-rails and it looks I have managed it. :) > > First of all I have tried to kill OSDs by pulling them off and then doing ceph > osd lost . Performed as expected. However ceph kept record of former OSDs even > so it did not try to use it. Looks correct. > > Then I have recreated OSDs and magically they just came back online and filled > up with data again. Again: that's what is expected. > > At last I have tried planned removal of OSDs: > > ceph osd crush remove 3 > ceph osd rm osd.3 > > Ceph complained that osd is still up. Shutdown OSD, tried again. Success. > Done the same with second OSD. Everything looked fine still. > > And then accidentally (and that's perhaps good test) I have rebooted box > running osd.3 and it had ceph osd in rc. So osd.3 started without having > knowledge that it was evicted from cluster. Cluster magically took it back and > osd.3 joined the culster (however it did not got any load as it was removed > from crush). I removed it from rc, shut it down, done ceph osd crush remove 3 > (just to be certain) and ceph osd rm osd.3 (both succeeded) but now I have > osd.3 still counted towards total cluster capacity, osd dump shows it as non > existent, pg dump shows it as it still member of a cluster: > > [root@x ceph]# ceph osd dump > dumped osdmap epoch 14892 > epoch 14892 > fsid 7719f573-4c48-4852-a27f-51c7a3fe1c1e > created 2012-03-31 04:47:12.130128 > modifed 2012-06-04 11:16:57.687645 > flags > > pool 0 'data' rep size 3 crush_ruleset 0 object_hash rjenkins pg_num 192 > pgp_num 192 last_change 13812 owner 0 crash_replay_interval 45 > pool 1 'metadata' rep size 3 crush_ruleset 1 object_hash rjenkins pg_num 192 > pgp_num 192 last_change 13815 owner 0 > pool 2 'rbd' rep size 3 crush_ruleset 2 object_hash rjenkins pg_num 192 > pgp_num 192 last_change 13817 owner 0 > > max_osd 6 > osd.0 up in weight 1 up_from 14407 up_thru 14890 down_at 14400 > last_clean_interval [14383,14399) 172.16.64.200:6801/25023 > 172.16.64.200:6802/25023 172.16.64.200:6803/25023 exists,up > osd.1 up in weight 1 up_from 14420 up_thru 14890 down_at 14413 > last_clean_interval [14388,14412) lost_at 11147 172.16.64.201:6800/5719 > 172.16.64.201:6801/5719 172.16.64.201:6802/5719 exists,up > 2c7ca892-e83c-4158-a3ae-7c4f96f040b0 > osd.4 up in weight 1 up_from 14432 up_thru 14890 down_at 14425 > last_clean_interval [14393,14424) lost_at 13373 172.16.64.204:6800/17419 > 172.16.64.204:6802/17419 172.16.64.204:6803/17419 exists,up > 19703275-74c3-403b-8647-85cc4f7ad870 > osd.5 up in weight 1 up_from 14448 up_thru 14890 down_at 14438 > last_clean_interval [14366,14437) 172.16.64.205:6800/7021 > 172.16.64.205:6801/7021 172.16.64.205:6802/7021 exists,up > 699a39ca-3806-4c4f-9cdc-76cbed61b2ab > > [root@x ceph]# ceph pg dump > dumped all in format plain > version 2223459 > last_osdmap_epoch 14892 > last_pg_scan 12769 > full_ratio 0.95 > nearfull_ratio 0.85 > <-snip-> > pool 0 21404 0 0 0 40635329394 23681638 23681638 > pool 1 114 0 0 0 234237438 4481899 4481899 > pool 2 113241 0 2 0 473699447111 27387426 27387426 > sum 134759 0 2 0 514569013943 55550963 55550963 > osdstat kbused kbavail kb hb in hb out > 0 399440780 316714764 744751104 [1,4,5] [] > 1 400369588 125798956 546603008 [0,4,5] [] > 3 130380 90124804 94470144 [0,1,4,5] [] > 4 387384720 132412912 540409856 [0,1,5] [] > 5 344705816 233764680 600997888 [0,1,4] [] > sum 1532031284 898816116 2527232000 > > Any idea how to get rid of it completely? That is a bug; testing a fix now. It's harmless, though, aside from slightly skewing the used/free stats you see from 'ceph pg stat' or 'df'. As a workaround, you can create the osd, and then mark it OUT (ceph osd out 3) first, and then delete it (ceph osd rm 3). Or wait for the next release, and then recreate and re-delete (ceph osd rm 3) it. Thanks! sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
