Hi Femi,
This sounds very strange.
Can you tar up the osdmap and osdmap_full directories from one of your
monitors and post it somewhere where a dev can take a look? I'd like to
see how the OSDs came to be removed. I can't think of anything in the
upgrade that would have affected this, but want to get to the bottom of it
either way!
Thanks-
sage
On Fri, 15 Feb 2013, femi anjorin wrote:
> Hi
>
> i didnt do a "ceph osd rm" command at all.
>
> For the upgrade this is what i did. "service ceph -a stop" command and
> did a parallel command to upgrade ceph.
>
> I checked each node with ceph -v to be sure they all got the update.
>
> I think the upgrade was ok because this is the second or third time im
> doing ceph upgrade, so i dont think its the upgrade process.
>
> Moreover if you noticed in the "service ceph -a restart" all the OSD
> reports gets up... so if a ceph osd rm command had been issued then
> it wont find the OSD.
>
> Still confused on how to solve it. I have stopd the service severally
> and restarted it ...its gives similar results.
>
>
> Regards.
>
>
>
>
> On Fri, Feb 15, 2013 at 3:11 PM, Patrick McGarry <patrick@xxxxxxxxxxx> wrote:
> > femi,
> >
> > CC'ing ceph-user as this discussion probably belongs there.
> >
> > Could you send a copy of your crushmap? DNE is typically what we see
> > when someone explicitly removes an osd with something like: 'ceph osd
> > rm 90' (Does Not Exist).
> >
> > Also, out of curiosity, how did you upgrade you cluster? One box at a
> > time? Take the whole thing down and upgrade everything? something
> > else? Just interested to see how you got to where you are. Feel free
> > to stop by irc and ask for scuttlemonkey if you want a more direct
> > discussion. Thanks.
> >
> >
> > Best Regards,
> >
> > Patrick
> >
> > On Fri, Feb 15, 2013 at 8:50 AM, femi anjorin <femi.anjorin@xxxxxxxxx> wrote:
> >> Hi All,
> >>
> >> Pls I got this result after i did an upgrade to 0.56.3. I'm not sure
> >> if its a problem with upgrade or some other things.
> >>
> >> # ceph osd tree
> >>
> >> # id weight type name up/down reweight
> >> -1 96 root default
> >> -3 96 rack unknownrack
> >> -2 4 host server109
> >> 0 1 osd.0 DNE
> >> 1 1 osd.1 DNE
> >> 2 1 osd.2 DNE
> >> 3 1 osd.3 up 1
> >> -4 4 host server111
> >> 10 1 osd.10 DNE
> >> 11 1 osd.11 DNE
> >> 8 1 osd.8 DNE
> >> 9 1 osd.9 up 1
> >> -5 4 host server112
> >> 12 1 osd.12 DNE
> >> 13 1 osd.13 DNE
> >> 14 1 osd.14 DNE
> >> 15 1 osd.15 up 1
> >> -6 4 host server113
> >> 16 1 osd.16 DNE
> >> 17 1 osd.17 DNE
> >> 18 1 osd.18 DNE
> >> 19 1 osd.19 up 1
> >> -7 4 host server114
> >> 20 1 osd.20 DNE
> >> 21 1 osd.21 DNE
> >> 22 1 osd.22 DNE
> >> 23 1 osd.23 up 1
> >> -8 4 host server115
> >> 24 1 osd.24 DNE
> >> 25 1 osd.25 DNE
> >> 26 1 osd.26 DNE
> >> 27 1 osd.27 up 1
> >> -9 4 host server116
> >> 28 1 osd.28 DNE
> >> 29 1 osd.29 DNE
> >> 30 1 osd.30 DNE
> >> 31 1 osd.31 up 1
> >> -10 4 host server209
> >> 32 1 osd.32 DNE
> >> 33 1 osd.33 DNE
> >> 34 1 osd.34 DNE
> >> 35 1 osd.35 up 1
> >> -11 4 host server210
> >> 36 1 osd.36 DNE
> >> 37 1 osd.37 DNE
> >> 38 1 osd.38 DNE
> >> 39 1 osd.39 up 1
> >> -12 4 host server110
> >> 4 1 osd.4 DNE
> >> 5 1 osd.5 DNE
> >> 6 1 osd.6 DNE
> >> 7 1 osd.7 up 1
> >> -13 4 host server211
> >> 40 1 osd.40 DNE
> >> 41 1 osd.41 DNE
> >> 42 1 osd.42 DNE
> >> 43 1 osd.43 up 1
> >> -14 4 host server212
> >> 44 1 osd.44 DNE
> >> 45 1 osd.45 DNE
> >> 46 1 osd.46 DNE
> >> 47 1 osd.47 up 1
> >> -15 4 host server213
> >> 48 1 osd.48 DNE
> >> 49 1 osd.49 DNE
> >> 50 1 osd.50 DNE
> >> 51 1 osd.51 up 1
> >> -16 4 host server214
> >> 52 1 osd.52 DNE
> >> 53 1 osd.53 DNE
> >> 54 1 osd.54 DNE
> >> 55 1 osd.55 up 1
> >> -17 4 host server215
> >> 56 1 osd.56 DNE
> >> 57 1 osd.57 DNE
> >> 58 1 osd.58 DNE
> >> 59 1 osd.59 up 1
> >> -18 4 host server216
> >> 60 1 osd.60 DNE
> >> 61 1 osd.61 DNE
> >> 62 1 osd.62 DNE
> >> 63 1 osd.63 up 1
> >> -19 4 host server309
> >> 64 1 osd.64 DNE
> >> 65 1 osd.65 DNE
> >> 66 1 osd.66 DNE
> >> 67 1 osd.67 up 1
> >> -20 4 host server310
> >> 68 1 osd.68 DNE
> >> 69 1 osd.69 DNE
> >> 70 1 osd.70 DNE
> >> 71 1 osd.71 up 1
> >> -21 4 host server311
> >> 72 1 osd.72 DNE
> >> 73 1 osd.73 DNE
> >> 74 1 osd.74 DNE
> >> 75 1 osd.75 up 1
> >> -22 4 host server312
> >> 76 1 osd.76 DNE
> >> 77 1 osd.77 DNE
> >> 78 1 osd.78 DNE
> >> 79 1 osd.79 up 1
> >> -23 4 host server313
> >> 80 1 osd.80 DNE
> >> 81 1 osd.81 DNE
> >> 82 1 osd.82 DNE
> >> 83 1 osd.83 up 1
> >> -24 4 host server314
> >> 84 1 osd.84 DNE
> >> 85 1 osd.85 DNE
> >> 86 1 osd.86 DNE
> >> 87 1 osd.87 up 1
> >> -25 4 host server315
> >> 88 1 osd.88 DNE
> >> 89 1 osd.89 DNE
> >> 90 1 osd.90 DNE
> >> 91 1 osd.91 up 1
> >> -26 4 host server316
> >> 92 1 osd.92 DNE
> >> 93 1 osd.93 DNE
> >> 94 1 osd.94 DNE
> >> 95 1 osd.95 up 1
> >>
> >>
> >> ALTHOUGH THE HEALTH IS GOOD....
> >>
> >> # ceph health
> >> HEALTH_OK
> >>
> >> # ceph status
> >> health HEALTH_OK
> >> monmap e1: 3 mons at
> >> {a=172.16.0.25:6789/0,b=172.16.0.26:6789/0,c=172.16.0.24:6789/0},
> >> election epoch 10, quorum 0,1,2 a,b,c
> >> osdmap e143: 24 osds: 24 up, 24 in
> >> pgmap v2080: 18624 pgs: 18624 active+clean; 8730 bytes data, 1315
> >> MB used, 168 TB / 168 TB avail
> >> mdsmap e9: 1/1/1 up {0=a=up:active}
> >>
> >> # ceph osd dump
> >>
> >> epoch 143
> >> fsid 16cf888d-a38f-4308-84ad-300b89b9fae9
> >> created 2013-02-15 13:05:29.465590
> >> modifed 2013-02-15 13:49:40.305081
> >> flags
> >>
> >> pool 0 'data' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num
> >> 6208 pgp_num 6208 last_change 1 owner 0 crash_replay_interval 45
> >> pool 1 'metadata' rep size 2 crush_ruleset 1 object_hash rjenkins
> >> pg_num 6208 pgp_num 6208 last_change 1 owner 0
> >> pool 2 'rbd' rep size 2 crush_ruleset 2 object_hash rjenkins pg_num
> >> 6208 pgp_num 6208 last_change 1 owner 0
> >>
> >> max_osd 96
> >> osd.3 up in weight 1 up_from 67 up_thru 139 down_at 66
> >> last_clean_interval [3,63) 172.16.1.9:6800/6880 172.16.1.9:6803/6880
> >> 172.16.1.9:6804/6880 exists,up 55a33287-62d3-47d7-8eca-b479dc74677e
> >> osd.7 up in weight 1 up_from 111 up_thru 139 down_at 110
> >> last_clean_interval [14,108) 172.16.1.10:6803/7175
> >> 172.16.1.10:6804/7175 172.16.1.10:6805/7175 exists,up
> >> b00cbbf6-b0f0-4aa2-a1fd-5cc5c5fa3af5
> >> osd.9 up in weight 1 up_from 135 up_thru 139 down_at 134
> >> last_clean_interval [32,131) 172.16.1.11:6800/8520
> >> 172.16.1.11:6803/8520 172.16.1.11:6804/8520 exists,up
> >> 3300850a-f32c-4c50-bf92-d85afb576e63
> >> osd.15 up in weight 1 up_from 53 up_thru 139 down_at 52
> >> last_clean_interval [2,51) 172.16.1.12:6800/6679 172.16.1.12:6803/6679
> >> 172.16.1.12:6804/6679 exists,up 1a970009-103d-4d55-8318-63d9b30c4c36
> >> osd.19 up in weight 1 up_from 57 up_thru 139 down_at 56
> >> last_clean_interval [4,54) 172.16.1.13:6800/11231
> >> 172.16.1.13:6803/11231 172.16.1.13:6804/11231 exists,up
> >> 16350f84-0472-479a-92f8-98a6a6f63998
> >> osd.23 up in weight 1 up_from 60 up_thru 139 down_at 59
> >> last_clean_interval [4,57) 172.16.1.14:6803/6835 172.16.1.14:6804/6835
> >> 172.16.1.14:6805/6835 exists,up 3c3551b0-79d7-49bc-b98b-5e412e56d791
> >> osd.27 up in weight 1 up_from 65 up_thru 139 down_at 64
> >> last_clean_interval [4,61) 172.16.1.15:6803/11582
> >> 172.16.1.15:6804/11582 172.16.1.15:6805/11582 exists,up
> >> 034b8935-be7f-4eda-a11b-b48e9673eab4
> >> osd.31 up in weight 1 up_from 71 up_thru 139 down_at 70
> >> last_clean_interval [5,67) 172.16.1.16:6803/7019 172.16.1.16:6804/7019
> >> 172.16.1.16:6805/7019 exists,up 216774f0-d9e6-4d07-9f19-1aea5392f832
> >> osd.35 up in weight 1 up_from 74 up_thru 139 down_at 73
> >> last_clean_interval [3,69) 172.16.2.9:6800/6877 172.16.2.9:6803/6877
> >> 172.16.2.9:6804/6877 exists,up 4ec66ccf-adbb-45cf-abed-a70eff93879e
> >> osd.39 up in weight 1 up_from 78 up_thru 139 down_at 77
> >> last_clean_interval [4,76) 172.16.2.10:6803/11465
> >> 172.16.2.10:6804/11465 172.16.2.10:6805/11465 exists,up
> >> 79f9026b-3c56-4d9f-9a0e-046902a4cfff
> >> osd.43 up in weight 1 up_from 81 up_thru 139 down_at 80
> >> last_clean_interval [4,78) 172.16.2.11:6803/7007 172.16.2.11:6804/7007
> >> 172.16.2.11:6805/7007 exists,up 961d7332-8df3-47cd-86f2-732920373145
> >> osd.47 up in weight 1 up_from 86 up_thru 139 down_at 85
> >> last_clean_interval [5,81) 172.16.2.12:6803/7057 172.16.2.12:6804/7057
> >> 172.16.2.12:6805/7057 exists,up c9a9b167-7454-456c-8a31-05de29e7bfd9
> >> osd.51 up in weight 1 up_from 91 up_thru 139 down_at 90
> >> last_clean_interval [6,88) 172.16.2.13:6800/7111 172.16.2.13:6803/7111
> >> 172.16.2.13:6804/7111 exists,up d78e9778-0eb0-48a1-bbb0-80326decfd88
> >> osd.55 up in weight 1 up_from 95 up_thru 139 down_at 94
> >> last_clean_interval [7,90) 172.16.2.14:6803/7189 172.16.2.14:6804/7189
> >> 172.16.2.14:6805/7189 exists,up 63701034-d4bf-47e5-aebf-fbce1c9997b1
> >> osd.59 up in weight 1 up_from 99 up_thru 139 down_at 98
> >> last_clean_interval [10,97) 172.16.2.15:6803/7242
> >> 172.16.2.15:6804/7242 172.16.2.15:6805/7242 exists,up
> >> 7fe841dd-ddb0-4e63-bd1c-4695c273d0d3
> >> osd.63 up in weight 1 up_from 103 up_thru 139 down_at 102
> >> last_clean_interval [11,100) 172.16.2.16:6803/8679
> >> 172.16.2.16:6804/8679 172.16.2.16:6805/8679 exists,up
> >> 3d3f2dd7-feb2-4720-b0b2-aeb68a4c3be
> >> f
> >> osd.67 up in weight 1 up_from 107 up_thru 139 down_at 106
> >> last_clean_interval [14,101) 172.16.3.9:6803/7320 172.16.3.9:6804/7320
> >> 172.16.3.9:6805/7320 exists,up fb86457e-8e2f-4e3f-8537-b0a21c928e81
> >> osd.71 up in weight 1 up_from 115 up_thru 139 down_at 114
> >> last_clean_interval [15,109) 172.16.3.10:6803/7375
> >> 172.16.3.10:6804/7375 172.16.3.10:6805/7375 exists,up
> >> b3fa6928-8495-49de-8bac-90f4c61c1f5
> >> 0
> >> osd.75 up in weight 1 up_from 119 up_thru 139 down_at 118
> >> last_clean_interval [19,117) 172.16.3.11:6803/7473
> >> 172.16.3.11:6804/7473 172.16.3.11:6805/7473 exists,up
> >> 9f39ee19-3c17-4816-975a-ad2b895b48e
> >> 8
> >> osd.79 up in weight 1 up_from 121 up_thru 139 down_at 120
> >> last_clean_interval [24,117) 172.16.3.12:6803/7512
> >> 172.16.3.12:6804/7512 172.16.3.12:6805/7512 exists,up
> >> f41eca77-6726-4872-981c-f309082f680
> >> b
> >> osd.83 up in weight 1 up_from 125 up_thru 139 down_at 124
> >> last_clean_interval [26,123) 172.16.3.13:6803/7581
> >> 172.16.3.13:6804/7581 172.16.3.13:6805/7581 exists,up
> >> e22b3b3e-296a-46fd-b313-96769179508
> >> 0
> >> osd.87 up in weight 1 up_from 129 up_thru 139 down_at 128
> >> last_clean_interval [30,123) 172.16.3.14:6800/7624
> >> 172.16.3.14:6803/7624 172.16.3.14:6804/7624 exists,up
> >> a3f86554-283c-426a-b3f9-a8e3da227fb
> >> 1
> >> osd.91 up in weight 1 up_from 135 up_thru 139 down_at 134
> >> last_clean_interval [35,133) 172.16.3.15:6803/7685
> >> 172.16.3.15:6804/7685 172.16.3.15:6805/7685 exists,up
> >> 805b77d5-89b9-4dec-b107-9615fdacbbd
> >> 0
> >> osd.95 up in weight 1 up_from 139 up_thru 139 down_at 138
> >> last_clean_interval [35,133) 172.16.3.16:6800/7756
> >> 172.16.3.16:6803/7756 172.16.3.16:6804/7756 exists,up
> >> c909eeba-d09d-40da-9100-a88c0cf60a7
> >> b
> >>
> >>
> >>
> >>
> >> # service ceph -a restart
> >> === mon.a ===
> >> === mon.a ===
> >> Stopping Ceph mon.a on PROXY2...kill 11651...done
> >> === mon.a ===
> >> Starting Ceph mon.a on PROXY2...
> >> starting mon.a rank 1 at 172.16.0.25:6789/0 mon_data
> >> /var/lib/ceph/mon/ceph-a fsid 16cf888d-a38f-4308-84ad-300b89b9fae9
> >> === mon.b ===
> >> === mon.b ===
> >> Stopping Ceph mon.b on PROXY3...kill 26928...done
> >> === mon.b ===
> >> Starting Ceph mon.b on PROXY3...
> >> starting mon.b rank 2 at 172.16.0.26:6789/0 mon_data
> >> /var/lib/ceph/mon/ceph-b fsid 16cf888d-a38f-4308-84ad-300b89b9fae9
> >> === mon.c ===
> >> === mon.c ===
> >> Stopping Ceph mon.c on PROXY1...kill 17719...done
> >> === mon.c ===
> >> Starting Ceph mon.c on PROXY1...
> >> starting mon.c rank 0 at 172.16.0.24:6789/0 mon_data
> >> /var/lib/ceph/mon/ceph-c fsid 16cf888d-a38f-4308-84ad-300b89b9fae9
> >> === mds.a ===
> >> === mds.a ===
> >> Stopping Ceph mds.a on PROXY2...kill 11805...done
> >> === mds.a ===
> >> Starting Ceph mds.a on PROXY2...
> >> starting mds.a at :/0
> >> === osd.0 ===
> >> === osd.0 ===
> >> Stopping Ceph osd.0 on server109...done
> >> === osd.0 ===
> >> Mounting xfs on server109:/var/lib/ceph/osd/ceph-0
> >> Starting Ceph osd.0 on server109...
> >> starting osd.0 at :/0 osd_data /var/lib/ceph/osd/ceph-0 /dev/sdf
> >> === osd.1 ===
> >> === osd.1 ===
> >> Stopping Ceph osd.1 on server109...done
> >> === osd.1 ===
> >> Mounting xfs on server109:/var/lib/ceph/osd/ceph-1
> >> Starting Ceph osd.1 on server109...
> >> starting osd.1 at :/0 osd_data /var/lib/ceph/osd/ceph-1 /dev/sdf
> >> ........
> >> ........
> >> ........
> >> === osd.93 ===
> >> === osd.93 ===
> >> Stopping Ceph osd.93 on server316...done
> >> === osd.93 ===
> >> Mounting xfs on server316:/var/lib/ceph/osd/ceph-93
> >> Starting Ceph osd.93 on server316...
> >> starting osd.93 at :/0 osd_data /var/lib/ceph/osd/ceph-93 /dev/sdf
> >> === osd.94 ===
> >> === osd.94 ===
> >> Stopping Ceph osd.94 on server316...done
> >> === osd.94 ===
> >> Mounting xfs on server316:/var/lib/ceph/osd/ceph-94
> >> Starting Ceph osd.94 on server316...
> >> starting osd.94 at :/0 osd_data /var/lib/ceph/osd/ceph-94 /dev/sdf
> >> === osd.95 ===
> >> === osd.95 ===
> >> Stopping Ceph osd.95 on server316...kill 7110...done
> >> === osd.95 ===
> >> Mounting xfs on server316:/var/lib/ceph/osd/ceph-95
> >> Starting Ceph osd.95 on server316...
> >> starting osd.95 at :/0 osd_data /var/lib/ceph/osd/ceph-95 /dev/sdf
> >>
> >> MEANING
> >> when i restart the service ...it restart all OSDs (osd.0 - osd.95)
> >> BUT when i check which OSDs are actually up. it is 24 NOT 96.
> >>
> >> Please how should i solve the problem???
> >>
> >> Regards.
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@xxxxxxxxxxxxxxx
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> >
> >
> > --
> > Patrick McGarry
> > Director, Community
> > Inktank
> >
> > @scuttlemonkey @inktank @ceph
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
