|
Hi, I’m looking for some help in figuring out why there are 2 pg’s in our cluster in 'active+undersized+degraded' state. They don’t seem to get assigned a 3rd osd to place data on. I’m not sure why,
everything looks ‘ok’ to me. Our ceph cluster consists of 3 nodes and has been upgraded from firefly to hammer to infernalis during its lifetime. Everything was fine until a few days ago when I created a new pool with # ceph osd pool create poolname 2048 replicated
and proceeded to create two rbd images # rbd create --size 2T poolname/image1 # rbd create --size 1600G poolname/image2 Since that moment ceph health shows a warning: cluster 6318a6a2-808b-45a1-9c89-31575c58de49 health HEALTH_WARN 2 pgs degraded 2 pgs stuck degraded 2 pgs stuck unclean 2 pgs stuck undersized 2 pgs undersized recovery 389/9308133 objects degraded (0.004%) monmap e7: 4 mons at {md002=172.19.20.2:6789/0,md005=172.19.20.5:6789/0,md008=172.19.20.8:6789/0,md010=172.19.20.10:6789/0} election epoch 18774, quorum 0,1,2,3 md002,md005,md008,md010 osdmap e105161: 30 osds: 30 up, 30 in pgmap v12776313: 2880 pgs, 5 pools, 12089 GB data, 3029 kobjects 36771 GB used, 24661 GB / 61433 GB avail 389/9308133 objects degraded (0.004%) 2878 active+clean 2 active+undersized+degraded client io 1883 kB/s rd, 15070 B/s wr, 1 op/s There is no recovery going on. We are on version 9.2.1 on centos 7 with kernel 4.4.9 except for the monitoring node which is still on 4.4.0. I’ve used crushtool to check whether the mapping should be ok, it seems to be fine (but I think this assumes all nodes in a cluster to be exactly the same, which they are not in our situation). There are no errors in the ceph logs. (zgrep –i err *gz in /var/log/ceph) Pg-num and pgp-num are both set to 3 for this pool. The details: HEALTH_WARN 2 pgs degraded; 2 pgs stuck degraded; 2 pgs stuck unclean; 2 pgs stuck undersized; 2 pgs undersized; recovery 389/9308133 objects degraded (0.004%) pg 24.17 is stuck unclean since forever, current state active+undersized+degraded, last acting [23,1] pg 24.54a is stuck unclean since forever, current state active+undersized+degraded, last acting [8,19] pg 24.17 is stuck undersized for 9653.439112, current state active+undersized+degraded, last acting [23,1] pg 24.54a is stuck undersized for 9659.961863, current state active+undersized+degraded, last acting [8,19] pg 24.17 is stuck degraded for 9653.439186, current state active+undersized+degraded, last acting [23,1] pg 24.54a is stuck degraded for 9659.961940, current state active+undersized+degraded, last acting [8,19] pg 24.54a is active+undersized+degraded, acting [8,19] pg 24.17 is active+undersized+degraded, acting [23,1] recovery 389/9308133 objects degraded (0.004%) # ceph pg dump_stuck degraded ok pg_stat state up up_primary acting acting_primary 24.17 active+undersized+degraded [23,1] 23 [23,1] 23 24.54a active+undersized+degraded [8,19] 8 [8,19] 8 # ceph pg map 24.17 osdmap e105161 pg 24.17 (24.17) -> up [23,1] acting [23,1] # ceph pg map 24.54a osdmap e105161 pg 24.54a (24.54a) -> up [8,19] acting [8,19] The osd tree and crushmap can be found here:
http://pastebin.com/i4BQq5Mi I’m hoping for some insight into why this is happening. I couldn’t find much out there on the net about undersized pg states other than that there are people trying to get a replication of 3 with less than
3 osd’s or less than 3 hosts when a host level has been specified in the crushmap hierarchy but that doesn’t apply here. Best regards, Max |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
