I have a lot of stuck and down+incomplete and incomplete, but on pg query doesn't show where is the fail
ceph health detail
HEALTH_WARN clock skew detected on mon.3; 3 pgs down; 6 pgs incomplete; 6 pgs stuck inactive; 6 pgs stuck unclean; 17 requests are blocked > 32 sec; 3 osds have slow requests; Monitor clock skew detected
pg 0.3 is stuck inactive since forever, current state down+incomplete, last acting [5,4,8]
pg 0.38 is stuck inactive for 308757.882019, current state incomplete, last acting [1,4,8]
pg 0.43 is stuck inactive for 308590.063291, current state incomplete, last acting [2,1,4]
pg 0.78 is stuck inactive since forever, current state down+incomplete, last acting [6,4,3]
pg 0.27 is stuck inactive for 308606.854986, current state down+incomplete, last acting [2,7,5]
pg 0.67 is stuck inactive for 308606.854992, current state incomplete, last acting [2,1,3]
pg 0.3 is stuck unclean since forever, current state down+incomplete, last acting [5,4,8]
pg 0.38 is stuck unclean for 308757.882075, current state incomplete, last acting [1,4,8]
pg 0.43 is stuck unclean for 308590.063345, current state incomplete, last acting [2,1,4]
pg 0.78 is stuck unclean since forever, current state down+incomplete, last acting [6,4,3]
pg 0.27 is stuck unclean for 308991.817516, current state down+incomplete, last acting [2,7,5]
pg 0.67 is stuck unclean for 308991.817523, current state incomplete, last acting [2,1,3]
pg 0.27 is down+incomplete, acting [2,7,5]
pg 0.3 is down+incomplete, acting [5,4,8]
pg 0.78 is down+incomplete, acting [6,4,3]
pg 0.67 is incomplete, acting [2,1,3]
pg 0.43 is incomplete, acting [2,1,4]
pg 0.38 is incomplete, acting [1,4,8]
3 ops are blocked > 2097.15 sec
14 ops are blocked > 131.072 sec
1 ops are blocked > 2097.15 sec on osd.1
1 ops are blocked > 2097.15 sec on osd.2
14 ops are blocked > 131.072 sec on osd.2
1 ops are blocked > 2097.15 sec on osd.6
3 osds have slow requests
mon.3 addr 172.20.20.13:6789/0 clock skew 0.0559069s > max 0.05s (latency 0.00118267s)
#ceph pg 0.27 query
"1",
"2",
"3",
"4",
"5",
"6",
"7",
"8"
],
"down_osds_we_would_probe": [],
"peering_blocked_by": []
},
#ceph pg 0.3 query
....
{
"first": 3318,
"last": 3320,
"maybe_went_rw": 1,
"up": [
5,
4
],
"acting": [
5,
4
],
"primary": 5,
"up_primary": 5
}
],
"probing_osds": [
"1",
"2",
"3",
"4",
"5",
"6",
"7",
"8"
],
"down_osds_we_would_probe": [],
"peering_blocked_by": []
},
What i can do to solve this?
By th way, the clock is sinchronized.
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
