The cluster healthy state is WARN:
health HEALTH_WARN 118 pgs degraded; 8 pgs down; 59 pgs
incomplete; 28 pgs peering; 292 pgs stale; 87 pgs stuck inactive;
292 pgs stuck stale; 205 pgs stuck unclean; 22 requests are blocked
> 32 sec; recovery 12474/46357 objects degraded (26.909%)
monmap e3: 3 mons at
{CVM-0-mon01=172.18.117.146:6789/0,CVM-0-mon02=172.18.117.152:6789/0,CVM-0-mon03=172.18.117.153:6789/0},
election epoch 24, quorum 0,1,2 CVM-0-mon01,CVM-0-mon02,CVM-0-mon03
osdmap e421: 9 osds: 9 up, 9 in
pgmap v2261: 292 pgs, 4 pools, 91532 MB data, 23178 objects
330 MB used, 3363 GB / 3363 GB avail
12474/46357 objects degraded (26.909%)
20 stale+peering
87 stale+active+clean
8 stale+down+peering
59 stale+incomplete
118 stale+active+degraded
What does these errors mean? Can these PGs be recovered?
2014-09-25 10:23:51.408102 mon.0 [INF] mdsmap e1: 0/0/1 up
2014-09-25 10:23:41.060536 osd.3 [WRN] 2 slow requests, 1 included
below; oldest blocked for > 240.436614 secs
2014-09-25 10:23:41.060553 osd.3 [WRN] slow request 240.436614
seconds old, received at 2014-09-25 10:19:40.623834:
osd_op(client.4625.1:9005787 kvtest.rbd [watch add cookie 1 ver 0]
2.eb238611 reassert_version=81'5 ondisk+write e414) v4 currently
waiting for pg to exist locally
2014-09-25 10:23:39.042514 osd.7 [WRN] 4 slow requests, 1 included
below; oldest blocked for > 240.121452 secs
2014-09-25 10:23:39.042529 osd.7 [WRN] slow request 240.121452
seconds old, received at 2014-09-25 10:19:38.921007:
osd_op(client.4625.1:9005791 kvtest.rbd [watch add cookie 1 ver 0]
2.eb238611 reassert_version=81'5 ondisk+write e418) v4 currently
waiting for pg to exist locally
2014-09-25 10:23:53.064475 osd.3 [WRN] 2 slow requests, 1 included
below; oldest blocked for > 252.440597 secs
2014-09-25 10:23:53.064484 osd.3 [WRN] slow request 240.871862
seconds old, received at 2014-09-25 10:19:52.192569:
osd_op(client.4625.1:9005790 kvtest.rbd [watch add cookie 1 ver 0]
2.eb238611 reassert_version=81'5 ondisk+write e417) v4 currently
waiting for pg to exist locally
2014-09-25 10:24:32.135858 osd.8 [WRN] 6 slow requests, 6 included
below; oldest blocked for > 240.211169 secs
2014-09-25 10:24:32.135867 osd.8 [WRN] slow request 240.211169
seconds old, received at 2014-09-25 10:20:31.924624:
osd_op(client.4625.1:9005132 rb.0.120e.2ae8944a.0000000009ba [write
180224~16384] 2.72dfe69d RETRY=1 ondisk+retry+write e421) v4
currently waiting for pg to exist locally
2014-09-25 10:24:32.135872 osd.8 [WRN] slow request 240.210789
seconds old, received at 2014-09-25 10:20:31.925004:
osd_op(client.4625.1:9005188 rb.0.120e.2ae8944a.000000000371 [write
1916928~16384] 2.3445541d RETRY=1 ondisk+retry+write e421) v4
currently waiting for pg to exist locally
2014-09-25 10:24:32.135876 osd.8 [WRN] slow request 240.210747
seconds old, received at 2014-09-25 10:20:31.925046:
osd_op(client.4625.1:9005367 rb.0.120e.2ae8944a.000000000427 [write
3817472~16384] 2.654ab5d RETRY=1 ondisk+retry+write e421) v4
currently waiting for pg to exist locally
2014-09-25 10:24:32.135880 osd.8 [WRN] slow request 240.210667
seconds old, received at 2014-09-25 10:20:31.925126:
osd_op(client.4625.1:9005444 rb.0.120e.2ae8944a.000000000bc6 [write
622592~16384] 2.2f36185d RETRY=1 ondisk+retry+write e421) v4
currently waiting for pg to exist locally
2014-09-25 10:24:32.135886 osd.8 [WRN] slow request 240.210488
seconds old, received at 2014-09-25 10:20:31.925305:
osd_op(client.4625.1:9005581 rb.0.120e.2ae8944a.000000000753 [write
884736~16384] 2.be66e1d RETRY=1 ondisk+retry+write e421) v4
currently waiting for pg to exist locally
2014-09-25 10:24:33.136321 osd.8 [WRN] 8 slow requests, 3 included
below; oldest blocked for > 241.211660 secs
2014-09-25 10:24:33.136330 osd.8 [WRN] slow request 241.210921
seconds old, received at 2014-09-25 10:20:31.925363:
osd_op(client.4625.1:9005606 rb.0.120e.2ae8944a.0000000003f0 [write
3391488~16384] 2.fdfdcfdd RETRY=1 ondisk+retry+write e421) v4
currently waiting for pg to exist locally
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140925/c43e5f78/attachment.htm>
