Hi,
We have ceph cluster (Ceph version 0.94.2) which
consists of four nodes with four disks on each node.
Ceph is configured to hold two replicas (size 2). We use
this cluster for ceph filesystem. Few days ago we had
power outage after which I had to replace three of our
cluster OSD disks. All OSD disks are now online, but I'm
unable to mount filesystem and constantly receive 'mount
error 5 = Input/output error'. Ceph status shows many
'incomplete' pgs and that 'mds cluster is degraded'.
According to 'ceph health detail' mds is replaying
journal.
[root@cnode0 ceph]# ceph -s
cluster 39c717a3-5e15-4e5e-bc54-7e7f1fd0ee24
health HEALTH_WARN
25 pgs backfill_toofull
10 pgs degraded
126 pgs down
263 pgs incomplete
54 pgs stale
10 pgs stuck degraded
263 pgs stuck inactive
54 pgs stuck stale
289 pgs stuck unclean
10 pgs stuck undersized
10 pgs undersized
4 requests are blocked > 32 sec
recovery 27139/10407227 objects degraded
(0.261%)
recovery 168597/10407227 objects misplaced
(1.620%)
4 near full osd(s)
too many PGs per OSD (312 > max 300)
mds cluster is degraded
monmap e6: 6 mons at
{0=x.x.70.1:6789/0,0m=x.x.71.1:6789/0,1=x.x.70.2:6789/0,1m=x.x.71.2:6789/0,2=x.x.70.3:6789/0,2m=x.x.71.3:6789/0}
election epoch 2958, quorum 0,1,2,3,4,5
0,1,2,0m,1m,2m
mdsmap e1236: 1/1/1 up {0=2=up:replay},
2 up:standby
osdmap e83705: 16 osds: 16 up, 16 in; 26 remapped
pgs
pgmap v40869228: 2496 pgs, 3 pools, 16952 GB data,
5046 kobjects
32825 GB used, 11698 GB / 44524 GB avail
27139/10407227 objects degraded (0.261%)
168597/10407227 objects misplaced (1.620%)
2153 active+clean
137 incomplete
126 down+incomplete
54 stale+active+clean
15 active+remapped+backfill_toofull
10
active+undersized+degraded+remapped+backfill_toofull
1 active+remapped
[root@cnode0 ceph]#
I wasn't able to find any solution in the Internet and I
worry I will make things even worse when continue to
troubleshoot this on my own. I'm stuck. Could you please
help?
Thanks.
Andrzej
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
