hello
I am running ceph hammer on debian jessie. using 6 old used
underwhelming servers
the cluster is a "in-migration" bastard mix of 3TB sata drives with on
disk journal partition, beeing migrated to 5 disk raid5 MD arrays with
ssd journals, for ram limitation reasons. There are about 18 raid5 sets
atm and the rest is 3TB spinners.
I have some challenges with scrub errors, that i am trying to sort out
using this http://ceph.com/planet/ceph-manually-repair-object/ method.
but they are quite stubborn/sticky
i do see that osd.8 is often represented in these inconsistencies. but
the broken objects are not allways on osd.8 itself
in the instructions at
http://ceph.com/planet/ceph-manually-repair-object/, one finds the
object name by grepping in the logs.
but some of these haven been here a while. so how can i identify the
broken object if the log file have been rotated away ?
in the end i move away the broken object with size 0 and run pg repair,
but the error is not removed.
does the pg need to scrub after the repair for it to clear the error. ?
any advice is appreciated
kind regards
Ronny Aasen
#ceph -s
cluster 3c229f54-bd12-4b4e-a143-1ec73dd0f12a
health HEALTH_ERR
3 pgs degraded
9 pgs inconsistent
3 pgs recovering
3 pgs stuck degraded
3 pgs stuck unclean
recovery 88/125583766 objects degraded (0.000%)
recovery 666778/125583766 objects misplaced (0.531%)
recovery 88/45311043 unfound (0.000%)
9 scrub errors
noout,noscrub,nodeep-scrub flag(s) set
monmap e1: 3 mons at
{mon1=10.24.11.11:6789/0,mon2=10.24.11.12:6789/0,mon3=10.24.11.13:6789/0}
election epoch 60, quorum 0,1,2 mon1,mon2,mon3
osdmap e105977: 92 osds: 92 up, 92 in; 2 remapped pgs
flags noout,noscrub,nodeep-scrub
pgmap v12896186: 4608 pgs, 3 pools, 117 TB data, 44249 kobjects
308 TB used, 107 TB / 416 TB avail
88/125583766 objects degraded (0.000%)
666778/125583766 objects misplaced (0.531%)
88/45311043 unfound (0.000%)
4593 active+clean
9 active+clean+inconsistent
3 active+clean+scrubbing
2 active+recovering+degraded+remapped
1 active+recovering+degraded
client io 4572 kB/s rd, 1141 op/s
# ceph health detail
HEALTH_ERR 3 pgs degraded; 9 pgs inconsistent; 3 pgs recovering; 3 pgs
stuck degraded; 3 pgs stuck unclean; recovery 88/125583766 objects
degraded (0.000%); recovery 666778/125583766 objects misplaced (0.531%);
recovery 88/45311043 unfound (0.000%); 9 scrub errors;
noout,noscrub,nodeep-scrub flag(s) set
pg 6.d4 is stuck unclean for 3770820.461291, current state
active+recovering+degraded+remapped, last acting [62,8]
pg 6.da is stuck unclean for 2420102.778679, current state
active+recovering+degraded, last acting [6,110]
pg 6.ab is stuck unclean for 3774233.330685, current state
active+recovering+degraded+remapped, last acting [12,8]
pg 6.d4 is stuck degraded for 304239.715211, current state
active+recovering+degraded+remapped, last acting [62,8]
pg 6.da is stuck degraded for 416210.309539, current state
active+recovering+degraded, last acting [6,110]
pg 6.ab is stuck degraded for 304239.779541, current state
active+recovering+degraded+remapped, last acting [12,8]
pg 1.356 is active+clean+inconsistent, acting [8,84,39]
pg 1.1a7 is active+clean+inconsistent, acting [8,36,34]
pg 1.11e is active+clean+inconsistent, acting [8,12,6]
pg 6.da is active+recovering+degraded, acting [6,110], 25 unfound
pg 6.d4 is active+recovering+degraded+remapped, acting [62,8], 25 unfound
pg 6.ab is active+recovering+degraded+remapped, acting [12,8], 38 unfound
pg 1.de4 is active+clean+inconsistent, acting [41,8,108]
pg 1.c90 is active+clean+inconsistent, acting [12,71,8]
pg 1.ae6 is active+clean+inconsistent, acting [8,36,49]
pg 1.8bc is active+clean+inconsistent, acting [59,8,107]
pg 1.806 is active+clean+inconsistent, acting [60,3,106]
pg 1.675 is active+clean+inconsistent, acting [37,106,62]
recovery 88/125583766 objects degraded (0.000%)
recovery 666778/125583766 objects misplaced (0.531%)
recovery 88/45311043 unfound (0.000%)
9 scrub errors
noout,noscrub,nodeep-scrub flag(s) set
NB: the 88 unfound objects are in a pool i experimented with size 2, so
not important in this context.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
