2018-01-31 19:20 GMT+01:00 Gregory Farnum <gfarnum@xxxxxxxxxx>:
On Wed, Jan 31, 2018 at 1:40 AM Philip Poten <philip.poten@xxxxxxxxx> wrote:Hello,i have this error message:2018-01-25 00:59:27.357916 7fd646ae1700 -1 osd.3 pg_epoch: 9393 pg[9.139s0( v 8799'82397 (5494'79049,8799'82397] local-lis/les=9392/9393 n=10003 ec=1478/1478 lis/c 9392/6304 les/c/f 9393/6307/807 9391/9392/9392) [3,6,12,9]/[3,6,2147483647,4] r=0 lpr=9392 pi=[6304,9392)/3 bft=9(3),12(2) crt=8799'82397 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+backfilling] recover_replicas: object added to missing set for backfill, but is not in recovering, error! The line prior to what you pasted should have the object name in it.
Ok, so since I was unable to find this information with the usual methods, I'll follow up on this:
-2> 2018-02-01 04:27:27.414658 7f0c7a521700 -1 osd.3 pg_epoch: 10329 pg[9.139s0( v 10329'83100 (5494'79049,10329'83100] local-lis/les=10321/10322 n=9979 ec=1478/1478 lis/c 10321/6304 les/c/f 10322/630
7/807 10318/10321/10318) [3,6,12,9]/[3,6,2147483647,4] r=0 lpr=10321 pi=[6304,10321)/3 bft=9(3),12(2) crt=10329'83099 lcod 10329'83099 mlcod 10329'83099 active+undersized+degraded+remapped+backfilling] re
cover_replicas: object 9:9ccec3b7:::1000021235e.000008dc:head last_backfill 9:9ccec1a8:::100000f6bd9.000001a3:head
-1> 2018-02-01 04:27:27.414774 7f0c7a521700 -1 osd.3 pg_epoch: 10329 pg[9.139s0( v 10329'83100 (5494'79049,10329'83100] local-lis/les=10321/10322 n=9979 ec=1478/1478 lis/c 10321/6304 les/c/f 10322/6307/807 10318/10321/10318) [3,6,12,9]/[3,6,2147483647,4] r=0 lpr=10321 pi=[6304,10321)/3 bft=9(3),12(2) crt=10329'83099 lcod 10329'83099 mlcod 10329'83099 active+undersized+degraded+remapped+backfilling] recover_replicas: object added to missing set for backfill, but is not in recovering, error!
0> 2018-02-01 04:27:27.421623 7f0c7a521700 -1 *** Caught signal (Aborted) **
in thread 7f0c7a521700 thread_name:tp_osd_tp
ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
So apparently, "cover_replicas: object 9:9ccec3b7:::1000021235e.000008dc:head last_backfill 9:9ccec1a8:::100000f6bd9.000001a3:head" is the interesting part. But what exactly does it mean?
First, I looked into the contents of the pool and what the keys looked like (rados -p cephfs-data ls). So I figured, the object key isn't the whole thing, but only the something.something part. Then I tried retrieving them manually:
root@lxt-prod-ceph-mon02:~# rados -p cephfs-data get "1000021235e.000008dc" foo
error getting cephfs-data/1000021235e.000008dc: (5) Input/output error
root@lxt-prod-ceph-mon02:~# rados -p cephfs-data get "100000f6bd9.000001a3" foo
root@lxt-prod-ceph-mon02:~#
Which suggested, that indeed the 08dc key is the culprit, and 01a3 is probably just the last object that was backfilled (?) - it also told me, that the offending object luckily was part of the cephfs-data not the cephfs-metadata pool. phew.
So I tried removing the offending key:
root@lxt-prod-ceph-mon02:~# rados -p cephfs-data rm "1000021235e.000008dc"
root@lxt-prod-ceph-mon02:~#
and restarted backfills. And wouldn't you believe it, it restarted backfilling without the OSD crashing!
I haven't found a way to find the cephfs file the object belongs to yet, so if you can guide me with this, please let me know. But I'm sure sooner or later it will make itself known anyway when someone attempts to read it *cough*.
Thanks for your very helpful hint Greg!
Philip
PS: it was one damaged object that prevented me from moving the last degraded pg completely off a broken harddisk and prevented the whole cluster from being maintainable... that really shouldn't happen in my opinion.
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
