ceph pg repair fails...?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Dear All,

I've just found two inconsistent pg that fail to repair.

This might be the same bug as shown here:

<https://tracker.ceph.com/journals/145118/diff?detail_id=147260>

Cluster is running Nautilus 14.2.2
OS is Scientific Linux 7.6
DB/WAL on NVMe, Data on 12TB HDD

Logs below cab also be seen here: <http://p.ip.fi/UOUc>

[root@ceph-s1 ~]# ceph health detail
HEALTH_ERR 22 scrub errors; Possible data damage: 2 pgs inconsistent
OSD_SCRUB_ERRORS 22 scrub errors
PG_DAMAGED Possible data damage: 2 pgs inconsistent
    pg 2.2a7 is active+clean+inconsistent+failed_repair, acting
[83,60,133,326,281,162,180,172,144,219]
    pg 2.36b is active+clean+inconsistent+failed_repair, acting
[254,268,10,262,32,280,211,114,169,53]

Issued "pg repair" commands, osd log shows:
[root@ceph-n10 ~]# grep "2.2a7" /var/log/ceph/ceph-osd.83.log
2019-10-01 07:05:02.459 7f9adab4b700  0 log_channel(cluster) log [DBG] :
2.2a7 repair starts
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 shard 83(0) soid 2:e5472cab:::1000702081f.00000000:head :
candidate size 4096 info size 0 mismatch
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 shard 60(1) soid 2:e5472cab:::1000702081f.00000000:head :
candidate size 4096 info size 0 mismatch
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 shard 133(2) soid 2:e5472cab:::1000702081f.00000000:head :
candidate size 4096 info size 0 mismatch
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 shard 144(8) soid 2:e5472cab:::1000702081f.00000000:head :
candidate size 4096 info size 0 mismatch
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 shard 162(5) soid 2:e5472cab:::1000702081f.00000000:head :
candidate size 4096 info size 0 mismatch
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 shard 172(7) soid 2:e5472cab:::1000702081f.00000000:head :
candidate size 4096 info size 0 mismatch
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 shard 180(6) soid 2:e5472cab:::1000702081f.00000000:head :
candidate size 4096 info size 0 mismatch
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 shard 219(9) soid 2:e5472cab:::1000702081f.00000000:head :
candidate size 4096 info size 0 mismatch
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 shard 281(4) soid 2:e5472cab:::1000702081f.00000000:head :
candidate size 4096 info size 0 mismatch
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 shard 326(3) soid 2:e5472cab:::1000702081f.00000000:head :
candidate size 4096 info size 0 mismatch
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 soid 2:e5472cab:::1000702081f.00000000:head : failed to pick
suitable object info
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
repair 2.2a7s0 2:e5472cab:::1000702081f.00000000:head : on disk size
(4096) does not match object info size (0) adjusted for ondisk to (0)
2019-10-01 07:19:47.060 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 repair 11 errors, 0 fixed
[root@ceph-n10 ~]#

[root@ceph-s1 ~]#  ceph pg repair 2.36b
instructing pg 2.36bs0 on osd.254 to repair

[root@ceph-n29 ~]# grep "2.36b" /var/log/ceph/ceph-osd.254.log
2019-10-01 11:15:12.215 7fa01f589700  0 log_channel(cluster) log [DBG] :
2.36b repair starts
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b shard 254(0) soid 2:d6cac754:::100070209f6.00000000:head :
candidate size 4096 info size 0 mismatch
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b shard 10(2) soid 2:d6cac754:::100070209f6.00000000:head :
candidate size 4096 info size 0 mismatch
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b shard 32(4) soid 2:d6cac754:::100070209f6.00000000:head :
candidate size 4096 info size 0 mismatch
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b shard 53(9) soid 2:d6cac754:::100070209f6.00000000:head :
candidate size 4096 info size 0 mismatch
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b shard 114(7) soid 2:d6cac754:::100070209f6.00000000:head :
candidate size 4096 info size 0 mismatch
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b shard 169(8) soid 2:d6cac754:::100070209f6.00000000:head :
candidate size 4096 info size 0 mismatch
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b shard 211(6) soid 2:d6cac754:::100070209f6.00000000:head :
candidate size 4096 info size 0 mismatch
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b shard 262(3) soid 2:d6cac754:::100070209f6.00000000:head :
candidate size 4096 info size 0 mismatch
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b shard 268(1) soid 2:d6cac754:::100070209f6.00000000:head :
candidate size 4096 info size 0 mismatch
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b shard 280(5) soid 2:d6cac754:::100070209f6.00000000:head :
candidate size 4096 info size 0 mismatch
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b soid 2:d6cac754:::100070209f6.00000000:head : failed to pick
suitable object info
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
repair 2.36bs0 2:d6cac754:::100070209f6.00000000:head : on disk size
(4096) does not match object info size (0) adjusted for ondisk to (0)
2019-10-01 11:30:10.573 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b repair 11 errors, 0 fixed

Any advice on fixing this would be very welcome!

Best regards,

Jake

-- 
Jake Grimmett
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux