Re: lot of inconsistent+failed_repair - failed to pick suitable auth object (14.2.3)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Does pool 6 have min_size = 1 set?

https://tracker.ceph.com/issues/24994#note-5 would possibly be helpful
here, depending on what the output of the following command looks
like.

# rados list-inconsistent-obj [pgid] --format=json-pretty

On Thu, Oct 10, 2019 at 8:16 PM Kenneth Waegeman
<kenneth.waegeman@xxxxxxxx> wrote:
>
> Hi all,
>
> After some node failure and rebalancing, we have a lot of pg's in
> inconsistent state. I tried to repair, but it din't work. This is also
> in the logs:
>
> > 2019-10-10 11:23:27.221 7ff54c9b0700  0 log_channel(cluster) log [DBG]
> > : 6.327 repair starts
> > 2019-10-10 11:23:27.431 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4c130fd:::20005f3b582.00000000:head :
> > omap_digest 0x334f57be != omap_digest 0xa8c4ce76 from auth oi
> > 6:e4c130fd:::20005f3b582.00000000:head(203789'1033530 osd.3.0:3666642
> > dirty|omap|data_digest|omap_digest s 0 uv 1032164 dd ffffffff od
> > a8c4ce76 alloc_hint [0 0 0])
> > 2019-10-10 11:23:27.431 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4c130fd:::20005f3b582.00000000:head :
> > omap_digest 0x334f57be != omap_digest 0xa8c4ce76 from auth oi
> > 6:e4c130fd:::20005f3b582.00000000:head(203789'1033530 osd.3.0:3666642
> > dirty|omap|data_digest|omap_digest s 0 uv 1032164 dd ffffffff od
> > a8c4ce76 alloc_hint [0 0 0])
> > 2019-10-10 11:23:27.431 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4c130fd:::20005f3b582.00000000:head :
> > omap_digest 0x334f57be != omap_digest 0xa8c4ce76 from auth oi
> > 6:e4c130fd:::20005f3b582.00000000:head(203789'1033530 osd.3.0:3666642
> > dirty|omap|data_digest|omap_digest s 0 uv 1032164 dd ffffffff od
> > a8c4ce76 alloc_hint [0 0 0])
> > 2019-10-10 11:23:27.431 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4c130fd:::20005f3b582.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:27.731 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4c2e57b:::20005f11daa.00000000:head :
> > omap_digest 0x6aafaf97 != omap_digest 0x56dd55a2 from auth oi
> > 6:e4c2e57b:::20005f11daa.00000000:head(203789'1033711 osd.3.0:3666823
> > dirty|omap|data_digest|omap_digest s 0 uv 1032158 dd ffffffff od
> > 56dd55a2 alloc_hint [0 0 0])
> > 2019-10-10 11:23:27.731 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4c2e57b:::20005f11daa.00000000:head :
> > omap_digest 0x6aafaf97 != omap_digest 0x56dd55a2 from auth oi
> > 6:e4c2e57b:::20005f11daa.00000000:head(203789'1033711 osd.3.0:3666823
> > dirty|omap|data_digest|omap_digest s 0 uv 1032158 dd ffffffff od
> > 56dd55a2 alloc_hint [0 0 0])
> > 2019-10-10 11:23:27.731 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4c2e57b:::20005f11daa.00000000:head :
> > omap_digest 0x6aafaf97 != omap_digest 0x56dd55a2 from auth oi
> > 6:e4c2e57b:::20005f11daa.00000000:head(203789'1033711 osd.3.0:3666823
> > dirty|omap|data_digest|omap_digest s 0 uv 1032158 dd ffffffff od
> > 56dd55a2 alloc_hint [0 0 0])
> > 2019-10-10 11:23:27.731 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4c2e57b:::20005f11daa.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:27.971 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4c40009:::20005f45f1b.00000000:head :
> > omap_digest 0x7ccf5cc9 != omap_digest 0xe048d29 from auth oi
> > 6:e4c40009:::20005f45f1b.00000000:head(203789'1033837 osd.3.0:3666949
> > dirty|omap|data_digest|omap_digest s 0 uv 1032168 dd ffffffff od
> > e048d29 alloc_hint [0 0 0])
> > 2019-10-10 11:23:27.971 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4c40009:::20005f45f1b.00000000:head :
> > omap_digest 0x7ccf5cc9 != omap_digest 0xe048d29 from auth oi
> > 6:e4c40009:::20005f45f1b.00000000:head(203789'1033837 osd.3.0:3666949
> > dirty|omap|data_digest|omap_digest s 0 uv 1032168 dd ffffffff od
> > e048d29 alloc_hint [0 0 0])
> > 2019-10-10 11:23:27.971 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4c40009:::20005f45f1b.00000000:head :
> > omap_digest 0x7ccf5cc9 != omap_digest 0xe048d29 from auth oi
> > 6:e4c40009:::20005f45f1b.00000000:head(203789'1033837 osd.3.0:3666949
> > dirty|omap|data_digest|omap_digest s 0 uv 1032168 dd ffffffff od
> > e048d29 alloc_hint [0 0 0])
> > 2019-10-10 11:23:27.971 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4c40009:::20005f45f1b.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:28.041 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4c4a042:::20005f389fb.00000000:head :
> > omap_digest 0xdd1558b8 != omap_digest 0xcf9af548 from auth oi
> > 6:e4c4a042:::20005f389fb.00000000:head(203789'1033899 osd.3.0:3667011
> > dirty|omap|data_digest|omap_digest s 0 uv 1031358 dd ffffffff od
> > cf9af548 alloc_hint [0 0 0])
> > 2019-10-10 11:23:28.041 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4c4a042:::20005f389fb.00000000:head :
> > omap_digest 0xdd1558b8 != omap_digest 0xcf9af548 from auth oi
> > 6:e4c4a042:::20005f389fb.00000000:head(203789'1033899 osd.3.0:3667011
> > dirty|omap|data_digest|omap_digest s 0 uv 1031358 dd ffffffff od
> > cf9af548 alloc_hint [0 0 0])
> > 2019-10-10 11:23:28.041 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4c4a042:::20005f389fb.00000000:head :
> > omap_digest 0xdd1558b8 != omap_digest 0xcf9af548 from auth oi
> > 6:e4c4a042:::20005f389fb.00000000:head(203789'1033899 osd.3.0:3667011
> > dirty|omap|data_digest|omap_digest s 0 uv 1031358 dd ffffffff od
> > cf9af548 alloc_hint [0 0 0])
> > 2019-10-10 11:23:28.041 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4c4a042:::20005f389fb.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:28.791 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4c91caa:::20005f36925.00000000:head :
> > omap_digest 0xbdd9c6d8 != omap_digest 0xd207a82c from auth oi
> > 6:e4c91caa:::20005f36925.00000000:head(203789'1034361 osd.3.0:3667473
> > dirty|omap|data_digest|omap_digest s 0 uv 1032163 dd ffffffff od
> > d207a82c alloc_hint [0 0 0])
> > 2019-10-10 11:23:28.791 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4c91caa:::20005f36925.00000000:head :
> > omap_digest 0xbdd9c6d8 != omap_digest 0xd207a82c from auth oi
> > 6:e4c91caa:::20005f36925.00000000:head(203789'1034361 osd.3.0:3667473
> > dirty|omap|data_digest|omap_digest s 0 uv 1032163 dd ffffffff od
> > d207a82c alloc_hint [0 0 0])
> > 2019-10-10 11:23:28.791 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4c91caa:::20005f36925.00000000:head :
> > omap_digest 0xbdd9c6d8 != omap_digest 0xd207a82c from auth oi
> > 6:e4c91caa:::20005f36925.00000000:head(203789'1034361 osd.3.0:3667473
> > dirty|omap|data_digest|omap_digest s 0 uv 1032163 dd ffffffff od
> > d207a82c alloc_hint [0 0 0])
> > 2019-10-10 11:23:28.791 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4c91caa:::20005f36925.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:28.961 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4ca4cc7:::20005f41d80.00000000:head :
> > omap_digest 0x66bea9d5 != omap_digest 0xa296af17 from auth oi
> > 6:e4ca4cc7:::20005f41d80.00000000:head(203789'1034508 osd.3.0:3667620
> > dirty|omap|data_digest|omap_digest s 0 uv 1032167 dd ffffffff od
> > a296af17 alloc_hint [0 0 0])
> > 2019-10-10 11:23:28.961 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4ca4cc7:::20005f41d80.00000000:head :
> > omap_digest 0x66bea9d5 != omap_digest 0xa296af17 from auth oi
> > 6:e4ca4cc7:::20005f41d80.00000000:head(203789'1034508 osd.3.0:3667620
> > dirty|omap|data_digest|omap_digest s 0 uv 1032167 dd ffffffff od
> > a296af17 alloc_hint [0 0 0])
> > 2019-10-10 11:23:28.961 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4ca4cc7:::20005f41d80.00000000:head :
> > omap_digest 0x66bea9d5 != omap_digest 0xa296af17 from auth oi
> > 6:e4ca4cc7:::20005f41d80.00000000:head(203789'1034508 osd.3.0:3667620
> > dirty|omap|data_digest|omap_digest s 0 uv 1032167 dd ffffffff od
> > a296af17 alloc_hint [0 0 0])
> > 2019-10-10 11:23:28.961 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4ca4cc7:::20005f41d80.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:30.001 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4d16e2c:::1001dd18807.00000000:head :
> > omap_digest 0x8b67426c != omap_digest 0x265813d7 from auth oi
> > 6:e4d16e2c:::1001dd18807.00000000:head(203789'1035272 osd.3.0:3668384
> > dirty|omap|data_digest|omap_digest s 0 uv 1032169 dd ffffffff od
> > 265813d7 alloc_hint [0 0 0])
> > 2019-10-10 11:23:30.001 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4d16e2c:::1001dd18807.00000000:head :
> > omap_digest 0x8b67426c != omap_digest 0x265813d7 from auth oi
> > 6:e4d16e2c:::1001dd18807.00000000:head(203789'1035272 osd.3.0:3668384
> > dirty|omap|data_digest|omap_digest s 0 uv 1032169 dd ffffffff od
> > 265813d7 alloc_hint [0 0 0])
> > 2019-10-10 11:23:30.001 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4d16e2c:::1001dd18807.00000000:head :
> > omap_digest 0x8b67426c != omap_digest 0x265813d7 from auth oi
> > 6:e4d16e2c:::1001dd18807.00000000:head(203789'1035272 osd.3.0:3668384
> > dirty|omap|data_digest|omap_digest s 0 uv 1032169 dd ffffffff od
> > 265813d7 alloc_hint [0 0 0])
> > 2019-10-10 11:23:30.001 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4d16e2c:::1001dd18807.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:30.181 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4d29b3d:::20005f332fb.00000000:head :
> > omap_digest 0xa6a8b00d != omap_digest 0x3f609d1d from auth oi
> > 6:e4d29b3d:::20005f332fb.00000000:head(203789'1035413 osd.3.0:3668525
> > dirty|omap|data_digest|omap_digest s 0 uv 1032160 dd ffffffff od
> > 3f609d1d alloc_hint [0 0 0])
> > 2019-10-10 11:23:30.181 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4d29b3d:::20005f332fb.00000000:head :
> > omap_digest 0xa6a8b00d != omap_digest 0x3f609d1d from auth oi
> > 6:e4d29b3d:::20005f332fb.00000000:head(203789'1035413 osd.3.0:3668525
> > dirty|omap|data_digest|omap_digest s 0 uv 1032160 dd ffffffff od
> > 3f609d1d alloc_hint [0 0 0])
> > 2019-10-10 11:23:30.181 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4d29b3d:::20005f332fb.00000000:head :
> > omap_digest 0xa6a8b00d != omap_digest 0x3f609d1d from auth oi
> > 6:e4d29b3d:::20005f332fb.00000000:head(203789'1035413 osd.3.0:3668525
> > dirty|omap|data_digest|omap_digest s 0 uv 1032160 dd ffffffff od
> > 3f609d1d alloc_hint [0 0 0])
> > 2019-10-10 11:23:30.181 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4d29b3d:::20005f332fb.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:30.551 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4d436d4:::20005f43997.00000000:head :
> > omap_digest 0xed0e46f != omap_digest 0xe480cd9a from auth oi
> > 6:e4d436d4:::20005f43997.00000000:head(203789'1035572 osd.3.0:3668684
> > dirty|omap|data_digest|omap_digest s 0 uv 1032166 dd ffffffff od
> > e480cd9a alloc_hint [0 0 0])
> > 2019-10-10 11:23:30.551 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4d436d4:::20005f43997.00000000:head :
> > omap_digest 0xed0e46f != omap_digest 0xe480cd9a from auth oi
> > 6:e4d436d4:::20005f43997.00000000:head(203789'1035572 osd.3.0:3668684
> > dirty|omap|data_digest|omap_digest s 0 uv 1032166 dd ffffffff od
> > e480cd9a alloc_hint [0 0 0])
> > 2019-10-10 11:23:30.551 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4d436d4:::20005f43997.00000000:head :
> > omap_digest 0xed0e46f != omap_digest 0xe480cd9a from auth oi
> > 6:e4d436d4:::20005f43997.00000000:head(203789'1035572 osd.3.0:3668684
> > dirty|omap|data_digest|omap_digest s 0 uv 1032166 dd ffffffff od
> > e480cd9a alloc_hint [0 0 0])
> > 2019-10-10 11:23:30.551 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4d436d4:::20005f43997.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:31.211 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4d8a4ca:::1001dd189d4.00000000:head :
> > omap_digest 0x4f413584 != omap_digest 0xb832d010 from auth oi
> > 6:e4d8a4ca:::1001dd189d4.00000000:head(203789'1036020 osd.3.0:3669132
> > dirty|omap|data_digest|omap_digest s 0 uv 1026527 dd ffffffff od
> > b832d010 alloc_hint [0 0 0])
> > 2019-10-10 11:23:31.211 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4d8a4ca:::1001dd189d4.00000000:head :
> > omap_digest 0x4f413584 != omap_digest 0xb832d010 from auth oi
> > 6:e4d8a4ca:::1001dd189d4.00000000:head(203789'1036020 osd.3.0:3669132
> > dirty|omap|data_digest|omap_digest s 0 uv 1026527 dd ffffffff od
> > b832d010 alloc_hint [0 0 0])
> > 2019-10-10 11:23:31.211 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4d8a4ca:::1001dd189d4.00000000:head :
> > omap_digest 0x4f413584 != omap_digest 0xb832d010 from auth oi
> > 6:e4d8a4ca:::1001dd189d4.00000000:head(203789'1036020 osd.3.0:3669132
> > dirty|omap|data_digest|omap_digest s 0 uv 1026527 dd ffffffff od
> > b832d010 alloc_hint [0 0 0])
> > 2019-10-10 11:23:31.211 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4d8a4ca:::1001dd189d4.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:31.541 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4dae402:::20005f4471c.00000000:head :
> > omap_digest 0x95196306 != omap_digest 0x78a1d6b0 from auth oi
> > 6:e4dae402:::20005f4471c.00000000:head(203789'1036246 osd.3.0:3669358
> > dirty|omap|data_digest|omap_digest s 0 uv 1031360 dd ffffffff od
> > 78a1d6b0 alloc_hint [0 0 0])
> > 2019-10-10 11:23:31.541 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4dae402:::20005f4471c.00000000:head :
> > omap_digest 0x95196306 != omap_digest 0x78a1d6b0 from auth oi
> > 6:e4dae402:::20005f4471c.00000000:head(203789'1036246 osd.3.0:3669358
> > dirty|omap|data_digest|omap_digest s 0 uv 1031360 dd ffffffff od
> > 78a1d6b0 alloc_hint [0 0 0])
> > 2019-10-10 11:23:31.541 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4dae402:::20005f4471c.00000000:head :
> > omap_digest 0x95196306 != omap_digest 0x78a1d6b0 from auth oi
> > 6:e4dae402:::20005f4471c.00000000:head(203789'1036246 osd.3.0:3669358
> > dirty|omap|data_digest|omap_digest s 0 uv 1031360 dd ffffffff od
> > 78a1d6b0 alloc_hint [0 0 0])
> > 2019-10-10 11:23:31.541 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4dae402:::20005f4471c.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:31.911 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4dd409f:::1001dd16f4c.00000000:head :
> > omap_digest 0xc0d797a6 != omap_digest 0x28ee2bfb from auth oi
> > 6:e4dd409f:::1001dd16f4c.00000000:head(203789'1036509 osd.3.0:3669621
> > dirty|omap|data_digest|omap_digest s 0 uv 1032170 dd ffffffff od
> > 28ee2bfb alloc_hint [0 0 0])
> > 2019-10-10 11:23:31.911 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4dd409f:::1001dd16f4c.00000000:head :
> > omap_digest 0xc0d797a6 != omap_digest 0x28ee2bfb from auth oi
> > 6:e4dd409f:::1001dd16f4c.00000000:head(203789'1036509 osd.3.0:3669621
> > dirty|omap|data_digest|omap_digest s 0 uv 1032170 dd ffffffff od
> > 28ee2bfb alloc_hint [0 0 0])
> > 2019-10-10 11:23:31.911 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4dd409f:::1001dd16f4c.00000000:head :
> > omap_digest 0xc0d797a6 != omap_digest 0x28ee2bfb from auth oi
> > 6:e4dd409f:::1001dd16f4c.00000000:head(203789'1036509 osd.3.0:3669621
> > dirty|omap|data_digest|omap_digest s 0 uv 1032170 dd ffffffff od
> > 28ee2bfb alloc_hint [0 0 0])
> > 2019-10-10 11:23:31.911 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4dd409f:::1001dd16f4c.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:32.131 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4de91a6:::20005f631c7.00000000:head :
> > omap_digest 0x9206e25b != omap_digest 0x4730506f from auth oi
> > 6:e4de91a6:::20005f631c7.00000000:head(203789'1036643 osd.3.0:3669755
> > dirty|omap|data_digest|omap_digest s 0 uv 1032783 dd ffffffff od
> > 4730506f alloc_hint [0 0 0])
> > 2019-10-10 11:23:32.131 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4de91a6:::20005f631c7.00000000:head :
> > omap_digest 0x9206e25b != omap_digest 0x4730506f from auth oi
> > 6:e4de91a6:::20005f631c7.00000000:head(203789'1036643 osd.3.0:3669755
> > dirty|omap|data_digest|omap_digest s 0 uv 1032783 dd ffffffff od
> > 4730506f alloc_hint [0 0 0])
> > 2019-10-10 11:23:32.131 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4de91a6:::20005f631c7.00000000:head :
> > omap_digest 0x9206e25b != omap_digest 0x4730506f from auth oi
> > 6:e4de91a6:::20005f631c7.00000000:head(203789'1036643 osd.3.0:3669755
> > dirty|omap|data_digest|omap_digest s 0 uv 1032783 dd ffffffff od
> > 4730506f alloc_hint [0 0 0])
> > 2019-10-10 11:23:32.131 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4de91a6:::20005f631c7.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:32.501 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4e106c1:::1001dd1895b.00000000:head :
> > omap_digest 0x8d24d314 != omap_digest 0x6df1ff43 from auth oi
> > 6:e4e106c1:::1001dd1895b.00000000:head(203789'1036895 osd.3.0:3670007
> > dirty|omap|data_digest|omap_digest s 0 uv 1026525 dd ffffffff od
> > 6df1ff43 alloc_hint [0 0 0])
> > 2019-10-10 11:23:32.501 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4e106c1:::1001dd1895b.00000000:head :
> > omap_digest 0x8d24d314 != omap_digest 0x6df1ff43 from auth oi
> > 6:e4e106c1:::1001dd1895b.00000000:head(203789'1036895 osd.3.0:3670007
> > dirty|omap|data_digest|omap_digest s 0 uv 1026525 dd ffffffff od
> > 6df1ff43 alloc_hint [0 0 0])
> > 2019-10-10 11:23:32.501 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4e106c1:::1001dd1895b.00000000:head :
> > omap_digest 0x8d24d314 != omap_digest 0x6df1ff43 from auth oi
> > 6:e4e106c1:::1001dd1895b.00000000:head(203789'1036895 osd.3.0:3670007
> > dirty|omap|data_digest|omap_digest s 0 uv 1026525 dd ffffffff od
> > 6df1ff43 alloc_hint [0 0 0])
> > 2019-10-10 11:23:32.501 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4e106c1:::1001dd1895b.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4e837c3:::20005f33142.00000000:head :
> > omap_digest 0x45cf8580 != omap_digest 0x7befa2c2 from auth oi
> > 6:e4e837c3:::20005f33142.00000000:head(203789'1037600 osd.3.0:3670712
> > dirty|omap|data_digest|omap_digest s 0 uv 1032159 dd ffffffff od
> > 7befa2c2 alloc_hint [0 0 0])
> > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4e837c3:::20005f33142.00000000:head :
> > omap_digest 0x45cf8580 != omap_digest 0x7befa2c2 from auth oi
> > 6:e4e837c3:::20005f33142.00000000:head(203789'1037600 osd.3.0:3670712
> > dirty|omap|data_digest|omap_digest s 0 uv 1032159 dd ffffffff od
> > 7befa2c2 alloc_hint [0 0 0])
> > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4e837c3:::20005f33142.00000000:head :
> > omap_digest 0x45cf8580 != omap_digest 0x7befa2c2 from auth oi
> > 6:e4e837c3:::20005f33142.00000000:head(203789'1037600 osd.3.0:3670712
> > dirty|omap|data_digest|omap_digest s 0 uv 1032159 dd ffffffff od
> > 7befa2c2 alloc_hint [0 0 0])
> > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4e837c3:::20005f33142.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4e842f7:::20005f33033.00000000:head :
> > omap_digest 0x33a64598 != omap_digest 0x1a45390a from auth oi
> > 6:e4e842f7:::20005f33033.00000000:head(203789'1037604 osd.3.0:3670716
> > dirty|omap|data_digest|omap_digest s 0 uv 1032161 dd ffffffff od
> > 1a45390a alloc_hint [0 0 0])
> > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4e842f7:::20005f33033.00000000:head :
> > omap_digest 0x33a64598 != omap_digest 0x1a45390a from auth oi
> > 6:e4e842f7:::20005f33033.00000000:head(203789'1037604 osd.3.0:3670716
> > dirty|omap|data_digest|omap_digest s 0 uv 1032161 dd ffffffff od
> > 1a45390a alloc_hint [0 0 0])
> > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4e842f7:::20005f33033.00000000:head :
> > omap_digest 0x33a64598 != omap_digest 0x1a45390a from auth oi
> > 6:e4e842f7:::20005f33033.00000000:head(203789'1037604 osd.3.0:3670716
> > dirty|omap|data_digest|omap_digest s 0 uv 1032161 dd ffffffff od
> > 1a45390a alloc_hint [0 0 0])
> > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4e842f7:::20005f33033.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:34.931 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4f15b8a:::20005f35cd7.00000000:head :
> > omap_digest 0xd196c582 != omap_digest 0xd3449697 from auth oi
> > 6:e4f15b8a:::20005f35cd7.00000000:head(203789'1038553 osd.3.0:3671665
> > dirty|omap|data_digest|omap_digest s 0 uv 1031357 dd ffffffff od
> > d3449697 alloc_hint [0 0 0])
> > 2019-10-10 11:23:34.931 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4f15b8a:::20005f35cd7.00000000:head :
> > omap_digest 0xd196c582 != omap_digest 0xd3449697 from auth oi
> > 6:e4f15b8a:::20005f35cd7.00000000:head(203789'1038553 osd.3.0:3671665
> > dirty|omap|data_digest|omap_digest s 0 uv 1031357 dd ffffffff od
> > d3449697 alloc_hint [0 0 0])
> > 2019-10-10 11:23:34.931 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4f15b8a:::20005f35cd7.00000000:head :
> > omap_digest 0xd196c582 != omap_digest 0xd3449697 from auth oi
> > 6:e4f15b8a:::20005f35cd7.00000000:head(203789'1038553 osd.3.0:3671665
> > dirty|omap|data_digest|omap_digest s 0 uv 1031357 dd ffffffff od
> > d3449697 alloc_hint [0 0 0])
> > 2019-10-10 11:23:34.931 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4f15b8a:::20005f35cd7.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:36.051 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4f8770e:::20005f3d629.00000000:head :
> > omap_digest 0x52a9267a != omap_digest 0xb2fbd549 from auth oi
> > 6:e4f8770e:::20005f3d629.00000000:head(203789'1039311 osd.3.0:3672423
> > dirty|omap|data_digest|omap_digest s 0 uv 1032165 dd ffffffff od
> > b2fbd549 alloc_hint [0 0 0])
> > 2019-10-10 11:23:36.051 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4f8770e:::20005f3d629.00000000:head :
> > omap_digest 0x52a9267a != omap_digest 0xb2fbd549 from auth oi
> > 6:e4f8770e:::20005f3d629.00000000:head(203789'1039311 osd.3.0:3672423
> > dirty|omap|data_digest|omap_digest s 0 uv 1032165 dd ffffffff od
> > b2fbd549 alloc_hint [0 0 0])
> > 2019-10-10 11:23:36.051 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4f8770e:::20005f3d629.00000000:head :
> > omap_digest 0x52a9267a != omap_digest 0xb2fbd549 from auth oi
> > 6:e4f8770e:::20005f3d629.00000000:head(203789'1039311 osd.3.0:3672423
> > dirty|omap|data_digest|omap_digest s 0 uv 1032165 dd ffffffff od
> > b2fbd549 alloc_hint [0 0 0])
> > 2019-10-10 11:23:36.051 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4f8770e:::20005f3d629.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:36.631 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4fc2d3e:::20005f570fb.00000000:head :
> > omap_digest 0xeac7763 != omap_digest 0x8965ec0b from auth oi
> > 6:e4fc2d3e:::20005f570fb.00000000:head(203789'1039698 osd.3.0:3672810
> > dirty|omap|data_digest|omap_digest s 0 uv 1032271 dd ffffffff od
> > 8965ec0b alloc_hint [0 0 0])
> > 2019-10-10 11:23:36.631 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4fc2d3e:::20005f570fb.00000000:head :
> > omap_digest 0xeac7763 != omap_digest 0x8965ec0b from auth oi
> > 6:e4fc2d3e:::20005f570fb.00000000:head(203789'1039698 osd.3.0:3672810
> > dirty|omap|data_digest|omap_digest s 0 uv 1032271 dd ffffffff od
> > 8965ec0b alloc_hint [0 0 0])
> > 2019-10-10 11:23:36.631 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4fc2d3e:::20005f570fb.00000000:head :
> > omap_digest 0xeac7763 != omap_digest 0x8965ec0b from auth oi
> > 6:e4fc2d3e:::20005f570fb.00000000:head(203789'1039698 osd.3.0:3672810
> > dirty|omap|data_digest|omap_digest s 0 uv 1032271 dd ffffffff od
> > 8965ec0b alloc_hint [0 0 0])
> > 2019-10-10 11:23:36.631 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4fc2d3e:::20005f570fb.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:36.821 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4fdb194:::20005f3759c.00000000:head :
> > omap_digest 0x1ef1fe3b != omap_digest 0x25f0dc6b from auth oi
> > 6:e4fdb194:::20005f3759c.00000000:head(203789'1039877 osd.3.0:3672989
> > dirty|omap|data_digest|omap_digest s 0 uv 1032162 dd ffffffff od
> > 25f0dc6b alloc_hint [0 0 0])
> > 2019-10-10 11:23:36.821 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4fdb194:::20005f3759c.00000000:head :
> > omap_digest 0x1ef1fe3b != omap_digest 0x25f0dc6b from auth oi
> > 6:e4fdb194:::20005f3759c.00000000:head(203789'1039877 osd.3.0:3672989
> > dirty|omap|data_digest|omap_digest s 0 uv 1032162 dd ffffffff od
> > 25f0dc6b alloc_hint [0 0 0])
> > 2019-10-10 11:23:36.821 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4fdb194:::20005f3759c.00000000:head :
> > omap_digest 0x1ef1fe3b != omap_digest 0x25f0dc6b from auth oi
> > 6:e4fdb194:::20005f3759c.00000000:head(203789'1039877 osd.3.0:3672989
> > dirty|omap|data_digest|omap_digest s 0 uv 1032162 dd ffffffff od
> > 25f0dc6b alloc_hint [0 0 0])
> > 2019-10-10 11:23:36.821 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4fdb194:::20005f3759c.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:36.901 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 19 soid 6:e4fe5018:::1001dd11fd4.00000000:head :
> > omap_digest 0xc0244623 != omap_digest 0xbaed0414 from auth oi
> > 6:e4fe5018:::1001dd11fd4.00000000:head(203789'1039928 osd.3.0:3673040
> > dirty|omap|data_digest|omap_digest s 0 uv 1031359 dd ffffffff od
> > baed0414 alloc_hint [0 0 0])
> > 2019-10-10 11:23:36.901 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 72 soid 6:e4fe5018:::1001dd11fd4.00000000:head :
> > omap_digest 0xc0244623 != omap_digest 0xbaed0414 from auth oi
> > 6:e4fe5018:::1001dd11fd4.00000000:head(203789'1039928 osd.3.0:3673040
> > dirty|omap|data_digest|omap_digest s 0 uv 1031359 dd ffffffff od
> > baed0414 alloc_hint [0 0 0])
> > 2019-10-10 11:23:36.901 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 shard 91 soid 6:e4fe5018:::1001dd11fd4.00000000:head :
> > omap_digest 0xc0244623 != omap_digest 0xbaed0414 from auth oi
> > 6:e4fe5018:::1001dd11fd4.00000000:head(203789'1039928 osd.3.0:3673040
> > dirty|omap|data_digest|omap_digest s 0 uv 1031359 dd ffffffff od
> > baed0414 alloc_hint [0 0 0])
> > 2019-10-10 11:23:36.901 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 soid 6:e4fe5018:::1001dd11fd4.00000000:head : failed to pick
> > suitable auth object
> > 2019-10-10 11:23:37.131 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > : 6.327 repair 63 errors, 0 fixed
> >
> >
> The cluster info, we are on ceph 14.2.3. :
> > [root@mds02 ~]# ceph health detail
> > HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 large
> > omap objects; 30967 scrub errors; Possible data damage: 219 pgs
> > inconsistent
> > FS_DEGRADED 1 filesystem is degraded
> >     fs ceph_fs is degraded
> > MDS_DAMAGE 1 mds daemon damaged
> >     fs ceph_fs mds.0 is damaged
> > LARGE_OMAP_OBJECTS 1 large omap objects
> >     1 large objects found in pool 'metadata'
> >     Search the cluster log for 'Large omap object found' for more details.
> > OSD_SCRUB_ERRORS 30967 scrub errors
> > PG_DAMAGED Possible data damage: 219 pgs inconsistent
> >     pg 6.23f is active+clean+inconsistent+failed_repair, acting
> > [18,55,541]
> >     pg 6.246 is active+clean+inconsistent+failed_repair, acting
> > [19,108,486]
> >     pg 6.24d is active+clean+inconsistent+failed_repair, acting
> > [109,91,442]
> >     pg 6.24e is active+clean+inconsistent+failed_repair, acting
> > [468,36,444]
> >     pg 6.255 is active+clean+inconsistent+failed_repair, acting
> > [37,513,442]
> >     pg 6.259 is active+clean+inconsistent+failed_repair, acting
> > [486,18,37]
> >     pg 6.262 is active+clean+inconsistent+failed_repair, acting
> > [449,108,90]
> >     pg 6.269 is active+clean+inconsistent+failed_repair, acting
> > [449,442,37]
> >     pg 6.26a is active+clean+inconsistent+failed_repair, acting
> > [36,513,449]
> >     pg 6.26b is active+clean+inconsistent+failed_repair, acting
> > [138,486,72]
> >     pg 6.27a is active+clean+inconsistent+failed_repair, acting
> > [520,515,468]
> >     pg 6.27c is active+clean+inconsistent+failed_repair, acting
> > [138,37,469]
> >     pg 6.282 is active+clean+inconsistent+failed_repair, acting
> > [37,54,109]
> >     pg 6.287 is active+clean+inconsistent+failed_repair, acting
> > [72,468,540]
> >     pg 6.288 is active+clean+inconsistent+failed_repair, acting
> > [541,140,449]
> >     pg 6.289 is active+clean+inconsistent+failed_repair, acting
> > [73,486,449]
> >     pg 6.28e is active+clean+inconsistent+failed_repair, acting
> > [449,36,109]
> >     pg 6.293 is active+clean+inconsistent+failed_repair, acting
> > [447,522,540]
> >     pg 6.294 is active+clean+inconsistent+failed_repair, acting
> > [513,444,37]
>
> > [root@mds02 ~]# ceph -s
> >   cluster:
> >     id:     92bfcf0a-1d39-43b3-b60f-44f01b630e47
> >     health: HEALTH_ERR
> >             1 filesystem is degraded
> >             1 mds daemon damaged
> >             1 large omap objects
> >             30967 scrub errors
> >             Possible data damage: 219 pgs inconsistent
> >
> >   services:
> >     mon: 3 daemons, quorum mds01,mds02,mds03 (age 2d)
> >     mgr: mds02(active, since 3w), standbys: mds01, mds03
> >     mds: ceph_fs:1/2 {1=mds03=up:resolve} 2 up:standby, 1 damaged
> >     osd: 535 osds: 533 up, 529 in
> >
> >   data:
> >     pools:   3 pools, 3328 pgs
> >     objects: 376.32M objects, 673 TiB
> >     usage:   1.0 PiB used, 2.2 PiB / 3.2 PiB avail
> >     pgs:     3097 active+clean
> >              160  active+clean+inconsistent
> >              59   active+clean+inconsistent
> >              9    active+clean+scrubbing+deep
> >              3    active+clean+scrubbing
> >
>
> Does somebody knows how to fix this?
>
> Thanks!
>
>
> Kenneth
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Cheers,
Brad

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux