Re: lot of inconsistent+failed_repair - failed to pick suitable auth object (14.2.3)

[Kenneth Waegeman <kenneth.waegeman@xxxxxxxx>]

On 11/10/2019 01:21, Brad Hubbard wrote:
> On Fri, Oct 11, 2019 at 12:27 AM Kenneth Waegeman
> <kenneth.waegeman@xxxxxxxx> wrote:
> > Hi Brad, all,
> >
> > Pool 6 has min_size 2:
> >
> > pool 6 'metadata' replicated size 3 min_size 2 crush_rule 1 object_hash
> > rjenkins pg_num 1024 pgp_num 1024 autoscale_mode warn last_change 172476
> > flags hashpspool stripe_width 0 application cephfs
> This looked like something min_size 1 could cause, but I guess that's
> not the cause here.
>
> > so inconsistens is empty, which is weird, no ?
> Try scrubbing the pg just before running the command.

Ah that worked! I could then do the trick with the temporary_key to 
solve the inconsistent errors.

Thanks!!

K

> > Thanks again!
> >
> > K
> >
> >
> > On 10/10/2019 12:52, Brad Hubbard wrote:
> > > Does pool 6 have min_size = 1 set?
> > >
> > > https://tracker.ceph.com/issues/24994#note-5 would possibly be helpful
> > > here, depending on what the output of the following command looks
> > > like.
> > >
> > > # rados list-inconsistent-obj [pgid] --format=json-pretty
> > >
> > > On Thu, Oct 10, 2019 at 8:16 PM Kenneth Waegeman
> > > <kenneth.waegeman@xxxxxxxx> wrote:
> > > > Hi all,
> > > >
> > > > After some node failure and rebalancing, we have a lot of pg's in
> > > > inconsistent state. I tried to repair, but it din't work. This is also
> > > > in the logs:
> > > >
> > > > > 2019-10-10 11:23:27.221 7ff54c9b0700  0 log_channel(cluster) log [DBG]
> > > > > : 6.327 repair starts
> > > > > 2019-10-10 11:23:27.431 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4c130fd:::20005f3b582.00000000:head :
> > > > > omap_digest 0x334f57be != omap_digest 0xa8c4ce76 from auth oi
> > > > > 6:e4c130fd:::20005f3b582.00000000:head(203789'1033530 osd.3.0:3666642
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032164 dd ffffffff od
> > > > > a8c4ce76 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:27.431 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4c130fd:::20005f3b582.00000000:head :
> > > > > omap_digest 0x334f57be != omap_digest 0xa8c4ce76 from auth oi
> > > > > 6:e4c130fd:::20005f3b582.00000000:head(203789'1033530 osd.3.0:3666642
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032164 dd ffffffff od
> > > > > a8c4ce76 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:27.431 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4c130fd:::20005f3b582.00000000:head :
> > > > > omap_digest 0x334f57be != omap_digest 0xa8c4ce76 from auth oi
> > > > > 6:e4c130fd:::20005f3b582.00000000:head(203789'1033530 osd.3.0:3666642
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032164 dd ffffffff od
> > > > > a8c4ce76 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:27.431 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4c130fd:::20005f3b582.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:27.731 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4c2e57b:::20005f11daa.00000000:head :
> > > > > omap_digest 0x6aafaf97 != omap_digest 0x56dd55a2 from auth oi
> > > > > 6:e4c2e57b:::20005f11daa.00000000:head(203789'1033711 osd.3.0:3666823
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032158 dd ffffffff od
> > > > > 56dd55a2 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:27.731 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4c2e57b:::20005f11daa.00000000:head :
> > > > > omap_digest 0x6aafaf97 != omap_digest 0x56dd55a2 from auth oi
> > > > > 6:e4c2e57b:::20005f11daa.00000000:head(203789'1033711 osd.3.0:3666823
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032158 dd ffffffff od
> > > > > 56dd55a2 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:27.731 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4c2e57b:::20005f11daa.00000000:head :
> > > > > omap_digest 0x6aafaf97 != omap_digest 0x56dd55a2 from auth oi
> > > > > 6:e4c2e57b:::20005f11daa.00000000:head(203789'1033711 osd.3.0:3666823
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032158 dd ffffffff od
> > > > > 56dd55a2 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:27.731 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4c2e57b:::20005f11daa.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:27.971 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4c40009:::20005f45f1b.00000000:head :
> > > > > omap_digest 0x7ccf5cc9 != omap_digest 0xe048d29 from auth oi
> > > > > 6:e4c40009:::20005f45f1b.00000000:head(203789'1033837 osd.3.0:3666949
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032168 dd ffffffff od
> > > > > e048d29 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:27.971 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4c40009:::20005f45f1b.00000000:head :
> > > > > omap_digest 0x7ccf5cc9 != omap_digest 0xe048d29 from auth oi
> > > > > 6:e4c40009:::20005f45f1b.00000000:head(203789'1033837 osd.3.0:3666949
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032168 dd ffffffff od
> > > > > e048d29 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:27.971 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4c40009:::20005f45f1b.00000000:head :
> > > > > omap_digest 0x7ccf5cc9 != omap_digest 0xe048d29 from auth oi
> > > > > 6:e4c40009:::20005f45f1b.00000000:head(203789'1033837 osd.3.0:3666949
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032168 dd ffffffff od
> > > > > e048d29 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:27.971 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4c40009:::20005f45f1b.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:28.041 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4c4a042:::20005f389fb.00000000:head :
> > > > > omap_digest 0xdd1558b8 != omap_digest 0xcf9af548 from auth oi
> > > > > 6:e4c4a042:::20005f389fb.00000000:head(203789'1033899 osd.3.0:3667011
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1031358 dd ffffffff od
> > > > > cf9af548 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:28.041 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4c4a042:::20005f389fb.00000000:head :
> > > > > omap_digest 0xdd1558b8 != omap_digest 0xcf9af548 from auth oi
> > > > > 6:e4c4a042:::20005f389fb.00000000:head(203789'1033899 osd.3.0:3667011
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1031358 dd ffffffff od
> > > > > cf9af548 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:28.041 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4c4a042:::20005f389fb.00000000:head :
> > > > > omap_digest 0xdd1558b8 != omap_digest 0xcf9af548 from auth oi
> > > > > 6:e4c4a042:::20005f389fb.00000000:head(203789'1033899 osd.3.0:3667011
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1031358 dd ffffffff od
> > > > > cf9af548 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:28.041 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4c4a042:::20005f389fb.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:28.791 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4c91caa:::20005f36925.00000000:head :
> > > > > omap_digest 0xbdd9c6d8 != omap_digest 0xd207a82c from auth oi
> > > > > 6:e4c91caa:::20005f36925.00000000:head(203789'1034361 osd.3.0:3667473
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032163 dd ffffffff od
> > > > > d207a82c alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:28.791 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4c91caa:::20005f36925.00000000:head :
> > > > > omap_digest 0xbdd9c6d8 != omap_digest 0xd207a82c from auth oi
> > > > > 6:e4c91caa:::20005f36925.00000000:head(203789'1034361 osd.3.0:3667473
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032163 dd ffffffff od
> > > > > d207a82c alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:28.791 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4c91caa:::20005f36925.00000000:head :
> > > > > omap_digest 0xbdd9c6d8 != omap_digest 0xd207a82c from auth oi
> > > > > 6:e4c91caa:::20005f36925.00000000:head(203789'1034361 osd.3.0:3667473
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032163 dd ffffffff od
> > > > > d207a82c alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:28.791 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4c91caa:::20005f36925.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:28.961 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4ca4cc7:::20005f41d80.00000000:head :
> > > > > omap_digest 0x66bea9d5 != omap_digest 0xa296af17 from auth oi
> > > > > 6:e4ca4cc7:::20005f41d80.00000000:head(203789'1034508 osd.3.0:3667620
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032167 dd ffffffff od
> > > > > a296af17 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:28.961 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4ca4cc7:::20005f41d80.00000000:head :
> > > > > omap_digest 0x66bea9d5 != omap_digest 0xa296af17 from auth oi
> > > > > 6:e4ca4cc7:::20005f41d80.00000000:head(203789'1034508 osd.3.0:3667620
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032167 dd ffffffff od
> > > > > a296af17 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:28.961 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4ca4cc7:::20005f41d80.00000000:head :
> > > > > omap_digest 0x66bea9d5 != omap_digest 0xa296af17 from auth oi
> > > > > 6:e4ca4cc7:::20005f41d80.00000000:head(203789'1034508 osd.3.0:3667620
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032167 dd ffffffff od
> > > > > a296af17 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:28.961 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4ca4cc7:::20005f41d80.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:30.001 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4d16e2c:::1001dd18807.00000000:head :
> > > > > omap_digest 0x8b67426c != omap_digest 0x265813d7 from auth oi
> > > > > 6:e4d16e2c:::1001dd18807.00000000:head(203789'1035272 osd.3.0:3668384
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032169 dd ffffffff od
> > > > > 265813d7 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:30.001 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4d16e2c:::1001dd18807.00000000:head :
> > > > > omap_digest 0x8b67426c != omap_digest 0x265813d7 from auth oi
> > > > > 6:e4d16e2c:::1001dd18807.00000000:head(203789'1035272 osd.3.0:3668384
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032169 dd ffffffff od
> > > > > 265813d7 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:30.001 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4d16e2c:::1001dd18807.00000000:head :
> > > > > omap_digest 0x8b67426c != omap_digest 0x265813d7 from auth oi
> > > > > 6:e4d16e2c:::1001dd18807.00000000:head(203789'1035272 osd.3.0:3668384
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032169 dd ffffffff od
> > > > > 265813d7 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:30.001 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4d16e2c:::1001dd18807.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:30.181 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4d29b3d:::20005f332fb.00000000:head :
> > > > > omap_digest 0xa6a8b00d != omap_digest 0x3f609d1d from auth oi
> > > > > 6:e4d29b3d:::20005f332fb.00000000:head(203789'1035413 osd.3.0:3668525
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032160 dd ffffffff od
> > > > > 3f609d1d alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:30.181 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4d29b3d:::20005f332fb.00000000:head :
> > > > > omap_digest 0xa6a8b00d != omap_digest 0x3f609d1d from auth oi
> > > > > 6:e4d29b3d:::20005f332fb.00000000:head(203789'1035413 osd.3.0:3668525
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032160 dd ffffffff od
> > > > > 3f609d1d alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:30.181 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4d29b3d:::20005f332fb.00000000:head :
> > > > > omap_digest 0xa6a8b00d != omap_digest 0x3f609d1d from auth oi
> > > > > 6:e4d29b3d:::20005f332fb.00000000:head(203789'1035413 osd.3.0:3668525
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032160 dd ffffffff od
> > > > > 3f609d1d alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:30.181 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4d29b3d:::20005f332fb.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:30.551 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4d436d4:::20005f43997.00000000:head :
> > > > > omap_digest 0xed0e46f != omap_digest 0xe480cd9a from auth oi
> > > > > 6:e4d436d4:::20005f43997.00000000:head(203789'1035572 osd.3.0:3668684
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032166 dd ffffffff od
> > > > > e480cd9a alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:30.551 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4d436d4:::20005f43997.00000000:head :
> > > > > omap_digest 0xed0e46f != omap_digest 0xe480cd9a from auth oi
> > > > > 6:e4d436d4:::20005f43997.00000000:head(203789'1035572 osd.3.0:3668684
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032166 dd ffffffff od
> > > > > e480cd9a alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:30.551 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4d436d4:::20005f43997.00000000:head :
> > > > > omap_digest 0xed0e46f != omap_digest 0xe480cd9a from auth oi
> > > > > 6:e4d436d4:::20005f43997.00000000:head(203789'1035572 osd.3.0:3668684
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032166 dd ffffffff od
> > > > > e480cd9a alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:30.551 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4d436d4:::20005f43997.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:31.211 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4d8a4ca:::1001dd189d4.00000000:head :
> > > > > omap_digest 0x4f413584 != omap_digest 0xb832d010 from auth oi
> > > > > 6:e4d8a4ca:::1001dd189d4.00000000:head(203789'1036020 osd.3.0:3669132
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1026527 dd ffffffff od
> > > > > b832d010 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:31.211 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4d8a4ca:::1001dd189d4.00000000:head :
> > > > > omap_digest 0x4f413584 != omap_digest 0xb832d010 from auth oi
> > > > > 6:e4d8a4ca:::1001dd189d4.00000000:head(203789'1036020 osd.3.0:3669132
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1026527 dd ffffffff od
> > > > > b832d010 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:31.211 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4d8a4ca:::1001dd189d4.00000000:head :
> > > > > omap_digest 0x4f413584 != omap_digest 0xb832d010 from auth oi
> > > > > 6:e4d8a4ca:::1001dd189d4.00000000:head(203789'1036020 osd.3.0:3669132
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1026527 dd ffffffff od
> > > > > b832d010 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:31.211 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4d8a4ca:::1001dd189d4.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:31.541 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4dae402:::20005f4471c.00000000:head :
> > > > > omap_digest 0x95196306 != omap_digest 0x78a1d6b0 from auth oi
> > > > > 6:e4dae402:::20005f4471c.00000000:head(203789'1036246 osd.3.0:3669358
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1031360 dd ffffffff od
> > > > > 78a1d6b0 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:31.541 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4dae402:::20005f4471c.00000000:head :
> > > > > omap_digest 0x95196306 != omap_digest 0x78a1d6b0 from auth oi
> > > > > 6:e4dae402:::20005f4471c.00000000:head(203789'1036246 osd.3.0:3669358
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1031360 dd ffffffff od
> > > > > 78a1d6b0 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:31.541 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4dae402:::20005f4471c.00000000:head :
> > > > > omap_digest 0x95196306 != omap_digest 0x78a1d6b0 from auth oi
> > > > > 6:e4dae402:::20005f4471c.00000000:head(203789'1036246 osd.3.0:3669358
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1031360 dd ffffffff od
> > > > > 78a1d6b0 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:31.541 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4dae402:::20005f4471c.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:31.911 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4dd409f:::1001dd16f4c.00000000:head :
> > > > > omap_digest 0xc0d797a6 != omap_digest 0x28ee2bfb from auth oi
> > > > > 6:e4dd409f:::1001dd16f4c.00000000:head(203789'1036509 osd.3.0:3669621
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032170 dd ffffffff od
> > > > > 28ee2bfb alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:31.911 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4dd409f:::1001dd16f4c.00000000:head :
> > > > > omap_digest 0xc0d797a6 != omap_digest 0x28ee2bfb from auth oi
> > > > > 6:e4dd409f:::1001dd16f4c.00000000:head(203789'1036509 osd.3.0:3669621
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032170 dd ffffffff od
> > > > > 28ee2bfb alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:31.911 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4dd409f:::1001dd16f4c.00000000:head :
> > > > > omap_digest 0xc0d797a6 != omap_digest 0x28ee2bfb from auth oi
> > > > > 6:e4dd409f:::1001dd16f4c.00000000:head(203789'1036509 osd.3.0:3669621
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032170 dd ffffffff od
> > > > > 28ee2bfb alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:31.911 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4dd409f:::1001dd16f4c.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:32.131 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4de91a6:::20005f631c7.00000000:head :
> > > > > omap_digest 0x9206e25b != omap_digest 0x4730506f from auth oi
> > > > > 6:e4de91a6:::20005f631c7.00000000:head(203789'1036643 osd.3.0:3669755
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032783 dd ffffffff od
> > > > > 4730506f alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:32.131 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4de91a6:::20005f631c7.00000000:head :
> > > > > omap_digest 0x9206e25b != omap_digest 0x4730506f from auth oi
> > > > > 6:e4de91a6:::20005f631c7.00000000:head(203789'1036643 osd.3.0:3669755
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032783 dd ffffffff od
> > > > > 4730506f alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:32.131 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4de91a6:::20005f631c7.00000000:head :
> > > > > omap_digest 0x9206e25b != omap_digest 0x4730506f from auth oi
> > > > > 6:e4de91a6:::20005f631c7.00000000:head(203789'1036643 osd.3.0:3669755
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032783 dd ffffffff od
> > > > > 4730506f alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:32.131 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4de91a6:::20005f631c7.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:32.501 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4e106c1:::1001dd1895b.00000000:head :
> > > > > omap_digest 0x8d24d314 != omap_digest 0x6df1ff43 from auth oi
> > > > > 6:e4e106c1:::1001dd1895b.00000000:head(203789'1036895 osd.3.0:3670007
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1026525 dd ffffffff od
> > > > > 6df1ff43 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:32.501 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4e106c1:::1001dd1895b.00000000:head :
> > > > > omap_digest 0x8d24d314 != omap_digest 0x6df1ff43 from auth oi
> > > > > 6:e4e106c1:::1001dd1895b.00000000:head(203789'1036895 osd.3.0:3670007
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1026525 dd ffffffff od
> > > > > 6df1ff43 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:32.501 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4e106c1:::1001dd1895b.00000000:head :
> > > > > omap_digest 0x8d24d314 != omap_digest 0x6df1ff43 from auth oi
> > > > > 6:e4e106c1:::1001dd1895b.00000000:head(203789'1036895 osd.3.0:3670007
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1026525 dd ffffffff od
> > > > > 6df1ff43 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:32.501 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4e106c1:::1001dd1895b.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4e837c3:::20005f33142.00000000:head :
> > > > > omap_digest 0x45cf8580 != omap_digest 0x7befa2c2 from auth oi
> > > > > 6:e4e837c3:::20005f33142.00000000:head(203789'1037600 osd.3.0:3670712
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032159 dd ffffffff od
> > > > > 7befa2c2 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4e837c3:::20005f33142.00000000:head :
> > > > > omap_digest 0x45cf8580 != omap_digest 0x7befa2c2 from auth oi
> > > > > 6:e4e837c3:::20005f33142.00000000:head(203789'1037600 osd.3.0:3670712
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032159 dd ffffffff od
> > > > > 7befa2c2 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4e837c3:::20005f33142.00000000:head :
> > > > > omap_digest 0x45cf8580 != omap_digest 0x7befa2c2 from auth oi
> > > > > 6:e4e837c3:::20005f33142.00000000:head(203789'1037600 osd.3.0:3670712
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032159 dd ffffffff od
> > > > > 7befa2c2 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4e837c3:::20005f33142.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4e842f7:::20005f33033.00000000:head :
> > > > > omap_digest 0x33a64598 != omap_digest 0x1a45390a from auth oi
> > > > > 6:e4e842f7:::20005f33033.00000000:head(203789'1037604 osd.3.0:3670716
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032161 dd ffffffff od
> > > > > 1a45390a alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4e842f7:::20005f33033.00000000:head :
> > > > > omap_digest 0x33a64598 != omap_digest 0x1a45390a from auth oi
> > > > > 6:e4e842f7:::20005f33033.00000000:head(203789'1037604 osd.3.0:3670716
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032161 dd ffffffff od
> > > > > 1a45390a alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4e842f7:::20005f33033.00000000:head :
> > > > > omap_digest 0x33a64598 != omap_digest 0x1a45390a from auth oi
> > > > > 6:e4e842f7:::20005f33033.00000000:head(203789'1037604 osd.3.0:3670716
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032161 dd ffffffff od
> > > > > 1a45390a alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4e842f7:::20005f33033.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:34.931 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4f15b8a:::20005f35cd7.00000000:head :
> > > > > omap_digest 0xd196c582 != omap_digest 0xd3449697 from auth oi
> > > > > 6:e4f15b8a:::20005f35cd7.00000000:head(203789'1038553 osd.3.0:3671665
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1031357 dd ffffffff od
> > > > > d3449697 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:34.931 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4f15b8a:::20005f35cd7.00000000:head :
> > > > > omap_digest 0xd196c582 != omap_digest 0xd3449697 from auth oi
> > > > > 6:e4f15b8a:::20005f35cd7.00000000:head(203789'1038553 osd.3.0:3671665
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1031357 dd ffffffff od
> > > > > d3449697 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:34.931 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4f15b8a:::20005f35cd7.00000000:head :
> > > > > omap_digest 0xd196c582 != omap_digest 0xd3449697 from auth oi
> > > > > 6:e4f15b8a:::20005f35cd7.00000000:head(203789'1038553 osd.3.0:3671665
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1031357 dd ffffffff od
> > > > > d3449697 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:34.931 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4f15b8a:::20005f35cd7.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:36.051 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4f8770e:::20005f3d629.00000000:head :
> > > > > omap_digest 0x52a9267a != omap_digest 0xb2fbd549 from auth oi
> > > > > 6:e4f8770e:::20005f3d629.00000000:head(203789'1039311 osd.3.0:3672423
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032165 dd ffffffff od
> > > > > b2fbd549 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:36.051 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4f8770e:::20005f3d629.00000000:head :
> > > > > omap_digest 0x52a9267a != omap_digest 0xb2fbd549 from auth oi
> > > > > 6:e4f8770e:::20005f3d629.00000000:head(203789'1039311 osd.3.0:3672423
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032165 dd ffffffff od
> > > > > b2fbd549 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:36.051 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4f8770e:::20005f3d629.00000000:head :
> > > > > omap_digest 0x52a9267a != omap_digest 0xb2fbd549 from auth oi
> > > > > 6:e4f8770e:::20005f3d629.00000000:head(203789'1039311 osd.3.0:3672423
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032165 dd ffffffff od
> > > > > b2fbd549 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:36.051 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4f8770e:::20005f3d629.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:36.631 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4fc2d3e:::20005f570fb.00000000:head :
> > > > > omap_digest 0xeac7763 != omap_digest 0x8965ec0b from auth oi
> > > > > 6:e4fc2d3e:::20005f570fb.00000000:head(203789'1039698 osd.3.0:3672810
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032271 dd ffffffff od
> > > > > 8965ec0b alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:36.631 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4fc2d3e:::20005f570fb.00000000:head :
> > > > > omap_digest 0xeac7763 != omap_digest 0x8965ec0b from auth oi
> > > > > 6:e4fc2d3e:::20005f570fb.00000000:head(203789'1039698 osd.3.0:3672810
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032271 dd ffffffff od
> > > > > 8965ec0b alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:36.631 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4fc2d3e:::20005f570fb.00000000:head :
> > > > > omap_digest 0xeac7763 != omap_digest 0x8965ec0b from auth oi
> > > > > 6:e4fc2d3e:::20005f570fb.00000000:head(203789'1039698 osd.3.0:3672810
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032271 dd ffffffff od
> > > > > 8965ec0b alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:36.631 7ff5509b8700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4fc2d3e:::20005f570fb.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:36.821 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4fdb194:::20005f3759c.00000000:head :
> > > > > omap_digest 0x1ef1fe3b != omap_digest 0x25f0dc6b from auth oi
> > > > > 6:e4fdb194:::20005f3759c.00000000:head(203789'1039877 osd.3.0:3672989
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032162 dd ffffffff od
> > > > > 25f0dc6b alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:36.821 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4fdb194:::20005f3759c.00000000:head :
> > > > > omap_digest 0x1ef1fe3b != omap_digest 0x25f0dc6b from auth oi
> > > > > 6:e4fdb194:::20005f3759c.00000000:head(203789'1039877 osd.3.0:3672989
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032162 dd ffffffff od
> > > > > 25f0dc6b alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:36.821 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4fdb194:::20005f3759c.00000000:head :
> > > > > omap_digest 0x1ef1fe3b != omap_digest 0x25f0dc6b from auth oi
> > > > > 6:e4fdb194:::20005f3759c.00000000:head(203789'1039877 osd.3.0:3672989
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1032162 dd ffffffff od
> > > > > 25f0dc6b alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:36.821 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4fdb194:::20005f3759c.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:36.901 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 19 soid 6:e4fe5018:::1001dd11fd4.00000000:head :
> > > > > omap_digest 0xc0244623 != omap_digest 0xbaed0414 from auth oi
> > > > > 6:e4fe5018:::1001dd11fd4.00000000:head(203789'1039928 osd.3.0:3673040
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1031359 dd ffffffff od
> > > > > baed0414 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:36.901 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 72 soid 6:e4fe5018:::1001dd11fd4.00000000:head :
> > > > > omap_digest 0xc0244623 != omap_digest 0xbaed0414 from auth oi
> > > > > 6:e4fe5018:::1001dd11fd4.00000000:head(203789'1039928 osd.3.0:3673040
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1031359 dd ffffffff od
> > > > > baed0414 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:36.901 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 shard 91 soid 6:e4fe5018:::1001dd11fd4.00000000:head :
> > > > > omap_digest 0xc0244623 != omap_digest 0xbaed0414 from auth oi
> > > > > 6:e4fe5018:::1001dd11fd4.00000000:head(203789'1039928 osd.3.0:3673040
> > > > > dirty|omap|data_digest|omap_digest s 0 uv 1031359 dd ffffffff od
> > > > > baed0414 alloc_hint [0 0 0])
> > > > > 2019-10-10 11:23:36.901 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 soid 6:e4fe5018:::1001dd11fd4.00000000:head : failed to pick
> > > > > suitable auth object
> > > > > 2019-10-10 11:23:37.131 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
> > > > > : 6.327 repair 63 errors, 0 fixed
> > > > The cluster info, we are on ceph 14.2.3. :
> > > > > [root@mds02 ~]# ceph health detail
> > > > > HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 large
> > > > > omap objects; 30967 scrub errors; Possible data damage: 219 pgs
> > > > > inconsistent
> > > > > FS_DEGRADED 1 filesystem is degraded
> > > > >       fs ceph_fs is degraded
> > > > > MDS_DAMAGE 1 mds daemon damaged
> > > > >       fs ceph_fs mds.0 is damaged
> > > > > LARGE_OMAP_OBJECTS 1 large omap objects
> > > > >       1 large objects found in pool 'metadata'
> > > > >       Search the cluster log for 'Large omap object found' for more details.
> > > > > OSD_SCRUB_ERRORS 30967 scrub errors
> > > > > PG_DAMAGED Possible data damage: 219 pgs inconsistent
> > > > >       pg 6.23f is active+clean+inconsistent+failed_repair, acting
> > > > > [18,55,541]
> > > > >       pg 6.246 is active+clean+inconsistent+failed_repair, acting
> > > > > [19,108,486]
> > > > >       pg 6.24d is active+clean+inconsistent+failed_repair, acting
> > > > > [109,91,442]
> > > > >       pg 6.24e is active+clean+inconsistent+failed_repair, acting
> > > > > [468,36,444]
> > > > >       pg 6.255 is active+clean+inconsistent+failed_repair, acting
> > > > > [37,513,442]
> > > > >       pg 6.259 is active+clean+inconsistent+failed_repair, acting
> > > > > [486,18,37]
> > > > >       pg 6.262 is active+clean+inconsistent+failed_repair, acting
> > > > > [449,108,90]
> > > > >       pg 6.269 is active+clean+inconsistent+failed_repair, acting
> > > > > [449,442,37]
> > > > >       pg 6.26a is active+clean+inconsistent+failed_repair, acting
> > > > > [36,513,449]
> > > > >       pg 6.26b is active+clean+inconsistent+failed_repair, acting
> > > > > [138,486,72]
> > > > >       pg 6.27a is active+clean+inconsistent+failed_repair, acting
> > > > > [520,515,468]
> > > > >       pg 6.27c is active+clean+inconsistent+failed_repair, acting
> > > > > [138,37,469]
> > > > >       pg 6.282 is active+clean+inconsistent+failed_repair, acting
> > > > > [37,54,109]
> > > > >       pg 6.287 is active+clean+inconsistent+failed_repair, acting
> > > > > [72,468,540]
> > > > >       pg 6.288 is active+clean+inconsistent+failed_repair, acting
> > > > > [541,140,449]
> > > > >       pg 6.289 is active+clean+inconsistent+failed_repair, acting
> > > > > [73,486,449]
> > > > >       pg 6.28e is active+clean+inconsistent+failed_repair, acting
> > > > > [449,36,109]
> > > > >       pg 6.293 is active+clean+inconsistent+failed_repair, acting
> > > > > [447,522,540]
> > > > >       pg 6.294 is active+clean+inconsistent+failed_repair, acting
> > > > > [513,444,37]
> > > > > [root@mds02 ~]# ceph -s
> > > > >     cluster:
> > > > >       id:     92bfcf0a-1d39-43b3-b60f-44f01b630e47
> > > > >       health: HEALTH_ERR
> > > > >               1 filesystem is degraded
> > > > >               1 mds daemon damaged
> > > > >               1 large omap objects
> > > > >               30967 scrub errors
> > > > >               Possible data damage: 219 pgs inconsistent
> > > > >
> > > > >     services:
> > > > >       mon: 3 daemons, quorum mds01,mds02,mds03 (age 2d)
> > > > >       mgr: mds02(active, since 3w), standbys: mds01, mds03
> > > > >       mds: ceph_fs:1/2 {1=mds03=up:resolve} 2 up:standby, 1 damaged
> > > > >       osd: 535 osds: 533 up, 529 in
> > > > >
> > > > >     data:
> > > > >       pools:   3 pools, 3328 pgs
> > > > >       objects: 376.32M objects, 673 TiB
> > > > >       usage:   1.0 PiB used, 2.2 PiB / 3.2 PiB avail
> > > > >       pgs:     3097 active+clean
> > > > >                160  active+clean+inconsistent
> > > > >                59   active+clean+inconsistent
> > > > >                9    active+clean+scrubbing+deep
> > > > >                3    active+clean+scrubbing
> > > > Does somebody knows how to fix this?
> > > >
> > > > Thanks!
> > > >
> > > >
> > > > Kenneth
> > > >
> > > > _______________________________________________
> > > > ceph-users mailing list
> > > > ceph-users@xxxxxxxxxxxxxx
> > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com