Does pool 6 have min_size = 1 set? https://tracker.ceph.com/issues/24994#note-5 would possibly be helpful here, depending on what the output of the following command looks like. # rados list-inconsistent-obj [pgid] --format=json-pretty On Thu, Oct 10, 2019 at 8:16 PM Kenneth Waegeman <kenneth.waegeman@xxxxxxxx> wrote: > > Hi all, > > After some node failure and rebalancing, we have a lot of pg's in > inconsistent state. I tried to repair, but it din't work. This is also > in the logs: > > > 2019-10-10 11:23:27.221 7ff54c9b0700 0 log_channel(cluster) log [DBG] > > : 6.327 repair starts > > 2019-10-10 11:23:27.431 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4c130fd:::20005f3b582.00000000:head : > > omap_digest 0x334f57be != omap_digest 0xa8c4ce76 from auth oi > > 6:e4c130fd:::20005f3b582.00000000:head(203789'1033530 osd.3.0:3666642 > > dirty|omap|data_digest|omap_digest s 0 uv 1032164 dd ffffffff od > > a8c4ce76 alloc_hint [0 0 0]) > > 2019-10-10 11:23:27.431 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4c130fd:::20005f3b582.00000000:head : > > omap_digest 0x334f57be != omap_digest 0xa8c4ce76 from auth oi > > 6:e4c130fd:::20005f3b582.00000000:head(203789'1033530 osd.3.0:3666642 > > dirty|omap|data_digest|omap_digest s 0 uv 1032164 dd ffffffff od > > a8c4ce76 alloc_hint [0 0 0]) > > 2019-10-10 11:23:27.431 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4c130fd:::20005f3b582.00000000:head : > > omap_digest 0x334f57be != omap_digest 0xa8c4ce76 from auth oi > > 6:e4c130fd:::20005f3b582.00000000:head(203789'1033530 osd.3.0:3666642 > > dirty|omap|data_digest|omap_digest s 0 uv 1032164 dd ffffffff od > > a8c4ce76 alloc_hint [0 0 0]) > > 2019-10-10 11:23:27.431 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4c130fd:::20005f3b582.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:27.731 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4c2e57b:::20005f11daa.00000000:head : > > omap_digest 0x6aafaf97 != omap_digest 0x56dd55a2 from auth oi > > 6:e4c2e57b:::20005f11daa.00000000:head(203789'1033711 osd.3.0:3666823 > > dirty|omap|data_digest|omap_digest s 0 uv 1032158 dd ffffffff od > > 56dd55a2 alloc_hint [0 0 0]) > > 2019-10-10 11:23:27.731 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4c2e57b:::20005f11daa.00000000:head : > > omap_digest 0x6aafaf97 != omap_digest 0x56dd55a2 from auth oi > > 6:e4c2e57b:::20005f11daa.00000000:head(203789'1033711 osd.3.0:3666823 > > dirty|omap|data_digest|omap_digest s 0 uv 1032158 dd ffffffff od > > 56dd55a2 alloc_hint [0 0 0]) > > 2019-10-10 11:23:27.731 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4c2e57b:::20005f11daa.00000000:head : > > omap_digest 0x6aafaf97 != omap_digest 0x56dd55a2 from auth oi > > 6:e4c2e57b:::20005f11daa.00000000:head(203789'1033711 osd.3.0:3666823 > > dirty|omap|data_digest|omap_digest s 0 uv 1032158 dd ffffffff od > > 56dd55a2 alloc_hint [0 0 0]) > > 2019-10-10 11:23:27.731 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4c2e57b:::20005f11daa.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:27.971 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4c40009:::20005f45f1b.00000000:head : > > omap_digest 0x7ccf5cc9 != omap_digest 0xe048d29 from auth oi > > 6:e4c40009:::20005f45f1b.00000000:head(203789'1033837 osd.3.0:3666949 > > dirty|omap|data_digest|omap_digest s 0 uv 1032168 dd ffffffff od > > e048d29 alloc_hint [0 0 0]) > > 2019-10-10 11:23:27.971 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4c40009:::20005f45f1b.00000000:head : > > omap_digest 0x7ccf5cc9 != omap_digest 0xe048d29 from auth oi > > 6:e4c40009:::20005f45f1b.00000000:head(203789'1033837 osd.3.0:3666949 > > dirty|omap|data_digest|omap_digest s 0 uv 1032168 dd ffffffff od > > e048d29 alloc_hint [0 0 0]) > > 2019-10-10 11:23:27.971 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4c40009:::20005f45f1b.00000000:head : > > omap_digest 0x7ccf5cc9 != omap_digest 0xe048d29 from auth oi > > 6:e4c40009:::20005f45f1b.00000000:head(203789'1033837 osd.3.0:3666949 > > dirty|omap|data_digest|omap_digest s 0 uv 1032168 dd ffffffff od > > e048d29 alloc_hint [0 0 0]) > > 2019-10-10 11:23:27.971 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4c40009:::20005f45f1b.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:28.041 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4c4a042:::20005f389fb.00000000:head : > > omap_digest 0xdd1558b8 != omap_digest 0xcf9af548 from auth oi > > 6:e4c4a042:::20005f389fb.00000000:head(203789'1033899 osd.3.0:3667011 > > dirty|omap|data_digest|omap_digest s 0 uv 1031358 dd ffffffff od > > cf9af548 alloc_hint [0 0 0]) > > 2019-10-10 11:23:28.041 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4c4a042:::20005f389fb.00000000:head : > > omap_digest 0xdd1558b8 != omap_digest 0xcf9af548 from auth oi > > 6:e4c4a042:::20005f389fb.00000000:head(203789'1033899 osd.3.0:3667011 > > dirty|omap|data_digest|omap_digest s 0 uv 1031358 dd ffffffff od > > cf9af548 alloc_hint [0 0 0]) > > 2019-10-10 11:23:28.041 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4c4a042:::20005f389fb.00000000:head : > > omap_digest 0xdd1558b8 != omap_digest 0xcf9af548 from auth oi > > 6:e4c4a042:::20005f389fb.00000000:head(203789'1033899 osd.3.0:3667011 > > dirty|omap|data_digest|omap_digest s 0 uv 1031358 dd ffffffff od > > cf9af548 alloc_hint [0 0 0]) > > 2019-10-10 11:23:28.041 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4c4a042:::20005f389fb.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:28.791 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4c91caa:::20005f36925.00000000:head : > > omap_digest 0xbdd9c6d8 != omap_digest 0xd207a82c from auth oi > > 6:e4c91caa:::20005f36925.00000000:head(203789'1034361 osd.3.0:3667473 > > dirty|omap|data_digest|omap_digest s 0 uv 1032163 dd ffffffff od > > d207a82c alloc_hint [0 0 0]) > > 2019-10-10 11:23:28.791 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4c91caa:::20005f36925.00000000:head : > > omap_digest 0xbdd9c6d8 != omap_digest 0xd207a82c from auth oi > > 6:e4c91caa:::20005f36925.00000000:head(203789'1034361 osd.3.0:3667473 > > dirty|omap|data_digest|omap_digest s 0 uv 1032163 dd ffffffff od > > d207a82c alloc_hint [0 0 0]) > > 2019-10-10 11:23:28.791 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4c91caa:::20005f36925.00000000:head : > > omap_digest 0xbdd9c6d8 != omap_digest 0xd207a82c from auth oi > > 6:e4c91caa:::20005f36925.00000000:head(203789'1034361 osd.3.0:3667473 > > dirty|omap|data_digest|omap_digest s 0 uv 1032163 dd ffffffff od > > d207a82c alloc_hint [0 0 0]) > > 2019-10-10 11:23:28.791 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4c91caa:::20005f36925.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:28.961 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4ca4cc7:::20005f41d80.00000000:head : > > omap_digest 0x66bea9d5 != omap_digest 0xa296af17 from auth oi > > 6:e4ca4cc7:::20005f41d80.00000000:head(203789'1034508 osd.3.0:3667620 > > dirty|omap|data_digest|omap_digest s 0 uv 1032167 dd ffffffff od > > a296af17 alloc_hint [0 0 0]) > > 2019-10-10 11:23:28.961 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4ca4cc7:::20005f41d80.00000000:head : > > omap_digest 0x66bea9d5 != omap_digest 0xa296af17 from auth oi > > 6:e4ca4cc7:::20005f41d80.00000000:head(203789'1034508 osd.3.0:3667620 > > dirty|omap|data_digest|omap_digest s 0 uv 1032167 dd ffffffff od > > a296af17 alloc_hint [0 0 0]) > > 2019-10-10 11:23:28.961 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4ca4cc7:::20005f41d80.00000000:head : > > omap_digest 0x66bea9d5 != omap_digest 0xa296af17 from auth oi > > 6:e4ca4cc7:::20005f41d80.00000000:head(203789'1034508 osd.3.0:3667620 > > dirty|omap|data_digest|omap_digest s 0 uv 1032167 dd ffffffff od > > a296af17 alloc_hint [0 0 0]) > > 2019-10-10 11:23:28.961 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4ca4cc7:::20005f41d80.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:30.001 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4d16e2c:::1001dd18807.00000000:head : > > omap_digest 0x8b67426c != omap_digest 0x265813d7 from auth oi > > 6:e4d16e2c:::1001dd18807.00000000:head(203789'1035272 osd.3.0:3668384 > > dirty|omap|data_digest|omap_digest s 0 uv 1032169 dd ffffffff od > > 265813d7 alloc_hint [0 0 0]) > > 2019-10-10 11:23:30.001 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4d16e2c:::1001dd18807.00000000:head : > > omap_digest 0x8b67426c != omap_digest 0x265813d7 from auth oi > > 6:e4d16e2c:::1001dd18807.00000000:head(203789'1035272 osd.3.0:3668384 > > dirty|omap|data_digest|omap_digest s 0 uv 1032169 dd ffffffff od > > 265813d7 alloc_hint [0 0 0]) > > 2019-10-10 11:23:30.001 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4d16e2c:::1001dd18807.00000000:head : > > omap_digest 0x8b67426c != omap_digest 0x265813d7 from auth oi > > 6:e4d16e2c:::1001dd18807.00000000:head(203789'1035272 osd.3.0:3668384 > > dirty|omap|data_digest|omap_digest s 0 uv 1032169 dd ffffffff od > > 265813d7 alloc_hint [0 0 0]) > > 2019-10-10 11:23:30.001 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4d16e2c:::1001dd18807.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:30.181 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4d29b3d:::20005f332fb.00000000:head : > > omap_digest 0xa6a8b00d != omap_digest 0x3f609d1d from auth oi > > 6:e4d29b3d:::20005f332fb.00000000:head(203789'1035413 osd.3.0:3668525 > > dirty|omap|data_digest|omap_digest s 0 uv 1032160 dd ffffffff od > > 3f609d1d alloc_hint [0 0 0]) > > 2019-10-10 11:23:30.181 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4d29b3d:::20005f332fb.00000000:head : > > omap_digest 0xa6a8b00d != omap_digest 0x3f609d1d from auth oi > > 6:e4d29b3d:::20005f332fb.00000000:head(203789'1035413 osd.3.0:3668525 > > dirty|omap|data_digest|omap_digest s 0 uv 1032160 dd ffffffff od > > 3f609d1d alloc_hint [0 0 0]) > > 2019-10-10 11:23:30.181 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4d29b3d:::20005f332fb.00000000:head : > > omap_digest 0xa6a8b00d != omap_digest 0x3f609d1d from auth oi > > 6:e4d29b3d:::20005f332fb.00000000:head(203789'1035413 osd.3.0:3668525 > > dirty|omap|data_digest|omap_digest s 0 uv 1032160 dd ffffffff od > > 3f609d1d alloc_hint [0 0 0]) > > 2019-10-10 11:23:30.181 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4d29b3d:::20005f332fb.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:30.551 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4d436d4:::20005f43997.00000000:head : > > omap_digest 0xed0e46f != omap_digest 0xe480cd9a from auth oi > > 6:e4d436d4:::20005f43997.00000000:head(203789'1035572 osd.3.0:3668684 > > dirty|omap|data_digest|omap_digest s 0 uv 1032166 dd ffffffff od > > e480cd9a alloc_hint [0 0 0]) > > 2019-10-10 11:23:30.551 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4d436d4:::20005f43997.00000000:head : > > omap_digest 0xed0e46f != omap_digest 0xe480cd9a from auth oi > > 6:e4d436d4:::20005f43997.00000000:head(203789'1035572 osd.3.0:3668684 > > dirty|omap|data_digest|omap_digest s 0 uv 1032166 dd ffffffff od > > e480cd9a alloc_hint [0 0 0]) > > 2019-10-10 11:23:30.551 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4d436d4:::20005f43997.00000000:head : > > omap_digest 0xed0e46f != omap_digest 0xe480cd9a from auth oi > > 6:e4d436d4:::20005f43997.00000000:head(203789'1035572 osd.3.0:3668684 > > dirty|omap|data_digest|omap_digest s 0 uv 1032166 dd ffffffff od > > e480cd9a alloc_hint [0 0 0]) > > 2019-10-10 11:23:30.551 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4d436d4:::20005f43997.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:31.211 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4d8a4ca:::1001dd189d4.00000000:head : > > omap_digest 0x4f413584 != omap_digest 0xb832d010 from auth oi > > 6:e4d8a4ca:::1001dd189d4.00000000:head(203789'1036020 osd.3.0:3669132 > > dirty|omap|data_digest|omap_digest s 0 uv 1026527 dd ffffffff od > > b832d010 alloc_hint [0 0 0]) > > 2019-10-10 11:23:31.211 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4d8a4ca:::1001dd189d4.00000000:head : > > omap_digest 0x4f413584 != omap_digest 0xb832d010 from auth oi > > 6:e4d8a4ca:::1001dd189d4.00000000:head(203789'1036020 osd.3.0:3669132 > > dirty|omap|data_digest|omap_digest s 0 uv 1026527 dd ffffffff od > > b832d010 alloc_hint [0 0 0]) > > 2019-10-10 11:23:31.211 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4d8a4ca:::1001dd189d4.00000000:head : > > omap_digest 0x4f413584 != omap_digest 0xb832d010 from auth oi > > 6:e4d8a4ca:::1001dd189d4.00000000:head(203789'1036020 osd.3.0:3669132 > > dirty|omap|data_digest|omap_digest s 0 uv 1026527 dd ffffffff od > > b832d010 alloc_hint [0 0 0]) > > 2019-10-10 11:23:31.211 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4d8a4ca:::1001dd189d4.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:31.541 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4dae402:::20005f4471c.00000000:head : > > omap_digest 0x95196306 != omap_digest 0x78a1d6b0 from auth oi > > 6:e4dae402:::20005f4471c.00000000:head(203789'1036246 osd.3.0:3669358 > > dirty|omap|data_digest|omap_digest s 0 uv 1031360 dd ffffffff od > > 78a1d6b0 alloc_hint [0 0 0]) > > 2019-10-10 11:23:31.541 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4dae402:::20005f4471c.00000000:head : > > omap_digest 0x95196306 != omap_digest 0x78a1d6b0 from auth oi > > 6:e4dae402:::20005f4471c.00000000:head(203789'1036246 osd.3.0:3669358 > > dirty|omap|data_digest|omap_digest s 0 uv 1031360 dd ffffffff od > > 78a1d6b0 alloc_hint [0 0 0]) > > 2019-10-10 11:23:31.541 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4dae402:::20005f4471c.00000000:head : > > omap_digest 0x95196306 != omap_digest 0x78a1d6b0 from auth oi > > 6:e4dae402:::20005f4471c.00000000:head(203789'1036246 osd.3.0:3669358 > > dirty|omap|data_digest|omap_digest s 0 uv 1031360 dd ffffffff od > > 78a1d6b0 alloc_hint [0 0 0]) > > 2019-10-10 11:23:31.541 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4dae402:::20005f4471c.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:31.911 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4dd409f:::1001dd16f4c.00000000:head : > > omap_digest 0xc0d797a6 != omap_digest 0x28ee2bfb from auth oi > > 6:e4dd409f:::1001dd16f4c.00000000:head(203789'1036509 osd.3.0:3669621 > > dirty|omap|data_digest|omap_digest s 0 uv 1032170 dd ffffffff od > > 28ee2bfb alloc_hint [0 0 0]) > > 2019-10-10 11:23:31.911 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4dd409f:::1001dd16f4c.00000000:head : > > omap_digest 0xc0d797a6 != omap_digest 0x28ee2bfb from auth oi > > 6:e4dd409f:::1001dd16f4c.00000000:head(203789'1036509 osd.3.0:3669621 > > dirty|omap|data_digest|omap_digest s 0 uv 1032170 dd ffffffff od > > 28ee2bfb alloc_hint [0 0 0]) > > 2019-10-10 11:23:31.911 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4dd409f:::1001dd16f4c.00000000:head : > > omap_digest 0xc0d797a6 != omap_digest 0x28ee2bfb from auth oi > > 6:e4dd409f:::1001dd16f4c.00000000:head(203789'1036509 osd.3.0:3669621 > > dirty|omap|data_digest|omap_digest s 0 uv 1032170 dd ffffffff od > > 28ee2bfb alloc_hint [0 0 0]) > > 2019-10-10 11:23:31.911 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4dd409f:::1001dd16f4c.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:32.131 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4de91a6:::20005f631c7.00000000:head : > > omap_digest 0x9206e25b != omap_digest 0x4730506f from auth oi > > 6:e4de91a6:::20005f631c7.00000000:head(203789'1036643 osd.3.0:3669755 > > dirty|omap|data_digest|omap_digest s 0 uv 1032783 dd ffffffff od > > 4730506f alloc_hint [0 0 0]) > > 2019-10-10 11:23:32.131 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4de91a6:::20005f631c7.00000000:head : > > omap_digest 0x9206e25b != omap_digest 0x4730506f from auth oi > > 6:e4de91a6:::20005f631c7.00000000:head(203789'1036643 osd.3.0:3669755 > > dirty|omap|data_digest|omap_digest s 0 uv 1032783 dd ffffffff od > > 4730506f alloc_hint [0 0 0]) > > 2019-10-10 11:23:32.131 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4de91a6:::20005f631c7.00000000:head : > > omap_digest 0x9206e25b != omap_digest 0x4730506f from auth oi > > 6:e4de91a6:::20005f631c7.00000000:head(203789'1036643 osd.3.0:3669755 > > dirty|omap|data_digest|omap_digest s 0 uv 1032783 dd ffffffff od > > 4730506f alloc_hint [0 0 0]) > > 2019-10-10 11:23:32.131 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4de91a6:::20005f631c7.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:32.501 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4e106c1:::1001dd1895b.00000000:head : > > omap_digest 0x8d24d314 != omap_digest 0x6df1ff43 from auth oi > > 6:e4e106c1:::1001dd1895b.00000000:head(203789'1036895 osd.3.0:3670007 > > dirty|omap|data_digest|omap_digest s 0 uv 1026525 dd ffffffff od > > 6df1ff43 alloc_hint [0 0 0]) > > 2019-10-10 11:23:32.501 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4e106c1:::1001dd1895b.00000000:head : > > omap_digest 0x8d24d314 != omap_digest 0x6df1ff43 from auth oi > > 6:e4e106c1:::1001dd1895b.00000000:head(203789'1036895 osd.3.0:3670007 > > dirty|omap|data_digest|omap_digest s 0 uv 1026525 dd ffffffff od > > 6df1ff43 alloc_hint [0 0 0]) > > 2019-10-10 11:23:32.501 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4e106c1:::1001dd1895b.00000000:head : > > omap_digest 0x8d24d314 != omap_digest 0x6df1ff43 from auth oi > > 6:e4e106c1:::1001dd1895b.00000000:head(203789'1036895 osd.3.0:3670007 > > dirty|omap|data_digest|omap_digest s 0 uv 1026525 dd ffffffff od > > 6df1ff43 alloc_hint [0 0 0]) > > 2019-10-10 11:23:32.501 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4e106c1:::1001dd1895b.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4e837c3:::20005f33142.00000000:head : > > omap_digest 0x45cf8580 != omap_digest 0x7befa2c2 from auth oi > > 6:e4e837c3:::20005f33142.00000000:head(203789'1037600 osd.3.0:3670712 > > dirty|omap|data_digest|omap_digest s 0 uv 1032159 dd ffffffff od > > 7befa2c2 alloc_hint [0 0 0]) > > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4e837c3:::20005f33142.00000000:head : > > omap_digest 0x45cf8580 != omap_digest 0x7befa2c2 from auth oi > > 6:e4e837c3:::20005f33142.00000000:head(203789'1037600 osd.3.0:3670712 > > dirty|omap|data_digest|omap_digest s 0 uv 1032159 dd ffffffff od > > 7befa2c2 alloc_hint [0 0 0]) > > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4e837c3:::20005f33142.00000000:head : > > omap_digest 0x45cf8580 != omap_digest 0x7befa2c2 from auth oi > > 6:e4e837c3:::20005f33142.00000000:head(203789'1037600 osd.3.0:3670712 > > dirty|omap|data_digest|omap_digest s 0 uv 1032159 dd ffffffff od > > 7befa2c2 alloc_hint [0 0 0]) > > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4e837c3:::20005f33142.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4e842f7:::20005f33033.00000000:head : > > omap_digest 0x33a64598 != omap_digest 0x1a45390a from auth oi > > 6:e4e842f7:::20005f33033.00000000:head(203789'1037604 osd.3.0:3670716 > > dirty|omap|data_digest|omap_digest s 0 uv 1032161 dd ffffffff od > > 1a45390a alloc_hint [0 0 0]) > > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4e842f7:::20005f33033.00000000:head : > > omap_digest 0x33a64598 != omap_digest 0x1a45390a from auth oi > > 6:e4e842f7:::20005f33033.00000000:head(203789'1037604 osd.3.0:3670716 > > dirty|omap|data_digest|omap_digest s 0 uv 1032161 dd ffffffff od > > 1a45390a alloc_hint [0 0 0]) > > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4e842f7:::20005f33033.00000000:head : > > omap_digest 0x33a64598 != omap_digest 0x1a45390a from auth oi > > 6:e4e842f7:::20005f33033.00000000:head(203789'1037604 osd.3.0:3670716 > > dirty|omap|data_digest|omap_digest s 0 uv 1032161 dd ffffffff od > > 1a45390a alloc_hint [0 0 0]) > > 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4e842f7:::20005f33033.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:34.931 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4f15b8a:::20005f35cd7.00000000:head : > > omap_digest 0xd196c582 != omap_digest 0xd3449697 from auth oi > > 6:e4f15b8a:::20005f35cd7.00000000:head(203789'1038553 osd.3.0:3671665 > > dirty|omap|data_digest|omap_digest s 0 uv 1031357 dd ffffffff od > > d3449697 alloc_hint [0 0 0]) > > 2019-10-10 11:23:34.931 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4f15b8a:::20005f35cd7.00000000:head : > > omap_digest 0xd196c582 != omap_digest 0xd3449697 from auth oi > > 6:e4f15b8a:::20005f35cd7.00000000:head(203789'1038553 osd.3.0:3671665 > > dirty|omap|data_digest|omap_digest s 0 uv 1031357 dd ffffffff od > > d3449697 alloc_hint [0 0 0]) > > 2019-10-10 11:23:34.931 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4f15b8a:::20005f35cd7.00000000:head : > > omap_digest 0xd196c582 != omap_digest 0xd3449697 from auth oi > > 6:e4f15b8a:::20005f35cd7.00000000:head(203789'1038553 osd.3.0:3671665 > > dirty|omap|data_digest|omap_digest s 0 uv 1031357 dd ffffffff od > > d3449697 alloc_hint [0 0 0]) > > 2019-10-10 11:23:34.931 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4f15b8a:::20005f35cd7.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:36.051 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4f8770e:::20005f3d629.00000000:head : > > omap_digest 0x52a9267a != omap_digest 0xb2fbd549 from auth oi > > 6:e4f8770e:::20005f3d629.00000000:head(203789'1039311 osd.3.0:3672423 > > dirty|omap|data_digest|omap_digest s 0 uv 1032165 dd ffffffff od > > b2fbd549 alloc_hint [0 0 0]) > > 2019-10-10 11:23:36.051 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4f8770e:::20005f3d629.00000000:head : > > omap_digest 0x52a9267a != omap_digest 0xb2fbd549 from auth oi > > 6:e4f8770e:::20005f3d629.00000000:head(203789'1039311 osd.3.0:3672423 > > dirty|omap|data_digest|omap_digest s 0 uv 1032165 dd ffffffff od > > b2fbd549 alloc_hint [0 0 0]) > > 2019-10-10 11:23:36.051 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4f8770e:::20005f3d629.00000000:head : > > omap_digest 0x52a9267a != omap_digest 0xb2fbd549 from auth oi > > 6:e4f8770e:::20005f3d629.00000000:head(203789'1039311 osd.3.0:3672423 > > dirty|omap|data_digest|omap_digest s 0 uv 1032165 dd ffffffff od > > b2fbd549 alloc_hint [0 0 0]) > > 2019-10-10 11:23:36.051 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4f8770e:::20005f3d629.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:36.631 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4fc2d3e:::20005f570fb.00000000:head : > > omap_digest 0xeac7763 != omap_digest 0x8965ec0b from auth oi > > 6:e4fc2d3e:::20005f570fb.00000000:head(203789'1039698 osd.3.0:3672810 > > dirty|omap|data_digest|omap_digest s 0 uv 1032271 dd ffffffff od > > 8965ec0b alloc_hint [0 0 0]) > > 2019-10-10 11:23:36.631 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4fc2d3e:::20005f570fb.00000000:head : > > omap_digest 0xeac7763 != omap_digest 0x8965ec0b from auth oi > > 6:e4fc2d3e:::20005f570fb.00000000:head(203789'1039698 osd.3.0:3672810 > > dirty|omap|data_digest|omap_digest s 0 uv 1032271 dd ffffffff od > > 8965ec0b alloc_hint [0 0 0]) > > 2019-10-10 11:23:36.631 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4fc2d3e:::20005f570fb.00000000:head : > > omap_digest 0xeac7763 != omap_digest 0x8965ec0b from auth oi > > 6:e4fc2d3e:::20005f570fb.00000000:head(203789'1039698 osd.3.0:3672810 > > dirty|omap|data_digest|omap_digest s 0 uv 1032271 dd ffffffff od > > 8965ec0b alloc_hint [0 0 0]) > > 2019-10-10 11:23:36.631 7ff5509b8700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4fc2d3e:::20005f570fb.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:36.821 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4fdb194:::20005f3759c.00000000:head : > > omap_digest 0x1ef1fe3b != omap_digest 0x25f0dc6b from auth oi > > 6:e4fdb194:::20005f3759c.00000000:head(203789'1039877 osd.3.0:3672989 > > dirty|omap|data_digest|omap_digest s 0 uv 1032162 dd ffffffff od > > 25f0dc6b alloc_hint [0 0 0]) > > 2019-10-10 11:23:36.821 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4fdb194:::20005f3759c.00000000:head : > > omap_digest 0x1ef1fe3b != omap_digest 0x25f0dc6b from auth oi > > 6:e4fdb194:::20005f3759c.00000000:head(203789'1039877 osd.3.0:3672989 > > dirty|omap|data_digest|omap_digest s 0 uv 1032162 dd ffffffff od > > 25f0dc6b alloc_hint [0 0 0]) > > 2019-10-10 11:23:36.821 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4fdb194:::20005f3759c.00000000:head : > > omap_digest 0x1ef1fe3b != omap_digest 0x25f0dc6b from auth oi > > 6:e4fdb194:::20005f3759c.00000000:head(203789'1039877 osd.3.0:3672989 > > dirty|omap|data_digest|omap_digest s 0 uv 1032162 dd ffffffff od > > 25f0dc6b alloc_hint [0 0 0]) > > 2019-10-10 11:23:36.821 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4fdb194:::20005f3759c.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:36.901 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 19 soid 6:e4fe5018:::1001dd11fd4.00000000:head : > > omap_digest 0xc0244623 != omap_digest 0xbaed0414 from auth oi > > 6:e4fe5018:::1001dd11fd4.00000000:head(203789'1039928 osd.3.0:3673040 > > dirty|omap|data_digest|omap_digest s 0 uv 1031359 dd ffffffff od > > baed0414 alloc_hint [0 0 0]) > > 2019-10-10 11:23:36.901 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 72 soid 6:e4fe5018:::1001dd11fd4.00000000:head : > > omap_digest 0xc0244623 != omap_digest 0xbaed0414 from auth oi > > 6:e4fe5018:::1001dd11fd4.00000000:head(203789'1039928 osd.3.0:3673040 > > dirty|omap|data_digest|omap_digest s 0 uv 1031359 dd ffffffff od > > baed0414 alloc_hint [0 0 0]) > > 2019-10-10 11:23:36.901 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 shard 91 soid 6:e4fe5018:::1001dd11fd4.00000000:head : > > omap_digest 0xc0244623 != omap_digest 0xbaed0414 from auth oi > > 6:e4fe5018:::1001dd11fd4.00000000:head(203789'1039928 osd.3.0:3673040 > > dirty|omap|data_digest|omap_digest s 0 uv 1031359 dd ffffffff od > > baed0414 alloc_hint [0 0 0]) > > 2019-10-10 11:23:36.901 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 soid 6:e4fe5018:::1001dd11fd4.00000000:head : failed to pick > > suitable auth object > > 2019-10-10 11:23:37.131 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > > : 6.327 repair 63 errors, 0 fixed > > > > > The cluster info, we are on ceph 14.2.3. : > > [root@mds02 ~]# ceph health detail > > HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 large > > omap objects; 30967 scrub errors; Possible data damage: 219 pgs > > inconsistent > > FS_DEGRADED 1 filesystem is degraded > > fs ceph_fs is degraded > > MDS_DAMAGE 1 mds daemon damaged > > fs ceph_fs mds.0 is damaged > > LARGE_OMAP_OBJECTS 1 large omap objects > > 1 large objects found in pool 'metadata' > > Search the cluster log for 'Large omap object found' for more details. > > OSD_SCRUB_ERRORS 30967 scrub errors > > PG_DAMAGED Possible data damage: 219 pgs inconsistent > > pg 6.23f is active+clean+inconsistent+failed_repair, acting > > [18,55,541] > > pg 6.246 is active+clean+inconsistent+failed_repair, acting > > [19,108,486] > > pg 6.24d is active+clean+inconsistent+failed_repair, acting > > [109,91,442] > > pg 6.24e is active+clean+inconsistent+failed_repair, acting > > [468,36,444] > > pg 6.255 is active+clean+inconsistent+failed_repair, acting > > [37,513,442] > > pg 6.259 is active+clean+inconsistent+failed_repair, acting > > [486,18,37] > > pg 6.262 is active+clean+inconsistent+failed_repair, acting > > [449,108,90] > > pg 6.269 is active+clean+inconsistent+failed_repair, acting > > [449,442,37] > > pg 6.26a is active+clean+inconsistent+failed_repair, acting > > [36,513,449] > > pg 6.26b is active+clean+inconsistent+failed_repair, acting > > [138,486,72] > > pg 6.27a is active+clean+inconsistent+failed_repair, acting > > [520,515,468] > > pg 6.27c is active+clean+inconsistent+failed_repair, acting > > [138,37,469] > > pg 6.282 is active+clean+inconsistent+failed_repair, acting > > [37,54,109] > > pg 6.287 is active+clean+inconsistent+failed_repair, acting > > [72,468,540] > > pg 6.288 is active+clean+inconsistent+failed_repair, acting > > [541,140,449] > > pg 6.289 is active+clean+inconsistent+failed_repair, acting > > [73,486,449] > > pg 6.28e is active+clean+inconsistent+failed_repair, acting > > [449,36,109] > > pg 6.293 is active+clean+inconsistent+failed_repair, acting > > [447,522,540] > > pg 6.294 is active+clean+inconsistent+failed_repair, acting > > [513,444,37] > > > [root@mds02 ~]# ceph -s > > cluster: > > id: 92bfcf0a-1d39-43b3-b60f-44f01b630e47 > > health: HEALTH_ERR > > 1 filesystem is degraded > > 1 mds daemon damaged > > 1 large omap objects > > 30967 scrub errors > > Possible data damage: 219 pgs inconsistent > > > > services: > > mon: 3 daemons, quorum mds01,mds02,mds03 (age 2d) > > mgr: mds02(active, since 3w), standbys: mds01, mds03 > > mds: ceph_fs:1/2 {1=mds03=up:resolve} 2 up:standby, 1 damaged > > osd: 535 osds: 533 up, 529 in > > > > data: > > pools: 3 pools, 3328 pgs > > objects: 376.32M objects, 673 TiB > > usage: 1.0 PiB used, 2.2 PiB / 3.2 PiB avail > > pgs: 3097 active+clean > > 160 active+clean+inconsistent > > 59 active+clean+inconsistent > > 9 active+clean+scrubbing+deep > > 3 active+clean+scrubbing > > > > Does somebody knows how to fix this? > > Thanks! > > > Kenneth > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Cheers, Brad _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |