On Fri, Oct 11, 2019 at 12:27 AM Kenneth Waegeman <kenneth.waegeman@xxxxxxxx> wrote: > > Hi Brad, all, > > Pool 6 has min_size 2: > > pool 6 'metadata' replicated size 3 min_size 2 crush_rule 1 object_hash > rjenkins pg_num 1024 pgp_num 1024 autoscale_mode warn last_change 172476 > flags hashpspool stripe_width 0 application cephfs This looked like something min_size 1 could cause, but I guess that's not the cause here. > so inconsistens is empty, which is weird, no ? Try scrubbing the pg just before running the command. > > Thanks again! > > K > > > On 10/10/2019 12:52, Brad Hubbard wrote: > > Does pool 6 have min_size = 1 set? > > > > https://tracker.ceph.com/issues/24994#note-5 would possibly be helpful > > here, depending on what the output of the following command looks > > like. > > > > # rados list-inconsistent-obj [pgid] --format=json-pretty > > > > On Thu, Oct 10, 2019 at 8:16 PM Kenneth Waegeman > > <kenneth.waegeman@xxxxxxxx> wrote: > >> Hi all, > >> > >> After some node failure and rebalancing, we have a lot of pg's in > >> inconsistent state. I tried to repair, but it din't work. This is also > >> in the logs: > >> > >>> 2019-10-10 11:23:27.221 7ff54c9b0700 0 log_channel(cluster) log [DBG] > >>> : 6.327 repair starts > >>> 2019-10-10 11:23:27.431 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4c130fd:::20005f3b582.00000000:head : > >>> omap_digest 0x334f57be != omap_digest 0xa8c4ce76 from auth oi > >>> 6:e4c130fd:::20005f3b582.00000000:head(203789'1033530 osd.3.0:3666642 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032164 dd ffffffff od > >>> a8c4ce76 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:27.431 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4c130fd:::20005f3b582.00000000:head : > >>> omap_digest 0x334f57be != omap_digest 0xa8c4ce76 from auth oi > >>> 6:e4c130fd:::20005f3b582.00000000:head(203789'1033530 osd.3.0:3666642 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032164 dd ffffffff od > >>> a8c4ce76 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:27.431 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4c130fd:::20005f3b582.00000000:head : > >>> omap_digest 0x334f57be != omap_digest 0xa8c4ce76 from auth oi > >>> 6:e4c130fd:::20005f3b582.00000000:head(203789'1033530 osd.3.0:3666642 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032164 dd ffffffff od > >>> a8c4ce76 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:27.431 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4c130fd:::20005f3b582.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:27.731 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4c2e57b:::20005f11daa.00000000:head : > >>> omap_digest 0x6aafaf97 != omap_digest 0x56dd55a2 from auth oi > >>> 6:e4c2e57b:::20005f11daa.00000000:head(203789'1033711 osd.3.0:3666823 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032158 dd ffffffff od > >>> 56dd55a2 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:27.731 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4c2e57b:::20005f11daa.00000000:head : > >>> omap_digest 0x6aafaf97 != omap_digest 0x56dd55a2 from auth oi > >>> 6:e4c2e57b:::20005f11daa.00000000:head(203789'1033711 osd.3.0:3666823 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032158 dd ffffffff od > >>> 56dd55a2 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:27.731 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4c2e57b:::20005f11daa.00000000:head : > >>> omap_digest 0x6aafaf97 != omap_digest 0x56dd55a2 from auth oi > >>> 6:e4c2e57b:::20005f11daa.00000000:head(203789'1033711 osd.3.0:3666823 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032158 dd ffffffff od > >>> 56dd55a2 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:27.731 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4c2e57b:::20005f11daa.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:27.971 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4c40009:::20005f45f1b.00000000:head : > >>> omap_digest 0x7ccf5cc9 != omap_digest 0xe048d29 from auth oi > >>> 6:e4c40009:::20005f45f1b.00000000:head(203789'1033837 osd.3.0:3666949 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032168 dd ffffffff od > >>> e048d29 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:27.971 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4c40009:::20005f45f1b.00000000:head : > >>> omap_digest 0x7ccf5cc9 != omap_digest 0xe048d29 from auth oi > >>> 6:e4c40009:::20005f45f1b.00000000:head(203789'1033837 osd.3.0:3666949 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032168 dd ffffffff od > >>> e048d29 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:27.971 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4c40009:::20005f45f1b.00000000:head : > >>> omap_digest 0x7ccf5cc9 != omap_digest 0xe048d29 from auth oi > >>> 6:e4c40009:::20005f45f1b.00000000:head(203789'1033837 osd.3.0:3666949 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032168 dd ffffffff od > >>> e048d29 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:27.971 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4c40009:::20005f45f1b.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:28.041 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4c4a042:::20005f389fb.00000000:head : > >>> omap_digest 0xdd1558b8 != omap_digest 0xcf9af548 from auth oi > >>> 6:e4c4a042:::20005f389fb.00000000:head(203789'1033899 osd.3.0:3667011 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1031358 dd ffffffff od > >>> cf9af548 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:28.041 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4c4a042:::20005f389fb.00000000:head : > >>> omap_digest 0xdd1558b8 != omap_digest 0xcf9af548 from auth oi > >>> 6:e4c4a042:::20005f389fb.00000000:head(203789'1033899 osd.3.0:3667011 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1031358 dd ffffffff od > >>> cf9af548 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:28.041 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4c4a042:::20005f389fb.00000000:head : > >>> omap_digest 0xdd1558b8 != omap_digest 0xcf9af548 from auth oi > >>> 6:e4c4a042:::20005f389fb.00000000:head(203789'1033899 osd.3.0:3667011 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1031358 dd ffffffff od > >>> cf9af548 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:28.041 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4c4a042:::20005f389fb.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:28.791 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4c91caa:::20005f36925.00000000:head : > >>> omap_digest 0xbdd9c6d8 != omap_digest 0xd207a82c from auth oi > >>> 6:e4c91caa:::20005f36925.00000000:head(203789'1034361 osd.3.0:3667473 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032163 dd ffffffff od > >>> d207a82c alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:28.791 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4c91caa:::20005f36925.00000000:head : > >>> omap_digest 0xbdd9c6d8 != omap_digest 0xd207a82c from auth oi > >>> 6:e4c91caa:::20005f36925.00000000:head(203789'1034361 osd.3.0:3667473 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032163 dd ffffffff od > >>> d207a82c alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:28.791 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4c91caa:::20005f36925.00000000:head : > >>> omap_digest 0xbdd9c6d8 != omap_digest 0xd207a82c from auth oi > >>> 6:e4c91caa:::20005f36925.00000000:head(203789'1034361 osd.3.0:3667473 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032163 dd ffffffff od > >>> d207a82c alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:28.791 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4c91caa:::20005f36925.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:28.961 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4ca4cc7:::20005f41d80.00000000:head : > >>> omap_digest 0x66bea9d5 != omap_digest 0xa296af17 from auth oi > >>> 6:e4ca4cc7:::20005f41d80.00000000:head(203789'1034508 osd.3.0:3667620 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032167 dd ffffffff od > >>> a296af17 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:28.961 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4ca4cc7:::20005f41d80.00000000:head : > >>> omap_digest 0x66bea9d5 != omap_digest 0xa296af17 from auth oi > >>> 6:e4ca4cc7:::20005f41d80.00000000:head(203789'1034508 osd.3.0:3667620 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032167 dd ffffffff od > >>> a296af17 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:28.961 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4ca4cc7:::20005f41d80.00000000:head : > >>> omap_digest 0x66bea9d5 != omap_digest 0xa296af17 from auth oi > >>> 6:e4ca4cc7:::20005f41d80.00000000:head(203789'1034508 osd.3.0:3667620 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032167 dd ffffffff od > >>> a296af17 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:28.961 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4ca4cc7:::20005f41d80.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:30.001 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4d16e2c:::1001dd18807.00000000:head : > >>> omap_digest 0x8b67426c != omap_digest 0x265813d7 from auth oi > >>> 6:e4d16e2c:::1001dd18807.00000000:head(203789'1035272 osd.3.0:3668384 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032169 dd ffffffff od > >>> 265813d7 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:30.001 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4d16e2c:::1001dd18807.00000000:head : > >>> omap_digest 0x8b67426c != omap_digest 0x265813d7 from auth oi > >>> 6:e4d16e2c:::1001dd18807.00000000:head(203789'1035272 osd.3.0:3668384 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032169 dd ffffffff od > >>> 265813d7 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:30.001 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4d16e2c:::1001dd18807.00000000:head : > >>> omap_digest 0x8b67426c != omap_digest 0x265813d7 from auth oi > >>> 6:e4d16e2c:::1001dd18807.00000000:head(203789'1035272 osd.3.0:3668384 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032169 dd ffffffff od > >>> 265813d7 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:30.001 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4d16e2c:::1001dd18807.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:30.181 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4d29b3d:::20005f332fb.00000000:head : > >>> omap_digest 0xa6a8b00d != omap_digest 0x3f609d1d from auth oi > >>> 6:e4d29b3d:::20005f332fb.00000000:head(203789'1035413 osd.3.0:3668525 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032160 dd ffffffff od > >>> 3f609d1d alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:30.181 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4d29b3d:::20005f332fb.00000000:head : > >>> omap_digest 0xa6a8b00d != omap_digest 0x3f609d1d from auth oi > >>> 6:e4d29b3d:::20005f332fb.00000000:head(203789'1035413 osd.3.0:3668525 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032160 dd ffffffff od > >>> 3f609d1d alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:30.181 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4d29b3d:::20005f332fb.00000000:head : > >>> omap_digest 0xa6a8b00d != omap_digest 0x3f609d1d from auth oi > >>> 6:e4d29b3d:::20005f332fb.00000000:head(203789'1035413 osd.3.0:3668525 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032160 dd ffffffff od > >>> 3f609d1d alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:30.181 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4d29b3d:::20005f332fb.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:30.551 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4d436d4:::20005f43997.00000000:head : > >>> omap_digest 0xed0e46f != omap_digest 0xe480cd9a from auth oi > >>> 6:e4d436d4:::20005f43997.00000000:head(203789'1035572 osd.3.0:3668684 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032166 dd ffffffff od > >>> e480cd9a alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:30.551 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4d436d4:::20005f43997.00000000:head : > >>> omap_digest 0xed0e46f != omap_digest 0xe480cd9a from auth oi > >>> 6:e4d436d4:::20005f43997.00000000:head(203789'1035572 osd.3.0:3668684 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032166 dd ffffffff od > >>> e480cd9a alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:30.551 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4d436d4:::20005f43997.00000000:head : > >>> omap_digest 0xed0e46f != omap_digest 0xe480cd9a from auth oi > >>> 6:e4d436d4:::20005f43997.00000000:head(203789'1035572 osd.3.0:3668684 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032166 dd ffffffff od > >>> e480cd9a alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:30.551 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4d436d4:::20005f43997.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:31.211 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4d8a4ca:::1001dd189d4.00000000:head : > >>> omap_digest 0x4f413584 != omap_digest 0xb832d010 from auth oi > >>> 6:e4d8a4ca:::1001dd189d4.00000000:head(203789'1036020 osd.3.0:3669132 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1026527 dd ffffffff od > >>> b832d010 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:31.211 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4d8a4ca:::1001dd189d4.00000000:head : > >>> omap_digest 0x4f413584 != omap_digest 0xb832d010 from auth oi > >>> 6:e4d8a4ca:::1001dd189d4.00000000:head(203789'1036020 osd.3.0:3669132 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1026527 dd ffffffff od > >>> b832d010 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:31.211 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4d8a4ca:::1001dd189d4.00000000:head : > >>> omap_digest 0x4f413584 != omap_digest 0xb832d010 from auth oi > >>> 6:e4d8a4ca:::1001dd189d4.00000000:head(203789'1036020 osd.3.0:3669132 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1026527 dd ffffffff od > >>> b832d010 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:31.211 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4d8a4ca:::1001dd189d4.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:31.541 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4dae402:::20005f4471c.00000000:head : > >>> omap_digest 0x95196306 != omap_digest 0x78a1d6b0 from auth oi > >>> 6:e4dae402:::20005f4471c.00000000:head(203789'1036246 osd.3.0:3669358 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1031360 dd ffffffff od > >>> 78a1d6b0 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:31.541 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4dae402:::20005f4471c.00000000:head : > >>> omap_digest 0x95196306 != omap_digest 0x78a1d6b0 from auth oi > >>> 6:e4dae402:::20005f4471c.00000000:head(203789'1036246 osd.3.0:3669358 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1031360 dd ffffffff od > >>> 78a1d6b0 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:31.541 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4dae402:::20005f4471c.00000000:head : > >>> omap_digest 0x95196306 != omap_digest 0x78a1d6b0 from auth oi > >>> 6:e4dae402:::20005f4471c.00000000:head(203789'1036246 osd.3.0:3669358 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1031360 dd ffffffff od > >>> 78a1d6b0 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:31.541 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4dae402:::20005f4471c.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:31.911 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4dd409f:::1001dd16f4c.00000000:head : > >>> omap_digest 0xc0d797a6 != omap_digest 0x28ee2bfb from auth oi > >>> 6:e4dd409f:::1001dd16f4c.00000000:head(203789'1036509 osd.3.0:3669621 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032170 dd ffffffff od > >>> 28ee2bfb alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:31.911 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4dd409f:::1001dd16f4c.00000000:head : > >>> omap_digest 0xc0d797a6 != omap_digest 0x28ee2bfb from auth oi > >>> 6:e4dd409f:::1001dd16f4c.00000000:head(203789'1036509 osd.3.0:3669621 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032170 dd ffffffff od > >>> 28ee2bfb alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:31.911 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4dd409f:::1001dd16f4c.00000000:head : > >>> omap_digest 0xc0d797a6 != omap_digest 0x28ee2bfb from auth oi > >>> 6:e4dd409f:::1001dd16f4c.00000000:head(203789'1036509 osd.3.0:3669621 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032170 dd ffffffff od > >>> 28ee2bfb alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:31.911 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4dd409f:::1001dd16f4c.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:32.131 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4de91a6:::20005f631c7.00000000:head : > >>> omap_digest 0x9206e25b != omap_digest 0x4730506f from auth oi > >>> 6:e4de91a6:::20005f631c7.00000000:head(203789'1036643 osd.3.0:3669755 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032783 dd ffffffff od > >>> 4730506f alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:32.131 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4de91a6:::20005f631c7.00000000:head : > >>> omap_digest 0x9206e25b != omap_digest 0x4730506f from auth oi > >>> 6:e4de91a6:::20005f631c7.00000000:head(203789'1036643 osd.3.0:3669755 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032783 dd ffffffff od > >>> 4730506f alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:32.131 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4de91a6:::20005f631c7.00000000:head : > >>> omap_digest 0x9206e25b != omap_digest 0x4730506f from auth oi > >>> 6:e4de91a6:::20005f631c7.00000000:head(203789'1036643 osd.3.0:3669755 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032783 dd ffffffff od > >>> 4730506f alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:32.131 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4de91a6:::20005f631c7.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:32.501 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4e106c1:::1001dd1895b.00000000:head : > >>> omap_digest 0x8d24d314 != omap_digest 0x6df1ff43 from auth oi > >>> 6:e4e106c1:::1001dd1895b.00000000:head(203789'1036895 osd.3.0:3670007 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1026525 dd ffffffff od > >>> 6df1ff43 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:32.501 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4e106c1:::1001dd1895b.00000000:head : > >>> omap_digest 0x8d24d314 != omap_digest 0x6df1ff43 from auth oi > >>> 6:e4e106c1:::1001dd1895b.00000000:head(203789'1036895 osd.3.0:3670007 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1026525 dd ffffffff od > >>> 6df1ff43 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:32.501 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4e106c1:::1001dd1895b.00000000:head : > >>> omap_digest 0x8d24d314 != omap_digest 0x6df1ff43 from auth oi > >>> 6:e4e106c1:::1001dd1895b.00000000:head(203789'1036895 osd.3.0:3670007 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1026525 dd ffffffff od > >>> 6df1ff43 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:32.501 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4e106c1:::1001dd1895b.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4e837c3:::20005f33142.00000000:head : > >>> omap_digest 0x45cf8580 != omap_digest 0x7befa2c2 from auth oi > >>> 6:e4e837c3:::20005f33142.00000000:head(203789'1037600 osd.3.0:3670712 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032159 dd ffffffff od > >>> 7befa2c2 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4e837c3:::20005f33142.00000000:head : > >>> omap_digest 0x45cf8580 != omap_digest 0x7befa2c2 from auth oi > >>> 6:e4e837c3:::20005f33142.00000000:head(203789'1037600 osd.3.0:3670712 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032159 dd ffffffff od > >>> 7befa2c2 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4e837c3:::20005f33142.00000000:head : > >>> omap_digest 0x45cf8580 != omap_digest 0x7befa2c2 from auth oi > >>> 6:e4e837c3:::20005f33142.00000000:head(203789'1037600 osd.3.0:3670712 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032159 dd ffffffff od > >>> 7befa2c2 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4e837c3:::20005f33142.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4e842f7:::20005f33033.00000000:head : > >>> omap_digest 0x33a64598 != omap_digest 0x1a45390a from auth oi > >>> 6:e4e842f7:::20005f33033.00000000:head(203789'1037604 osd.3.0:3670716 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032161 dd ffffffff od > >>> 1a45390a alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4e842f7:::20005f33033.00000000:head : > >>> omap_digest 0x33a64598 != omap_digest 0x1a45390a from auth oi > >>> 6:e4e842f7:::20005f33033.00000000:head(203789'1037604 osd.3.0:3670716 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032161 dd ffffffff od > >>> 1a45390a alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4e842f7:::20005f33033.00000000:head : > >>> omap_digest 0x33a64598 != omap_digest 0x1a45390a from auth oi > >>> 6:e4e842f7:::20005f33033.00000000:head(203789'1037604 osd.3.0:3670716 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032161 dd ffffffff od > >>> 1a45390a alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4e842f7:::20005f33033.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:34.931 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4f15b8a:::20005f35cd7.00000000:head : > >>> omap_digest 0xd196c582 != omap_digest 0xd3449697 from auth oi > >>> 6:e4f15b8a:::20005f35cd7.00000000:head(203789'1038553 osd.3.0:3671665 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1031357 dd ffffffff od > >>> d3449697 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:34.931 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4f15b8a:::20005f35cd7.00000000:head : > >>> omap_digest 0xd196c582 != omap_digest 0xd3449697 from auth oi > >>> 6:e4f15b8a:::20005f35cd7.00000000:head(203789'1038553 osd.3.0:3671665 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1031357 dd ffffffff od > >>> d3449697 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:34.931 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4f15b8a:::20005f35cd7.00000000:head : > >>> omap_digest 0xd196c582 != omap_digest 0xd3449697 from auth oi > >>> 6:e4f15b8a:::20005f35cd7.00000000:head(203789'1038553 osd.3.0:3671665 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1031357 dd ffffffff od > >>> d3449697 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:34.931 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4f15b8a:::20005f35cd7.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:36.051 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4f8770e:::20005f3d629.00000000:head : > >>> omap_digest 0x52a9267a != omap_digest 0xb2fbd549 from auth oi > >>> 6:e4f8770e:::20005f3d629.00000000:head(203789'1039311 osd.3.0:3672423 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032165 dd ffffffff od > >>> b2fbd549 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:36.051 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4f8770e:::20005f3d629.00000000:head : > >>> omap_digest 0x52a9267a != omap_digest 0xb2fbd549 from auth oi > >>> 6:e4f8770e:::20005f3d629.00000000:head(203789'1039311 osd.3.0:3672423 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032165 dd ffffffff od > >>> b2fbd549 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:36.051 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4f8770e:::20005f3d629.00000000:head : > >>> omap_digest 0x52a9267a != omap_digest 0xb2fbd549 from auth oi > >>> 6:e4f8770e:::20005f3d629.00000000:head(203789'1039311 osd.3.0:3672423 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032165 dd ffffffff od > >>> b2fbd549 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:36.051 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4f8770e:::20005f3d629.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:36.631 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4fc2d3e:::20005f570fb.00000000:head : > >>> omap_digest 0xeac7763 != omap_digest 0x8965ec0b from auth oi > >>> 6:e4fc2d3e:::20005f570fb.00000000:head(203789'1039698 osd.3.0:3672810 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032271 dd ffffffff od > >>> 8965ec0b alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:36.631 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4fc2d3e:::20005f570fb.00000000:head : > >>> omap_digest 0xeac7763 != omap_digest 0x8965ec0b from auth oi > >>> 6:e4fc2d3e:::20005f570fb.00000000:head(203789'1039698 osd.3.0:3672810 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032271 dd ffffffff od > >>> 8965ec0b alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:36.631 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4fc2d3e:::20005f570fb.00000000:head : > >>> omap_digest 0xeac7763 != omap_digest 0x8965ec0b from auth oi > >>> 6:e4fc2d3e:::20005f570fb.00000000:head(203789'1039698 osd.3.0:3672810 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032271 dd ffffffff od > >>> 8965ec0b alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:36.631 7ff5509b8700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4fc2d3e:::20005f570fb.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:36.821 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4fdb194:::20005f3759c.00000000:head : > >>> omap_digest 0x1ef1fe3b != omap_digest 0x25f0dc6b from auth oi > >>> 6:e4fdb194:::20005f3759c.00000000:head(203789'1039877 osd.3.0:3672989 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032162 dd ffffffff od > >>> 25f0dc6b alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:36.821 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4fdb194:::20005f3759c.00000000:head : > >>> omap_digest 0x1ef1fe3b != omap_digest 0x25f0dc6b from auth oi > >>> 6:e4fdb194:::20005f3759c.00000000:head(203789'1039877 osd.3.0:3672989 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032162 dd ffffffff od > >>> 25f0dc6b alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:36.821 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4fdb194:::20005f3759c.00000000:head : > >>> omap_digest 0x1ef1fe3b != omap_digest 0x25f0dc6b from auth oi > >>> 6:e4fdb194:::20005f3759c.00000000:head(203789'1039877 osd.3.0:3672989 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1032162 dd ffffffff od > >>> 25f0dc6b alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:36.821 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4fdb194:::20005f3759c.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:36.901 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 19 soid 6:e4fe5018:::1001dd11fd4.00000000:head : > >>> omap_digest 0xc0244623 != omap_digest 0xbaed0414 from auth oi > >>> 6:e4fe5018:::1001dd11fd4.00000000:head(203789'1039928 osd.3.0:3673040 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1031359 dd ffffffff od > >>> baed0414 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:36.901 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 72 soid 6:e4fe5018:::1001dd11fd4.00000000:head : > >>> omap_digest 0xc0244623 != omap_digest 0xbaed0414 from auth oi > >>> 6:e4fe5018:::1001dd11fd4.00000000:head(203789'1039928 osd.3.0:3673040 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1031359 dd ffffffff od > >>> baed0414 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:36.901 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 shard 91 soid 6:e4fe5018:::1001dd11fd4.00000000:head : > >>> omap_digest 0xc0244623 != omap_digest 0xbaed0414 from auth oi > >>> 6:e4fe5018:::1001dd11fd4.00000000:head(203789'1039928 osd.3.0:3673040 > >>> dirty|omap|data_digest|omap_digest s 0 uv 1031359 dd ffffffff od > >>> baed0414 alloc_hint [0 0 0]) > >>> 2019-10-10 11:23:36.901 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 soid 6:e4fe5018:::1001dd11fd4.00000000:head : failed to pick > >>> suitable auth object > >>> 2019-10-10 11:23:37.131 7ff54c9b0700 -1 log_channel(cluster) log [ERR] > >>> : 6.327 repair 63 errors, 0 fixed > >>> > >>> > >> The cluster info, we are on ceph 14.2.3. : > >>> [root@mds02 ~]# ceph health detail > >>> HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 large > >>> omap objects; 30967 scrub errors; Possible data damage: 219 pgs > >>> inconsistent > >>> FS_DEGRADED 1 filesystem is degraded > >>> fs ceph_fs is degraded > >>> MDS_DAMAGE 1 mds daemon damaged > >>> fs ceph_fs mds.0 is damaged > >>> LARGE_OMAP_OBJECTS 1 large omap objects > >>> 1 large objects found in pool 'metadata' > >>> Search the cluster log for 'Large omap object found' for more details. > >>> OSD_SCRUB_ERRORS 30967 scrub errors > >>> PG_DAMAGED Possible data damage: 219 pgs inconsistent > >>> pg 6.23f is active+clean+inconsistent+failed_repair, acting > >>> [18,55,541] > >>> pg 6.246 is active+clean+inconsistent+failed_repair, acting > >>> [19,108,486] > >>> pg 6.24d is active+clean+inconsistent+failed_repair, acting > >>> [109,91,442] > >>> pg 6.24e is active+clean+inconsistent+failed_repair, acting > >>> [468,36,444] > >>> pg 6.255 is active+clean+inconsistent+failed_repair, acting > >>> [37,513,442] > >>> pg 6.259 is active+clean+inconsistent+failed_repair, acting > >>> [486,18,37] > >>> pg 6.262 is active+clean+inconsistent+failed_repair, acting > >>> [449,108,90] > >>> pg 6.269 is active+clean+inconsistent+failed_repair, acting > >>> [449,442,37] > >>> pg 6.26a is active+clean+inconsistent+failed_repair, acting > >>> [36,513,449] > >>> pg 6.26b is active+clean+inconsistent+failed_repair, acting > >>> [138,486,72] > >>> pg 6.27a is active+clean+inconsistent+failed_repair, acting > >>> [520,515,468] > >>> pg 6.27c is active+clean+inconsistent+failed_repair, acting > >>> [138,37,469] > >>> pg 6.282 is active+clean+inconsistent+failed_repair, acting > >>> [37,54,109] > >>> pg 6.287 is active+clean+inconsistent+failed_repair, acting > >>> [72,468,540] > >>> pg 6.288 is active+clean+inconsistent+failed_repair, acting > >>> [541,140,449] > >>> pg 6.289 is active+clean+inconsistent+failed_repair, acting > >>> [73,486,449] > >>> pg 6.28e is active+clean+inconsistent+failed_repair, acting > >>> [449,36,109] > >>> pg 6.293 is active+clean+inconsistent+failed_repair, acting > >>> [447,522,540] > >>> pg 6.294 is active+clean+inconsistent+failed_repair, acting > >>> [513,444,37] > >>> [root@mds02 ~]# ceph -s > >>> cluster: > >>> id: 92bfcf0a-1d39-43b3-b60f-44f01b630e47 > >>> health: HEALTH_ERR > >>> 1 filesystem is degraded > >>> 1 mds daemon damaged > >>> 1 large omap objects > >>> 30967 scrub errors > >>> Possible data damage: 219 pgs inconsistent > >>> > >>> services: > >>> mon: 3 daemons, quorum mds01,mds02,mds03 (age 2d) > >>> mgr: mds02(active, since 3w), standbys: mds01, mds03 > >>> mds: ceph_fs:1/2 {1=mds03=up:resolve} 2 up:standby, 1 damaged > >>> osd: 535 osds: 533 up, 529 in > >>> > >>> data: > >>> pools: 3 pools, 3328 pgs > >>> objects: 376.32M objects, 673 TiB > >>> usage: 1.0 PiB used, 2.2 PiB / 3.2 PiB avail > >>> pgs: 3097 active+clean > >>> 160 active+clean+inconsistent > >>> 59 active+clean+inconsistent > >>> 9 active+clean+scrubbing+deep > >>> 3 active+clean+scrubbing > >>> > >> Does somebody knows how to fix this? > >> > >> Thanks! > >> > >> > >> Kenneth > >> > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@xxxxxxxxxxxxxx > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- Cheers, Brad _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |