Hi Brad, all,
Pool 6 has min_size 2:
pool 6 'metadata' replicated size 3 min_size 2 crush_rule 1 object_hash
rjenkins pg_num 1024 pgp_num 1024 autoscale_mode warn last_change 172476
flags hashpspool stripe_width 0 application cephfs
The output for all the inconsistent pgs is this:
{
"epoch": 207682,
"inconsistents": []
}
{
"epoch": 207627,
"inconsistents": []
}
{
"epoch": 207700,
"inconsistents": []
}
{
"epoch": 207720,
"inconsistents": []
}
{
"epoch": 207657,
"inconsistents": []
}
{
"epoch": 207652,
"inconsistents": []
}
{
"epoch": 207750,
"inconsistents": []
}
{
"epoch": 208021,
"inconsistents": []
}
{
"epoch": 207726,
"inconsistents": []
}
{
"epoch": 207645,
"inconsistents": []
}
{
"epoch": 207347,
"inconsistents": []
}
{
"epoch": 207649,
"inconsistents": []
}
{
"epoch": 207727,
"inconsistents": []
}
{
"epoch": 207676,
"inconsistents": []
}
{
"epoch": 207373,
"inconsistents": []
}
{
"epoch": 207736,
"inconsistents": []
}
{
"epoch": 207641,
"inconsistents": []
}
{
"epoch": 207750,
"inconsistents": []
}
{
"epoch": 207573,
"inconsistents": []
}
{
"epoch": 207658,
"inconsistents": []
}
{
"epoch": 207616,
"inconsistents": []
}
{
"epoch": 207387,
"inconsistents": []
}
{
"epoch": 207991,
"inconsistents": []
}
{
"epoch": 207648,
"inconsistents": []
}
{
"epoch": 207614,
"inconsistents": []
}
{
"epoch": 207287,
"inconsistents": []
}
{
"epoch": 207663,
"inconsistents": []
}
{
"epoch": 207643,
"inconsistents": []
}
{
"epoch": 207701,
"inconsistents": []
}
{
"epoch": 207693,
"inconsistents": []
}
{
"epoch": 207632,
"inconsistents": []
}
{
"epoch": 207389,
"inconsistents": []
}
{
"epoch": 207692,
"inconsistents": []
}
{
"epoch": 207634,
"inconsistents": []
}
{
"epoch": 207309,
"inconsistents": []
}
{
"epoch": 207651,
"inconsistents": []
}
{
"epoch": 207643,
"inconsistents": []
}
{
"epoch": 207656,
"inconsistents": []
}
{
"epoch": 207729,
"inconsistents": []
}
{
"epoch": 207196,
"inconsistents": []
}
{
"epoch": 207626,
"inconsistents": []
}
{
"epoch": 207432,
"inconsistents": []
}
{
"epoch": 207652,
"inconsistents": []
}
{
"epoch": 207427,
"inconsistents": []
}
{
"epoch": 207676,
"inconsistents": []
}
{
"epoch": 207624,
"inconsistents": []
}
{
"epoch": 207658,
"inconsistents": []
}
{
"epoch": 207628,
"inconsistents": []
}
{
"epoch": 207546,
"inconsistents": []
}
{
"epoch": 207655,
"inconsistents": []
}
{
"epoch": 207602,
"inconsistents": []
}
so inconsistens is empty, which is weird, no ?
Thanks again!
K
On 10/10/2019 12:52, Brad Hubbard wrote:
Does pool 6 have min_size = 1 set?
https://tracker.ceph.com/issues/24994#note-5 would possibly be helpful
here, depending on what the output of the following command looks
like.
# rados list-inconsistent-obj [pgid] --format=json-pretty
On Thu, Oct 10, 2019 at 8:16 PM Kenneth Waegeman
<kenneth.waegeman@xxxxxxxx> wrote:
Hi all,
After some node failure and rebalancing, we have a lot of pg's in
inconsistent state. I tried to repair, but it din't work. This is also
in the logs:
2019-10-10 11:23:27.221 7ff54c9b0700 0 log_channel(cluster) log [DBG]
: 6.327 repair starts
2019-10-10 11:23:27.431 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4c130fd:::20005f3b582.00000000:head :
omap_digest 0x334f57be != omap_digest 0xa8c4ce76 from auth oi
6:e4c130fd:::20005f3b582.00000000:head(203789'1033530 osd.3.0:3666642
dirty|omap|data_digest|omap_digest s 0 uv 1032164 dd ffffffff od
a8c4ce76 alloc_hint [0 0 0])
2019-10-10 11:23:27.431 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4c130fd:::20005f3b582.00000000:head :
omap_digest 0x334f57be != omap_digest 0xa8c4ce76 from auth oi
6:e4c130fd:::20005f3b582.00000000:head(203789'1033530 osd.3.0:3666642
dirty|omap|data_digest|omap_digest s 0 uv 1032164 dd ffffffff od
a8c4ce76 alloc_hint [0 0 0])
2019-10-10 11:23:27.431 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4c130fd:::20005f3b582.00000000:head :
omap_digest 0x334f57be != omap_digest 0xa8c4ce76 from auth oi
6:e4c130fd:::20005f3b582.00000000:head(203789'1033530 osd.3.0:3666642
dirty|omap|data_digest|omap_digest s 0 uv 1032164 dd ffffffff od
a8c4ce76 alloc_hint [0 0 0])
2019-10-10 11:23:27.431 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4c130fd:::20005f3b582.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:27.731 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4c2e57b:::20005f11daa.00000000:head :
omap_digest 0x6aafaf97 != omap_digest 0x56dd55a2 from auth oi
6:e4c2e57b:::20005f11daa.00000000:head(203789'1033711 osd.3.0:3666823
dirty|omap|data_digest|omap_digest s 0 uv 1032158 dd ffffffff od
56dd55a2 alloc_hint [0 0 0])
2019-10-10 11:23:27.731 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4c2e57b:::20005f11daa.00000000:head :
omap_digest 0x6aafaf97 != omap_digest 0x56dd55a2 from auth oi
6:e4c2e57b:::20005f11daa.00000000:head(203789'1033711 osd.3.0:3666823
dirty|omap|data_digest|omap_digest s 0 uv 1032158 dd ffffffff od
56dd55a2 alloc_hint [0 0 0])
2019-10-10 11:23:27.731 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4c2e57b:::20005f11daa.00000000:head :
omap_digest 0x6aafaf97 != omap_digest 0x56dd55a2 from auth oi
6:e4c2e57b:::20005f11daa.00000000:head(203789'1033711 osd.3.0:3666823
dirty|omap|data_digest|omap_digest s 0 uv 1032158 dd ffffffff od
56dd55a2 alloc_hint [0 0 0])
2019-10-10 11:23:27.731 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4c2e57b:::20005f11daa.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:27.971 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4c40009:::20005f45f1b.00000000:head :
omap_digest 0x7ccf5cc9 != omap_digest 0xe048d29 from auth oi
6:e4c40009:::20005f45f1b.00000000:head(203789'1033837 osd.3.0:3666949
dirty|omap|data_digest|omap_digest s 0 uv 1032168 dd ffffffff od
e048d29 alloc_hint [0 0 0])
2019-10-10 11:23:27.971 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4c40009:::20005f45f1b.00000000:head :
omap_digest 0x7ccf5cc9 != omap_digest 0xe048d29 from auth oi
6:e4c40009:::20005f45f1b.00000000:head(203789'1033837 osd.3.0:3666949
dirty|omap|data_digest|omap_digest s 0 uv 1032168 dd ffffffff od
e048d29 alloc_hint [0 0 0])
2019-10-10 11:23:27.971 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4c40009:::20005f45f1b.00000000:head :
omap_digest 0x7ccf5cc9 != omap_digest 0xe048d29 from auth oi
6:e4c40009:::20005f45f1b.00000000:head(203789'1033837 osd.3.0:3666949
dirty|omap|data_digest|omap_digest s 0 uv 1032168 dd ffffffff od
e048d29 alloc_hint [0 0 0])
2019-10-10 11:23:27.971 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4c40009:::20005f45f1b.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:28.041 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4c4a042:::20005f389fb.00000000:head :
omap_digest 0xdd1558b8 != omap_digest 0xcf9af548 from auth oi
6:e4c4a042:::20005f389fb.00000000:head(203789'1033899 osd.3.0:3667011
dirty|omap|data_digest|omap_digest s 0 uv 1031358 dd ffffffff od
cf9af548 alloc_hint [0 0 0])
2019-10-10 11:23:28.041 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4c4a042:::20005f389fb.00000000:head :
omap_digest 0xdd1558b8 != omap_digest 0xcf9af548 from auth oi
6:e4c4a042:::20005f389fb.00000000:head(203789'1033899 osd.3.0:3667011
dirty|omap|data_digest|omap_digest s 0 uv 1031358 dd ffffffff od
cf9af548 alloc_hint [0 0 0])
2019-10-10 11:23:28.041 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4c4a042:::20005f389fb.00000000:head :
omap_digest 0xdd1558b8 != omap_digest 0xcf9af548 from auth oi
6:e4c4a042:::20005f389fb.00000000:head(203789'1033899 osd.3.0:3667011
dirty|omap|data_digest|omap_digest s 0 uv 1031358 dd ffffffff od
cf9af548 alloc_hint [0 0 0])
2019-10-10 11:23:28.041 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4c4a042:::20005f389fb.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:28.791 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4c91caa:::20005f36925.00000000:head :
omap_digest 0xbdd9c6d8 != omap_digest 0xd207a82c from auth oi
6:e4c91caa:::20005f36925.00000000:head(203789'1034361 osd.3.0:3667473
dirty|omap|data_digest|omap_digest s 0 uv 1032163 dd ffffffff od
d207a82c alloc_hint [0 0 0])
2019-10-10 11:23:28.791 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4c91caa:::20005f36925.00000000:head :
omap_digest 0xbdd9c6d8 != omap_digest 0xd207a82c from auth oi
6:e4c91caa:::20005f36925.00000000:head(203789'1034361 osd.3.0:3667473
dirty|omap|data_digest|omap_digest s 0 uv 1032163 dd ffffffff od
d207a82c alloc_hint [0 0 0])
2019-10-10 11:23:28.791 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4c91caa:::20005f36925.00000000:head :
omap_digest 0xbdd9c6d8 != omap_digest 0xd207a82c from auth oi
6:e4c91caa:::20005f36925.00000000:head(203789'1034361 osd.3.0:3667473
dirty|omap|data_digest|omap_digest s 0 uv 1032163 dd ffffffff od
d207a82c alloc_hint [0 0 0])
2019-10-10 11:23:28.791 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4c91caa:::20005f36925.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:28.961 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4ca4cc7:::20005f41d80.00000000:head :
omap_digest 0x66bea9d5 != omap_digest 0xa296af17 from auth oi
6:e4ca4cc7:::20005f41d80.00000000:head(203789'1034508 osd.3.0:3667620
dirty|omap|data_digest|omap_digest s 0 uv 1032167 dd ffffffff od
a296af17 alloc_hint [0 0 0])
2019-10-10 11:23:28.961 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4ca4cc7:::20005f41d80.00000000:head :
omap_digest 0x66bea9d5 != omap_digest 0xa296af17 from auth oi
6:e4ca4cc7:::20005f41d80.00000000:head(203789'1034508 osd.3.0:3667620
dirty|omap|data_digest|omap_digest s 0 uv 1032167 dd ffffffff od
a296af17 alloc_hint [0 0 0])
2019-10-10 11:23:28.961 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4ca4cc7:::20005f41d80.00000000:head :
omap_digest 0x66bea9d5 != omap_digest 0xa296af17 from auth oi
6:e4ca4cc7:::20005f41d80.00000000:head(203789'1034508 osd.3.0:3667620
dirty|omap|data_digest|omap_digest s 0 uv 1032167 dd ffffffff od
a296af17 alloc_hint [0 0 0])
2019-10-10 11:23:28.961 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4ca4cc7:::20005f41d80.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:30.001 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4d16e2c:::1001dd18807.00000000:head :
omap_digest 0x8b67426c != omap_digest 0x265813d7 from auth oi
6:e4d16e2c:::1001dd18807.00000000:head(203789'1035272 osd.3.0:3668384
dirty|omap|data_digest|omap_digest s 0 uv 1032169 dd ffffffff od
265813d7 alloc_hint [0 0 0])
2019-10-10 11:23:30.001 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4d16e2c:::1001dd18807.00000000:head :
omap_digest 0x8b67426c != omap_digest 0x265813d7 from auth oi
6:e4d16e2c:::1001dd18807.00000000:head(203789'1035272 osd.3.0:3668384
dirty|omap|data_digest|omap_digest s 0 uv 1032169 dd ffffffff od
265813d7 alloc_hint [0 0 0])
2019-10-10 11:23:30.001 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4d16e2c:::1001dd18807.00000000:head :
omap_digest 0x8b67426c != omap_digest 0x265813d7 from auth oi
6:e4d16e2c:::1001dd18807.00000000:head(203789'1035272 osd.3.0:3668384
dirty|omap|data_digest|omap_digest s 0 uv 1032169 dd ffffffff od
265813d7 alloc_hint [0 0 0])
2019-10-10 11:23:30.001 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4d16e2c:::1001dd18807.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:30.181 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4d29b3d:::20005f332fb.00000000:head :
omap_digest 0xa6a8b00d != omap_digest 0x3f609d1d from auth oi
6:e4d29b3d:::20005f332fb.00000000:head(203789'1035413 osd.3.0:3668525
dirty|omap|data_digest|omap_digest s 0 uv 1032160 dd ffffffff od
3f609d1d alloc_hint [0 0 0])
2019-10-10 11:23:30.181 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4d29b3d:::20005f332fb.00000000:head :
omap_digest 0xa6a8b00d != omap_digest 0x3f609d1d from auth oi
6:e4d29b3d:::20005f332fb.00000000:head(203789'1035413 osd.3.0:3668525
dirty|omap|data_digest|omap_digest s 0 uv 1032160 dd ffffffff od
3f609d1d alloc_hint [0 0 0])
2019-10-10 11:23:30.181 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4d29b3d:::20005f332fb.00000000:head :
omap_digest 0xa6a8b00d != omap_digest 0x3f609d1d from auth oi
6:e4d29b3d:::20005f332fb.00000000:head(203789'1035413 osd.3.0:3668525
dirty|omap|data_digest|omap_digest s 0 uv 1032160 dd ffffffff od
3f609d1d alloc_hint [0 0 0])
2019-10-10 11:23:30.181 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4d29b3d:::20005f332fb.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:30.551 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4d436d4:::20005f43997.00000000:head :
omap_digest 0xed0e46f != omap_digest 0xe480cd9a from auth oi
6:e4d436d4:::20005f43997.00000000:head(203789'1035572 osd.3.0:3668684
dirty|omap|data_digest|omap_digest s 0 uv 1032166 dd ffffffff od
e480cd9a alloc_hint [0 0 0])
2019-10-10 11:23:30.551 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4d436d4:::20005f43997.00000000:head :
omap_digest 0xed0e46f != omap_digest 0xe480cd9a from auth oi
6:e4d436d4:::20005f43997.00000000:head(203789'1035572 osd.3.0:3668684
dirty|omap|data_digest|omap_digest s 0 uv 1032166 dd ffffffff od
e480cd9a alloc_hint [0 0 0])
2019-10-10 11:23:30.551 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4d436d4:::20005f43997.00000000:head :
omap_digest 0xed0e46f != omap_digest 0xe480cd9a from auth oi
6:e4d436d4:::20005f43997.00000000:head(203789'1035572 osd.3.0:3668684
dirty|omap|data_digest|omap_digest s 0 uv 1032166 dd ffffffff od
e480cd9a alloc_hint [0 0 0])
2019-10-10 11:23:30.551 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4d436d4:::20005f43997.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:31.211 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4d8a4ca:::1001dd189d4.00000000:head :
omap_digest 0x4f413584 != omap_digest 0xb832d010 from auth oi
6:e4d8a4ca:::1001dd189d4.00000000:head(203789'1036020 osd.3.0:3669132
dirty|omap|data_digest|omap_digest s 0 uv 1026527 dd ffffffff od
b832d010 alloc_hint [0 0 0])
2019-10-10 11:23:31.211 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4d8a4ca:::1001dd189d4.00000000:head :
omap_digest 0x4f413584 != omap_digest 0xb832d010 from auth oi
6:e4d8a4ca:::1001dd189d4.00000000:head(203789'1036020 osd.3.0:3669132
dirty|omap|data_digest|omap_digest s 0 uv 1026527 dd ffffffff od
b832d010 alloc_hint [0 0 0])
2019-10-10 11:23:31.211 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4d8a4ca:::1001dd189d4.00000000:head :
omap_digest 0x4f413584 != omap_digest 0xb832d010 from auth oi
6:e4d8a4ca:::1001dd189d4.00000000:head(203789'1036020 osd.3.0:3669132
dirty|omap|data_digest|omap_digest s 0 uv 1026527 dd ffffffff od
b832d010 alloc_hint [0 0 0])
2019-10-10 11:23:31.211 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4d8a4ca:::1001dd189d4.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:31.541 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4dae402:::20005f4471c.00000000:head :
omap_digest 0x95196306 != omap_digest 0x78a1d6b0 from auth oi
6:e4dae402:::20005f4471c.00000000:head(203789'1036246 osd.3.0:3669358
dirty|omap|data_digest|omap_digest s 0 uv 1031360 dd ffffffff od
78a1d6b0 alloc_hint [0 0 0])
2019-10-10 11:23:31.541 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4dae402:::20005f4471c.00000000:head :
omap_digest 0x95196306 != omap_digest 0x78a1d6b0 from auth oi
6:e4dae402:::20005f4471c.00000000:head(203789'1036246 osd.3.0:3669358
dirty|omap|data_digest|omap_digest s 0 uv 1031360 dd ffffffff od
78a1d6b0 alloc_hint [0 0 0])
2019-10-10 11:23:31.541 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4dae402:::20005f4471c.00000000:head :
omap_digest 0x95196306 != omap_digest 0x78a1d6b0 from auth oi
6:e4dae402:::20005f4471c.00000000:head(203789'1036246 osd.3.0:3669358
dirty|omap|data_digest|omap_digest s 0 uv 1031360 dd ffffffff od
78a1d6b0 alloc_hint [0 0 0])
2019-10-10 11:23:31.541 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4dae402:::20005f4471c.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:31.911 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4dd409f:::1001dd16f4c.00000000:head :
omap_digest 0xc0d797a6 != omap_digest 0x28ee2bfb from auth oi
6:e4dd409f:::1001dd16f4c.00000000:head(203789'1036509 osd.3.0:3669621
dirty|omap|data_digest|omap_digest s 0 uv 1032170 dd ffffffff od
28ee2bfb alloc_hint [0 0 0])
2019-10-10 11:23:31.911 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4dd409f:::1001dd16f4c.00000000:head :
omap_digest 0xc0d797a6 != omap_digest 0x28ee2bfb from auth oi
6:e4dd409f:::1001dd16f4c.00000000:head(203789'1036509 osd.3.0:3669621
dirty|omap|data_digest|omap_digest s 0 uv 1032170 dd ffffffff od
28ee2bfb alloc_hint [0 0 0])
2019-10-10 11:23:31.911 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4dd409f:::1001dd16f4c.00000000:head :
omap_digest 0xc0d797a6 != omap_digest 0x28ee2bfb from auth oi
6:e4dd409f:::1001dd16f4c.00000000:head(203789'1036509 osd.3.0:3669621
dirty|omap|data_digest|omap_digest s 0 uv 1032170 dd ffffffff od
28ee2bfb alloc_hint [0 0 0])
2019-10-10 11:23:31.911 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4dd409f:::1001dd16f4c.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:32.131 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4de91a6:::20005f631c7.00000000:head :
omap_digest 0x9206e25b != omap_digest 0x4730506f from auth oi
6:e4de91a6:::20005f631c7.00000000:head(203789'1036643 osd.3.0:3669755
dirty|omap|data_digest|omap_digest s 0 uv 1032783 dd ffffffff od
4730506f alloc_hint [0 0 0])
2019-10-10 11:23:32.131 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4de91a6:::20005f631c7.00000000:head :
omap_digest 0x9206e25b != omap_digest 0x4730506f from auth oi
6:e4de91a6:::20005f631c7.00000000:head(203789'1036643 osd.3.0:3669755
dirty|omap|data_digest|omap_digest s 0 uv 1032783 dd ffffffff od
4730506f alloc_hint [0 0 0])
2019-10-10 11:23:32.131 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4de91a6:::20005f631c7.00000000:head :
omap_digest 0x9206e25b != omap_digest 0x4730506f from auth oi
6:e4de91a6:::20005f631c7.00000000:head(203789'1036643 osd.3.0:3669755
dirty|omap|data_digest|omap_digest s 0 uv 1032783 dd ffffffff od
4730506f alloc_hint [0 0 0])
2019-10-10 11:23:32.131 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4de91a6:::20005f631c7.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:32.501 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4e106c1:::1001dd1895b.00000000:head :
omap_digest 0x8d24d314 != omap_digest 0x6df1ff43 from auth oi
6:e4e106c1:::1001dd1895b.00000000:head(203789'1036895 osd.3.0:3670007
dirty|omap|data_digest|omap_digest s 0 uv 1026525 dd ffffffff od
6df1ff43 alloc_hint [0 0 0])
2019-10-10 11:23:32.501 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4e106c1:::1001dd1895b.00000000:head :
omap_digest 0x8d24d314 != omap_digest 0x6df1ff43 from auth oi
6:e4e106c1:::1001dd1895b.00000000:head(203789'1036895 osd.3.0:3670007
dirty|omap|data_digest|omap_digest s 0 uv 1026525 dd ffffffff od
6df1ff43 alloc_hint [0 0 0])
2019-10-10 11:23:32.501 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4e106c1:::1001dd1895b.00000000:head :
omap_digest 0x8d24d314 != omap_digest 0x6df1ff43 from auth oi
6:e4e106c1:::1001dd1895b.00000000:head(203789'1036895 osd.3.0:3670007
dirty|omap|data_digest|omap_digest s 0 uv 1026525 dd ffffffff od
6df1ff43 alloc_hint [0 0 0])
2019-10-10 11:23:32.501 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4e106c1:::1001dd1895b.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4e837c3:::20005f33142.00000000:head :
omap_digest 0x45cf8580 != omap_digest 0x7befa2c2 from auth oi
6:e4e837c3:::20005f33142.00000000:head(203789'1037600 osd.3.0:3670712
dirty|omap|data_digest|omap_digest s 0 uv 1032159 dd ffffffff od
7befa2c2 alloc_hint [0 0 0])
2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4e837c3:::20005f33142.00000000:head :
omap_digest 0x45cf8580 != omap_digest 0x7befa2c2 from auth oi
6:e4e837c3:::20005f33142.00000000:head(203789'1037600 osd.3.0:3670712
dirty|omap|data_digest|omap_digest s 0 uv 1032159 dd ffffffff od
7befa2c2 alloc_hint [0 0 0])
2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4e837c3:::20005f33142.00000000:head :
omap_digest 0x45cf8580 != omap_digest 0x7befa2c2 from auth oi
6:e4e837c3:::20005f33142.00000000:head(203789'1037600 osd.3.0:3670712
dirty|omap|data_digest|omap_digest s 0 uv 1032159 dd ffffffff od
7befa2c2 alloc_hint [0 0 0])
2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4e837c3:::20005f33142.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4e842f7:::20005f33033.00000000:head :
omap_digest 0x33a64598 != omap_digest 0x1a45390a from auth oi
6:e4e842f7:::20005f33033.00000000:head(203789'1037604 osd.3.0:3670716
dirty|omap|data_digest|omap_digest s 0 uv 1032161 dd ffffffff od
1a45390a alloc_hint [0 0 0])
2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4e842f7:::20005f33033.00000000:head :
omap_digest 0x33a64598 != omap_digest 0x1a45390a from auth oi
6:e4e842f7:::20005f33033.00000000:head(203789'1037604 osd.3.0:3670716
dirty|omap|data_digest|omap_digest s 0 uv 1032161 dd ffffffff od
1a45390a alloc_hint [0 0 0])
2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4e842f7:::20005f33033.00000000:head :
omap_digest 0x33a64598 != omap_digest 0x1a45390a from auth oi
6:e4e842f7:::20005f33033.00000000:head(203789'1037604 osd.3.0:3670716
dirty|omap|data_digest|omap_digest s 0 uv 1032161 dd ffffffff od
1a45390a alloc_hint [0 0 0])
2019-10-10 11:23:33.491 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4e842f7:::20005f33033.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:34.931 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4f15b8a:::20005f35cd7.00000000:head :
omap_digest 0xd196c582 != omap_digest 0xd3449697 from auth oi
6:e4f15b8a:::20005f35cd7.00000000:head(203789'1038553 osd.3.0:3671665
dirty|omap|data_digest|omap_digest s 0 uv 1031357 dd ffffffff od
d3449697 alloc_hint [0 0 0])
2019-10-10 11:23:34.931 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4f15b8a:::20005f35cd7.00000000:head :
omap_digest 0xd196c582 != omap_digest 0xd3449697 from auth oi
6:e4f15b8a:::20005f35cd7.00000000:head(203789'1038553 osd.3.0:3671665
dirty|omap|data_digest|omap_digest s 0 uv 1031357 dd ffffffff od
d3449697 alloc_hint [0 0 0])
2019-10-10 11:23:34.931 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4f15b8a:::20005f35cd7.00000000:head :
omap_digest 0xd196c582 != omap_digest 0xd3449697 from auth oi
6:e4f15b8a:::20005f35cd7.00000000:head(203789'1038553 osd.3.0:3671665
dirty|omap|data_digest|omap_digest s 0 uv 1031357 dd ffffffff od
d3449697 alloc_hint [0 0 0])
2019-10-10 11:23:34.931 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4f15b8a:::20005f35cd7.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:36.051 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4f8770e:::20005f3d629.00000000:head :
omap_digest 0x52a9267a != omap_digest 0xb2fbd549 from auth oi
6:e4f8770e:::20005f3d629.00000000:head(203789'1039311 osd.3.0:3672423
dirty|omap|data_digest|omap_digest s 0 uv 1032165 dd ffffffff od
b2fbd549 alloc_hint [0 0 0])
2019-10-10 11:23:36.051 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4f8770e:::20005f3d629.00000000:head :
omap_digest 0x52a9267a != omap_digest 0xb2fbd549 from auth oi
6:e4f8770e:::20005f3d629.00000000:head(203789'1039311 osd.3.0:3672423
dirty|omap|data_digest|omap_digest s 0 uv 1032165 dd ffffffff od
b2fbd549 alloc_hint [0 0 0])
2019-10-10 11:23:36.051 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4f8770e:::20005f3d629.00000000:head :
omap_digest 0x52a9267a != omap_digest 0xb2fbd549 from auth oi
6:e4f8770e:::20005f3d629.00000000:head(203789'1039311 osd.3.0:3672423
dirty|omap|data_digest|omap_digest s 0 uv 1032165 dd ffffffff od
b2fbd549 alloc_hint [0 0 0])
2019-10-10 11:23:36.051 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4f8770e:::20005f3d629.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:36.631 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4fc2d3e:::20005f570fb.00000000:head :
omap_digest 0xeac7763 != omap_digest 0x8965ec0b from auth oi
6:e4fc2d3e:::20005f570fb.00000000:head(203789'1039698 osd.3.0:3672810
dirty|omap|data_digest|omap_digest s 0 uv 1032271 dd ffffffff od
8965ec0b alloc_hint [0 0 0])
2019-10-10 11:23:36.631 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4fc2d3e:::20005f570fb.00000000:head :
omap_digest 0xeac7763 != omap_digest 0x8965ec0b from auth oi
6:e4fc2d3e:::20005f570fb.00000000:head(203789'1039698 osd.3.0:3672810
dirty|omap|data_digest|omap_digest s 0 uv 1032271 dd ffffffff od
8965ec0b alloc_hint [0 0 0])
2019-10-10 11:23:36.631 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4fc2d3e:::20005f570fb.00000000:head :
omap_digest 0xeac7763 != omap_digest 0x8965ec0b from auth oi
6:e4fc2d3e:::20005f570fb.00000000:head(203789'1039698 osd.3.0:3672810
dirty|omap|data_digest|omap_digest s 0 uv 1032271 dd ffffffff od
8965ec0b alloc_hint [0 0 0])
2019-10-10 11:23:36.631 7ff5509b8700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4fc2d3e:::20005f570fb.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:36.821 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4fdb194:::20005f3759c.00000000:head :
omap_digest 0x1ef1fe3b != omap_digest 0x25f0dc6b from auth oi
6:e4fdb194:::20005f3759c.00000000:head(203789'1039877 osd.3.0:3672989
dirty|omap|data_digest|omap_digest s 0 uv 1032162 dd ffffffff od
25f0dc6b alloc_hint [0 0 0])
2019-10-10 11:23:36.821 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4fdb194:::20005f3759c.00000000:head :
omap_digest 0x1ef1fe3b != omap_digest 0x25f0dc6b from auth oi
6:e4fdb194:::20005f3759c.00000000:head(203789'1039877 osd.3.0:3672989
dirty|omap|data_digest|omap_digest s 0 uv 1032162 dd ffffffff od
25f0dc6b alloc_hint [0 0 0])
2019-10-10 11:23:36.821 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4fdb194:::20005f3759c.00000000:head :
omap_digest 0x1ef1fe3b != omap_digest 0x25f0dc6b from auth oi
6:e4fdb194:::20005f3759c.00000000:head(203789'1039877 osd.3.0:3672989
dirty|omap|data_digest|omap_digest s 0 uv 1032162 dd ffffffff od
25f0dc6b alloc_hint [0 0 0])
2019-10-10 11:23:36.821 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4fdb194:::20005f3759c.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:36.901 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 19 soid 6:e4fe5018:::1001dd11fd4.00000000:head :
omap_digest 0xc0244623 != omap_digest 0xbaed0414 from auth oi
6:e4fe5018:::1001dd11fd4.00000000:head(203789'1039928 osd.3.0:3673040
dirty|omap|data_digest|omap_digest s 0 uv 1031359 dd ffffffff od
baed0414 alloc_hint [0 0 0])
2019-10-10 11:23:36.901 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 72 soid 6:e4fe5018:::1001dd11fd4.00000000:head :
omap_digest 0xc0244623 != omap_digest 0xbaed0414 from auth oi
6:e4fe5018:::1001dd11fd4.00000000:head(203789'1039928 osd.3.0:3673040
dirty|omap|data_digest|omap_digest s 0 uv 1031359 dd ffffffff od
baed0414 alloc_hint [0 0 0])
2019-10-10 11:23:36.901 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 shard 91 soid 6:e4fe5018:::1001dd11fd4.00000000:head :
omap_digest 0xc0244623 != omap_digest 0xbaed0414 from auth oi
6:e4fe5018:::1001dd11fd4.00000000:head(203789'1039928 osd.3.0:3673040
dirty|omap|data_digest|omap_digest s 0 uv 1031359 dd ffffffff od
baed0414 alloc_hint [0 0 0])
2019-10-10 11:23:36.901 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 soid 6:e4fe5018:::1001dd11fd4.00000000:head : failed to pick
suitable auth object
2019-10-10 11:23:37.131 7ff54c9b0700 -1 log_channel(cluster) log [ERR]
: 6.327 repair 63 errors, 0 fixed
The cluster info, we are on ceph 14.2.3. :
[root@mds02 ~]# ceph health detail
HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 large
omap objects; 30967 scrub errors; Possible data damage: 219 pgs
inconsistent
FS_DEGRADED 1 filesystem is degraded
fs ceph_fs is degraded
MDS_DAMAGE 1 mds daemon damaged
fs ceph_fs mds.0 is damaged
LARGE_OMAP_OBJECTS 1 large omap objects
1 large objects found in pool 'metadata'
Search the cluster log for 'Large omap object found' for more details.
OSD_SCRUB_ERRORS 30967 scrub errors
PG_DAMAGED Possible data damage: 219 pgs inconsistent
pg 6.23f is active+clean+inconsistent+failed_repair, acting
[18,55,541]
pg 6.246 is active+clean+inconsistent+failed_repair, acting
[19,108,486]
pg 6.24d is active+clean+inconsistent+failed_repair, acting
[109,91,442]
pg 6.24e is active+clean+inconsistent+failed_repair, acting
[468,36,444]
pg 6.255 is active+clean+inconsistent+failed_repair, acting
[37,513,442]
pg 6.259 is active+clean+inconsistent+failed_repair, acting
[486,18,37]
pg 6.262 is active+clean+inconsistent+failed_repair, acting
[449,108,90]
pg 6.269 is active+clean+inconsistent+failed_repair, acting
[449,442,37]
pg 6.26a is active+clean+inconsistent+failed_repair, acting
[36,513,449]
pg 6.26b is active+clean+inconsistent+failed_repair, acting
[138,486,72]
pg 6.27a is active+clean+inconsistent+failed_repair, acting
[520,515,468]
pg 6.27c is active+clean+inconsistent+failed_repair, acting
[138,37,469]
pg 6.282 is active+clean+inconsistent+failed_repair, acting
[37,54,109]
pg 6.287 is active+clean+inconsistent+failed_repair, acting
[72,468,540]
pg 6.288 is active+clean+inconsistent+failed_repair, acting
[541,140,449]
pg 6.289 is active+clean+inconsistent+failed_repair, acting
[73,486,449]
pg 6.28e is active+clean+inconsistent+failed_repair, acting
[449,36,109]
pg 6.293 is active+clean+inconsistent+failed_repair, acting
[447,522,540]
pg 6.294 is active+clean+inconsistent+failed_repair, acting
[513,444,37]
[root@mds02 ~]# ceph -s
cluster:
id: 92bfcf0a-1d39-43b3-b60f-44f01b630e47
health: HEALTH_ERR
1 filesystem is degraded
1 mds daemon damaged
1 large omap objects
30967 scrub errors
Possible data damage: 219 pgs inconsistent
services:
mon: 3 daemons, quorum mds01,mds02,mds03 (age 2d)
mgr: mds02(active, since 3w), standbys: mds01, mds03
mds: ceph_fs:1/2 {1=mds03=up:resolve} 2 up:standby, 1 damaged
osd: 535 osds: 533 up, 529 in
data:
pools: 3 pools, 3328 pgs
objects: 376.32M objects, 673 TiB
usage: 1.0 PiB used, 2.2 PiB / 3.2 PiB avail
pgs: 3097 active+clean
160 active+clean+inconsistent
59 active+clean+inconsistent
9 active+clean+scrubbing+deep
3 active+clean+scrubbing
Does somebody knows how to fix this?
Thanks!
Kenneth
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
