Hi everyone, i'm having issues with one of our clusters, regarding a
seemingly unfixable inconsistent pg. We are running ubuntu 16.04, ceph
10.2.7, 96 osds on 8 nodes. After a power outage, we had some
inconsistent pgs, i managed to fix all of them but this one, here's an
excerpt from the logs(it's outputting this everytime i issue a ceph pg
repair command)
2018-01-29 12:49:35.126066 7f09ffd1e700 -1 log_channel(cluster) log
[ERR] : 3.c04 shard 44: soid
3:203d2906:::benchmark_data_mon3_3417_object685:head data_digest
0x8d3f3b5b != data_digest 0xdbdd31f0 from auth oi
3:203d2906:::benchmark_data_mon3_3417_object685:head(112873'834220
client.79854137.0:686 dirty|data_digest|omap_digest s 65536 uv 834220 dd
dbdd31f0 od ffffffff)
2018-01-29 12:49:35.126087 7f09ffd1e700 -1 log_channel(cluster) log
[ERR] : 3.c04 shard 97: soid
3:203d2906:::benchmark_data_mon3_3417_object685:head data_digest
0x8d3f3b5b != data_digest 0xdbdd31f0 from auth oi
3:203d2906:::benchmark_data_mon3_3417_object685:head(112873'834220
client.79854137.0:686 dirty|data_digest|omap_digest s 65536 uv 834220 dd
dbdd31f0 od ffffffff), attr name mismatch '_', attr name mismatch 'snapset'
2018-01-29 12:49:35.126091 7f09ffd1e700 -1 log_channel(cluster) log
[ERR] : 3.c04 soid 3:203d2906:::benchmark_data_mon3_3417_object685:head:
failed to pick suitable auth object
2018-01-29 12:49:35.126164 7f09ffd1e700 -1 log_channel(cluster) log
[ERR] : deep-scrub 3.c04
3:203d2906:::benchmark_data_mon3_3417_object685:head no '_' attr
2018-01-29 12:49:35.126170 7f09ffd1e700 -1 log_channel(cluster) log
[ERR] : deep-scrub 3.c04
3:203d2906:::benchmark_data_mon3_3417_object685:head no 'snapset' attr
2018-01-29 12:50:11.670123 7f09f3d06700 -1 log_channel(cluster) log
[ERR] : 3.c04 deep-scrub 5 errors
2018-01-29 13:30:13.839317 7f596c5d2700 -1 log_channel(cluster) log
[ERR] : 3.c04 shard 44: soid
3:203d2906:::benchmark_data_mon3_3417_object685:head data_digest
0x8d3f3b5b != data_digest 0xdbdd31f0 from auth oi
3:203d2906:::benchmark_data_mon3_3417_object685:head(112873'834220
client.79854137.0:686 dirty|data_digest|omap_digest s 65536 uv 834220 dd
dbdd31f0 od ffffffff)
2018-01-29 13:30:13.839335 7f596c5d2700 -1 log_channel(cluster) log
[ERR] : 3.c04 shard 97 missing
3:203d2906:::benchmark_data_mon3_3417_object685:head
2018-01-29 13:30:13.839339 7f596c5d2700 -1 log_channel(cluster) log
[ERR] : 3.c04 soid 3:203d2906:::benchmark_data_mon3_3417_object685:head:
failed to pick suitable auth object
2018-01-29 13:30:52.850323 7f596c5d2700 -1 log_channel(cluster) log
[ERR] : 3.c04 repair stat mismatch, got 4084/4085 objects, 0/0 clones,
4084/4084 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0
whiteouts, 16824119169/16824119169 bytes, 0/0 hit_set_archive bytes.
2018-01-29 13:30:52.850379 7f596c5d2700 -1 log_channel(cluster) log
[ERR] : 3.c04 repair 3 errors, 1 fixed
2018-01-29 13:51:33.138881 7f59605ba700 -1 log_channel(cluster) log
[ERR] : 3.c04 shard 44: soid
3:203d2906:::benchmark_data_mon3_3417_object685:head data_digest
0x8d3f3b5b != data_digest 0xdbdd31f0 from auth oi
3:203d2906:::benchmark_data_mon3_3417_object685:head(112873'834220
client.79854137.0:686 dirty|data_digest|omap_digest s 65536 uv 834220 dd
dbdd31f0 od ffffffff)
2018-01-29 13:51:33.138895 7f59605ba700 -1 log_channel(cluster) log
[ERR] : 3.c04 shard 97 missing
3:203d2906:::benchmark_data_mon3_3417_object685:head
2018-01-29 13:51:33.138898 7f59605ba700 -1 log_channel(cluster) log
[ERR] : 3.c04 soid 3:203d2906:::benchmark_data_mon3_3417_object685:head:
failed to pick suitable auth object
when i try to find info about the object itself, i get this(after a deep
scrub)
rados list-inconsistent-obj 3.c04 --format=json-pretty
{
"epoch": 114466,
"inconsistents": []
}
i tried deleting the object from the primary and repairing, truncating
the object to the same size on both primary and secondary and even
copying the identical object from the secondary to the primary, but
nothing seems to work. any pointers regarding this?
thanks
Josef Zelenka
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
