Hi,
ceph pg repair is currently not fixing three "inconsistent" objects
on one of our pg on a replica 3 pool.
The 3 replica data objets are identical (we checked them on disk on the
3 OSD), the error says "head data_digest != data_digest from auth oi",
see below.
The data in question are used on rbd volumes from KVM and we did a read
from /dev/sdX at the right place on the VM and got a good looking
result : text file, uncorrupted according to our user, so data
currently returned by ceph and replicated 3 times seems fine.
Now the question is how to tell ceph that the replica data is correct
so that the inconsistent message disappears?
We're thinking of doing manually rados get/put but may be this is not
a good idea or there is another way.
Thanks in advance for your help,
Sincerely,
Laurent
# ceph --version
ceph version 10.2.9 (2ee413f77150c0f375ff6f10edd6c8f9c7d060d0)
# ceph health detail
HEALTH_ERR 1 pgs inconsistent; 9 scrub errors;
pg 58.6c1 is active+clean+inconsistent, acting [46,44,19]
...
# rados list-inconsistent-obj 58.6c1 --format=json-pretty
{
"epoch": 277681,
"inconsistents": []
}
# ceph pg repair 58.6c1
... osd 46 /var/log :
shard 19: soid 58:83772424:::rbd_data.30fce9e39dad7a6.000000000007f027:head data_digest 0x783cd2c5 != data_digest 0x501f846c from auth oi 58:83772424:::rbd_data.30fce9e39dad7a6.000000000007f027:head(252707'3100507 osd.2.0:710926 dirty|data_digest|omap_digest s 4194304 uv 3481755 dd 501f846c od ffffffff alloc_hint [0 0])
shard 44: soid 58:83772424:::rbd_data.30fce9e39dad7a6.000000000007f027:head data_digest 0x783cd2c5 != data_digest 0x501f846c from auth oi 58:83772424:::rbd_data.30fce9e39dad7a6.000000000007f027:head(252707'3100507 osd.2.0:710926 dirty|data_digest|omap_digest s 4194304 uv 3481755 dd 501f846c od ffffffff alloc_hint [0 0])
shard 46: soid 58:83772424:::rbd_data.30fce9e39dad7a6.000000000007f027:head data_digest 0x783cd2c5 != data_digest 0x501f846c from auth oi 58:83772424:::rbd_data.30fce9e39dad7a6.000000000007f027:head(252707'3100507 osd.2.0:710926 dirty|data_digest|omap_digest s 4194304 uv 3481755 dd 501f846c od ffffffff alloc_hint [0 0])
soid 58:83772424:::rbd_data.30fce9e39dad7a6.000000000007f027:head: failed to pick suitable auth object
shard 19: soid 58:83772d9e:::rbd_data.68cb7f74b0dc51.000000000000181e:head data_digest 0xd8f6895a != data_digest 0x4edc70a3 from auth oi 58:83772d9e:::rbd_data.68cb7f74b0dc51.000000000000181e:head(77394'2047065 osd.16.0:4500125 dirty|data_digest|omap_digest s 4194304 uv 1895034 dd 4edc70a3 od ffffffff alloc_hint [0 0])
shard 44: soid 58:83772d9e:::rbd_data.68cb7f74b0dc51.000000000000181e:head data_digest 0xd8f6895a != data_digest 0x4edc70a3 from auth oi 58:83772d9e:::rbd_data.68cb7f74b0dc51.000000000000181e:head(77394'2047065 osd.16.0:4500125 dirty|data_digest|omap_digest s 4194304 uv 1895034 dd 4edc70a3 od ffffffff alloc_hint [0 0])
shard 46: soid 58:83772d9e:::rbd_data.68cb7f74b0dc51.000000000000181e:head data_digest 0xd8f6895a != data_digest 0x4edc70a3 from auth oi 58:83772d9e:::rbd_data.68cb7f74b0dc51.000000000000181e:head(77394'2047065 osd.16.0:4500125 dirty|data_digest|omap_digest s 4194304 uv 1895034 dd 4edc70a3 od ffffffff alloc_hint [0 0])
soid 58:83772d9e:::rbd_data.68cb7f74b0dc51.000000000000181e:head: failed to pick suitable auth object
shard 19: soid 58:8377bf9a:::rbd_data.2ef7e1a528b30ea.00000000000254f6:head data_digest 0xdf8916bf != data_digest 0x47b79db8 from auth oi 58:8377bf9a:::rbd_data.2ef7e1a528b30ea.00000000000254f6:head(252707'3100535 osd.2.0:710954 dirty|data_digest|omap_digest s 4194304 uv 3298154 dd 47b79db8 od ffffffff alloc_hint [0 0])
shard 44: soid 58:8377bf9a:::rbd_data.2ef7e1a528b30ea.00000000000254f6:head data_digest 0xdf8916bf != data_digest 0x47b79db8 from auth oi 58:8377bf9a:::rbd_data.2ef7e1a528b30ea.00000000000254f6:head(252707'3100535 osd.2.0:710954 dirty|data_digest|omap_digest s 4194304 uv 3298154 dd 47b79db8 od ffffffff alloc_hint [0 0])
shard 46: soid 58:8377bf9a:::rbd_data.2ef7e1a528b30ea.00000000000254f6:head data_digest 0xdf8916bf != data_digest 0x47b79db8 from auth oi 58:8377bf9a:::rbd_data.2ef7e1a528b30ea.00000000000254f6:head(252707'3100535 osd.2.0:710954 dirty|data_digest|omap_digest s 4194304 uv 3298154 dd 47b79db8 od ffffffff alloc_hint [0 0])
soid 58:8377bf9a:::rbd_data.2ef7e1a528b30ea.00000000000254f6:head: failed to pick suitable auth object
repair 9 errors, 0 fixed
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
