> >Try something like the following on each OSD that holds a copy of >rbd_data.1f114174b0dc51.0000000000000974 and see what output you get. >Note that you can drop the bluestore flag if they are not bluestore >osds and you will need the osd stopped at the time (set noout). Also >note, snapids are displayed in hexadecimal in the output (but then '4' >is '4' so not a big issues here). > >$ ceph-objectstore-tool --type bluestore --data-path >/var/lib/ceph/osd/ceph-XX/ --pgid 17.36 --op list >rbd_data.1f114174b0dc51.0000000000000974 I got these results osd.7 Error getting attr on : 17.36_head,#-19:6c000000:::scrub_17.36:head#, (61) No data available ["17.36",{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","sna pid":63,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}] ["17.36",{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","sna pid":-2,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}] osd.12 ["17.36",{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","sna pid":63,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}] ["17.36",{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","sna pid":-2,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}] osd.29 ["17.36",{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","sna pid":63,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}] ["17.36",{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","sna pid":-2,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}] > >The likely issue here is the primary believes snapshot 4 is gone but >there is still data and/or metadata on one of the replicas which is >confusing the issue. If that is the case you can use the the >ceph-objectstore-tool to delete the relevant snapshot(s) > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com