On Fri, Oct 4, 2019 at 6:09 PM Marc Roos <M.Roos@xxxxxxxxxxxxxxxxx> wrote:
>
> >
> >Try something like the following on each OSD that holds a copy of
> >rbd_data.1f114174b0dc51.0000000000000974 and see what output you get.
> >Note that you can drop the bluestore flag if they are not bluestore
> >osds and you will need the osd stopped at the time (set noout). Also
> >note, snapids are displayed in hexadecimal in the output (but then '4'
> >is '4' so not a big issues here).
> >
> >$ ceph-objectstore-tool --type bluestore --data-path
> >/var/lib/ceph/osd/ceph-XX/ --pgid 17.36 --op list
> >rbd_data.1f114174b0dc51.0000000000000974
>
> I got these results
>
> osd.7
> Error getting attr on : 17.36_head,#-19:6c000000:::scrub_17.36:head#,
> (61) No data available
> ["17.36",{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","sna
> pid":63,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}]
> ["17.36",{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","sna
> pid":-2,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}]
Ah, so of course the problem is the snapshot is missing. You may need
to try something like the following on each of those osds.
$ ceph-objectstore-tool --type bluestore --data-path
/var/lib/ceph/osd/ceph-XX/ --pgid 17.36
'{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","snapid":-2,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}'
remove-clone-metadata 4
>
> osd.12
> ["17.36",{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","sna
> pid":63,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}]
> ["17.36",{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","sna
> pid":-2,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}]
>
> osd.29
> ["17.36",{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","sna
> pid":63,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}]
> ["17.36",{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","sna
> pid":-2,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}]
>
>
> >
> >The likely issue here is the primary believes snapshot 4 is gone but
> >there is still data and/or metadata on one of the replicas which is
> >confusing the issue. If that is the case you can use the the
> >ceph-objectstore-tool to delete the relevant snapshot(s)
> >
--
Cheers,
Brad
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
