On 10/07/12 14:32, Dan Mick wrote:
On 07/09/2012 08:27 PM, Vladimir Bashkirtsev wrote:
On 10/07/12 03:17, Dan Mick wrote:
Well, it's not so much those; those are the objects that hold data
blocks. You're more interested in the objects whose names end in
'.rbd'. These are the header objects, one per image, and are
interpreted by rbd info, but I'm concerned that one of them may not
exist.
Right on the ball: .rbd for image concerned just does not exist. So how
can we recover from this? And why it has disappeared in first place? (I
guess latter may be related to some sort of bug)
Don't know why it might have disappeared. Recovery: no easy way. It's
possible that image header could be reconstructed, but there aren't
any tools written to do it (the header format is pretty uncomplicated).
Well... Then somehow either I need to rebuild it manually or clean up
image remains to free up space. Given that rbd tool refuses to do
anything without .rbd object then clean up appears to be manual as well.
I have run rbd info on the rest of images and excluded rb.* objects
belonging to good images. Now I know broken image has prefix of rb.0.1
and technically I can clean out objects belonging to this image. But rbd
ls seems to pull the list of rbd images from somewhere: broken image
must be removed from there as well. Not sure where it is stored.
Alternatively how hard it would be to throw together a quick tool which
picks up these objects and reconstructs .rbd header? Something tells me
that it should be relatively straight forward.
I have no pressing need to recover this image - I have pulled the backup
and now it is on its merry way. But just for future sake we need to get
this one resolved: another day someone else will hit it.
----------------------------------
30 minutes later:
I have looked at structure of rbd_obj_header_ondisk and really it is
quite simple. Image has no snapshots and so it makes everything straight
forward. Order is default 22, size - well, unknown but finding object
with highest index provides some guidance. get rbd header from another
image, using hexedit changed name and size, put it back and viola -
image is back and running. Not quite sure about integrity but at least
now it will allow to remove image cleanly.
It certainly shouldn't have just happened. Any idea what operations
might have been in progress when it did?
Obviously not. I am running ceph over last few months trying to get it
off track and till now had no major issues. VM concerned was running
while I did upgrade from 0.47.3 to 0.48. After that point I have asked
the list if it is safe to live migrate VM with rbd cache on. Josh
confirmed that it is safe to do so. So I have live migrated VM to
another host. No dramas. Still everything runs. Then I have updated
hosts (rolling update again - migrating VMs away while rebooting hosts).
I have around 10 VMs (including heavily loaded) and all of them migrated
around without any issues. Then suddenly this VM refused to migrate.
While I was typing it I remembered that there was one issue between
upgrade of ceph and failure to migrate: one of pgs turned inconsistent.
pg repair fixed it and I immediately forgot about it. Could it be the
reason why this .rbd disappeared? (Went to check logs but logrotate
already removed it).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
