Hi,
On 08.08.2016 09:58, Georgios Dimitrakakis wrote:
Dear all,
I would like your help with an emergency issue but first let me
describe our environment.
Our environment consists of 2OSD nodes with 10x 2TB HDDs each and 3MON
nodes (2 of them are the OSD nodes as well) all with ceph version
0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047)
This environment provides RBD volumes to an OpenStack Icehouse
installation.
Although not a state of the art environment is working well and within
our expectations.
The issue now is that one of our users accidentally deleted one of the
volumes without keeping its data first!
Is there any way (since the data are considered critical and very
important) to recover them from CEPH?
Short answer: no
Long answer: no, but....
Consider the way Ceph stores data... each RBD is striped into chunks
(RADOS objects with 4MB size by default); the chunks are distributed
among the OSDs with the configured number of replicates (probably two in
your case since you use 2 OSD hosts). RBD uses thin provisioning, so
chunks are allocated upon first write access.
If an RBD is deleted all of its chunks are deleted on the corresponding
OSDs. If you want to recover a deleted RBD, you need to recover all
individual chunks. Whether this is possible depends on your filesystem
and whether the space of a former chunk is already assigned to other
RADOS objects. The RADOS object names are composed of the RBD name and
the offset position of the chunk, so if an undelete mechanism exists for
the OSDs' filesystem, you have to be able to recover file by their
filename, otherwise you might end up mixing the content of various
deleted RBDs. Due to the thin provisioning there might be some chunks
missing (e.g. never allocated before).
Given the fact that
- you probably use XFS on the OSDs since it is the preferred filesystem
for OSDs (there is RDR-XFS, but I've never had to use it)
- you would need to stop the complete ceph cluster (recovery tools do
not work on mounted filesystems)
- your cluster has been in use after the RBD was deleted and thus parts
of its former space might already have been overwritten (replication
might help you here, since there are two OSDs to try)
- XFS undelete does not work well on fragmented files (and OSDs tend to
introduce fragmentation...)
the answer is no, since it might not be feasible and the chance of
success are way too low.
If you want to spend time on it I would propose the stop the ceph
cluster as soon as possible, create copies of all involved OSDs, start
the cluster again and attempt the recovery on the copies.
Regards,
Burkhard
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
