On 08/10/2018 06:24 PM, Gregory Farnum wrote:
On Fri, Aug 10, 2018 at 4:53 AM, Paweł Sadowsk <ceph@xxxxxxxxx> wrote:
On 08/09/2018 04:39 PM, Alex Elder wrote:
On 08/09/2018 08:15 AM, Sage Weil wrote:
On Thu, 9 Aug 2018, Piotr Dałek wrote:
Hello,
At OVH we're heavily utilizing snapshots for our backup system. We think
there's an interesting optimization opportunity regarding snapshots I'd like
to discuss here.
The idea is to introduce a concept of a "lightweight" snapshots - such
snapshot would not contain data but only the information about what has
changed on the image since it was created (so basically only the object map
part of snapshots).
Our backup solution (which seems to be a pretty common practice) is as
follows:
1. Create snapshot of the image we want to backup
2. If there's a previous backup snapshot, export diff and apply it on the
backup image
3. If there's no older snapshot, just do a full backup of image
This introduces one big issue: it enforces COW snapshot on image, meaning that
original image access latencies and consumed space increases. "Lightweight"
snapshots would remove these inefficiencies - no COW performance and storage
overhead.
The snapshot in 1 would be lightweight you mean? And you'd do the backup
some (short) time later based on a diff with changed extents?
I'm pretty sure this will export a garbage image. I mean, it will usually
be non-garbage, but the result won't be crash consistent, and in some
(many?) cases won't be usable.
Consider:
- take reference snapshot
- back up this image (assume for now it is perfect)
- write A to location 1
- take lightweight snapshot
- write B to location 1
- backup process copie location 1 (B) to target
The way I (we) see it working is a bit different:
- take snapshot (1)
- data write might occur, it's ok - CoW kicks in here to preserve data
- export data
- convert snapshot (1) to a lightweight one (not create new):
* from now on just remember which blocks has been modified instead
of doing CoW
* you can get rid on previously CoW data blocks (they've been
exported already)
- more writes
- take snapshot (2)
- export diff - only blocks modified since snap (1)
- convert snapshot (2) to a lightweight one
- ...
That way I don't see a place for data corruption. Of course this has
some drawbacks - you can't rollback/export data from such lightweight
snapshot anymore. But on the other hand we are reducing need for CoW -
and that's the main goal with this idea. Instead of making CoW ~all the
time it's needed only for the time of exporting image/modified blocks.
What's the advantage of remembering the blocks changed for a
"lightweight snapshot" once the actual data diff is no longer there?
Is there a meaningful difference between this and just immediately
deleting a snapshot after doing the export?
-Greg
Advantage is that when I need to export diff I know which blocks changed,
without checking (reading) others so I can just export them for backup.
If i delete snapshot after export, next time I'll have to read whole image
again - no possibility to do differential backup.
But as Sage wrote, we are doing this on Filestore. I don't know how Bluestore
works with snapshots (are whole 4MB chunks copied or only area of current write)
so performance might be much better - need to test it.
Our main goal with this idea is to improve performance in case where all images
have at least one snapshot taken every *backup period* (24h or lower).
--
PS
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
