-----------------------------------------------------------
Never put off until tomorrow what you can do the day after tomorrow.
Mark Twain
Allen Samuels
Chief Software Architect, Emerging Storage Solutions
951 SanDisk Drive, Milpitas, CA 95035
T: +1 408 801 7030| M: +1 408 780 6416
allen.samuels@xxxxxxxxxxx
-----Original Message-----
From: Sage Weil [mailto:sage@xxxxxxxxxxx]
Sent: Wednesday, May 07, 2014 9:24 AM
To: Allen Samuels
Cc: ceph-devel@xxxxxxxxxxxxxxx
Subject: RE: RBD thoughts
On Wed, 7 May 2014, Allen Samuels wrote:
Sage wrote:
Allen wrote:
I was looking over the CDS for Giant and was paying particular
attention to the rbd journaling stuff. Asynchronous
geo-replications for block devices is really a key for enterprise
deployment and this is the foundational element of that. It?s an
area that we are keenly interested in and would be willing to
devote development resources toward. It wasn?t clear from the
recording whether this was just musings or would actually be
development for Giant, but when you get your head above water
w.r.t. the acquisition I?d like to investigate how we (Sandisk) could help turn this into a real project. IMO, this is MUCH more important than CephFS stuff for penetrating enterprises.
The blueprint suggests the creation of an additional journal for
the block device and that this journal would track metadata
changes and potentially record overwritten data (without the
overwritten data you can only sync to snapshots ? which will be
reasonable functionality for some use-cases). It seems to me that
this probably doesn?t work too well. Wouldn?t it be the case that
you really want to commit to the journal AND to the block device
atomically? That?s really problematic with the current RADOS
design as the separate journal would be in a separate PG from the
target block and likely on a separate OSD. Now you have all sorts of cases of crashes/updates where the journal and the target block are out of sync.
The idea is to make it a write-ahead journal, which avoids any need
for atomicity. The writes are streamed to the journal, and applied
to the rbd image proper only after they commit there. Since block
operations are effeictively idempotent (you can replay the journal
from any point and the end result is always the same) the recovery
case is pretty simple.
Who is responsible for the block device part of the commit?. If it's
the RBD code rather than the OSD, then I think there's a dangerous
failure case where the journal commits and then the client crashes and
the journal-based replication system ends up replicating the last
(un-performed) write operation. If it's the OSDs that are responsible,
then this is not an issue.
The idea is to use the usual set of write-ahead journaling tricks: we write first to the journal, then to the device, and lazily update a pointer indicating which journal events have been applied. After a crash, the new client will reapply anything in the journal after that point to ensure the device is in sync.
While the device is in active use, we'd need to track which writes have not yet been applied to the device so we can delay a read following a recent write until it is applied. (This should be very rare, given that the file system sitting on top of the device is generally doing all sorts of caching.)
This only works, of course, for use-cases where there is a single active writer for the device. That means it's usable for local file systems like
ext3/4 and xfs, but not for someting like ocfs2.
Similarly, I don't think the snapshot limitation is there; you can
simply note the journal offset, then copy the image (in a racy way),
and then replay the journal from that position to capture the recent
updates.
w.r.t. snapshots and non-old-data-preserving journaling mode, How will
you deal with the race between reading the head of the journal and
reading the data referenced by that head of the journal that could be
over-written by a write operation before you can actually read it?
Oh, I think I'm using different terminology. I'm assuming that the journal includes the *new* data (ala data=journal mode for ext*). We talked a bit at CDS about an optional separate journal with overwritten data so that you could 'rewind' activity on an image, but that is probably not what you were talking about :).
Even past the functional level issues this probably creates a
performance hot-spot too ? also undesirable.
For a naive journal implementation and busy block device, yes. What
I'd like to do, though, is make a journal abstraction on top of
librados that can eventually also replace the current MDS journaler
and do things a bit more intelligently. The main thing would be to
stripe events over a set of objects to distribute the load. For the
MDS, there are a bunch of other minor things we want to do to
streamline the implementation and to improve the ability to inspect and repair the journal.
Note that the 'old data' would be an optional thing that would only
be enabled if the user wanted the ability to rewind.
It seems to me that the extra journal isn?t necessary, i.e., that
the current PG log already has most of the information that?s
needed (it doesn?t have the ?old data?, but that?s easily added ?
in fact it?s cheaper to add it in with a special transaction token
because you don?t have to send the ?old data? over the wire twice?
the OSD can read it locally to put into the PG log). Of course, PG
logs aren?t synchronized across the pool but that?s easy [...]
I don't think the pg log can be sanely repurposed for this. It is a
metadata journal only, and needs to be in order to make peering work
effectively, whereas the rbd journal needs to be a data journal to
work well. Also, if the updates are spread across all of the rbd
image blocks/objects, then it becomes impractical to stream them to
another cluster because you'll need to watch for those updates on
all objects (vs just the journal objects)...
I don't see the difference between the pg-log "metadata" journal and
the rbd journal (when running in the 'non-old-data-preserving' mode).
Essentially, the pg-log allows a local replica to "catch up", how is
that different then allowing a non-local rbd to "catch up"??
The PG log only indicates which objects were touched and which versions are (now) the latest. When recovery happens, we go get the latest version of the object from the usual location. If there are two updates to the same object the log tells us that happens but we don't preserved the intermediate version. The rbd data journal, on the other hand, would preserve the full update timeline, ensuring that we have a fully-coherent view of the image at any point in the timeline.
--
In any case, this is the proposal we originally discussed at CDS. I'm not sure if it's the best or most efficient, but I think it is relatively simple to implement and takes advantage of the existing abstractions and interfaces. Input is definitely welcome! I'm skeptical that the pg log will be useful in this case, but you're right that the overhead with the proposed approach is non-trivial...
sage
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
