On Fri, 10 Jun 2011, Amir G. wrote: > CC'ing lvm-devel and fsdevel > > > On Wed, Jun 8, 2011 at 9:26 PM, Amir G. <amir73il@xxxxxxxxxxxxxxxxxxxxx> wrote: > > On Wed, Jun 8, 2011 at 7:19 PM, Mike Snitzer <snitzer@xxxxxxxxxx> wrote: > >> On Wed, Jun 8, 2011 at 11:59 AM, Amir G. <amir73il@xxxxxxxxxxxxxxxxxxxxx> wrote: > >>> On Wed, Jun 8, 2011 at 6:38 PM, Lukas Czerner <lczerner@xxxxxxxxxx> wrote: > >>>> Amir said: > >> > >>>>> The question of whether the world needs ext4 snapshots is > >>>>> perfectly valid, but going back to the food analogy, I think it's > >>>>> a case of "the proof of the pudding is in the eating". > >>>>> I have no doubt that if ext4 snapshots are merged, many people will use it. > >>>> > >>>> Well, I would like to have your confidence. Why do you think so ? They > >>>> will use it for what ? Doing backups ? We can do this easily with LVM > >>>> without any risk of compromising existing filesystem at all. On desktop > >>> > >>> LVM snapshots are not meant to be long lived snapshots. > >>> As temporary snapshots they are fine, but with ext4 snapshots > >>> you can easily retain monthly/weekly snapshots without the > >>> need to allocate the space for it in advance and without the > >>> 'vanish' quality of LVM snapshots. > >> > >> In that old sf.net wiki you say: > >> Why use Next3 snapshots and not LVM snapshots? > >> * Performance: only small overhead to write performance with snapshots > >> > >> Fair claim against current LVM snapshot (but not multisnap). > >> > >> In this thread you're being very terse on the performance hit you > >> assert multisnap has that ext4 snapshots does not. Can you please be > >> more specific? > >> > >> In your most recent post it seems you're focusing on "LVM snapshots" > >> and attributing the deficiencies of old-style LVM snapshots > >> (non-shared exception store causing N-way copy-out) to dm-multisnap? > >> > >> Again, nobody will dispute that the existing dm-snapshot target has > >> poor performance that requires snapshots be short-lived. But > >> multisnap does _not_ suffer from those performance problems. > >> > >> Mike > >> > > > > Hi Mike, > > > > I am glad that you joined the debate and I am going to start a fresh > > thread for that occasion, to give your question the proper attention. > > > > In my old next3.sf.net wiki, which I do update from time to time, > > I listed 4 advantages of Ext4 (then next3) snapshots over LVM: > > * Performance: only small overhead to write performance with snapshots > > * Scalability: no extra overhead per snapshot > > * Maintenance: no need to pre-allocate disk space for snapshots > > * Persistence: snapshots don't vanish when disk is full > > > > As far as I know, the only thing that has changed from dm-snap > > to dm-multisnap is the Scalability. > > > > Did you resolve the Maintenance and Persistence issues? > > > > With Regards to Performance, Ext4 snapshots are inherently different > > then LVM snapshots and have near zero overhead to write performance > > as the following benchmark, which I presented on LSF, demonstrates: > > http://global.phoronix-test-suite.com/index.php?k=profile&u=amir73il-4632-11284-26560 > > > > There are several reasons for the near zero overhead: > > > > 1. Metadata buffers are always in cache when performing COW, > > so there is no extra read I/O and write I/O of the copied pages is handled > > by the journal (when flushing the snapshot file dirty pages). > > > > 2. Data blocks are never copied > > The move-on-write technique is used to re-allocate data blocks on rewrite > > instead of copying them. > > This is not something that can be done when the snapshot is stored on > > external storage, but it can done when the snapshot file lives in the fs. > > > > 3. New (= after last snapshot take) allocated blocks are never copied > > nor reallocated on rewrite. > > Ext4 snapshots uses the fs block bitmap, to know which blocks were allocated > > at the time the last snapshot was taken, so new blocks are just out of the game. > > For example, in the workload of a fresh kernel build and daily snapshots, > > the creation and deletion of temp files causes no extra I/O overhead whatsoever. > > > > So, yes, I know. I need to run a benchmark of Ext4 snapshots vs. LVM multisnap > > and post the results. When I'll get around to it I'll do it. > > But I really don't think that performance is how the 2 solutions > > should be compared. > > > > The way I see it, LVM snapshots are a complementary solution and they > > have several advantages over Ext4 snapshots, like: > > * Work with any FS > > * Writable snapshots and snapshots of snapshots > > * Merge a snapshot back to the main vol > > > > We actually have one Google summer of code project that is going to export > > an Ext4 snapshot to an LVM snapshot, in order to implement the "revert > > to snapshot" > > functionality, which Ext4 snapshots is lacking. > > > > I'll be happy to answer more question regarding Ext4 snapshots. > > > > Thanks, > > Amir. > > > Adding ejt into discussion. > > Hi Mike, > > In the beginning of this thread I wrote that "competition is good > because it makes us modest", > so now I have to live up to this standard and apologize for not > learning the new LVM > implementation properly before passing judgment. > > To my defense, I could not find any design papers and benchmarks on multisnap > until Christoph had pointed me to some (and was too lazy to read the code...) > > Anyway, it was never my intention to bad mouth LVM. I think LVM is a very useful > tool and the new multisnap and thinp targets look very promising. > > For the sake of letting everyone understand the differences and trade > offs between > LVM and ext4 snapshots, so ext4 snapshots can get a fair trial, I need > to ask you > some questions about the implementation, which I could not figure out by myself > from reading the documents. > > 1. Crash resistance > How is multisnap handling system crashes? > Ext4 snapshots are journaled along with data, so they are fully > resistant to crashes. > Do you need to keep origin target writes pending in batches and issue FUA/flush > request for the metadata and data store devices? > > 2. Performance > In the presentation from LinuxTag, there are 2 "meaningless benchmarks". > I suppose they are meaningless because the metadata is linear mapping > and therefor all disk writes and read are sequential. > Do you have any "real world" benchmarks? > I am guessing that without the filesystem level knowledge in the thin > provisioned target, > files and filesystem metadata are not really laid out on the hard > drive as the filesystem > designer intended. > Wouldn't that be causing a large seek overhead on spinning media? > > 3. ENOSPC > Ext4 snapshots will get into readonly mode on unexpected ENOSPC situation. > That is not perfect and the best practice is to avoid getting to > ENOSPC situation. > But most application do know how to deal with ENOSPC and EROFS gracefully. > Do you have any "real life" experience of how applications deal with > blocking the > write request in ENOSPC situation? > Or what is the outcome if someone presses the reset button because of an > unexplained (to him) system halt? > > 4. Cache size > At the time, I examined using ZFS on an embedded system with 512MB RAM. > I wasn't able to find any official requirements, but there were > several reports around > the net saying that running ZFS with less that 1GB RAM is a performance killer. > Do you have any information about recommended cache sizes to prevent > the metadata store from being a performance bottleneck? > > Thank you! > Amir. > --
