Re: [ANNOUNCE] Btrfs: a copy on write, snapshotting FS

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Philipp Matthias Hahn wrote:
I would also suggest one more feature: support for block level
de-duplication. I mean:

...

That would be very usable feature, which in most cases would allow to
shrink occupied disk space on 50-90%.

Have you references for this number?
In my experience one gets a lot of benefit from
the much simpler process of "de-duplication" of files.

Yes, I would expect simple hard links to be a better solution for this,
but the feature request is not that out of line.  I actually had plans
on implementing auto duplicate block reuse earlier in btrfs.


One problem with hard-links for me is, they also share the meta-data,
especially file permissions and owners.

Take a Subversion checkout for example: For each file "$A" Subversion
saves a backup under ".svm/text-base/$A.svn-base" for file comparison
and diff generation. The user controls the file permissions of "$A",
Subversion protects its backup with 0444. You can't hard-link them,
because than "svn diff" doesn't work anymore if your editor doesn't
break the hard-link, or worse, your permissions can get wrong.

If previous versions Subversion also had an extra file for file
attributes (mime-type, permissions, to-be-ignored, etc.) Since most
files had no special attributes, each had a file only containing "END".
Those you could hard-link by hand to save space.

If somebody want to research this further:

There is this nice little package called "perforate", which contains
"finddup" to find duplicate files. Run it two times, once with "-i" to
ignore permissions while comparing file contents
	finddup -i -d /
and once without "-i" for "content and permissions must match"
	finddup -d /
This will give you a hint on how many files you could hard-link or how
many files share their content.

So, seems ever for file based de-duplication some support from the FS, including some kind of ability for different inodes point to the same data blocks to store the meta-data, would be needed anyway.

Vlad
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[Index of Archives]     [Linux Ext4 Filesystem]     [Union Filesystem]     [Filesystem Testing]     [Ceph Users]     [Ecryptfs]     [AutoFS]     [Kernel Newbies]     [Share Photos]     [Security]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux Cachefs]     [Reiser Filesystem]     [Linux RAID]     [Samba]     [Device Mapper]     [CEPH Development]
  Powered by Linux