Re: Understanding VDO vs ZFS

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]



Hi David,

in my opinion, VDO isn't worth the effort. I tried VDO for the same use case: backups. My dataset is 2-3TB and I backup daily. Even with a smaller dataset, VDO couldn't stand up to it's promises. It used tons of CPU and memory and with a lot of tuning I could get it to kind of work, but it became corrupted at the slightest problem (even a shutdown could do this, and shutdowns could also take hours).

I have tried a number of things and I use a combination of two things now:
1. a btrfs volume with force-compress enabled to store the intermediate data - it compresses my data to about 60% and that's enough for me
2. use of bup (https://bup.github.io/) to store long-term backups.

bup is incredibly efficient for my use case (full VM backups). Over the course of a whole month, the dataset only increases by about 30% from the initial size (I create a new full backup each month) - and this is with FULL backups of all VMs every day. bup backupsets can also be mounted via FUSE, giving you access to all stored versions in a filesystem-like manner.

If you can backup at will you can probably forego the btrfs volume for intermediate storage - that is just a band-aid to work around a specific issue here.


Stefan


--

________________________________
From: CentOS <centos-bounces@xxxxxxxxxx> on behalf of david <david@xxxxxxxx>
Sent: Sunday, May 3, 2020 2:50 AM
To: centos@xxxxxxxxxx <centos@xxxxxxxxxx>
Subject:  Understanding VDO vs ZFS

Folks

I'm looking for a solution for backups because ZFS has failed on me
too many times.  In my environment, I have a large amount of data
(around 2tb) that I periodically back up.  I keep the last 5
"snapshots".  I use rsync so that when I overwrite the oldest backup,
most of the data is already there and the backup completes quickly,
because only a small number of files have actually changed.

Because of this low change rate, I have used ZFS with its
deduplication feature to store the data.  I started using a Centos-6
installation, and upgraded years ago to Centos7.  Centos 8 is on my
agenda.  However, I've had several data-loss events with ZFS where
because of a combination of errors and/or mistakes, the entire store
was lost.  I've also noticed that ZFS is maintained separately from
Centos.  At this moment, the Centos 8 update causes ZFS to
fail.  Looking for an alternate, I'm trying VDO.

In the VDO installation, I created a logical volume containing two
hard-drives, and defined VDO on top of that logical volume.  It
appears to be running, yet I find the deduplication numbers don't
pass the smell test.  I would expect that if the logical volume
contains three copies of essentially identical data, I should see
deduplication numbers close to 3.00, but instead I'm seeing numbers
like 1.15.  I compute the compression number as follows:
  Use df and extract the value for "1k blocks used" from the third column
  use vdostats --verbose and extract the number titled "1K-blocks used"

Divide the first by the second.

Can you provide any advice on my use of ZFS or VDO without telling me
that I should be doing backups differently?

Thanks

David

_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
https://lists.centos.org/mailman/listinfo/centos
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
https://lists.centos.org/mailman/listinfo/centos



[Index of Archives]     [CentOS]     [CentOS Announce]     [CentOS Development]     [CentOS ARM Devel]     [CentOS Docs]     [CentOS Virtualization]     [Carrier Grade Linux]     [Linux Media]     [Asterisk]     [DCCP]     [Netdev]     [Xorg]     [Linux USB]


  Powered by Linux