On May 3, 2020 8:33:33 AM GMT+03:00, Erick Perez - Quadrian Enterprises <eperez@xxxxxxxxxxxxxxx> wrote:
>sorry corrections:
>For this test I created a 40GB lvm volume group with /dev/sdb and
>/dev/sdc
>then a 40GB LV
>then a 60GB VDO vol (for testing purposes)
>
>vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent'
>output from just created vdoas
>
>[root@localhost ~]# vdostats --verbose /dev/mapper/vdoas | grep -B6
>'saving
>percent'
>physical blocks : 10483712
> logical blocks : 15728640
> 1K-blocks : 41934848
> 1K-blocks used : 4212024
> 1K-blocks available : 37722824
> used percent : 10
> saving percent : 99
>[root@localhost ~]#
>
>FIRST copy CentOS-7-x86_64-Minimal-2003.iso (1.1G) to vdoas from source
>outside vdo volume
>[root@localhost ~]# vdostats --verbose /dev/mapper/vdoas | grep -B6
>'saving
>percent'
> 1K-blocks used : 4721348
> 1K-blocks available : 37213500
> used percent : 11
> saving percent : 9
>
>SECOND copy CentOS-7-x86_64-Minimal-2003.iso (1.1G) to vdoas form
>source
>outside vdo volume
>#cp /root/CentOS-7-x86_64-Minimal-2003.iso
>/mnt/vdomounts/CentOS-7-x86_64-Minimal-2003-version2.iso
> 1K-blocks used : 5239012
> 1K-blocks available : 36695836
> used percent : 12
> saving percent : 52
>
>THIRD copy CentOS-7-x86_64-Minimal-2003.iso (1.1G) to
>vdoas form inside vdo volume to inside vdo volume
> 1K-blocks used : 5248060
> 1K-blocks available : 36686788
> used percent : 12
> saving percent : 67
>
>Then I did this a total of 9 more times to have 10 ISOs copied. Total
>data
>copied 10.6GB.
>
>
>Do note this:
>When using DF, it will show the VDO size, in my case 60G
>when using vdostats it will show the size of the LV, in my case 40G
>Remeber dedupe AND compression are enabled.
>
>The df -hT output shows the logical space occupied by these iso files
>as
>seen by the filesystem on the VDO volume.
>Since VDO manages a logical to physical block map, df sees logical
>space
>consumed according to the file system that resides on top of the VDO
>volume.
>vdostats --hu is viewing the physical block device as managed by VDO.
>Physically a single .ISO image is residing on the disk, but logically
>the
>file system thinks there are 10 copies, occupying 10.6GB.
>
>So at the end I have 10 .ISOs of 1086 1MB blocks (total 10860 1MB
>blocks)
>that yield these results:
> 1K-blocks used : 5248212
> 1K-blocks available : 36686636
> used percent : 12
> saving percent : 89
>
>So at the end it is using 5248212 1K blocks minus 4212024 initial
>used 1K
>blocks, gives (5248212 - 4212024) = 1036188 1K blocks / 1024 = about
>1012MB
>total.
>
>Hope this helps understanding where the space goes.
>
>BTW: Testing system is CentOS Linux release 7.8.2003 stock. with only
>"yum
>install vdo kmod-kvdo"
>
>History of commands:
>[root@localhost vdomounts]# history
> 2 pvcreate /dev/sdb
> 3 pvcreate /dev/sdc
> 8 vgcreate -v -A y vgvol01 /dev/sdb /dev/sdc
> 9 vgdisplay
> 13 lvcreate -l 100%FREE -n lvvdo01 vgvol01
> 14 yum install vdo kmod-kvdo
> 18 vdo create --name=vdoas --device=/dev/vgvol01/lvvdo01
>--vdoLogicalSize=60G --writePolicy=async
> 19 mkfs.xfs -K /dev/mapper/vdoas
> 20 ls /mnt
> 21 mkdir /mnt/vdomounts
> 22 mount /dev/mapper/vdoas /mnt//vdomounts/
> 26 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent'
> 28 cp /root/CentOS-7-x86_64-Minimal-2003.iso /mnt/vdomounts/ -vvv
> 29 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent'
> 30 cp /root/CentOS-7-x86_64-Minimal-2003.iso
>/mnt/vdomounts/CentOS-7-x86_64-Minimal-2003-version2.iso
> 31 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent'
> 33 cd /mnt/vdomounts/
> 35 cp CentOS-7-x86_64-Minimal-2003-version2.iso
>./CentOS-7-x86_64-Minimal-2003-version3.iso
> 36 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent'
> 37 df
> 39 vdostats --hu
> 40 ls -l --block-size=1MB /root/CentOS-7-x86_64-Minimal-2003.iso
> 41 df -hT
> 42 vdo status | grep Dedupl
> 43 vdostats --hu
> 44 vdostats
> 48 cp CentOS-7-x86_64-Minimal-2003-version2.iso
>./CentOS-7-x86_64-Minimal-2003-version4.iso
> 49 cp CentOS-7-x86_64-Minimal-2003-version2.iso
>./CentOS-7-x86_64-Minimal-2003-version5.iso
> 50 cp CentOS-7-x86_64-Minimal-2003-version2.iso
>./CentOS-7-x86_64-Minimal-2003-version6.iso
> 51 cp CentOS-7-x86_64-Minimal-2003-version2.iso
>./CentOS-7-x86_64-Minimal-2003-version7.iso
> 52 cp CentOS-7-x86_64-Minimal-2003-version2.iso
>./CentOS-7-x86_64-Minimal-2003-version8.iso
> 53 cp CentOS-7-x86_64-Minimal-2003-version2.iso
>./CentOS-7-x86_64-Minimal-2003-version9.iso
> 54 df -hT
> 55 ls -l --block-size=1MB
> 56 vdostats --hu
> 57 df -hT
> 58 df
> 59 vdostats --hu
> 60 vdostats
> 61 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent'
> 62 cat /etc/centos-release
> 63 history
>[root@localhost vdomounts]#
>
>
>
>
>
>On Sat, May 2, 2020 at 10:07 PM Erick Perez - Quadrian Enterprises <
>eperez@xxxxxxxxxxxxxxx> wrote:
>
>> My two cents:
>> 1- Do you have an encrypted filesystem on top of VDO? If yes, you
>will see
>> no benefit from dedupe.
>> 2- can you post the stats of vdostats –verbose /dev/mapper/xxxxx
>(replace
>> with your device)
>>
>> you can do something like: "vdostats -verbose /dev/mapper/xxxxxxxx |
>grep
>> -B6 'save percentage'
>>
>>
>>
>>
>> On Sat, May 2, 2020 at 9:54 PM david <david@xxxxxxxx> wrote:
>>
>>> Folks
>>>
>>> I'm looking for a solution for backups because ZFS has failed on me
>>> too many times. In my environment, I have a large amount of data
>>> (around 2tb) that I periodically back up. I keep the last 5
>>> "snapshots". I use rsync so that when I overwrite the oldest
>backup,
>>> most of the data is already there and the backup completes quickly,
>>> because only a small number of files have actually changed.
>>>
>>> Because of this low change rate, I have used ZFS with its
>>> deduplication feature to store the data. I started using a Centos-6
>>> installation, and upgraded years ago to Centos7. Centos 8 is on my
>>> agenda. However, I've had several data-loss events with ZFS where
>>> because of a combination of errors and/or mistakes, the entire store
>>> was lost. I've also noticed that ZFS is maintained separately from
>>> Centos. At this moment, the Centos 8 update causes ZFS to
>>> fail. Looking for an alternate, I'm trying VDO.
>>>
>>> In the VDO installation, I created a logical volume containing two
>>> hard-drives, and defined VDO on top of that logical volume. It
>>> appears to be running, yet I find the deduplication numbers don't
>>> pass the smell test. I would expect that if the logical volume
>>> contains three copies of essentially identical data, I should see
>>> deduplication numbers close to 3.00, but instead I'm seeing numbers
>>> like 1.15. I compute the compression number as follows:
>>> Use df and extract the value for "1k blocks used" from the third
>column
>>> use vdostats --verbose and extract the number titled "1K-blocks
>used"
>>>
>>> Divide the first by the second.
>>>
>>> Can you provide any advice on my use of ZFS or VDO without telling
>me
>>> that I should be doing backups differently?
>>>
>>> Thanks
>>>
>>> David
>>>
>>> _______________________________________________
>>> CentOS mailing list
>>> CentOS@xxxxxxxxxx
>>> https://lists.centos.org/mailman/listinfo/centos
>>>
>>
>>
>> --
>>
>> ---------------------
>> Erick Perez
>>
>>
>
>--
>
>---------------------
>Erick Perez
>Quadrian Enterprises S.A. - Panama, Republica de Panama
>Skype chat: eaperezh
>WhatsApp IM: +507-6675-5083
>---------------------
>_______________________________________________
>CentOS mailing list
>CentOS@xxxxxxxxxx
>https://lists.centos.org/mailman/listinfo/centos
Hi Erick,
In the end how much space is used as per vdostats ?
Keep in mind that VDO keeps one slab for itself (default is 2GB) and also some metadata is written : https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/vdo-qs-requirements#vdo-ig-storage-reqts
Best Regards,
Strahil Nikolov
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
https://lists.centos.org/mailman/listinfo/centos
