Re: Ceph RBD object-map and discard in VM

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



There appears to be a hole in the documentation.  You know have to set
a configuration option to enable tracing:

rbd_tracing = true

This will causes librbd.so to dynamically load the tracing module
librbd_tp.so (which has linkage to LTTng-UST).

On Fri, Jul 15, 2016 at 1:47 PM, Vaibhav Bhembre
<vaibhav@xxxxxxxxxxxxxxxx> wrote:
> I followed the steps mentioned in [1] but somehow I am unable to see any
> traces to continue with its step 2. There are no errors seen when performing
> operations mentioned in step 1. In my setup I am running lttng commands on
> the HV where my VM has the RBD device attached.
>
> My lttng version is as follows:
>
> $ lttng --version
> lttng (LTTng Trace Control) 2.4.0 - Époque Opaque
> r$ lttng-sessiond --version
> 2.4.0
>
> My uname -r looks like follows:
> Linux infra1node71 3.13.0-65-generic #106-Ubuntu SMP Fri Oct 2 22:08:27 UTC
> 2015 x86_64 x86_64 x86_64 GNU/Linux
>
> The kern.log is clear of any apparmor denials as well.
>
> Would I need to have my librbd linked with lttng-ust by any chance? I don't
> see it linked as seen below:
>
> $ ldd /usr/lib/librbd.so.1.0.0 | grep lttng
> $
>
> Any idea what I might be missing here to get lttng running successfully?
>
> [1] http://docs.ceph.com/docs/master/rbd/rbd-replay/
>
>
> On 07/14, Jason Dillaman wrote:
>>
>> I would probably be able to resolve the issue fairly quickly if it
>> would be possible for you to provide a RBD replay trace from a slow
>> and fast mkfs.xfs test run and attach it to the tracker ticket I just
>> opened for this issue [1]. You can follow the instructions here [2]
>> but would only need to perform steps 1 and 2 (attaching to output from
>> step 2 to the ticket).
>>
>> Thanks,
>>
>> [1] http://tracker.ceph.com/issues/16689
>> [2] http://docs.ceph.com/docs/master/rbd/rbd-replay/
>>
>> On Thu, Jul 14, 2016 at 2:55 PM, Vaibhav Bhembre
>> <vaibhav@xxxxxxxxxxxxxxxx> wrote:
>>>
>>> We have been observing this similar behavior. Usually it is the case
>>> where
>>> we create a new rbd image, expose it into the guest and perform any
>>> operation that issues discard to the device.
>>>
>>> A typical command that's first run on a given device is mkfs, usually
>>> with
>>> discard on.
>>>
>>> # time mkfs.xfs -s size=4096 -f /dev/sda
>>> meta-data=/dev/sda               isize=256    agcount=4, agsize=6553600
>>> blks
>>>          =                       sectsz=4096  attr=2, projid32bit=0
>>> data     =                       bsize=4096   blocks=26214400, imaxpct=25
>>>          =                       sunit=0      swidth=0 blks
>>> naming   =version 2              bsize=4096   ascii-ci=0
>>> log      =internal log           bsize=4096   blocks=12800, version=2
>>>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
>>> realtime =none                   extsz=4096   blocks=0, rtextents=0
>>>
>>> real 9m10.882s
>>> user 0m0.000s
>>> sys 0m0.012s
>>>
>>> When we issue this same command with object-map feature disabled on the
>>> image it completes much faster.
>>>
>>> # time mkfs.xfs -s size=4096 -f /dev/sda
>>> meta-data=/dev/sda               isize=256    agcount=4, agsize=6553600
>>> blks
>>>          =                       sectsz=4096  attr=2, projid32bit=0
>>> data     =                       bsize=4096   blocks=26214400, imaxpct=25
>>>          =                       sunit=0      swidth=0 blks
>>> naming   =version 2              bsize=4096   ascii-ci=0
>>> log      =internal log           bsize=4096   blocks=12800, version=2
>>>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
>>> realtime =none                   extsz=4096   blocks=0, rtextents=0
>>>
>>> real 0m1.780s
>>> user 0m0.000s
>>> sys 0m0.012s
>>>
>>> Also from what I am seeing the slowness seems to be proportional to the
>>> size
>>> of the image rather than the amount of data written into it. Issuing mkfs
>>> without discard doesn't reproduce this issue. The above values were for
>>> 100G
>>> rbd image. The 250G takes slightly more than twice the time taken for
>>> 100G
>>> one.
>>>
>>> # time mkfs.xfs -s size=4096 -f /dev/sda
>>> meta-data=/dev/sda               isize=256    agcount=4, agsize=16384000
>>> blks
>>>          =                       sectsz=4096  attr=2, projid32bit=0
>>> data     =                       bsize=4096   blocks=65536000, imaxpct=25
>>>          =                       sunit=0      swidth=0 blks
>>> naming   =version 2              bsize=4096   ascii-ci=0
>>> log      =internal log           bsize=4096   blocks=32000, version=2
>>>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
>>> realtime =none                   extsz=4096   blocks=0, rtextents=0
>>>
>>> real 22m58.076s
>>> user 0m0.000s
>>> sys 0m0.024s
>>>
>>> Let me know if you need any more information regarding this. We would
>>> like
>>> to enable object-map (and fast-diff) on our images once this gets
>>> resolved.
>>>
>>>
>>> On Wed, Jun 22, 2016 at 5:39 PM, Jason Dillaman <jdillama@xxxxxxxxxx>
>>> wrote:
>>>>
>>>>
>>>> I'm not sure why I never received the original list email, so I
>>>> apologize for the delay. Is /dev/sda1, from your example, fresh with
>>>> no data to actually discard or does it actually have lots of data to
>>>> discard?
>>>>
>>>> Thanks,
>>>>
>>>> On Wed, Jun 22, 2016 at 1:56 PM, Brian Andrus <bandrus@xxxxxxxxxx>
>>>> wrote:
>>>> > I've created a downstream bug for this same issue.
>>>> >
>>>> > https://bugzilla.redhat.com/show_bug.cgi?id=1349116
>>>> >
>>>> > On Wed, Jun 15, 2016 at 6:23 AM, <list@xxxxxxxxxxxxxxx> wrote:
>>>> >>
>>>> >> Hello guys,
>>>> >>
>>>> >> We are currently testing Ceph Jewel with object-map feature enabled:
>>>> >>
>>>> >> rbd image 'disk-22920':
>>>> >>         size 102400 MB in 25600 objects
>>>> >>         order 22 (4096 kB objects)
>>>> >>         block_name_prefix: rbd_data.7cfa2238e1f29
>>>> >>         format: 2
>>>> >>         features: layering, exclusive-lock, object-map, fast-diff,
>>>> >> deep-flatten
>>>> >>         flags:
>>>> >>
>>>> >> We use this RBD as disk for a kvm virtual machine with virtio-scsi
>>>> >> and
>>>> >> discard=unmap. We noticed the following paremeters in /sys/block:
>>>> >>
>>>> >> # cat /sys/block/sda/queue/discard_*
>>>> >> 4096
>>>> >> 1073741824
>>>> >> 0 <- discard_zeroes_data
>>>> >>
>>>> >> While trying to do a mkfs.ext4 on the disk in VM we noticed a low
>>>> >> performance with using discard.
>>>> >>
>>>> >> mkfs.ext4 -E nodiscard /dev/sda1 - tooks 5 seconds to complete
>>>> >> mkfs.ext4 -E discard /dev/sda1 - tooks around 3 monutes
>>>> >>
>>>> >> When disabling the object-map the mkfs with discard tooks just 5
>>>> >> seconds.
>>>> >>
>>>> >> Do you have any idea what might cause this issue?
>>>> >>
>>>> >> Kernel: 4.2.0-35-generic #40~14.04.1-Ubuntu
>>>> >> Ceph: 10.2.0
>>>> >> Libvirt: 1.3.1
>>>> >> QEMU: 2.5.0
>>>> >>
>>>> >> Thanks!
>>>> >>
>>>> >> Best regards,
>>>> >> Jonas
>>>> >> _______________________________________________
>>>> >> ceph-users mailing list
>>>> >> ceph-users@xxxxxxxxxxxxxx
>>>> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Brian Andrus
>>>> > Red Hat, Inc.
>>>> > Storage Consultant, Global Storage Practice
>>>> > Mobile +1 (530) 903-8487
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > ceph-users mailing list
>>>> > ceph-users@xxxxxxxxxxxxxx
>>>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Jason
>>>> _______________________________________________
>>>> ceph-users mailing list
>>>> ceph-users@xxxxxxxxxxxxxx
>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>>
>>>
>>
>>
>>
>> --
>> Jason
>
>
> --
> Vaibhav Bhembre



-- 
Jason
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux