Re: How to improve single thread sequential reads?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



> On 18 Aug 2015, at 13:58, Nick Fisk <nick@xxxxxxxxxx> wrote:
> 
> 
> 
> 
> 
>> -----Original Message-----
>> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of
>> Jan Schermer
>> Sent: 18 August 2015 12:41
>> To: Nick Fisk <nick@xxxxxxxxxx>
>> Cc: ceph-users@xxxxxxxxxxxxxx
>> Subject: Re:  How to improve single thread sequential reads?
>> 
>> Reply in text
>> 
>>> On 18 Aug 2015, at 12:59, Nick Fisk <nick@xxxxxxxxxx> wrote:
>>> 
>>> 
>>> 
>>>> -----Original Message-----
>>>> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf
>>>> Of Jan Schermer
>>>> Sent: 18 August 2015 11:50
>>>> To: Benedikt Fraunhofer <given.to.lists.ceph-
>>>> users.ceph.com.toasta.001@xxxxxxxxxx>
>>>> Cc: ceph-users@xxxxxxxxxxxxxx; Nick Fisk <nick@xxxxxxxxxx>
>>>> Subject: Re:  How to improve single thread sequential
> reads?
>>>> 
>>>> I'm not sure if I missed that but are you testing in a VM backed by
>>>> RBD device, or using the device directly?
>>>> 
>>>> I don't see how blk-mq would help if it's not a VM, it just passes
>>>> the
>>> request
>>>> to the underlying block device, and in case of RBD there is no real
>>>> block device from the host perspective...? Enlighten me if I'm wrong
>>>> please. I
>>> have
>>>> some Ubuntu VMs that use blk-mq for virtio-blk devices and makes me
>>>> cringe because I'm unable to tune the scheduler and it just makes no
>>>> sense at all...?
>>> 
>>> Since 4.0 (I think) the Kernel RBD client now uses the blk-mq
>>> infrastructure, but there is a bug which limits max IO sizes to 128kb,
>>> which is why for large block/sequential that testing kernel is
>>> essential. I think this bug fix should make it to 4.2 hopefully.
>> 
>> blk-mq is supposed to remove redundancy of having
>> 
>> IO scheduler in VM -> VM block device -> host IO scheduler -> block device
>> 
>> it's a paravirtualized driver that just moves requests from inside the VM
> to
>> the host queue (and this is why inside the VM you have no IO scheduler
>> options - it effectively becomes noop).
>> 
>> But this just doesn't make sense if you're using qemu with librbd -
> there's no
>> host queue.
>> It would make sense if the qemu drive was krbd device with a queue.
>> 
>> If there's no VM there should be no blk-mq?
> 
> I think you might be thinking about the virtio-blk driver for blk-mq. Blk-mq
> itself seems to be a lot more about enhancing the overall block layer
> performance in Linux
> 
> https://www.thomas-krenn.com/en/wiki/Linux_Multi-Queue_Block_IO_Queueing_Mec
> hanism_(blk-mq)
> 
> 
> 
>> 
>> So what was added to the kernel was probably the host-side infrastructure
>> to handle blk-mq in guest passthrough to the krdb device, but that's
> probably
>> not your case, is it?
>> 
>>> 
>>>> 
>>>> Anyway I'd try to bump up read_ahead_kb first, and max_hw_sectors_kb
>>>> (to make sure it gets into readahead), also try (if you're not using
>>>> blk-mq)
>>> to a
>>>> cfq scheduler and set it to rotational=1. I see you've also tried
>>>> this,
>>> but I think
>>>> blk-mq is the limiting factor here now.
>>> 
>>> I'm pretty sure you can't adjust the max_hw_sectors_kb (which equals
>>> object size, from what I can tell) and the max_sectors_kb is already
>>> set at the hw_max. But it would sure be nice if the max_hw_sectors_kb
>>> could be set higher though, but I'm not sure if there is a reason for
> this
>> limit.
>>> 
>>>> 
>>>> If you are running a single-threaded benchmark like rados bench then
>>> what's
>>>> limiting you is latency - it's not surprising it scales up with more
>>> threads.
>>> 
>>> Agreed, but with sequential workloads, if you can get readahead
>>> working properly then you can easily remove this limitation as a
>>> single threaded op effectively becomes multithreaded.
>> 
>> Thinking on this more - I don't know if this will help after all, it will
> still be a
>> single thread, just trying to get ahead of the client IO - and that's not
> likely to
>> happen unless you can read the data in userspace slower than what Ceph
>> can read...
>> 
>> I think striping multiple device could be the answer after all. But have
> you
>> tried creating the RBD volume as striped in Ceph?
> 
> Yes striping would probably give amazing performance, but the kernel client
> currently doesn't support it, which leaves us in the position of trying to
> find work arounds to boost performance.
> 
> Although the client read is single threaded, the RBD/RADOS layer would split
> these larger readahead IOs into 4MB requests that would then be processed in
> parallel by the OSD's. This is much the same way as sequential access
> performance varies with a RAID array. If your IO size matches the stripe
> size of the array then you get nearly the bandwidth of all disks involved. I
> think in Ceph the effective stripe size is the   object size * #OSDS.
> 

Hmmm...

RBD -> PG -> objects

stripe_unit (more commonly called "stride") bytes are put into strip_count objects - not OSDs, but it's possible you'll hit all OSDs with a small enough stride and large enough stripe_count... 
I have no idea how well that works in practice on current Ceph releases, my Dumpling experience is probably useless here.

So we're back at striping with mdraid I guess ... :)

> 
>> 
>>> 
>>>> It should run nicely with a real workload once readahead kicks in and
>>>> the queue fills up. But again - not sure how that works with blk-mq
>>>> and I've never used the RBD device directly (the kernel client). Does
>>>> it show in /sys/block ? Can you dump "find /sys/block/$rbd" in here?
>>>> 
>>>> Jan
>>>> 
>>>> 
>>>>> On 18 Aug 2015, at 12:25, Benedikt Fraunhofer <given.to.lists.ceph-
>>>> users.ceph.com.toasta.001@xxxxxxxxxx> wrote:
>>>>> 
>>>>> Hi Nick,
>>>>> 
>>>>> did you do anything fancy to get to ~90MB/s in the first place?
>>>>> I'm stuck at ~30MB/s reading cold data. single-threaded-writes are
>>>>> quite speedy, around 600MB/s.
>>>>> 
>>>>> radosgw for cold data is around the 90MB/s, which is imho limitted
>>>>> by the speed of a single disk.
>>>>> 
>>>>> Data already present on the osd-os-buffers arrive with around
>>>>> 400-700MB/s so I don't think the network is the culprit.
>>>>> 
>>>>> (20 node cluster, 12x4TB 7.2k disks, 2 ssds for journals for 6 osds
>>>>> each, lacp 2x10g bonds)
>>>>> 
>>>>> rados bench single-threaded performs equally bad, but with its
>>>>> default multithreaded settings it generates wonderful numbers,
>>>>> usually only limiited by linerate and/or interrupts/s.
>>>>> 
>>>>> I just gave kernel 4.0 with its rbd-blk-mq feature a shot, hoping to
>>>>> get to "your wonderful" numbers, but it's staying below 30 MB/s.
>>>>> 
>>>>> I was thinking about using a software raid0 like you did but that's
>>>>> imho really ugly.
>>>>> When I know I needed something speedy, I usually just started dd-ing
>>>>> the file to /dev/null and wait for about  three minutes before
>>>>> starting the actual job; some sort of hand-made read-ahead for
>>>>> dummies.
>>>>> 
>>>>> Thx in advance
>>>>> Benedikt
>>>>> 
>>>>> 
>>>>> 2015-08-17 13:29 GMT+02:00 Nick Fisk <nick@xxxxxxxxxx>:
>>>>>> Thanks for the replies guys.
>>>>>> 
>>>>>> The client is set to 4MB, I haven't played with the OSD side yet as
>>>>>> I wasn't sure if it would make much difference, but I will give it
>>>>>> a go. If the client is already passing a 4MB request down through
>>>>>> to the OSD, will it be able to readahead any further? The next 4MB
>>>>>> object in theory will be on another OSD and so I'm not sure if
>>>>>> reading ahead any further on the OSD side would help.
>>>>>> 
>>>>>> How I see the problem is that the RBD client will only read 1 OSD
>>>>>> at a time as the RBD readahead can't be set any higher than
>>>>>> max_hw_sectors_kb, which is the object size of the RBD. Please
>>>>>> correct
>>>> me if I'm wrong on this.
>>>>>> 
>>>>>> If you could set the RBD readahead to much higher than the object
>>>>>> size, then this would probably give the desired effect where the
>>>>>> buffer could be populated by reading from several OSD's in advance
>>>>>> to give much higher performance. That or wait for striping to
>>>>>> appear in
>>> the
>>>> Kernel client.
>>>>>> 
>>>>>> I've also found that BareOS (fork of Bacula) seems to has a direct
>>>>>> RADOS feature that supports radosstriper. I might try this and see
>>>>>> how it performs as well.
>>>>>> 
>>>>>> 
>>>>>>> -----Original Message-----
>>>>>>> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On
>>>>>>> Behalf Of Somnath Roy
>>>>>>> Sent: 17 August 2015 03:36
>>>>>>> To: Alex Gorbachev <ag@xxxxxxxxxxxxxxxxxxx>; Nick Fisk
>>>>>>> <nick@xxxxxxxxxx>
>>>>>>> Cc: ceph-users@xxxxxxxxxxxxxx
>>>>>>> Subject: Re:  How to improve single thread sequential
>>>> reads?
>>>>>>> 
>>>>>>> Have you tried setting read_ahead_kb to bigger number for both
>>>>>>> client/OSD side if you are using krbd ?
>>>>>>> In case of librbd, try the different config options for rbd cache..
>>>>>>> 
>>>>>>> Thanks & Regards
>>>>>>> Somnath
>>>>>>> 
>>>>>>> -----Original Message-----
>>>>>>> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On
>>>>>>> Behalf Of Alex Gorbachev
>>>>>>> Sent: Sunday, August 16, 2015 7:07 PM
>>>>>>> To: Nick Fisk
>>>>>>> Cc: ceph-users@xxxxxxxxxxxxxx
>>>>>>> Subject: Re:  How to improve single thread sequential
>>>> reads?
>>>>>>> 
>>>>>>> Hi Nick,
>>>>>>> 
>>>>>>> On Thu, Aug 13, 2015 at 4:37 PM, Nick Fisk <nick@xxxxxxxxxx> wrote:
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On
>>>>>>>>> Behalf Of Nick Fisk
>>>>>>>>> Sent: 13 August 2015 18:04
>>>>>>>>> To: ceph-users@xxxxxxxxxxxxxx
>>>>>>>>> Subject:  How to improve single thread sequential
>> reads?
>>>>>>>>> 
>>>>>>>>> Hi,
>>>>>>>>> 
>>>>>>>>> I'm trying to use a RBD to act as a staging area for some data
>>>>>>>>> before
>>>>>>>> pushing
>>>>>>>>> it down to some LTO6 tapes. As I cannot use striping with the
>>>>>>>>> kernel
>>>>>>>> client I
>>>>>>>>> tend to be maxing out at around 80MB/s reads testing with DD.
>>>>>>>>> Has anyone got any clever suggestions of giving this a bit of a
>>>>>>>>> boost, I think I need
>>>>>>>> to get it
>>>>>>>>> up to around 200MB/s to make sure there is always a steady flow
>>>>>>>>> of data to the tape drive.
>>>>>>>> 
>>>>>>>> I've just tried the testing kernel with the blk-mq fixes in it
>>>>>>>> for full size IO's, this combined with bumping readahead up to
>>>>>>>> 4MB, is now getting me on average 150MB/s to 200MB/s so this
>> might suffice.
>>>>>>>> 
>>>>>>>> On a personal interest, I would still like to know if anyone has
>>>>>>>> ideas on how to really push much higher bandwidth through a RBD.
>>>>>>> 
>>>>>>> Some settings in our ceph.conf that may help:
>>>>>>> 
>>>>>>> osd_op_threads = 20
>>>>>>> osd_mount_options_xfs = rw,noatime,inode64,logbsize=256k
>>>>>>> filestore_queue_max_ops = 90000 filestore_flusher = false
>>>>>>> filestore_max_sync_interval = 10 filestore_sync_flush = false
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Alex
>>>>>>> 
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Rbd-fuse seems to top out at 12MB/s, so there goes that option.
>>>>>>>>> 
>>>>>>>>> I'm thinking mapping multiple RBD's and then combining them into
>>>>>>>>> a mdadm
>>>>>>>>> RAID0 stripe might work, but seems a bit messy.
>>>>>>>>> 
>>>>>>>>> Any suggestions?
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Nick
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> _______________________________________________
>>>>>>>> ceph-users mailing list
>>>>>>>> ceph-users@xxxxxxxxxxxxxx
>>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>> _______________________________________________
>>>>>>> ceph-users mailing list
>>>>>>> ceph-users@xxxxxxxxxxxxxx
>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>> 
>>>>>>> ________________________________
>>>>>>> 
>>>>>>> PLEASE NOTE: The information contained in this electronic mail
>>>>>>> message is intended only for the use of the designated
>>>>>>> recipient(s) named above. If
>>>>>> the
>>>>>>> reader of this message is not the intended recipient, you are
>>>>>>> hereby
>>>>>> notified
>>>>>>> that you have received this message in error and that any review,
>>>>>>> dissemination, distribution, or copying of this message is
>>>>>>> strictly
>>>>>> prohibited. If
>>>>>>> you have received this communication in error, please notify the
>>>>>>> sender by telephone or e-mail (as shown above) immediately and
>>>>>>> destroy any and all copies of this message in your possession
>>>>>>> (whether hard copies or electronically stored copies).
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> ceph-users mailing list
>>>>>>> ceph-users@xxxxxxxxxxxxxx
>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list
>>>>>> ceph-users@xxxxxxxxxxxxxx
>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>> _______________________________________________
>>>>> ceph-users mailing list
>>>>> ceph-users@xxxxxxxxxxxxxx
>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>> 
>>>> _______________________________________________
>>>> ceph-users mailing list
>>>> ceph-users@xxxxxxxxxxxxxx
>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> 
>>> 
>>> 
>>> 
>> 
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
> 

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux