Thanks for the replies guys. The client is set to 4MB, I haven't played with the OSD side yet as I wasn't sure if it would make much difference, but I will give it a go. If the client is already passing a 4MB request down through to the OSD, will it be able to readahead any further? The next 4MB object in theory will be on another OSD and so I'm not sure if reading ahead any further on the OSD side would help. How I see the problem is that the RBD client will only read 1 OSD at a time as the RBD readahead can't be set any higher than max_hw_sectors_kb, which is the object size of the RBD. Please correct me if I'm wrong on this. If you could set the RBD readahead to much higher than the object size, then this would probably give the desired effect where the buffer could be populated by reading from several OSD's in advance to give much higher performance. That or wait for striping to appear in the Kernel client. I've also found that BareOS (fork of Bacula) seems to has a direct RADOS feature that supports radosstriper. I might try this and see how it performs as well. > -----Original Message----- > From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of > Somnath Roy > Sent: 17 August 2015 03:36 > To: Alex Gorbachev <ag@xxxxxxxxxxxxxxxxxxx>; Nick Fisk <nick@xxxxxxxxxx> > Cc: ceph-users@xxxxxxxxxxxxxx > Subject: Re: How to improve single thread sequential reads? > > Have you tried setting read_ahead_kb to bigger number for both client/OSD > side if you are using krbd ? > In case of librbd, try the different config options for rbd cache.. > > Thanks & Regards > Somnath > > -----Original Message----- > From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of > Alex Gorbachev > Sent: Sunday, August 16, 2015 7:07 PM > To: Nick Fisk > Cc: ceph-users@xxxxxxxxxxxxxx > Subject: Re: How to improve single thread sequential reads? > > Hi Nick, > > On Thu, Aug 13, 2015 at 4:37 PM, Nick Fisk <nick@xxxxxxxxxx> wrote: > >> -----Original Message----- > >> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf > >> Of Nick Fisk > >> Sent: 13 August 2015 18:04 > >> To: ceph-users@xxxxxxxxxxxxxx > >> Subject: How to improve single thread sequential reads? > >> > >> Hi, > >> > >> I'm trying to use a RBD to act as a staging area for some data before > > pushing > >> it down to some LTO6 tapes. As I cannot use striping with the kernel > > client I > >> tend to be maxing out at around 80MB/s reads testing with DD. Has > >> anyone got any clever suggestions of giving this a bit of a boost, I > >> think I need > > to get it > >> up to around 200MB/s to make sure there is always a steady flow of > >> data to the tape drive. > > > > I've just tried the testing kernel with the blk-mq fixes in it for > > full size IO's, this combined with bumping readahead up to 4MB, is now > > getting me on average 150MB/s to 200MB/s so this might suffice. > > > > On a personal interest, I would still like to know if anyone has ideas > > on how to really push much higher bandwidth through a RBD. > > Some settings in our ceph.conf that may help: > > osd_op_threads = 20 > osd_mount_options_xfs = rw,noatime,inode64,logbsize=256k > filestore_queue_max_ops = 90000 filestore_flusher = false > filestore_max_sync_interval = 10 filestore_sync_flush = false > > Regards, > Alex > > > > >> > >> Rbd-fuse seems to top out at 12MB/s, so there goes that option. > >> > >> I'm thinking mapping multiple RBD's and then combining them into a > >> mdadm > >> RAID0 stripe might work, but seems a bit messy. > >> > >> Any suggestions? > >> > >> Thanks, > >> Nick > >> > > > > > > > > > > > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ________________________________ > > PLEASE NOTE: The information contained in this electronic mail message is > intended only for the use of the designated recipient(s) named above. If the > reader of this message is not the intended recipient, you are hereby notified > that you have received this message in error and that any review, > dissemination, distribution, or copying of this message is strictly prohibited. If > you have received this communication in error, please notify the sender by > telephone or e-mail (as shown above) immediately and destroy any and all > copies of this message in your possession (whether hard copies or > electronically stored copies). > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |