On Wed, Jun 10, 2015 at 6:18 PM, Nick Fisk <nick@xxxxxxxxxx> wrote: >> -----Original Message----- >> From: Ilya Dryomov [mailto:idryomov@xxxxxxxxx] >> Sent: 10 June 2015 14:06 >> To: Nick Fisk >> Cc: ceph-users >> Subject: Re: krbd splitting large IO's into smaller IO's >> >> On Wed, Jun 10, 2015 at 2:47 PM, Nick Fisk <nick@xxxxxxxxxx> wrote: >> > Hi, >> > >> > Using Kernel RBD client with Kernel 4.03 (I have also tried some older >> > kernels with the same effect) and IO is being split into smaller IO's >> > which is having a negative impact on performance. >> > >> > cat /sys/block/sdc/queue/max_hw_sectors_kb >> > 4096 >> > >> > cat /sys/block/rbd0/queue/max_sectors_kb >> > 4096 >> > >> > Using DD >> > dd if=/dev/rbd0 of=/dev/null bs=4M >> > >> > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz >> > avgqu-sz await r_await w_await svctm %util >> > rbd0 0.00 0.00 201.50 0.00 25792.00 0.00 256.00 >> > 1.99 10.15 10.15 0.00 4.96 100.00 >> > >> > >> > Using FIO with 4M blocks >> > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz >> > avgqu-sz await r_await w_await svctm %util >> > rbd0 0.00 0.00 232.00 0.00 118784.00 0.00 1024.00 >> > 11.29 48.58 48.58 0.00 4.31 100.00 >> > >> > Any ideas why IO sizes are limited to 128k (256 blocks) in DD's case >> > and 512k in Fio's case? >> >> 128k vs 512k is probably buffered vs direct IO - add iflag=direct to your dd >> invocation. > > Yes, thanks for this, that was the case > >> >> As for the 512k - I'm pretty sure it's a regression in our switch to blk-mq. I >> tested it around 3.18-3.19 and saw steady 4M IOs. I hope we are just missing >> a knob - I'll take a look. > > I've tested both 4.03 and 3.16 and both seem to be split into 512k. Let me know if you need me to test any other particular version. With 3.16 you are going to need to adjust max_hw_sectors_kb / max_sectors_kb as discussed in Dan's thread. The patch that fixed that in the block layer went into 3.19, blk-mq into 4.0 - try 3.19. Thanks, Ilya _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |