On Fri, 18 Oct 2013, Ugis wrote: > > Ugis, please provide the output of: > > > > RBD_DEVICE=<rbd device name> > > pvs -o pe_start $RBD_DEVICE > > cat /sys/block/$RBD_DEVICE/queue/minimum_io_size > > cat /sys/block/$RBD_DEVICE/queue/optimal_io_size > > > > The 'pvs' command will tell you where LVM aligned the start of the data > > area (which follows the LVM metadata area). Hopefully it reflects what > > was published in sysfs for rbd's striping. > > output follows: > #pvs -o pe_start /dev/rbd1p1 > 1st PE > 4.00m > # cat /sys/block/rbd1/queue/minimum_io_size > 4194304 > # cat /sys/block/rbd1/queue/optimal_io_size > 4194304 Well, the parameters are being set at least. Mike, is it possible that having minimum_io_size set to 4m is causing some read amplification in LVM, translating a small read into a complete fetch of the PE (or somethinga long those lines)? Ugis, if your cluster is on the small side, it might be interesting to see what requests the client is generated in the LVM and non-LVM case by setting 'debug ms = 1' on the osds (e.g., ceph tell osd.* injectargs '--debug-ms 1') and then looking at the osd_op messages that appear in /var/log/ceph/ceph-osd*.log. It may be obvious that the IO pattern is different. > Seems correct in terms of ceph-LVM io parameter negotiation? I wonded > about gpt header+PV metadata - it makes some shift starting from ceph > 1st block beginning. Does this mean that all following LVM 4m data > blocks are shifted by this part and span 2 ceph objects? > If so, performance will be affected. I'm no LVM expert, but I would guess that LVM is aligning things properly based on the above device properties... sage _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |