|
Sumit, I think random read/write will always outperform sequential read/write in Ceph if we don’t have any kind of cache in front or you have proper striping enabled
in the image. The reason is the following. 1. If you are trying with the default image option, the object size is 4 MB and the stripe size = 4MB and stripe unit = 1. 2. You didn’t mention your write size, so, if it is less than 4 MB , 2 seq write will always land on a same PG and it will be serialized within the OSD. 3. But, if we have 2 random writes it will always (more probable) to land on different PGs and it will be processed in parallel. 4. Same will happen in case of random vs seq read as well. Increasing read_ahead_kb to a reasonable big number will improve the seq read speed. If you are using
librbd, rbd_cache will help you both for read/write I guess. 5. Another option you may want to try to set the strip_size/object_size/stripe_unit to your io_size so that seq read/write can land on different object and
in that case the difference should go away. Hope this is helpful. Thanks & Regards Somnath From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx]
On Behalf Of Sumit Gaur Hi All, What I saw after enabling RBD cache it is working as expected, means sequential write has better MBps than random write. can somebody explain this behaviour ? Is RBD cache setting must for ceph cluster to behave normally ? Thanks sumit On Mon, Feb 2, 2015 at 9:59 AM, Sumit Gaur <sumitkgaur@xxxxxxxxx> wrote: Hi Florent, Cache tiering , No . ** Our Architecture : vdbench/FIO inside VM <--> RBD without cache <-> Ceph Cluster (6 OSDs + 3 Mons) Thanks sumit [root@ceph-mon01 ~]# ceph -s cluster 47b3b559-f93c-4259-a6fb-97b00d87c55a health HEALTH_WARN clock skew detected on mon.ceph-mon02, mon.ceph-mon03 monmap e1: 3 mons at {ceph-mon01=192.168.10.19:6789/0,ceph-mon02=192.168.10.20:6789/0,ceph-mon03=192.168.10.21:6789/0},
election epoch 14, quorum 0,1,2 ceph-mon01,ceph-mon02,ceph-mon03 osdmap e603: 36 osds: 36 up, 36 in pgmap v40812: 5120 pgs, 2 pools, 179 GB data, 569 kobjects 522 GB used, 9349 GB / 9872 GB avail 5120 active+clean On Mon, Feb 2, 2015 at 12:21 AM, Florent MONTHEL <fmonthel@xxxxxxxxxxxxx> wrote: Hi Sumit PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
