As we use submit_transaction(instead of submit_transaction_sync) in DBObjectmap, and we also don't use a kv_sync_thread for DB. Seems we need to rely on the syncfs(2) at commit time for persist everything? If that is the case, moving db out of the same FS as Data may cause issue? > -----Original Message----- > From: ceph-devel-owner@xxxxxxxxxxxxxxx [mailto:ceph-devel- > owner@xxxxxxxxxxxxxxx] On Behalf Of Xue, Chendi > Sent: Friday, October 30, 2015 10:05 AM > To: 'Samuel Just' > Cc: ceph-devel@xxxxxxxxxxxxxxx > Subject: Specify omap path for filestore > > Hi, Sam > > Last week I introduced about how we saw the benefit of moving omap to a > separate device. > > And here is the pull request: > https://github.com/ceph/ceph/pull/6421 > > I had tested redeploy and restart ceph cluster at my setup, the codes works > fine. > one problem is do you think I should *DELETE* all the files under the > omap_path firstly? Because I notice if old pg data leaves there, osd daemon > may run into chaos. But I am not sure if it should leave to users to DELETE. > > Any thoughts? > > Also I paste some data I talked , which is about the rbd and osd write iops > ratio when doing randwrite to a rbd device. > > ======Here is some data===== > We uses 4 clients , 35 vm each to test on rbd randwrite. > 4 osd physical nodes, each has 10 HDD as osd and 2 ssd as journal > 2 replica > filestore_max_inline_xattr_xfs=0 > filestore_max_inline_xattr_size_xfs=0 > > Before moving omap to separate ssd, we saw a frontend and backend iops > ratio of 1:5.8, rbd side total iops 1206, hdd total iops 7034 Like we talked, 5.8 > consists of 2 replica write, inode and omap writes > runid op_size op_type QD engine serverNum clie > ntNum rbdNum runtime fio_iops fio_bw fio_latency > osd_iops osd_bw osd_latency > 332 4k randwrite qd8 qemurbd 4 4 > 140 400 sec 1206.000 4.987 MB/s 884.617 > msec 7034.975 47.407 MB/s 242.620 msec > > And after moving omap to a separate ssd, we saw a frontend vs. backend > ratio drops to 1:2.6, rbd side total iops 5006, hdd total iops 13089 > runid op_size op_type QD engine serverNum clie > ntNum rbdNum runtime fio_iops fio_bw fio_latency > osd_iops osd_bw osd_latency > 326 4k randwrite qd8 qemurbd 4 4 > 140 400 sec 5006.000 19.822 MB/s 222.296 > msec 13089.020 82.897 MB/s 482.203 msec > > > Best regards, > Chendi > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the > body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at > http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
