Maybe, I figured that the call to DBObjectMap::sync in FileStore::sync should take care of it though? -Sam On Sat, Oct 31, 2015 at 11:41 PM, Chen, Xiaoxi <xiaoxi.chen@xxxxxxxxx> wrote: > As we use submit_transaction(instead of submit_transaction_sync) in DBObjectmap, and we also don't use a kv_sync_thread for DB. Seems we need to rely on the syncfs(2) at commit time for persist everything? > > If that is the case, moving db out of the same FS as Data may cause issue? > > >> -----Original Message----- >> From: ceph-devel-owner@xxxxxxxxxxxxxxx [mailto:ceph-devel- >> owner@xxxxxxxxxxxxxxx] On Behalf Of Xue, Chendi >> Sent: Friday, October 30, 2015 10:05 AM >> To: 'Samuel Just' >> Cc: ceph-devel@xxxxxxxxxxxxxxx >> Subject: Specify omap path for filestore >> >> Hi, Sam >> >> Last week I introduced about how we saw the benefit of moving omap to a >> separate device. >> >> And here is the pull request: >> https://github.com/ceph/ceph/pull/6421 >> >> I had tested redeploy and restart ceph cluster at my setup, the codes works >> fine. >> one problem is do you think I should *DELETE* all the files under the >> omap_path firstly? Because I notice if old pg data leaves there, osd daemon >> may run into chaos. But I am not sure if it should leave to users to DELETE. >> >> Any thoughts? >> >> Also I paste some data I talked , which is about the rbd and osd write iops >> ratio when doing randwrite to a rbd device. >> >> ======Here is some data===== >> We uses 4 clients , 35 vm each to test on rbd randwrite. >> 4 osd physical nodes, each has 10 HDD as osd and 2 ssd as journal >> 2 replica >> filestore_max_inline_xattr_xfs=0 >> filestore_max_inline_xattr_size_xfs=0 >> >> Before moving omap to separate ssd, we saw a frontend and backend iops >> ratio of 1:5.8, rbd side total iops 1206, hdd total iops 7034 Like we talked, 5.8 >> consists of 2 replica write, inode and omap writes >> runid op_size op_type QD engine serverNum clie >> ntNum rbdNum runtime fio_iops fio_bw fio_latency >> osd_iops osd_bw osd_latency >> 332 4k randwrite qd8 qemurbd 4 4 >> 140 400 sec 1206.000 4.987 MB/s 884.617 >> msec 7034.975 47.407 MB/s 242.620 msec >> >> And after moving omap to a separate ssd, we saw a frontend vs. backend >> ratio drops to 1:2.6, rbd side total iops 5006, hdd total iops 13089 >> runid op_size op_type QD engine serverNum clie >> ntNum rbdNum runtime fio_iops fio_bw fio_latency >> osd_iops osd_bw osd_latency >> 326 4k randwrite qd8 qemurbd 4 4 >> 140 400 sec 5006.000 19.822 MB/s 222.296 >> msec 13089.020 82.897 MB/s 482.203 msec >> >> >> Best regards, >> Chendi >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the >> body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at >> http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
