On Tue, 25 Oct 2011, Christian Brunner wrote: > 2011/10/25 Sage Weil <sage@xxxxxxxxxxxx>: > > On Tue, 25 Oct 2011, Josef Bacik wrote: > >> At this point it seems like the biggest problem with latency in ceph-osd > >> is not related to btrfs, the latency seems to all be from the fact that > >> ceph-osd is fsyncing a block dev for whatever reason. > > > > There is one place where we sync_file_range() on the journal block device, > > but that should only happen if directio is disabled (it's on by default). > > > > Christian, have you tweaked those settings in your ceph.conf? It would be > > something like 'journal dio = false'. If not, can you verify that > > directio shows true when the journal is initialized from your osd log? > > E.g., > > > > 2011-10-21 15:21:02.026789 7ff7e5c54720 journal _open dev/osd0.journal fd 14: 104857600 bytes, block size 4096 bytes, directio = 1 > > > > If directio = 1 for you, something else funky is causing those > > blkdev_fsync's... > > I've looked it up in the logs - directio is 1: > > Oct 25 17:20:16 os00 osd.000[1696]: 7f0016841740 journal _open > /dev/vg01/lv_osd_journal_0 fd 15: 17179869184 bytes, block size 4096 > bytes, directio = 1 Do you mind capturing an strace? I'd like to see where that blkdev_fsync is coming from. thanks! sage
