On 03/06/2015 06:51 AM, Jake Young wrote: > > > On Thursday, March 5, 2015, Nick Fisk <nick@xxxxxxxxxx > <mailto:nick@xxxxxxxxxx>> wrote: > > Hi All,____ > > __ __ > > Just a heads up after a day’s experimentation.____ > > __ __ > > I believe tgt with its default settings has a small write cache when > exporting a kernel mapped RBD. Doing some write tests I saw 4 times > the write throughput when using tgt aio + krbd compared to tgt with > the builtin librbd.____ > > __ __ > > After running the following command against the LUN, which > apparently disables write cache, Performance dropped back to what I > am seeing using tgt+librbd and also the same as fio.____ > > __ __ > > tgtadm --op update --mode logicalunit --tid 2 --lun 3 -P > mode_page=8:0:18:0x10:0:0xff:0xff:0:0:0xff:0xff:0xff:0xff:0x80:0x14:0:0:0:0:0:0____ > > __ __ > > From that I can only deduce that using tgt + krbd in its default > state is not 100% safe to use, especially in an HA environment.____ > > __ __ > > Nick____ > > > > > Hey Nick, > > tgt actually does not have any caches. No read, no write. tgt's design > is to passthrough all commands to the backend as efficiently as possible. > > http://lists.wpkg.org/pipermail/stgt/2013-May/005788.html > tgt itself does not do any type of caching, but depending on how you have tgt access the underlying block device you might end up using the normal old linux page cache like you would if you did dd if=/dev/rbd0 of=/dev/null bs=4K count=1 dd if=/dev/rbd0 of=/dev/null bs=4K count=1 This is what Ronnie meant in that thread when he was saying there might be caching in the underlying device. If you use tgt bs_rdwr.c (--bstype=rdwr) with the default settings and with krbd then you will end up doing caching, because the krbd's block device will be accessed like in the dd example above (no direct bits set). You can tell tgt bs_rdwr devices to use O_DIRECT or O_SYNC. When you create the lun pass in the "--bsoflags {direct | sync }". Here is an example from the man page: tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 --bsoflags="sync" --backing-store=/data/100m_image.raw If you use bs_aio.c then we always set O_DIRECT when opening the krbd device, so no page caching is done. I think linux aio might require this or at least it did at the time it was written. Also the cache settings exported to the other OS's initiator with that modepage command might affect performance then too. It might change how that OS does writes like send cache syncs down or do some sort of barrier or FUA. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |