From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Jake Young Sent: 06 March 2015 12:52 To: Nick Fisk Cc: ceph-users@xxxxxxxxxxxxxx Subject: Re: tgt and krbd On Thursday, March 5, 2015, Nick Fisk <nick@xxxxxxxxxx> wrote: Hi All, Just a heads up after a day’s experimentation. I believe tgt with its default settings has a small write cache when exporting a kernel mapped RBD. Doing some write tests I saw 4 times the write throughput when using tgt aio + krbd compared to tgt with the builtin librbd. After running the following command against the LUN, which apparently disables write cache, Performance dropped back to what I am seeing using tgt+librbd and also the same as fio. tgtadm --op update --mode logicalunit --tid 2 --lun 3 -P mode_page=8:0:18:0x10:0:0xff:0xff:0:0:0xff:0xff:0xff:0xff:0x80:0x14:0:0:0:0:0:0 >From that I can only deduce that using tgt + krbd in its default state is not 100% safe to use, especially in an HA environment. Nick Hey Nick, tgt actually does not have any caches. No read, no write. tgt's design is to passthrough all commands to the backend as efficiently as possible. http://lists.wpkg.org/pipermail/stgt/2013-May/005788.html The configuration parameters just inform the initiators whether the backend storage has a cache. Clearly this makes a big difference for you. What initiator are you using with this test? Maybe the kernel is doing the caching. What tuning parameters do you have on the krbd disk? It could be that using aio is much more efficient. Maybe built in lib rbd isn't doing aio? Jake Hi Jake, Hmm that’s interesting, it’s definitely effecting write behaviour though. I was running iometer doing single io depth writes in a windows VM on ESXi using its software initiator, which as far as I’m aware should be sending sync writes for each request. I saw in iostat on the tgt server that my 128kb writes were being coalesced into ~1024kb writes, which would explain the performance increase. So something somewhere is doing caching, albeit on a small scale. The krbd disk was all using default settings. I know the RBD support for tgt is using the librbd sync writes which I suppose might explain the default difference, but this should be the expected behaviour. Nick _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |