On 11/26/16 09:52, Peter Maloney wrote: > On 11/18/16 23:15, Peter Maloney wrote: >> BTW, my rebalance finished, and I guess the performance is a bit >> better, with load distributed a bit better, but blocked requests still >> happen if I use snapshot create + export-diff + delete snapshot, and >> make qemu clients hang the same. A 30s sleep between image backups >> prevents most, but not large exports (or snap creates or snap rm?). I >> plan to test and see if really heavy client IO can cause it too >> (without creating more objects, like in place modification rather than >> using less space like trim and using more space), or only other io >> (rbd commands, rebalancing, cow, scrub). So far I suspect it has to do >> with either poor xattr performance, or something to do with a large >> amount of splitting or otherwise reorganizing many pg/object >> directories all at once. > A new update... major but not full success for my issue > > using these settings on the qemu clients: >> rbd cache writethrough until flush = true >> rbd_cache = true > and restarting the qemu processes with `-drive ....,cache=writeback` > > Now it blocks less, but still a bit, but so far I can run anything > without qemu processes hanging. I wrote too soon... they still hang. But it's lots harder to make them hang I think. > > And the next things on my list to try: > - bcache > - moving xattrs to omap > - ext4 > > so far in my testing, bcache and moving xattrs makes everything small > 9-15% faster, and larger blocks can sometimes be the same, but seems > nothing is slower (later I'll test moving xattrs without bcache) > > I hope somehow that helps you, Thomas, but I think you didn't have > hangs, only lots of blocks, so I am doubtful. > > (And also I tried setting filestore_split_multiple to 1, 2(default),4,8 > and didn't really notce any effect... maybe larger numbers just delay > the problem, but don't fix anything; I'll try again later since the hang > issue is fixed) > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |