On 11/18/16 23:15, Peter Maloney wrote: > > BTW, my rebalance finished, and I guess the performance is a bit > better, with load distributed a bit better, but blocked requests still > happen if I use snapshot create + export-diff + delete snapshot, and > make qemu clients hang the same. A 30s sleep between image backups > prevents most, but not large exports (or snap creates or snap rm?). I > plan to test and see if really heavy client IO can cause it too > (without creating more objects, like in place modification rather than > using less space like trim and using more space), or only other io > (rbd commands, rebalancing, cow, scrub). So far I suspect it has to do > with either poor xattr performance, or something to do with a large > amount of splitting or otherwise reorganizing many pg/object > directories all at once. A new update... major but not full success for my issue using these settings on the qemu clients: > rbd cache writethrough until flush = true > rbd_cache = true and restarting the qemu processes with `-drive ....,cache=writeback` Now it blocks less, but still a bit, but so far I can run anything without qemu processes hanging. And the next things on my list to try: - bcache - moving xattrs to omap - ext4 so far in my testing, bcache and moving xattrs makes everything small 9-15% faster, and larger blocks can sometimes be the same, but seems nothing is slower (later I'll test moving xattrs without bcache) I hope somehow that helps you, Thomas, but I think you didn't have hangs, only lots of blocks, so I am doubtful. (And also I tried setting filestore_split_multiple to 1, 2(default),4,8 and didn't really notce any effect... maybe larger numbers just delay the problem, but don't fix anything; I'll try again later since the hang issue is fixed) _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |