|
Hello Jason, thanks for your response. Am 10.05.2018 um 21:18 schrieb Jason
Dillaman:
XENServer directly utilizes nbd devices which are connected in my understanding by blkback (dom-0) and blkfront (dom-U) to the virtual machines.If i configure caches like described at http://docs.ceph.com/docs/luminous/rbd/rbd-config-ref/, are there dedicated caches per rbd-nbd/krbd device or is there a only a single cache area.The librbd cache is per device, but if you aren't performing direct IOs to the device, you would also have the unified Linux pagecache on top of all the devices. In my understanding pagecache is only part of the game if i use data on mounted filesystems (VFS usage). Therefore it would be a good thing to use rbd cache for rbd-nbd (/dev/nbdX). How can i identify the rbd cache with the tools provided by the operating system?Identify how? You can enable the admin sockets and use "ceph --admin-deamon config show" to display the in-use settings. Ah ok, i discovered that i can gather configuration settings by executing: (xen_test is the identity of the xen rbd_nbd user) ceph --id xen_test --admin-daemon /var/run/ceph/ceph-client.xen_test.asok config show | less -p rbd_cache Sorry, my question was a bit unprecice: I was searching for usage statistics of the rbd cache. Is there also a possibility to gather rbd_cache usage statistics as a source of verification for optimizing the cache settings? Due to the fact that a rbd cache is created for every device, i assume that the rbd cache simply part of the rbd-nbd process memory. As described, xenserver directly utilizes the nbd devices.Can you provide some hints how to about adequate cache settings for a write intensive environment (70% write, 30% read)? Is it a good idea to specify a huge rbd cache of 1 GB with a max dirty age of 10 seconds?The librbd cache is really only useful for sequential read-ahead and for small writes (assuming writeback is enabled). Assuming you aren't using direct IO, I'd suspect your best performance would be to disable the librbd cache and rely on the Linux pagecache to work its magic. Our typical workload is originated over 70 percent in database write operations in the virtual machines. Therefore collecting write operations with rbd cache and writing them in chunks to ceph might be a good thing. A higher limit for "rbd cache max dirty" might be a adequate here. At the other side our read workload typically reads huge files in sequential manner. Therefore it might be useful to do start with a configuration like that: rbd cache size = 64MB What is the strategy of librbd to write data to the storage from
rbd_cache if "rbd cache max dirty = 48MB" is reached? Additionally, i would do no non-default settings for readahead on nbd level to have the possibility to configure this at operating system level of the vms. Our operating
systems in the virtual machines use currently a readahead of
256 (256*512 = 128KB). What do you
think about this? Marc |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
