Greg is right, you need to enable RBD admin sockets. This can be a bit
tricky though, so here are a few tips:
1) In ceph.conf on the compute node, explicitly set a location for the
admin socket:
[client.volumes]
admin socket = /var/run/ceph/rbd-$pid.asok
In this example, libvirt/qemu is running with permissions from
ceph.client.volumes.keyring. If you use something different, adjust
accordingly. You can put this under a more generic [client] section, but
there are some downsides (like a new admin socket for each ceph cli
command).
2) Watch for permissions issues creating the admin socket at the path
you used above. For me, I needed to explicitly grant some permissions in
/etc/apparmor.d/abstractions/libvirt-qemu, specifically I had to add:
# for rbd
capability mknod,
and
# for rbd
/etc/ceph/ceph.conf r,
/var/log/ceph/* rw,
/{,var/}run/ceph/** rw,
3) Be aware that if you have multiple rbd volumes attached to a single
rbd image, you'll only get an admin socket to the volume mounted last.
If you can set admin_socket via the libvirt xml for each volume, you can
avoid this issue. This thread will explain better:
http://www.mail-archive.com/ceph-devel@xxxxxxxxxxxxxxx/msg16168.html
4) Once you get an RBD admin socket, query it like:
ceph --admin-daemon /var/run/ceph/rbd-29050.asok config show | grep rbd
Cheers,
Mike Dawson
On 11/25/2013 11:12 AM, Gregory Farnum wrote:
On Mon, Nov 25, 2013 at 5:58 AM, Mark Nelson <mark.nelson@xxxxxxxxxxx> wrote:
On 11/25/2013 07:21 AM, Shu, Xinxin wrote:
Recently , I want to enable rbd cache to identify performance benefit. I
add rbd_cache=true option in my ceph configure file, I use ’virsh
attach-device’ to attach rbd to vm, below is my vdb xml file.
Ceph configuration files are a bit confusing because sometimes you'll see
something like "rbd_cache" listed somewhere but in the ceph.conf file you'll
want a space instead:
rbd cache = true
with no underscore. That should (hopefully) fix it for you!
I believe the config file will take either format.
The RBD cache is a client-side thing, though, so it's not ever going
to show up in the OSD! You want to look at the admin socket created by
QEMU (via librbd) to see if it's working. :)
-Greg
-Greg
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='writeback'/>
<source protocol='rbd'
name='rbd/node12_2:rbd_cache=true:rbd_cache_writethrough_until_flush=true'/>
<target dev='vdb' bus='virtio'/>
<serial>6b5ff6f4-9f8c-4fe0-84d6-9d795967c7dd</serial>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>i
</disk>
I do not know this is ok to enable rbd cache. I see perf counter for rbd
cache in source code, but when I used admin daemon to check rbd cache
statistics,
Ceph –admin-daemon /var/run/ceph/ceph-osd.0.asok perf dump
But I did not get any rbd cahce flags.
My question is how to enable rbd cahce and check rbd cache perf counter,
or how can I make sure rbd cache is enabled, any tips will be
appreciated? Thanks in advanced.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
