Re: fuse access seems ok but native qemu access is too slow

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Many oVirt users are using qemu with libgfapi for Virtualization needs.

Have you checked this one :  
https://github.com/gluster/glusterfs-specs/blob/master/done/GlusterFS%203.5/libgfapi%20with%20qemu%20libvirt.md

I've used only oVirt's way (which obscures the details), but the article looks legit.


Best Regards,
Strahil Nikolov
Hi,

I have just configure a distributed-replicate 2x3 gluster volume.
It seems to be working fine when I mount the volume with fuse but native gluster access with qemu-img is much slower and outputs errors. Log files are filled with warning and errors.
Starting vm with the created images is possible but it takes much more time when using gluster native access. Nevertheless, after starting up, there seems to be no penalty in using the gluster native access image being some times faster.

Can anyone, please, help me in figuring out if there is and where is the problem?

Thanks
rm



It seems to be working fine when I mount it with fuse:
root@srv-31:~# mount -t glusterfs srv-32.lan.example.com/ssd-volume/ /mnt/ssd-volume/
root@srv-31:~# echo 'This is a test' > /mnt/ssd-volume/test/text.txt
root@srv-31:~# ll /data/glusterfs/ssd/brick[12]/brick/test/*
-rw-r--r-- 2 root root 15 mar 17 14:08 /data/glusterfs/ssd/brick1/brick/test/text.txt

root@srv-31:~# time qemu-img create -f qcow2 /mnt/ssd-volume/libvirt/images/test_5.qcow2.img 64G
Formatting '/mnt/ssd-volume/libvirt/images/test_5.qcow2.img', fmt=qcow2 size=68719476736 cluster_size=65536 lazy_refcounts=off refcount_bits=16

real    0m0,204s
user    0m0,009s
sys     0m0,009s


Creating a image file onteh mouted gluster volume is quick but I get some warnings on log files:
root@srv-31:~# tail -f /var/log/glusterfs/glusterd.log  /var/log/glusterfs/glustershd.log  /var/log/glusterfs/bricks/data-glusterfs-ssd-brick[12]-brick.log
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:37:11.301388 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.303694 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.305808 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.307868 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.309863 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.312139 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.317780 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.319007 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.321096 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.323111 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.325284 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.327319 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.329488 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.331792 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.333597 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.335864 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.339551 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.341110 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.343079 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.345174 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.349569 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.351920 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.353282 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.355272 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.357467 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.359584 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.361771 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.364013 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.366094 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.368094 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.370261 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.372389 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.375550 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
(...)
[2021-03-19 11:37:11.477472 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.479897 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.482318 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.484489 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]


Native gluster access with qemu-img is much slower and outputs errors:
root@srv-31:~# time qemu-img create -f qcow2 gluster://srv-31.lan.example.com/ssd-volume/libvirt/images/test_6.qcow2.img 64G
Formatting 'gluster://srv-31.lan.example.com/ssd-volume/libvirt/images/test_6.qcow2.img', fmt=qcow2 size=68719476736 cluster_size=65536 lazy_refcounts=off refcount_bits=16
[2021-03-19 11:47:42.490844 +0000] I [io-stats.c:3708:ios_sample_buf_size_configure] 0-ssd-volume: Configure ios_sample_buf  size is 1024 because ios_sample_interval is 0
[2021-03-19 11:47:42.615942 +0000] E [MSGID: 108006] [afr-common.c:6146:__afr_handle_child_down_event] 0-ssd-volume-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up.
[2021-03-19 11:47:42.617033 +0000] E [MSGID: 108006] [afr-common.c:6146:__afr_handle_child_down_event] 0-ssd-volume-replicate-1: All subvolumes are down. Going offline until at least one of them comes back up.
[2021-03-19 11:47:52.502174 +0000] I [io-stats.c:4038:fini] 0-ssd-volume: io-stats translator unloaded
[2021-03-19 11:47:53.514364 +0000] I [io-stats.c:3708:ios_sample_buf_size_configure] 0-ssd-volume: Configure ios_sample_buf  size is 1024 because ios_sample_interval is 0
[2021-03-19 11:47:53.654336 +0000] E [MSGID: 108006] [afr-common.c:6146:__afr_handle_child_down_event] 0-ssd-volume-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up.
[2021-03-19 11:47:53.655251 +0000] E [MSGID: 108006] [afr-common.c:6146:__afr_handle_child_down_event] 0-ssd-volume-replicate-1: All subvolumes are down. Going offline until at least one of them comes back up.
[2021-03-19 11:48:03.525947 +0000] I [io-stats.c:4038:fini] 0-ssd-volume: io-stats translator unloaded

real    0m22,068s
user    0m0,055s
sys     0m0,043s


In both cases images are written correctly (and usable):
root@srv-31:~# l /data/glusterfs/ssd/brick[12]/brick/libvirt/images/test_[56]*
-rw-r--r-- 2 root root 193K mar 19 11:37 /data/glusterfs/ssd/brick2/brick/libvirt/images/test_5.qcow2.img
-rw------- 2 root root 193K mar 19 11:47 /data/glusterfs/ssd/brick2/brick/libvirt/images/test_6.qcow2.img

But I get lots of warnings on log files:
root@srv-31:~# tail -f /var/log/glusterfs/glusterd.log  /var/log/glusterfs/glustershd.log  /var/log/glusterfs/bricks/data-glusterfs-ssd-brick[12]-brick.log
==> /var/log/glusterfs/glusterd.log <==
[2021-03-19 11:47:42.483736 +0000] I [MSGID: 106496] [glusterd-handshake.c:969:__server_getspec] 0-management: Received mount request for volume ssd-volume

==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:47:42.498908 +0000] I [addr.c:54:compare_addr_and_update] 0-/data/glusterfs/ssd/brick1/brick: allowed = "*", received addr = "127.0.0.1"
[2021-03-19 11:47:42.498967 +0000] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 80c9a2bf-6fad-4023-8458-a175cef7f681
[2021-03-19 11:47:42.498989 +0000] I [MSGID: 115029] [server-handshake.c:561:server_setvolume] 0-ssd-volume-server: accepted client from CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0 (version: 9.0) with subvol /data/glusterfs/ssd/brick1/brick

==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:47:42.502029 +0000] I [addr.c:54:compare_addr_and_update] 0-/data/glusterfs/ssd/brick2/brick: allowed = "*", received addr = "127.0.0.1"
[2021-03-19 11:47:42.502087 +0000] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 80c9a2bf-6fad-4023-8458-a175cef7f681
[2021-03-19 11:47:42.502109 +0000] I [MSGID: 115029] [server-handshake.c:561:server_setvolume] 0-ssd-volume-server: accepted client from CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0 (version: 9.0) with subvol /data/glusterfs/ssd/brick2/brick

==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:47:42.615182 +0000] W [socket.c:767:__socket_rwv] 0-tcp.ssd-volume-server: readv on 127.0.0.1:49151 failed (No data available)
[2021-03-19 11:47:42.615258 +0000] I [MSGID: 115036] [server.c:500:server_rpc_notify] 0-ssd-volume-server: disconnecting connection [{client-uid=CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0}]

==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:47:42.615215 +0000] W [socket.c:767:__socket_rwv] 0-tcp.ssd-volume-server: readv on 127.0.0.1:49150 failed (No data available)
[2021-03-19 11:47:42.615316 +0000] I [MSGID: 115036] [server.c:500:server_rpc_notify] 0-ssd-volume-server: disconnecting connection [{client-uid=CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0}]

==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:47:42.615459 +0000] I [MSGID: 101055] [client_t.c:397:gf_client_unref] 0-ssd-volume-server: Shutting down connection CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0

==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:47:42.615524 +0000] I [MSGID: 101055] [client_t.c:397:gf_client_unref] 0-ssd-volume-server: Shutting down connection CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0

==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:47:53.522422 +0000] I [addr.c:54:compare_addr_and_update] 0-/data/glusterfs/ssd/brick1/brick: allowed = "*", received addr = "127.0.0.1"
[2021-03-19 11:47:53.522480 +0000] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 80c9a2bf-6fad-4023-8458-a175cef7f681
[2021-03-19 11:47:53.522502 +0000] I [MSGID: 115029] [server-handshake.c:561:server_setvolume] 0-ssd-volume-server: accepted client from CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0 (version: 9.0) with subvol /data/glusterfs/ssd/brick1/brick

==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:47:53.525344 +0000] I [addr.c:54:compare_addr_and_update] 0-/data/glusterfs/ssd/brick2/brick: allowed = "*", received addr = "127.0.0.1"
[2021-03-19 11:47:53.525396 +0000] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 80c9a2bf-6fad-4023-8458-a175cef7f681
[2021-03-19 11:47:53.525419 +0000] I [MSGID: 115029] [server-handshake.c:561:server_setvolume] 0-ssd-volume-server: accepted client from CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0 (version: 9.0) with subvol /data/glusterfs/ssd/brick2/brick

==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:47:53.653652 +0000] W [socket.c:767:__socket_rwv] 0-tcp.ssd-volume-server: readv on 127.0.0.1:49146 failed (No data available)
[2021-03-19 11:47:53.653731 +0000] I [MSGID: 115036] [server.c:500:server_rpc_notify] 0-ssd-volume-server: disconnecting connection [{client-uid=CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0}]

==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:47:53.653752 +0000] W [socket.c:767:__socket_rwv] 0-tcp.ssd-volume-server: readv on 127.0.0.1:49144 failed (No data available)
[2021-03-19 11:47:53.653840 +0000] I [MSGID: 115036] [server.c:500:server_rpc_notify] 0-ssd-volume-server: disconnecting connection [{client-uid=CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0}]

==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:47:53.653928 +0000] I [MSGID: 101055] [client_t.c:397:gf_client_unref] 0-ssd-volume-server: Shutting down connection CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0

==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:47:53.654040 +0000] I [MSGID: 101055] [client_t.c:397:gf_client_unref] 0-ssd-volume-server: Shutting down connection CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0

==> /var/log/glusterfs/glusterd.log <==
[2021-03-19 11:47:53.507207 +0000] I [MSGID: 106496] [glusterd-handshake.c:969:__server_getspec] 0-management: Received mount request for volume ssd-volume


Starting a vm with a virtual disk on the mounted gluster volume (-drive file=/mnt/ssd-volume/libvirt/images/test_6.qcow2.img)is quick:
root@srv-31:~# time virsh start vm-11
Domain vm-11 started

real    0m0,277s
user    0m0,020s
sys     0m0,004s

Statring a vm with a virtual disk on gluster volume with native access (-drive file=gluster://srv-31.lan.example.com:24007/ssd-volume/libvirt/images/test_7.qcow2.img)is much slower:
root@srv-31:~# time virsh start vm-11
Domain vm-11 started

real    0m22,339s
user    0m0,021s
sys     0m0,007s



From time to time, even without user operations, log files report errors:
root@srv-31:~# tail -f /var/log/glusterfs/glusterd.log  /var/log/glusterfs/glustershd.log  /var/log/glusterfs/bricks/data-glusterfs-ssd-brick[12]-brick.log
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:45:09.525079 +0000] E [MSGID: 113002] [posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null for (null) [Invalid argument]
[2021-03-19 11:45:09.525212 +0000] E [MSGID: 115056] [server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server: MKDIR info [{frame=11817}, {MKDIR_path=}, {uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=}, {client=CTX_ID:4b47408f-323c-4c6a-9a20-2ae2a3a2cdb8-GRAPH_ID:3-PID:2291-HOST:srv-32-PC_NAME:ssd-volume-client-0-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid argument}]

==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:45:13.525581 +0000] E [MSGID: 113002] [posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null for (null) [Invalid argument]
[2021-03-19 11:45:13.525711 +0000] E [MSGID: 115056] [server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server: MKDIR info [{frame=10055}, {MKDIR_path=}, {uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=}, {client=CTX_ID:4b47408f-323c-4c6a-9a20-2ae2a3a2cdb8-GRAPH_ID:3-PID:2291-HOST:srv-32-PC_NAME:ssd-volume-client-3-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid argument}]

==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:45:38.829803 +0000] E [MSGID: 115056] [server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server: MKDIR info [{frame=11820}, {MKDIR_path=}, {uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=}, {client=CTX_ID:974f4637-64ef-42e6-afad-1dc9c67c4a43-GRAPH_ID:3-PID:2062-HOST:srv-33-PC_NAME:ssd-volume-client-0-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid argument}]

==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:45:38.829970 +0000] E [MSGID: 115056] [server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server: MKDIR info [{frame=10058}, {MKDIR_path=}, {uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=}, {client=CTX_ID:974f4637-64ef-42e6-afad-1dc9c67c4a43-GRAPH_ID:3-PID:2062-HOST:srv-33-PC_NAME:ssd-volume-client-3-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid argument}]

==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:45:38.829721 +0000] E [MSGID: 113002] [posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null for (null) [Invalid argument]

==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:45:38.829883 +0000] E [MSGID: 113002] [posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null for (null) [Invalid argument]
[2021-03-19 11:45:50.005995 +0000] E [MSGID: 113002] [posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null for (null) [Invalid argument]
[2021-03-19 11:45:50.006115 +0000] E [MSGID: 115056] [server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server: MKDIR info [{frame=10082}, {MKDIR_path=}, {uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=}, {client=CTX_ID:4945546f-f368-4fa7-8bfc-3dd7abda5d1b-GRAPH_ID:3-PID:2486-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid argument}]

==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:45:50.006096 +0000] E [MSGID: 113002] [posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null for (null) [Invalid argument]
[2021-03-19 11:45:50.006212 +0000] E [MSGID: 115056] [server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server: MKDIR info [{frame=11844}, {MKDIR_path=}, {uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=}, {client=CTX_ID:4945546f-f368-4fa7-8bfc-3dd7abda5d1b-GRAPH_ID:3-PID:2486-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid argument}]

==> /var/log/glusterfs/glustershd.log <==
[2021-03-19 11:45:50.006255 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-3: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]
[2021-03-19 11:45:50.006352 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-0: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]
[2021-03-19 11:45:50.006408 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-4: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]
[2021-03-19 11:45:50.006505 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-5: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]
[2021-03-19 11:45:50.006562 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-1: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]
[2021-03-19 11:45:50.006645 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-2: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]



Everything seems to be ok
root@srv-31:~# gluster volume status ssd-volume
Status of volume: ssd-volume
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick srv-31.lan.example.com:/data/glusterfs/ssd/brick1/brick               49156     0          Y       9242
Brick srv-32.lan.example.com:/data/glusterfs/ssd/brick1/brick               49156     0          Y       7968
Brick srv-33.lan.example.com:/data/glusterfs/ssd/brick1/brick               49156     0          Y       8211
Brick srv-31.lan.example.com:/data/glusterfs/ssd/brick2/brick               49157     0          Y       9258
Brick srv-32.lan.example.com:/data/glusterfs/ssd/brick2/brick               49157     0          Y       7984
Brick srv-33.lan.example.com:/data/glusterfs/ssd/brick2/brick               49157     0          Y       8227
Self-heal Daemon on localhost               N/A       N/A        Y       2486
Self-heal Daemon on srv-32.lan.example.com                                  N/A       N/A        Y       2291
Self-heal Daemon on srv-33.lan.example.com                                  N/A       N/A        Y       2062

Task Status of Volume ssd-volume
------------------------------------------------------------------------------
There are no active volume tasks




root@srv-31:~# gluster volume info ssd-volume
Volume Name: ssd-volume
Type: Distributed-Replicate
Volume ID: a6f3426f-5b33-404b-ab61-581e24e0c36d
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: srv-31.lan.example.com:/data/glusterfs/ssd/brick1/brick
Brick2: srv-32.lan.example.com:/data/glusterfs/ssd/brick1/brick
Brick3: srv-33.lan.example.com:/data/glusterfs/ssd/brick1/brick
Brick4: srv-31.lan.example.com:/data/glusterfs/ssd/brick2/brick
Brick5: srv-32.lan.example.com:/data/glusterfs/ssd/brick2/brick
Brick6: srv-33.lan.example.com:/data/glusterfs/ssd/brick2/brick
Options Reconfigured:
server.allow-insecure: on
storage.owner-gid: 64055
storage.owner-uid: 64055
ssl.cipher-list: HIGH:!SSLv2
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
storage.fips-mode-rchecksum: on
client.ssl: off
server.ssl: off
auth.ssl-allow: *


root@srv-31:~# gluster peer status
Number of Peers: 2

Hostname: srv-32.lan.example.com
Uuid: 0a11bfb9-5821-42f0-8ef3-18b36bfcfa8a
State: Peer in Cluster (Connected)

Hostname: srv-33.lan.example.com
Uuid: 111d93d2-64aa-463f-8f33-7b53a3f05cab
State: Peer in Cluster (Connected)


root@srv-31:~# gluster pool list
UUID                                    Hostname                                State
0a11bfb9-5821-42f0-8ef3-18b36bfcfa8a    srv-32.lan.example.com                  Connected
111d93d2-64aa-463f-8f33-7b53a3f05cab    srv-33.lan.example.com                  Connected
67e9da67-90c5-486b-acb1-eb91abebcefb    localhost                               Connected

root@srv-32:~# gluster pool list
UUID                                    Hostname                                State
111d93d2-64aa-463f-8f33-7b53a3f05cab    srv-33.lan.example.com                  Connected
67e9da67-90c5-486b-acb1-eb91abebcefb    10.0.0.31                               Connected
0a11bfb9-5821-42f0-8ef3-18b36bfcfa8a    localhost                               Connected

root@srv-33:~# gluster pool list
UUID                                    Hostname                                State
0a11bfb9-5821-42f0-8ef3-18b36bfcfa8a    srv-32.lan.example.com                  Connected
67e9da67-90c5-486b-acb1-eb91abebcefb    10.0.0.31                               Connected
111d93d2-64aa-463f-8f33-7b53a3f05cab    localhost                               Connected


root@srv-31:~# cat /etc/glusterfs/glusterd.vol
volume management
    type mgmt/glusterd
    option working-directory /var/lib/glusterd
    option transport-type socket
    option transport.socket.keepalive-time 10
    option transport.socket.keepalive-interval 2
    option transport.socket.read-fail-log off
    option transport.socket.listen-port 24007
    option ping-timeout 0
    option event-threads 1
#   option lock-timer 180
#   option transport.address-family inet6
#   option base-port 49152
    option max-port  60999
    option rpc-auth-allow-insecure on
end-volume

root@srv-31:~# ss -lntp
State     Recv-Q    Send-Q       Local Address:Port        Peer Address:Port                                                                                   
LISTEN    0         128                0.0.0.0:49153            0.0.0.0:*        users:(("glusterfsd",pid=2455,fd=11))                                         
LISTEN    0         128                0.0.0.0:49156            0.0.0.0:*        users:(("glusterfsd",pid=9242,fd=11))                                         
LISTEN    0         128                0.0.0.0:49157            0.0.0.0:*        users:(("glusterfsd",pid=9258,fd=11))                                         
LISTEN    0         128                0.0.0.0:24007            0.0.0.0:*        users:(("glusterd",pid=1954,fd=10))                                           
LISTEN    0         128                0.0.0.0:111              0.0.0.0:*        users:(("rpcbind",pid=1716,fd=4),("systemd",pid=1,fd=36))                     
LISTEN    0         128                0.0.0.0:2224             0.0.0.0:*        users:(("pcsd",pid=2126,fd=4))                                                
LISTEN    0         128                0.0.0.0:22               0.0.0.0:*        users:(("sshd",pid=1963,fd=3))                                                
LISTEN    0         20               127.0.0.1:25               0.0.0.0:*        users:(("exim4",pid=2377,fd=3))                                               
LISTEN    0         128                0.0.0.0:49152            0.0.0.0:*        users:(("glusterfsd",pid=2446,fd=11))                                         
LISTEN    0         128                   [::]:111                 [::]:*        users:(("rpcbind",pid=1716,fd=6),("systemd",pid=1,fd=38))                     
LISTEN    0         128                   [::]:2224                [::]:*        users:(("pcsd",pid=2126,fd=5))

root@srv-31:~# ss -tnp | grep gluster
ESTAB   0         0                127.0.0.1:49149            127.0.1.1:24007    users:(("glusterfsd",pid=2446,fd=9))                                          
ESTAB   0         0                10.0.0.31:49144            10.0.0.33:49153    users:(("glusterfs",pid=2486,fd=17))                                          
ESTAB   0         0                127.0.0.1:24007            127.0.0.1:49145    users:(("glusterd",pid=1954,fd=26))                                           
ESTAB   0         0                127.0.1.1:49157            127.0.0.1:49140    users:(("glusterfsd",pid=9258,fd=10))                                         
ESTAB   0         0                127.0.0.1:49137            127.0.1.1:49156    users:(("glusterfs",pid=2917,fd=10))                                          
ESTAB   0         0                10.0.0.31:49146            10.0.0.32:49152    users:(("glusterfs",pid=2486,fd=25))                                          
ESTAB   0         0                10.0.0.31:49153            10.0.0.33:49147    users:(("glusterfsd",pid=2455,fd=270))                                        
ESTAB   0         0                127.0.1.1:24007            127.0.0.1:49148    users:(("glusterd",pid=1954,fd=12))                                           
ESTAB   0         0                10.0.0.31:49157            10.0.0.33:49141    users:(("glusterfsd",pid=9258,fd=271))                                        
ESTAB   0         0                127.0.1.1:24007            127.0.0.1:49131    users:(("glusterd",pid=1954,fd=31))                                           
ESTAB   0         0                10.0.0.31:49152            10.0.0.32:49146    users:(("glusterfsd",pid=2446,fd=271))                                        
ESTAB   0         0                10.0.0.31:49134            10.0.0.33:24007    users:(("glusterd",pid=1954,fd=33))                                           
ESTAB   0         0                10.0.0.31:49127            10.0.0.33:49157    users:(("glusterfs",pid=2917,fd=11))                                          
ESTAB   0         0                127.0.0.1:49142            127.0.1.1:49156    users:(("glusterfs",pid=2486,fd=15))                                          
ESTAB   0         0                127.0.0.1:49135            127.0.1.1:49157    users:(("glusterfs",pid=2917,fd=14))                                          
ESTAB   0         0                127.0.0.1:49145            127.0.0.1:24007    users:(("glusterfs",pid=2486,fd=9))                                           
ESTAB   0         0                127.0.1.1:49157            127.0.0.1:49135    users:(("glusterfsd",pid=9258,fd=272))                                        
ESTAB   0         0                127.0.1.1:49152            127.0.0.1:49141    users:(("glusterfsd",pid=2446,fd=10))                                         
ESTAB   0         0                10.0.0.31:49157            10.0.0.32:49144    users:(("glusterfsd",pid=9258,fd=270))                                        
ESTAB   0         0                127.0.1.1:24007            127.0.0.1:49132    users:(("glusterd",pid=1954,fd=25))                                           
ESTAB   0         0                127.0.0.1:49148            127.0.1.1:24007    users:(("glusterfsd",pid=2455,fd=9))                                          
ESTAB   0         0                10.0.0.31:49118            10.0.0.32:49156    users:(("glusterfs",pid=2486,fd=13))                                          
ESTAB   0         0                127.0.1.1:49156            127.0.0.1:49137    users:(("glusterfsd",pid=9242,fd=272))                                        
ESTAB   0         0                10.0.0.31:49121            10.0.0.33:49156    users:(("glusterfs",pid=2917,fd=13))                                          
ESTAB   0         0                127.0.0.1:49132            127.0.1.1:24007    users:(("glusterfsd",pid=9242,fd=9))                                          
ESTAB   0         0                127.0.0.1:49141            127.0.1.1:49152    users:(("glusterfs",pid=2486,fd=22))                                          
ESTAB   0         0                10.0.0.31:49156            10.0.0.32:49148    users:(("glusterfsd",pid=9242,fd=270))                                        
ESTAB   0         0                10.0.0.31:49152            10.0.0.33:49149    users:(("glusterfsd",pid=2446,fd=270))                                        
ESTAB   0         0                127.0.0.1:49140            127.0.1.1:49157    users:(("glusterfs",pid=2486,fd=11))                                          
ESTAB   0         0                10.0.0.31:49123            10.0.0.32:49157    users:(("glusterfs",pid=2917,fd=12))                                          
ESTAB   0         0                10.0.0.31:49141            10.0.0.33:49152    users:(("glusterfs",pid=2486,fd=23))                                          
ESTAB   0         0                127.0.0.1:49131            127.0.1.1:24007    users:(("glusterfsd",pid=9258,fd=9))                                          
ESTAB   0         0                10.0.0.31:49153            10.0.0.32:49145    users:(("glusterfsd",pid=2455,fd=271))                                        
ESTAB   0         0                10.0.0.31:49117            10.0.0.33:49156    users:(("glusterfs",pid=2486,fd=5))                                           
ESTAB   0         0                10.0.0.31:49114            10.0.0.32:49157    users:(("glusterfs",pid=2486,fd=16))                                          
ESTAB   0         0                127.0.1.1:49153            127.0.0.1:49138    users:(("glusterfsd",pid=2455,fd=10))                                         
ESTAB   0         0                10.0.0.31:49143            10.0.0.32:24007    users:(("glusterfs",pid=2917,fd=9))                                           
ESTAB   0         0                10.0.0.31:49113            10.0.0.33:49157    users:(("glusterfs",pid=2486,fd=18))                                          
ESTAB   0         0                10.0.0.31:49156            10.0.0.33:49143    users:(("glusterfsd",pid=9242,fd=271))                                        
ESTAB   0         0                127.0.0.1:49138            127.0.1.1:49153    users:(("glusterfs",pid=2486,fd=19))                                          
ESTAB   0         0                10.0.0.31:24007            10.0.0.33:49150    users:(("glusterd",pid=1954,fd=32))                                           
ESTAB   0         0                10.0.0.31:49131            10.0.0.32:49156    users:(("glusterfs",pid=2917,fd=7))                                           
ESTAB   0         0                10.0.0.31:49145            10.0.0.32:49153    users:(("glusterfs",pid=2486,fd=8))                                           
ESTAB   0         0                127.0.1.1:24007            127.0.0.1:49149    users:(("glusterd",pid=1954,fd=13))                                           
ESTAB   0         0                10.0.0.31:24007            10.0.0.32:49150    users:(("glusterd",pid=1954,fd=7))                                            
ESTAB   0         0                10.0.0.31:49151            10.0.0.32:24007    users:(("glusterd",pid=1954,fd=8))                                            
ESTAB   0         0                127.0.1.1:49156            127.0.0.1:49142    users:(("glusterfsd",pid=9242,fd=10))

hostnames resolve correctly:
root@srv-31:~# for i in {31..33}; do host srv-$i.lan.example.com; done
srv-31.lan.example.com has address 10.0.0.31
srv-32.lan.example.com has address 10.0.0.32
srv-33.lan.example.com has address 10.0.0.33


And network conectivity is ok:
root@srv-31:~# for i in {31..33}; do ping -c 1 -q srv-$i.lan.example.com | echo "srv-$i ok"; done
srv-31 ok
srv-32 ok
srv-33 ok

root@srv-31:~# for i in {31..33}; do nc -v srv-$i.lan.example.com 24007; done
Connection to srv-31.lan.example.com 24007 port [tcp/*] succeeded!

Connection to srv-32.lan.example.com 24007 port [tcp/*] succeeded!

Connection to srv-33.lan.example.com 24007 port [tcp/*] succeeded!


root@srv-31:~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination        

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination



________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux