Hi,
I have just configure a distributed-replicate 2x3 gluster volume.
It seems to be working fine when I mount the volume with fuse but native gluster access with qemu-img is much slower and outputs errors. Log files are filled with warning and errors.
Starting vm with the created images is possible but it takes much more time when using gluster native access. Nevertheless, after starting up, there seems to be no penalty in using the gluster native access image being some times faster.
Can anyone, please, help me in figuring out if there is and where is the problem?
Thanks
rm
It seems to be working fine when I mount it with fuse:
root@srv-31:~# echo 'This is a test' > /mnt/ssd-volume/test/text.txt
root@srv-31:~# ll /data/glusterfs/ssd/brick[12]/brick/test/*
-rw-r--r-- 2 root root 15 mar 17 14:08 /data/glusterfs/ssd/brick1/brick/test/text.txt
root@srv-31:~# time qemu-img create -f qcow2 /mnt/ssd-volume/libvirt/images/test_5.qcow2.img 64G
Formatting '/mnt/ssd-volume/libvirt/images/test_5.qcow2.img', fmt=qcow2 size=68719476736 cluster_size=65536 lazy_refcounts=off refcount_bits=16
real 0m0,204s
user 0m0,009s
sys 0m0,009s
Creating a image file onteh mouted gluster volume is quick but I get some warnings on log files:
root@srv-31:~# tail -f /var/log/glusterfs/glusterd.log /var/log/glusterfs/glustershd.log /var/log/glusterfs/bricks/data-glusterfs-ssd-brick[12]-brick.log
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:37:11.301388 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.303694 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.305808 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.307868 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.309863 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.312139 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.317780 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.319007 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.321096 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.323111 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.325284 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.327319 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.329488 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.331792 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.333597 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.335864 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.339551 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.341110 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.343079 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.345174 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.349569 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.351920 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.353282 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.355272 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.357467 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.359584 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.361771 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.364013 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.366094 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.368094 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.370261 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.372389 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.375550 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
(...)
[2021-03-19 11:37:11.477472 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.479897 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.482318 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
[2021-03-19 11:37:11.484489 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]
Native gluster access with qemu-img is much slower and outputs errors:
root@srv-31:~# time qemu-img create -f qcow2 gluster://srv-31.lan.example.com/ssd-volume/libvirt/images/test_6.qcow2.img 64G
[2021-03-19 11:47:42.490844 +0000] I [io-stats.c:3708:ios_sample_buf_size_configure] 0-ssd-volume: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0
[2021-03-19 11:47:42.615942 +0000] E [MSGID: 108006] [afr-common.c:6146:__afr_handle_child_down_event] 0-ssd-volume-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up.
[2021-03-19 11:47:42.617033 +0000] E [MSGID: 108006] [afr-common.c:6146:__afr_handle_child_down_event] 0-ssd-volume-replicate-1: All subvolumes are down. Going offline until at least one of them comes back up.
[2021-03-19 11:47:52.502174 +0000] I [io-stats.c:4038:fini] 0-ssd-volume: io-stats translator unloaded
[2021-03-19 11:47:53.514364 +0000] I [io-stats.c:3708:ios_sample_buf_size_configure] 0-ssd-volume: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0
[2021-03-19 11:47:53.654336 +0000] E [MSGID: 108006] [afr-common.c:6146:__afr_handle_child_down_event] 0-ssd-volume-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up.
[2021-03-19 11:47:53.655251 +0000] E [MSGID: 108006] [afr-common.c:6146:__afr_handle_child_down_event] 0-ssd-volume-replicate-1: All subvolumes are down. Going offline until at least one of them comes back up.
[2021-03-19 11:48:03.525947 +0000] I [io-stats.c:4038:fini] 0-ssd-volume: io-stats translator unloaded
real 0m22,068s
user 0m0,055s
sys 0m0,043s
In both cases images are written correctly (and usable):
root@srv-31:~# l /data/glusterfs/ssd/brick[12]/brick/libvirt/images/test_[56]*
-rw-r--r-- 2 root root 193K mar 19 11:37 /data/glusterfs/ssd/brick2/brick/libvirt/images/test_5.qcow2.img
-rw------- 2 root root 193K mar 19 11:47 /data/glusterfs/ssd/brick2/brick/libvirt/images/test_6.qcow2.img
But I get lots of warnings on log files:
root@srv-31:~# tail -f /var/log/glusterfs/glusterd.log /var/log/glusterfs/glustershd.log /var/log/glusterfs/bricks/data-glusterfs-ssd-brick[12]-brick.log
==> /var/log/glusterfs/glusterd.log <==
[2021-03-19 11:47:42.483736 +0000] I [MSGID: 106496] [glusterd-handshake.c:969:__server_getspec] 0-management: Received mount request for volume ssd-volume
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:47:42.498908 +0000] I [addr.c:54:compare_addr_and_update] 0-/data/glusterfs/ssd/brick1/brick: allowed = "*", received addr = "127.0.0.1"
[2021-03-19 11:47:42.498967 +0000] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 80c9a2bf-6fad-4023-8458-a175cef7f681
[2021-03-19 11:47:42.498989 +0000] I [MSGID: 115029] [server-handshake.c:561:server_setvolume] 0-ssd-volume-server: accepted client from CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0 (version: 9.0) with subvol /data/glusterfs/ssd/brick1/brick
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:47:42.502029 +0000] I [addr.c:54:compare_addr_and_update] 0-/data/glusterfs/ssd/brick2/brick: allowed = "*", received addr = "127.0.0.1"
[2021-03-19 11:47:42.502087 +0000] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 80c9a2bf-6fad-4023-8458-a175cef7f681
[2021-03-19 11:47:42.502109 +0000] I [MSGID: 115029] [server-handshake.c:561:server_setvolume] 0-ssd-volume-server: accepted client from CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0 (version: 9.0) with subvol /data/glusterfs/ssd/brick2/brick
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:47:42.615182 +0000] W [socket.c:767:__socket_rwv] 0-tcp.ssd-volume-server: readv on 127.0.0.1:49151 failed (No data available)
[2021-03-19 11:47:42.615258 +0000] I [MSGID: 115036] [server.c:500:server_rpc_notify] 0-ssd-volume-server: disconnecting connection [{client-uid=CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0}]
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:47:42.615215 +0000] W [socket.c:767:__socket_rwv] 0-tcp.ssd-volume-server: readv on 127.0.0.1:49150 failed (No data available)
[2021-03-19 11:47:42.615316 +0000] I [MSGID: 115036] [server.c:500:server_rpc_notify] 0-ssd-volume-server: disconnecting connection [{client-uid=CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0}]
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:47:42.615459 +0000] I [MSGID: 101055] [client_t.c:397:gf_client_unref] 0-ssd-volume-server: Shutting down connection CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:47:42.615524 +0000] I [MSGID: 101055] [client_t.c:397:gf_client_unref] 0-ssd-volume-server: Shutting down connection CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:47:53.522422 +0000] I [addr.c:54:compare_addr_and_update] 0-/data/glusterfs/ssd/brick1/brick: allowed = "*", received addr = "127.0.0.1"
[2021-03-19 11:47:53.522480 +0000] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 80c9a2bf-6fad-4023-8458-a175cef7f681
[2021-03-19 11:47:53.522502 +0000] I [MSGID: 115029] [server-handshake.c:561:server_setvolume] 0-ssd-volume-server: accepted client from CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0 (version: 9.0) with subvol /data/glusterfs/ssd/brick1/brick
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:47:53.525344 +0000] I [addr.c:54:compare_addr_and_update] 0-/data/glusterfs/ssd/brick2/brick: allowed = "*", received addr = "127.0.0.1"
[2021-03-19 11:47:53.525396 +0000] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 80c9a2bf-6fad-4023-8458-a175cef7f681
[2021-03-19 11:47:53.525419 +0000] I [MSGID: 115029] [server-handshake.c:561:server_setvolume] 0-ssd-volume-server: accepted client from CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0 (version: 9.0) with subvol /data/glusterfs/ssd/brick2/brick
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:47:53.653652 +0000] W [socket.c:767:__socket_rwv] 0-tcp.ssd-volume-server: readv on 127.0.0.1:49146 failed (No data available)
[2021-03-19 11:47:53.653731 +0000] I [MSGID: 115036] [server.c:500:server_rpc_notify] 0-ssd-volume-server: disconnecting connection [{client-uid=CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0}]
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:47:53.653752 +0000] W [socket.c:767:__socket_rwv] 0-tcp.ssd-volume-server: readv on 127.0.0.1:49144 failed (No data available)
[2021-03-19 11:47:53.653840 +0000] I [MSGID: 115036] [server.c:500:server_rpc_notify] 0-ssd-volume-server: disconnecting connection [{client-uid=CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0}]
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:47:53.653928 +0000] I [MSGID: 101055] [client_t.c:397:gf_client_unref] 0-ssd-volume-server: Shutting down connection CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:47:53.654040 +0000] I [MSGID: 101055] [client_t.c:397:gf_client_unref] 0-ssd-volume-server: Shutting down connection CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0
==> /var/log/glusterfs/glusterd.log <==
[2021-03-19 11:47:53.507207 +0000] I [MSGID: 106496] [glusterd-handshake.c:969:__server_getspec] 0-management: Received mount request for volume ssd-volume
Starting a vm with a virtual disk on the mounted gluster volume (-drive file=/mnt/ssd-volume/libvirt/images/test_6.qcow2.img)is quick:
root@srv-31:~# time virsh start vm-11
Domain vm-11 started
real 0m0,277s
user 0m0,020s
sys 0m0,004s
Statring a vm with a virtual disk on gluster volume with native access (-drive file=gluster://srv-31.lan.example.com:24007/ssd-volume/libvirt/images/test_7.qcow2.img)is much slower:
root@srv-31:~# time virsh start vm-11
Domain vm-11 started
real 0m22,339s
user 0m0,021s
sys 0m0,007s
From time to time, even without user operations, log files report errors:
root@srv-31:~# tail -f /var/log/glusterfs/glusterd.log /var/log/glusterfs/glustershd.log /var/log/glusterfs/bricks/data-glusterfs-ssd-brick[12]-brick.log
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:45:09.525079 +0000] E [MSGID: 113002] [posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null for (null) [Invalid argument]
[2021-03-19 11:45:09.525212 +0000] E [MSGID: 115056] [server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server: MKDIR info [{frame=11817}, {MKDIR_path=}, {uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=}, {client=CTX_ID:4b47408f-323c-4c6a-9a20-2ae2a3a2cdb8-GRAPH_ID:3-PID:2291-HOST:srv-32-PC_NAME:ssd-volume-client-0-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid argument}]
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:45:13.525581 +0000] E [MSGID: 113002] [posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null for (null) [Invalid argument]
[2021-03-19 11:45:13.525711 +0000] E [MSGID: 115056] [server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server: MKDIR info [{frame=10055}, {MKDIR_path=}, {uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=}, {client=CTX_ID:4b47408f-323c-4c6a-9a20-2ae2a3a2cdb8-GRAPH_ID:3-PID:2291-HOST:srv-32-PC_NAME:ssd-volume-client-3-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid argument}]
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:45:38.829803 +0000] E [MSGID: 115056] [server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server: MKDIR info [{frame=11820}, {MKDIR_path=}, {uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=}, {client=CTX_ID:974f4637-64ef-42e6-afad-1dc9c67c4a43-GRAPH_ID:3-PID:2062-HOST:srv-33-PC_NAME:ssd-volume-client-0-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid argument}]
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:45:38.829970 +0000] E [MSGID: 115056] [server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server: MKDIR info [{frame=10058}, {MKDIR_path=}, {uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=}, {client=CTX_ID:974f4637-64ef-42e6-afad-1dc9c67c4a43-GRAPH_ID:3-PID:2062-HOST:srv-33-PC_NAME:ssd-volume-client-3-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid argument}]
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:45:38.829721 +0000] E [MSGID: 113002] [posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null for (null) [Invalid argument]
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:45:38.829883 +0000] E [MSGID: 113002] [posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null for (null) [Invalid argument]
[2021-03-19 11:45:50.005995 +0000] E [MSGID: 113002] [posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null for (null) [Invalid argument]
[2021-03-19 11:45:50.006115 +0000] E [MSGID: 115056] [server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server: MKDIR info [{frame=10082}, {MKDIR_path=}, {uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=}, {client=CTX_ID:4945546f-f368-4fa7-8bfc-3dd7abda5d1b-GRAPH_ID:3-PID:2486-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid argument}]
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:45:50.006096 +0000] E [MSGID: 113002] [posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null for (null) [Invalid argument]
[2021-03-19 11:45:50.006212 +0000] E [MSGID: 115056] [server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server: MKDIR info [{frame=11844}, {MKDIR_path=}, {uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=}, {client=CTX_ID:4945546f-f368-4fa7-8bfc-3dd7abda5d1b-GRAPH_ID:3-PID:2486-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid argument}]
==> /var/log/glusterfs/glustershd.log <==
[2021-03-19 11:45:50.006255 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-3: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]
[2021-03-19 11:45:50.006352 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-0: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]
[2021-03-19 11:45:50.006408 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-4: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]
[2021-03-19 11:45:50.006505 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-5: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]
[2021-03-19 11:45:50.006562 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-1: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]
[2021-03-19 11:45:50.006645 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-2: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]
Everything seems to be ok
root@srv-31:~# gluster volume status ssd-volume
Status of volume: ssd-volume
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick srv-31.lan.example.com:/data/glusterfs/ssd/brick1/brick 49156 0 Y 9242
Brick srv-32.lan.example.com:/data/glusterfs/ssd/brick1/brick 49156 0 Y 7968
Brick srv-33.lan.example.com:/data/glusterfs/ssd/brick1/brick 49156 0 Y 8211
Brick srv-31.lan.example.com:/data/glusterfs/ssd/brick2/brick 49157 0 Y 9258
Brick srv-32.lan.example.com:/data/glusterfs/ssd/brick2/brick 49157 0 Y 7984
Brick srv-33.lan.example.com:/data/glusterfs/ssd/brick2/brick 49157 0 Y 8227
Self-heal Daemon on localhost N/A N/A Y 2486
Self-heal Daemon on srv-32.lan.example.com N/A N/A Y 2291
Self-heal Daemon on srv-33.lan.example.com N/A N/A Y 2062
Task Status of Volume ssd-volume
------------------------------------------------------------------------------
There are no active volume tasks
root@srv-31:~# gluster volume info ssd-volume
Volume Name: ssd-volume
Type: Distributed-Replicate
Volume ID: a6f3426f-5b33-404b-ab61-581e24e0c36d
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: srv-31.lan.example.com:/data/glusterfs/ssd/brick1/brick
Brick2: srv-32.lan.example.com:/data/glusterfs/ssd/brick1/brick
Brick3: srv-33.lan.example.com:/data/glusterfs/ssd/brick1/brick
Brick4: srv-31.lan.example.com:/data/glusterfs/ssd/brick2/brick
Brick5: srv-32.lan.example.com:/data/glusterfs/ssd/brick2/brick
Brick6: srv-33.lan.example.com:/data/glusterfs/ssd/brick2/brick
Options Reconfigured:
server.allow-insecure: on
storage.owner-gid: 64055
storage.owner-uid: 64055
ssl.cipher-list: HIGH:!SSLv2
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
storage.fips-mode-rchecksum: on
client.ssl: off
server.ssl: off
auth.ssl-allow: *
root@srv-31:~# gluster peer status
Number of Peers: 2
Hostname: srv-32.lan.example.com
Uuid: 0a11bfb9-5821-42f0-8ef3-18b36bfcfa8a
State: Peer in Cluster (Connected)
Hostname: srv-33.lan.example.com
Uuid: 111d93d2-64aa-463f-8f33-7b53a3f05cab
State: Peer in Cluster (Connected)
root@srv-31:~# gluster pool list
UUID Hostname State
0a11bfb9-5821-42f0-8ef3-18b36bfcfa8a srv-32.lan.example.com Connected
111d93d2-64aa-463f-8f33-7b53a3f05cab srv-33.lan.example.com Connected
67e9da67-90c5-486b-acb1-eb91abebcefb localhost Connected
root@srv-32:~# gluster pool list
UUID Hostname State
111d93d2-64aa-463f-8f33-7b53a3f05cab srv-33.lan.example.com Connected
67e9da67-90c5-486b-acb1-eb91abebcefb 10.0.0.31 Connected
0a11bfb9-5821-42f0-8ef3-18b36bfcfa8a localhost Connected
root@srv-33:~# gluster pool list
UUID Hostname State
0a11bfb9-5821-42f0-8ef3-18b36bfcfa8a srv-32.lan.example.com Connected
67e9da67-90c5-486b-acb1-eb91abebcefb 10.0.0.31 Connected
111d93d2-64aa-463f-8f33-7b53a3f05cab localhost Connected
root@srv-31:~# cat /etc/glusterfs/glusterd.vol
volume management
type mgmt/glusterd
option working-directory /var/lib/glusterd
option transport-type socket
option transport.socket.keepalive-time 10
option transport.socket.keepalive-interval 2
option transport.socket.read-fail-log off
option transport.socket.listen-port 24007
option ping-timeout 0
option event-threads 1
# option lock-timer 180
# option transport.address-family inet6
# option base-port 49152
option max-port 60999
option rpc-auth-allow-insecure on
end-volume
root@srv-31:~# ss -lntp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 0.0.0.0:49153 0.0.0.0:* users:(("glusterfsd",pid=2455,fd=11))
LISTEN 0 128 0.0.0.0:49156 0.0.0.0:* users:(("glusterfsd",pid=9242,fd=11))
LISTEN 0 128 0.0.0.0:49157 0.0.0.0:* users:(("glusterfsd",pid=9258,fd=11))
LISTEN 0 128 0.0.0.0:24007 0.0.0.0:* users:(("glusterd",pid=1954,fd=10))
LISTEN 0 128 0.0.0.0:111 0.0.0.0:* users:(("rpcbind",pid=1716,fd=4),("systemd",pid=1,fd=36))
LISTEN 0 128 0.0.0.0:2224 0.0.0.0:* users:(("pcsd",pid=2126,fd=4))
LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=1963,fd=3))
LISTEN 0 20 127.0.0.1:25 0.0.0.0:* users:(("exim4",pid=2377,fd=3))
LISTEN 0 128 0.0.0.0:49152 0.0.0.0:* users:(("glusterfsd",pid=2446,fd=11))
LISTEN 0 128 [::]:111 [::]:* users:(("rpcbind",pid=1716,fd=6),("systemd",pid=1,fd=38))
LISTEN 0 128 [::]:2224 [::]:* users:(("pcsd",pid=2126,fd=5))
root@srv-31:~# ss -tnp | grep gluster
ESTAB 0 0 127.0.0.1:49149 127.0.1.1:24007 users:(("glusterfsd",pid=2446,fd=9))
ESTAB 0 0 10.0.0.31:49144 10.0.0.33:49153 users:(("glusterfs",pid=2486,fd=17))
ESTAB 0 0 127.0.0.1:24007 127.0.0.1:49145 users:(("glusterd",pid=1954,fd=26))
ESTAB 0 0 127.0.1.1:49157 127.0.0.1:49140 users:(("glusterfsd",pid=9258,fd=10))
ESTAB 0 0 127.0.0.1:49137 127.0.1.1:49156 users:(("glusterfs",pid=2917,fd=10))
ESTAB 0 0 10.0.0.31:49146 10.0.0.32:49152 users:(("glusterfs",pid=2486,fd=25))
ESTAB 0 0 10.0.0.31:49153 10.0.0.33:49147 users:(("glusterfsd",pid=2455,fd=270))
ESTAB 0 0 127.0.1.1:24007 127.0.0.1:49148 users:(("glusterd",pid=1954,fd=12))
ESTAB 0 0 10.0.0.31:49157 10.0.0.33:49141 users:(("glusterfsd",pid=9258,fd=271))
ESTAB 0 0 127.0.1.1:24007 127.0.0.1:49131 users:(("glusterd",pid=1954,fd=31))
ESTAB 0 0 10.0.0.31:49152 10.0.0.32:49146 users:(("glusterfsd",pid=2446,fd=271))
ESTAB 0 0 10.0.0.31:49134 10.0.0.33:24007 users:(("glusterd",pid=1954,fd=33))
ESTAB 0 0 10.0.0.31:49127 10.0.0.33:49157 users:(("glusterfs",pid=2917,fd=11))
ESTAB 0 0 127.0.0.1:49142 127.0.1.1:49156 users:(("glusterfs",pid=2486,fd=15))
ESTAB 0 0 127.0.0.1:49135 127.0.1.1:49157 users:(("glusterfs",pid=2917,fd=14))
ESTAB 0 0 127.0.0.1:49145 127.0.0.1:24007 users:(("glusterfs",pid=2486,fd=9))
ESTAB 0 0 127.0.1.1:49157 127.0.0.1:49135 users:(("glusterfsd",pid=9258,fd=272))
ESTAB 0 0 127.0.1.1:49152 127.0.0.1:49141 users:(("glusterfsd",pid=2446,fd=10))
ESTAB 0 0 10.0.0.31:49157 10.0.0.32:49144 users:(("glusterfsd",pid=9258,fd=270))
ESTAB 0 0 127.0.1.1:24007 127.0.0.1:49132 users:(("glusterd",pid=1954,fd=25))
ESTAB 0 0 127.0.0.1:49148 127.0.1.1:24007 users:(("glusterfsd",pid=2455,fd=9))
ESTAB 0 0 10.0.0.31:49118 10.0.0.32:49156 users:(("glusterfs",pid=2486,fd=13))
ESTAB 0 0 127.0.1.1:49156 127.0.0.1:49137 users:(("glusterfsd",pid=9242,fd=272))
ESTAB 0 0 10.0.0.31:49121 10.0.0.33:49156 users:(("glusterfs",pid=2917,fd=13))
ESTAB 0 0 127.0.0.1:49132 127.0.1.1:24007 users:(("glusterfsd",pid=9242,fd=9))
ESTAB 0 0 127.0.0.1:49141 127.0.1.1:49152 users:(("glusterfs",pid=2486,fd=22))
ESTAB 0 0 10.0.0.31:49156 10.0.0.32:49148 users:(("glusterfsd",pid=9242,fd=270))
ESTAB 0 0 10.0.0.31:49152 10.0.0.33:49149 users:(("glusterfsd",pid=2446,fd=270))
ESTAB 0 0 127.0.0.1:49140 127.0.1.1:49157 users:(("glusterfs",pid=2486,fd=11))
ESTAB 0 0 10.0.0.31:49123 10.0.0.32:49157 users:(("glusterfs",pid=2917,fd=12))
ESTAB 0 0 10.0.0.31:49141 10.0.0.33:49152 users:(("glusterfs",pid=2486,fd=23))
ESTAB 0 0 127.0.0.1:49131 127.0.1.1:24007 users:(("glusterfsd",pid=9258,fd=9))
ESTAB 0 0 10.0.0.31:49153 10.0.0.32:49145 users:(("glusterfsd",pid=2455,fd=271))
ESTAB 0 0 10.0.0.31:49117 10.0.0.33:49156 users:(("glusterfs",pid=2486,fd=5))
ESTAB 0 0 10.0.0.31:49114 10.0.0.32:49157 users:(("glusterfs",pid=2486,fd=16))
ESTAB 0 0 127.0.1.1:49153 127.0.0.1:49138 users:(("glusterfsd",pid=2455,fd=10))
ESTAB 0 0 10.0.0.31:49143 10.0.0.32:24007 users:(("glusterfs",pid=2917,fd=9))
ESTAB 0 0 10.0.0.31:49113 10.0.0.33:49157 users:(("glusterfs",pid=2486,fd=18))
ESTAB 0 0 10.0.0.31:49156 10.0.0.33:49143 users:(("glusterfsd",pid=9242,fd=271))
ESTAB 0 0 127.0.0.1:49138 127.0.1.1:49153 users:(("glusterfs",pid=2486,fd=19))
ESTAB 0 0 10.0.0.31:24007 10.0.0.33:49150 users:(("glusterd",pid=1954,fd=32))
ESTAB 0 0 10.0.0.31:49131 10.0.0.32:49156 users:(("glusterfs",pid=2917,fd=7))
ESTAB 0 0 10.0.0.31:49145 10.0.0.32:49153 users:(("glusterfs",pid=2486,fd=8))
ESTAB 0 0 127.0.1.1:24007 127.0.0.1:49149 users:(("glusterd",pid=1954,fd=13))
ESTAB 0 0 10.0.0.31:24007 10.0.0.32:49150 users:(("glusterd",pid=1954,fd=7))
ESTAB 0 0 10.0.0.31:49151 10.0.0.32:24007 users:(("glusterd",pid=1954,fd=8))
ESTAB 0 0 127.0.1.1:49156 127.0.0.1:49142 users:(("glusterfsd",pid=9242,fd=10))
hostnames resolve correctly:
root@srv-31:~# for i in {31..33}; do host srv-$i.lan.example.com; done
srv-31.lan.example.com has address 10.0.0.31
srv-32.lan.example.com has address 10.0.0.32
srv-33.lan.example.com has address 10.0.0.33
And network conectivity is ok:
root@srv-31:~# for i in {31..33}; do ping -c 1 -q srv-$i.lan.example.com | echo "srv-$i ok"; done
srv-31 ok
srv-32 ok
srv-33 ok
root@srv-31:~# for i in {31..33}; do nc -v srv-$i.lan.example.com 24007; done
Connection to srv-31.lan.example.com 24007 port [tcp/*] succeeded!
Connection to srv-32.lan.example.com 24007 port [tcp/*] succeeded!
Connection to srv-33.lan.example.com 24007 port [tcp/*] succeeded!
root@srv-31:~# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
