---------- Forwarded message ---------- From: Guido De Rosa <guido.derosa@xxxxxxxxxxx> Date: 2013/8/30 Subject: Re: QEMU (and other libgfapi client?) crashes on add-brick / replace-brick To: Anand Avati <anand.avati@xxxxxxxxx> Well, the add-brick issue is still here. Updated/recompiled GlusterFS to master 3a3441e and qemu to release 1.5.3. Qemu doesn't crash but the guest behaves as if the hard drive had been physically hot-detached: lots of I/O kernel errors, filesystem autoremounted readonly, non-cached data unavailable etc. Here are the logs: In the machine which runs qemu -drive file=gluster://localhost/gv/... and the gluster add-brick command (hostname: mgy-virt) : /usr/local/var/log/glusterfs/bricks/var-export-gluster-gv.log: Final graph: +------------------------------------------------------------------------------+ 1: volume gv-posix 2: type storage/posix 3: option glusterd-uuid 42ff1e51-7c77-4c70-9e1b-3e6207935bee 4: option directory /var/export/gluster/gv 5: option volume-id a562cb7c-0edf-4efa-afc6-80ea4e3fe978 6: end-volume 7: 8: volume gv-changelog 9: type features/changelog 10: option changelog-brick /var/export/gluster/gv 11: option changelog-dir /var/export/gluster/gv/.glusterfs/changelogs 12: subvolumes gv-posix 13: end-volume 14: 15: volume gv-access-control 16: type features/access-control 17: subvolumes gv-changelog 18: end-volume 19: 20: volume gv-locks 21: type features/locks 22: subvolumes gv-access-control 23: end-volume 24: 25: volume gv-io-threads 26: type performance/io-threads 27: subvolumes gv-locks 28: end-volume 29: 30: volume gv-index 31: type features/index 32: option index-base /var/export/gluster/gv/.glusterfs/indices 33: subvolumes gv-io-threads 34: end-volume 35: 36: volume gv-marker 37: type features/marker 38: option volume-uuid a562cb7c-0edf-4efa-afc6-80ea4e3fe978 39: option timestamp-file /var/lib/glusterd/vols/gv/marker.tstamp 40: option xtime off 41: option quota off 42: subvolumes gv-index 43: end-volume 44: 45: volume /var/export/gluster/gv 46: type debug/io-stats 47: option latency-measurement off 48: option count-fop-hits off 49: subvolumes gv-marker 50: end-volume 51: 52: volume gv-server 53: type protocol/server 54: option transport.socket.listen-port 49152 55: option rpc-auth.auth-glusterfs on 56: option rpc-auth.auth-unix on 57: option rpc-auth.auth-null on 58: option transport-type tcp 59: option auth.login./var/export/gluster/gv.allow ae4ffb2b-75fb-4b5a-b9d3-6c9e390fee03 60: option auth.login.ae4ffb2b-75fb-4b5a-b9d3-6c9e390fee03.password 041ee2e7-e8cf-4ecd-bba6-655348721610 61: option auth.addr./var/export/gluster/gv.allow * 62: subvolumes /var/export/gluster/gv 63: end-volume 64: +------------------------------------------------------------------------------+ Final graph: +------------------------------------------------------------------------------+ 1: volume gv-posix 2: type storage/posix 3: option glusterd-uuid 42ff1e51-7c77-4c70-9e1b-3e6207935bee 4: option directory /var/export/gluster/gv 5: option volume-id a562cb7c-0edf-4efa-afc6-80ea4e3fe978 6: end-volume 7: 8: volume gv-changelog 9: type features/changelog 10: option changelog-brick /var/export/gluster/gv 11: option changelog-dir /var/export/gluster/gv/.glusterfs/changelogs 12: subvolumes gv-posix 13: end-volume 14: 15: volume gv-access-control 16: type features/access-control 17: subvolumes gv-changelog 18: end-volume 19: 20: volume gv-locks 21: type features/locks 22: subvolumes gv-access-control 23: end-volume 24: 25: volume gv-io-threads 26: type performance/io-threads 27: subvolumes gv-locks 28: end-volume 29: 30: volume gv-index 31: type features/index 32: option index-base /var/export/gluster/gv/.glusterfs/indices 33: subvolumes gv-io-threads 34: end-volume 35: 36: volume gv-marker 37: type features/marker 38: option volume-uuid a562cb7c-0edf-4efa-afc6-80ea4e3fe978 39: option timestamp-file /var/lib/glusterd/vols/gv/marker.tstamp 40: option xtime off 41: option quota off 42: subvolumes gv-index 43: end-volume 44: 45: volume /var/export/gluster/gv 46: type debug/io-stats 47: option latency-measurement off 48: option count-fop-hits off 49: subvolumes gv-marker 50: end-volume 51: 52: volume gv-server 53: type protocol/server 54: option transport.socket.listen-port 49152 55: option rpc-auth.auth-glusterfs on 56: option rpc-auth.auth-unix on 57: option rpc-auth.auth-null on 58: option transport-type tcp 59: option auth.login./var/export/gluster/gv.allow ae4ffb2b-75fb-4b5a-b9d3-6c9e390fee03 60: option auth.login.ae4ffb2b-75fb-4b5a-b9d3-6c9e390fee03.password 041ee2e7-e8cf-4ecd-bba6-655348721610 61: option auth.addr./var/export/gluster/gv.allow * 62: subvolumes /var/export/gluster/gv 63: end-volume 64: +------------------------------------------------------------------------------+ Final graph: +------------------------------------------------------------------------------+ 1: volume gv-posix 2: type storage/posix 3: option glusterd-uuid 42ff1e51-7c77-4c70-9e1b-3e6207935bee 4: option directory /var/export/gluster/gv 5: option volume-id a562cb7c-0edf-4efa-afc6-80ea4e3fe978 6: end-volume 7: 8: volume gv-changelog 9: type features/changelog 10: option changelog-brick /var/export/gluster/gv 11: option changelog-dir /var/export/gluster/gv/.glusterfs/changelogs 12: subvolumes gv-posix 13: end-volume 14: 15: volume gv-access-control 16: type features/access-control 17: subvolumes gv-changelog 18: end-volume 19: 20: volume gv-locks 21: type features/locks 22: subvolumes gv-access-control 23: end-volume 24: 25: volume gv-io-threads 26: type performance/io-threads 27: subvolumes gv-locks 28: end-volume 29: 30: volume gv-index 31: type features/index 32: option index-base /var/export/gluster/gv/.glusterfs/indices 33: subvolumes gv-io-threads 34: end-volume 35: 36: volume gv-marker 37: type features/marker 38: option volume-uuid a562cb7c-0edf-4efa-afc6-80ea4e3fe978 39: option timestamp-file /var/lib/glusterd/vols/gv/marker.tstamp 40: option xtime off 41: option quota off 42: subvolumes gv-index 43: end-volume 44: 45: volume /var/export/gluster/gv 46: type debug/io-stats 47: option latency-measurement off 48: option count-fop-hits off 49: subvolumes gv-marker 50: end-volume 51: 52: volume gv-server 53: type protocol/server 54: option transport.socket.listen-port 49152 55: option rpc-auth.auth-glusterfs on 56: option rpc-auth.auth-unix on 57: option rpc-auth.auth-null on 58: option transport-type tcp 59: option auth.login./var/export/gluster/gv.allow ae4ffb2b-75fb-4b5a-b9d3-6c9e390fee03 60: option auth.login.ae4ffb2b-75fb-4b5a-b9d3-6c9e390fee03.password 041ee2e7-e8cf-4ecd-bba6-655348721610 61: option auth.addr./var/export/gluster/gv.allow * 62: subvolumes /var/export/gluster/gv 63: end-volume 64: +------------------------------------------------------------------------------+ Final graph: +------------------------------------------------------------------------------+ 1: volume gv-posix 2: type storage/posix 3: option glusterd-uuid 42ff1e51-7c77-4c70-9e1b-3e6207935bee 4: option directory /var/export/gluster/gv 5: option volume-id a562cb7c-0edf-4efa-afc6-80ea4e3fe978 6: end-volume 7: 8: volume gv-changelog 9: type features/changelog 10: option changelog-brick /var/export/gluster/gv 11: option changelog-dir /var/export/gluster/gv/.glusterfs/changelogs 12: subvolumes gv-posix 13: end-volume 14: 15: volume gv-access-control 16: type features/access-control 17: subvolumes gv-changelog 18: end-volume 19: 20: volume gv-locks 21: type features/locks 22: subvolumes gv-access-control 23: end-volume 24: 25: volume gv-io-threads 26: type performance/io-threads 27: subvolumes gv-locks 28: end-volume 29: 30: volume gv-index 31: type features/index 32: option index-base /var/export/gluster/gv/.glusterfs/indices 33: subvolumes gv-io-threads 34: end-volume 35: 36: volume gv-marker 37: type features/marker 38: option volume-uuid a562cb7c-0edf-4efa-afc6-80ea4e3fe978 39: option timestamp-file /var/lib/glusterd/vols/gv/marker.tstamp 40: option xtime off 41: option quota off 42: subvolumes gv-index 43: end-volume 44: 45: volume /var/export/gluster/gv 46: type debug/io-stats 47: option latency-measurement off 48: option count-fop-hits off 49: subvolumes gv-marker 50: end-volume 51: 52: volume gv-server 53: type protocol/server 54: option transport.socket.listen-port 49152 55: option rpc-auth.auth-glusterfs on 56: option rpc-auth.auth-unix on 57: option rpc-auth.auth-null on 58: option transport-type tcp 59: option auth.login./var/export/gluster/gv.allow ae4ffb2b-75fb-4b5a-b9d3-6c9e390fee03 60: option auth.login.ae4ffb2b-75fb-4b5a-b9d3-6c9e390fee03.password 041ee2e7-e8cf-4ecd-bba6-655348721610 61: option auth.addr./var/export/gluster/gv.allow * 62: subvolumes /var/export/gluster/gv 63: end-volume 64: +------------------------------------------------------------------------------+ [2013-08-29 15:12:28.897382] I [glusterfsd.c:1947:main] 0-/usr/local/sbin/glusterfsd: Started running /usr/local/sbin/glusterfsd version 3git (/usr/local/sbin/glusterfsd -s 192.168.232.179 --volfile-id gv.192.168.232.179.var-export-gluster-gv -p /var/lib/glusterd/vols/gv/run/192.168.232.179-var-export-gluster-gv.pid -S /var/run/ac988524335693210524c8d1d9acc6cd.socket --brick-name /var/export/gluster/gv -l /usr/local/var/log/glusterfs/bricks/var-export-gluster-gv.log --xlator-option *-posix.glusterd-uuid=42ff1e51-7c77-4c70-9e1b-3e6207935bee --brick-port 49152 --xlator-option gv-server.listen-port=49152) [2013-08-29 15:12:28.909949] I [socket.c:3515:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled [2013-08-29 15:12:28.909997] I [socket.c:3530:socket_init] 0-socket.glusterfsd: using system polling thread [2013-08-29 15:12:28.910139] I [socket.c:3515:socket_init] 0-glusterfs: SSL support is NOT enabled [2013-08-29 15:12:28.910150] I [socket.c:3530:socket_init] 0-glusterfs: using system polling thread [2013-08-29 15:12:28.998457] I [graph.c:254:gf_add_cmdline_options] 0-gv-server: adding option 'listen-port' for volume 'gv-server' with value '49152' [2013-08-29 15:12:28.998480] I [graph.c:254:gf_add_cmdline_options] 0-gv-posix: adding option 'glusterd-uuid' for volume 'gv-posix' with value '42ff1e51-7c77-4c70-9e1b-3e6207935bee' [2013-08-29 15:12:29.000012] W [options.c:848:xl_opt_validate] 0-gv-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction [2013-08-29 15:12:29.000099] I [socket.c:3515:socket_init] 0-tcp.gv-server: SSL support is NOT enabled [2013-08-29 15:12:29.000109] I [socket.c:3530:socket_init] 0-tcp.gv-server: using system polling thread [2013-08-29 15:12:29.025832] E [posix.c:4596:init] 0-gv-posix: Extended attribute trusted.glusterfs.volume-id is absent [2013-08-29 15:12:29.025857] E [xlator.c:425:xlator_init] 0-gv-posix: Initialization of volume 'gv-posix' failed, review your volfile again [2013-08-29 15:12:29.025868] E [graph.c:307:glusterfs_graph_init] 0-gv-posix: initializing translator failed [2013-08-29 15:12:29.025876] E [graph.c:502:glusterfs_graph_activate] 0-graph: init failed [2013-08-29 15:12:29.026073] W [glusterfsd.c:1022:cleanup_and_exit] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x1ae) [0x7f3727037f51] (-->/usr/local/sbin/glusterfsd(mgmt_getspec_cbk+0x32e) [0x40d02c] (-->/usr/local/sbin/glusterfsd(glusterfs_process_volfp+0x17a) [0x408c14]))) 0-: received signum (0), shutting down [2013-08-29 15:18:20.349821] I [glusterfsd.c:1947:main] 0-/usr/local/sbin/glusterfsd: Started running /usr/local/sbin/glusterfsd version 3git (/usr/local/sbin/glusterfsd -s 192.168.232.179 --volfile-id gv.192.168.232.179.var-export-gluster-gv -p /var/lib/glusterd/vols/gv/run/192.168.232.179-var-export-gluster-gv.pid -S /var/run/ac988524335693210524c8d1d9acc6cd.socket --brick-name /var/export/gluster/gv -l /usr/local/var/log/glusterfs/bricks/var-export-gluster-gv.log --xlator-option *-posix.glusterd-uuid=42ff1e51-7c77-4c70-9e1b-3e6207935bee --brick-port 49153 --xlator-option gv-server.listen-port=49153) [2013-08-29 15:18:20.351662] I [socket.c:3515:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled [2013-08-29 15:18:20.351708] I [socket.c:3530:socket_init] 0-socket.glusterfsd: using system polling thread [2013-08-29 15:18:20.351859] I [socket.c:3515:socket_init] 0-glusterfs: SSL support is NOT enabled [2013-08-29 15:18:20.351870] I [socket.c:3530:socket_init] 0-glusterfs: using system polling thread [2013-08-29 15:18:20.354717] I [graph.c:254:gf_add_cmdline_options] 0-gv-server: adding option 'listen-port' for volume 'gv-server' with value '49153' [2013-08-29 15:18:20.354742] I [graph.c:254:gf_add_cmdline_options] 0-gv-posix: adding option 'glusterd-uuid' for volume 'gv-posix' with value '42ff1e51-7c77-4c70-9e1b-3e6207935bee' [2013-08-29 15:18:20.356005] W [options.c:848:xl_opt_validate] 0-gv-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction [2013-08-29 15:18:20.356075] I [socket.c:3515:socket_init] 0-tcp.gv-server: SSL support is NOT enabled [2013-08-29 15:18:20.356085] I [socket.c:3530:socket_init] 0-tcp.gv-server: using system polling thread Final graph: +------------------------------------------------------------------------------+ 1: volume gv-posix 2: type storage/posix 3: option glusterd-uuid 42ff1e51-7c77-4c70-9e1b-3e6207935bee 4: option directory /var/export/gluster/gv 5: option volume-id cd8ac995-6014-4a49-b215-fbf4deb07320 6: end-volume 7: 8: volume gv-changelog 9: type features/changelog 10: option changelog-brick /var/export/gluster/gv 11: option changelog-dir /var/export/gluster/gv/.glusterfs/changelogs 12: subvolumes gv-posix 13: end-volume 14: 15: volume gv-access-control 16: type features/access-control 17: subvolumes gv-changelog 18: end-volume 19: 20: volume gv-locks 21: type features/locks 22: subvolumes gv-access-control 23: end-volume 24: 25: volume gv-io-threads 26: type performance/io-threads 27: subvolumes gv-locks 28: end-volume 29: 30: volume gv-index 31: type features/index 32: option index-base /var/export/gluster/gv/.glusterfs/indices 33: subvolumes gv-io-threads 34: end-volume 35: 36: volume gv-marker 37: type features/marker 38: option volume-uuid cd8ac995-6014-4a49-b215-fbf4deb07320 39: option timestamp-file /var/lib/glusterd/vols/gv/marker.tstamp 40: option xtime off 41: option quota off 42: subvolumes gv-index 43: end-volume 44: 45: volume /var/export/gluster/gv 46: type debug/io-stats 47: option latency-measurement off 48: option count-fop-hits off 49: subvolumes gv-marker 50: end-volume 51: 52: volume gv-server 53: type protocol/server 54: option transport.socket.listen-port 49153 55: option rpc-auth.auth-glusterfs on 56: option rpc-auth.auth-unix on 57: option rpc-auth.auth-null on 58: option transport-type tcp 59: option auth.login./var/export/gluster/gv.allow 12bcb54f-3cb9-4944-89f0-530cddaf22bc 60: option auth.login.12bcb54f-3cb9-4944-89f0-530cddaf22bc.password 7e467025-2ab6-4737-a36a-0adc95a8b282 61: option auth.addr./var/export/gluster/gv.allow * 62: subvolumes /var/export/gluster/gv 63: end-volume 64: +------------------------------------------------------------------------------+ [2013-08-29 15:18:20.918680] I [client_t.c:170:gf_client_get] 0-gv-server: client_uid=mgy-virt-6087-2013/08/29-15:18:20:860476-gv-client-0-0 [2013-08-29 15:18:20.918756] I [server-handshake.c:569:server_setvolume] 0-gv-server: accepted client from mgy-virt-6087-2013/08/29-15:18:20:860476-gv-client-0-0 (version: 3git) [2013-08-29 15:18:20.923199] I [client_t.c:170:gf_client_get] 0-gv-server: client_uid=mgy-virt-6087-2013/08/29-15:18:20:860476-gv-client-0-0 [2013-08-29 15:18:42.247347] I [client_t.c:170:gf_client_get] 0-gv-server: client_uid=mgy-virt-6118-2013/08/29-15:18:42:231523-gv-client-0-0 [2013-08-29 15:18:42.247398] I [server-handshake.c:569:server_setvolume] 0-gv-server: accepted client from mgy-virt-6118-2013/08/29-15:18:42:231523-gv-client-0-0 (version: 3git) [2013-08-29 15:18:42.247591] I [client_t.c:170:gf_client_get] 0-gv-server: client_uid=mgy-virt-6118-2013/08/29-15:18:42:231523-gv-client-0-0 [2013-08-29 17:53:52.936685] I [client_t.c:170:gf_client_get] 0-gv-server: client_uid=mgy-virt-11753-2013/08/29-17:53:52:686096-gv-client-0-0 [2013-08-29 17:53:52.963373] I [server-handshake.c:569:server_setvolume] 0-gv-server: accepted client from mgy-virt-11753-2013/08/29-17:53:52:686096-gv-client-0-0 (version: 3git) [2013-08-29 17:53:52.963615] I [client_t.c:170:gf_client_get] 0-gv-server: client_uid=mgy-virt-11753-2013/08/29-17:53:52:686096-gv-client-0-0 [2013-08-29 17:54:06.855900] I [server.c:514:server_rpc_notify] 0-gv-server: disconnecting connectionfrom mgy-virt-6118-2013/08/29-15:18:42:231523-gv-client-0-0 [2013-08-29 17:54:06.855936] I [client_t.c:304:gf_client_put] 0-gv-server: Shutting down connection mgy-virt-6118-2013/08/29-15:18:42:231523-gv-client-0-0 [2013-08-29 17:57:42.378267] I [client_t.c:170:gf_client_get] 0-gv-server: client_uid=mgy-virt-12105-2013/08/29-17:57:42:324032-gv-client-0-0 [2013-08-29 17:57:42.378341] I [server-handshake.c:569:server_setvolume] 0-gv-server: accepted client from mgy-virt-12105-2013/08/29-17:57:42:324032-gv-client-0-0 (version: 3git) [2013-08-29 17:57:42.378475] I [client_t.c:170:gf_client_get] 0-gv-server: client_uid=mgy-virt-12105-2013/08/29-17:57:42:324032-gv-client-0-0 [2013-08-29 18:37:49.614103] I [client_t.c:170:gf_client_get] 0-gv-server: client_uid=margay-virt-tv-2660-2013/08/29-18:37:49:178965-gv-client-0-0 [2013-08-29 18:37:49.614185] I [server-handshake.c:569:server_setvolume] 0-gv-server: accepted client from margay-virt-tv-2660-2013/08/29-18:37:49:178965-gv-client-0-0 (version: 3git) [2013-08-29 18:37:49.614807] I [client_t.c:170:gf_client_get] 0-gv-server: client_uid=margay-virt-tv-2660-2013/08/29-18:37:49:178965-gv-client-0-0 [2013-08-29 18:39:40.672083] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed [2013-08-29 18:39:41.400711] I [server.c:514:server_rpc_notify] 0-gv-server: disconnecting connectionfrom mgy-virt-6087-2013/08/29-15:18:20:860476-gv-client-0-0 [2013-08-29 18:39:41.400767] I [client_t.c:304:gf_client_put] 0-gv-server: Shutting down connection mgy-virt-6087-2013/08/29-15:18:20:860476-gv-client-0-0 [2013-08-29 18:39:42.423786] I [glusterfsd-mgmt.c:1554:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2013-08-29 18:39:42.617446] I [client_t.c:170:gf_client_get] 0-gv-server: client_uid=mgy-virt-11753-2013/08/29-17:53:52:686096-gv-client-0-1 [2013-08-29 18:39:42.617508] I [server-handshake.c:569:server_setvolume] 0-gv-server: accepted client from mgy-virt-11753-2013/08/29-17:53:52:686096-gv-client-0-1 (version: 3git) [2013-08-29 18:39:42.617766] I [client_t.c:170:gf_client_get] 0-gv-server: client_uid=mgy-virt-11753-2013/08/29-17:53:52:686096-gv-client-0-1 [2013-08-29 18:39:42.648154] I [client_t.c:170:gf_client_get] 0-gv-server: client_uid=mgy-virt-13113-2013/08/29-18:39:42:457984-gv-client-0-0 [2013-08-29 18:39:42.648204] I [server-handshake.c:569:server_setvolume] 0-gv-server: accepted client from mgy-virt-13113-2013/08/29-18:39:42:457984-gv-client-0-0 (version: 3git) [2013-08-29 18:39:42.648395] I [client_t.c:170:gf_client_get] 0-gv-server: client_uid=mgy-virt-13113-2013/08/29-18:39:42:457984-gv-client-0-0 [2013-08-29 18:39:42.946955] I [client_t.c:170:gf_client_get] 0-gv-server: client_uid=mgy-virt-13111-2013/08/29-18:39:42:457982-gv-client-0-0 [2013-08-29 18:39:42.947016] I [server-handshake.c:569:server_setvolume] 0-gv-server: accepted client from mgy-virt-13111-2013/08/29-18:39:42:457982-gv-client-0-0 (version: 3git) [2013-08-29 18:39:42.947226] I [client_t.c:170:gf_client_get] 0-gv-server: client_uid=mgy-virt-13111-2013/08/29-18:39:42:457982-gv-client-0-0 [2013-08-29 18:39:43.971274] I [server.c:514:server_rpc_notify] 0-gv-server: disconnecting connectionfrom margay-virt-tv-2660-2013/08/29-18:37:49:178965-gv-client-0-0 [2013-08-29 18:39:43.971319] I [client_t.c:304:gf_client_put] 0-gv-server: Shutting down connection margay-virt-tv-2660-2013/08/29-18:37:49:178965-gv-client-0-0 [2013-08-29 18:39:44.990773] I [client_t.c:170:gf_client_get] 0-gv-server: client_uid=margay-virt-tv-2685-2013/08/29-18:39:44:974559-gv-client-0-0 [2013-08-29 18:39:44.990865] I [server-handshake.c:569:server_setvolume] 0-gv-server: accepted client from margay-virt-tv-2685-2013/08/29-18:39:44:974559-gv-client-0-0 (version: 3git) [2013-08-29 18:39:44.991272] I [client_t.c:170:gf_client_get] 0-gv-server: client_uid=margay-virt-tv-2685-2013/08/29-18:39:44:974559-gv-client-0-0 [2013-08-29 18:39:45.008437] I [client_t.c:170:gf_client_get] 0-gv-server: client_uid=margay-virt-tv-2683-2013/08/29-18:39:44:970731-gv-client-0-0 [2013-08-29 18:39:45.008512] I [server-handshake.c:569:server_setvolume] 0-gv-server: accepted client from margay-virt-tv-2683-2013/08/29-18:39:44:970731-gv-client-0-0 (version: 3git) [2013-08-29 18:39:45.009137] I [client_t.c:170:gf_client_get] 0-gv-server: client_uid=margay-virt-tv-2683-2013/08/29-18:39:44:970731-gv-client-0-0 [2013-08-29 18:43:38.919744] I [server.c:514:server_rpc_notify] 0-gv-server: disconnecting connectionfrom mgy-virt-12105-2013/08/29-17:57:42:324032-gv-client-0-0 [2013-08-29 18:43:38.919806] I [client_t.c:304:gf_client_put] 0-gv-server: Shutting down connection mgy-virt-12105-2013/08/29-17:57:42:324032-gv-client-0-0 [2013-08-29 18:43:38.919830] I [server-helpers.c:354:do_fd_cleanup] 0-gv-server: fd cleanup on /QEMU/deb-on-gluster/disk0.qcow2 [2013-08-29 18:43:39.419318] I [server.c:514:server_rpc_notify] 0-gv-server: disconnecting connectionfrom mgy-virt-11753-2013/08/29-17:53:52:686096-gv-client-0-0 [2013-08-29 18:43:39.419371] I [client_t.c:304:gf_client_put] 0-gv-server: Shutting down connection mgy-virt-11753-2013/08/29-17:53:52:686096-gv-client-0-0 glustershd.log: [2013-08-29 18:39:42.498589] I [glusterfsd.c:1947:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3git (/usr/local/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /usr/local/var/log/glusterfs/glustershd.log -S /var/run/c56834fc4e2a1c6cf16b556509b26807.socket --xlator-option *replicate*.node-uuid=42ff1e51-7c77-4c70-9e1b-3e6207935bee) [2013-08-29 18:39:42.500392] I [socket.c:3515:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled [2013-08-29 18:39:42.500438] I [socket.c:3530:socket_init] 0-socket.glusterfsd: using system polling thread [2013-08-29 18:39:42.500569] I [socket.c:3515:socket_init] 0-glusterfs: SSL support is NOT enabled [2013-08-29 18:39:42.500580] I [socket.c:3530:socket_init] 0-glusterfs: using system polling thread [2013-08-29 18:39:42.641985] I [graph.c:254:gf_add_cmdline_options] 0-gv-replicate-0: adding option 'node-uuid' for volume 'gv-replicate-0' with value '42ff1e51-7c77-4c70-9e1b-3e6207935bee' [2013-08-29 18:39:42.645404] I [socket.c:3515:socket_init] 0-gv-client-1: SSL support is NOT enabled [2013-08-29 18:39:42.645449] I [socket.c:3530:socket_init] 0-gv-client-1: using system polling thread [2013-08-29 18:39:42.646195] I [socket.c:3515:socket_init] 0-gv-client-0: SSL support is NOT enabled [2013-08-29 18:39:42.646216] I [socket.c:3530:socket_init] 0-gv-client-0: using system polling thread [2013-08-29 18:39:42.646253] I [client.c:2236:notify] 0-gv-client-0: parent translators are ready, attempting connect on transport [2013-08-29 18:39:42.646670] I [client.c:2236:notify] 0-gv-client-1: parent translators are ready, attempting connect on transport Final graph: +------------------------------------------------------------------------------+ 1: volume gv-client-0 2: type protocol/client 3: option remote-host 192.168.232.179 4: option remote-subvolume /var/export/gluster/gv 5: option transport-type socket 6: option username 12bcb54f-3cb9-4944-89f0-530cddaf22bc 7: option password 7e467025-2ab6-4737-a36a-0adc95a8b282 8: end-volume 9: 10: volume gv-client-1 11: type protocol/client 12: option remote-host 192.168.232.101 13: option remote-subvolume /var/export/gluster/gv 14: option transport-type socket 15: option username 12bcb54f-3cb9-4944-89f0-530cddaf22bc 16: option password 7e467025-2ab6-4737-a36a-0adc95a8b282 17: end-volume 18: 19: volume gv-replicate-0 20: type cluster/replicate 21: option node-uuid 42ff1e51-7c77-4c70-9e1b-3e6207935bee 22: option background-self-heal-count 0 23: option metadata-self-heal on 24: option data-self-heal on 25: option entry-self-heal on 26: option self-heal-daemon on 27: option iam-self-heal-daemon yes 28: subvolumes gv-client-0 gv-client-1 29: end-volume 30: 31: volume glustershd 32: type debug/io-stats 33: subvolumes gv-replicate-0 34: end-volume 35: +------------------------------------------------------------------------------+ [2013-08-29 18:39:42.647465] I [rpc-clnt.c:1675:rpc_clnt_reconfig] 0-gv-client-0: changing port to 49153 (from 0) [2013-08-29 18:39:42.648007] I [client-handshake.c:1658:select_server_supported_programs] 0-gv-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2013-08-29 18:39:42.648292] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv-client-0: Connected to 192.168.232.179:49153, attached to remote volume '/var/export/gluster/gv'. [2013-08-29 18:39:42.648307] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv-client-0: Server and Client lk-version numbers are not same, reopening the fds [2013-08-29 18:39:42.648371] I [afr-common.c:4312:afr_notify] 0-gv-replicate-0: Subvolume 'gv-client-0' came back up; going online. [2013-08-29 18:39:42.695288] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv-client-0: Server lk version = 1 [2013-08-29 18:39:42.961473] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv-replicate-0: Another crawl is in progress for gv-client-0 [2013-08-29 18:39:42.961528] E [afr-self-heald.c:1067:afr_find_child_position] 0-gv-replicate-0: getxattr failed on gv-client-1 - (Transport endpoint is not connected) [2013-08-29 18:39:44.974824] I [rpc-clnt.c:1675:rpc_clnt_reconfig] 0-gv-client-1: changing port to 49152 (from 0) [2013-08-29 18:39:44.976348] I [client-handshake.c:1658:select_server_supported_programs] 0-gv-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2013-08-29 18:39:44.981325] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv-client-1: Connected to 192.168.232.101:49152, attached to remote volume '/var/export/gluster/gv'. [2013-08-29 18:39:44.981345] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv-client-1: Server and Client lk-version numbers are not same, reopening the fds [2013-08-29 18:39:44.982118] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv-client-1: Server lk version = 1 usr-local-etc-glusterfs-glusterd.vol.log: Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option transport.socket.listen-backlog 128 7: option transport.socket.read-fail-log off 8: option transport.socket.keepalive-interval 2 9: option transport.socket.keepalive-time 10 10: option transport-type rdma 11: option working-directory /var/lib/glusterd 12: end-volume 13: +------------------------------------------------------------------------------+ Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option transport.socket.listen-backlog 128 7: option transport.socket.read-fail-log off 8: option transport.socket.keepalive-interval 2 9: option transport.socket.keepalive-time 10 10: option transport-type rdma 11: option working-directory /var/lib/glusterd 12: end-volume 13: +------------------------------------------------------------------------------+ Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option transport.socket.listen-backlog 128 7: option transport.socket.read-fail-log off 8: option transport.socket.keepalive-interval 2 9: option transport.socket.keepalive-time 10 10: option transport-type rdma 11: option working-directory /var/lib/glusterd 12: end-volume 13: +------------------------------------------------------------------------------+ Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option transport.socket.listen-backlog 128 7: option transport.socket.read-fail-log off 8: option transport.socket.keepalive-interval 2 9: option transport.socket.keepalive-time 10 10: option transport-type rdma 11: option working-directory /var/lib/glusterd 12: end-volume 13: +------------------------------------------------------------------------------+ [2013-08-29 15:12:25.620777] I [glusterfsd.c:1947:main] 0-/usr/local/sbin/glusterd: Started running /usr/local/sbin/glusterd version 3git (/usr/local/sbin/glusterd -p /var/run/glusterd.pid) [2013-08-29 15:12:25.631435] I [glusterd.c:1003:init] 0-management: Using /var/lib/glusterd as working directory [2013-08-29 15:12:25.644612] I [socket.c:3515:socket_init] 0-socket.management: SSL support is NOT enabled [2013-08-29 15:12:25.644633] I [socket.c:3530:socket_init] 0-socket.management: using system polling thread [2013-08-29 15:12:25.644729] E [rpc-transport.c:253:rpc_transport_load] 0-rpc-transport: /usr/local/lib/glusterfs/3git/rpc-transport/rdma.so: cannot open shared object file: No such file or directory [2013-08-29 15:12:25.644742] W [rpc-transport.c:257:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine [2013-08-29 15:12:25.644752] W [rpcsvc.c:1437:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2013-08-29 15:12:28.422807] I [glusterd-store.c:1357:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 3 [2013-08-29 15:12:28.434272] E [glusterd-store.c:1874:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2013-08-29 15:12:28.866722] I [glusterd.c:128:glusterd_uuid_init] 0-management: retrieved UUID: 42ff1e51-7c77-4c70-9e1b-3e6207935bee [2013-08-29 15:12:28.870166] I [rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2013-08-29 15:12:28.870283] I [socket.c:3515:socket_init] 0-management: SSL support is NOT enabled [2013-08-29 15:12:28.870296] I [socket.c:3530:socket_init] 0-management: using system polling thread Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option transport.socket.listen-backlog 128 7: option transport.socket.read-fail-log off 8: option transport.socket.keepalive-interval 2 9: option transport.socket.keepalive-time 10 10: option transport-type rdma 11: option working-directory /var/lib/glusterd 12: end-volume 13: +------------------------------------------------------------------------------+ [2013-08-29 15:12:28.931968] I [rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2013-08-29 15:12:28.932063] I [socket.c:3515:socket_init] 0-management: SSL support is NOT enabled [2013-08-29 15:12:28.932074] I [socket.c:3530:socket_init] 0-management: using system polling thread [2013-08-29 15:12:28.932970] I [socket.c:2240:socket_event_handler] 0-transport: disconnecting now [2013-08-29 15:12:28.933833] I [glusterd-handler.c:3667:__glusterd_brick_rpc_notify] 0-management: Disconnected from 192.168.232.179:/var/export/gluster/gv [2013-08-29 15:12:28.933897] I [socket.c:2240:socket_event_handler] 0-transport: disconnecting now [2013-08-29 15:12:29.026302] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick (null) on port 49152 [2013-08-29 15:15:02.130169] I [glusterd-handler.c:1067:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2013-08-29 15:15:02.132182] I [glusterd-handler.c:1067:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2013-08-29 15:15:20.371233] I [glusterd-handler.c:1012:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2013-08-29 15:15:51.913885] E [glusterd-volume-ops.c:1121:glusterd_op_stage_delete_volume] 0-management: Volume gv has been started.Volume needs to be stopped before deletion. [2013-08-29 15:15:51.913926] E [glusterd-syncop.c:896:gd_stage_op_phase] 0-management: Staging of operation 'Volume Delete' failed on localhost : Volume gv has been started.Volume needs to be stopped before deletion. [2013-08-29 15:15:59.283416] E [glusterd-volume-ops.c:966:glusterd_op_stage_start_volume] 0-management: Volume gv already started [2013-08-29 15:15:59.283450] E [glusterd-syncop.c:896:gd_stage_op_phase] 0-management: Staging of operation 'Volume Start' failed on localhost : Volume gv already started [2013-08-29 15:16:06.185926] I [mem-pool.c:539:mem_pool_destroy] 0-management: size=2236 max=0 total=0 [2013-08-29 15:16:06.185965] I [mem-pool.c:539:mem_pool_destroy] 0-management: size=124 max=0 total=0 [2013-08-29 15:16:06.185997] I [glusterd-utils.c:1031:glusterd_service_stop] 0-management: brick already stopped [2013-08-29 15:16:06.186034] E [glusterd-utils.c:1339:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/ac988524335693210524c8d1d9acc6cd.socket error: No such file or directory [2013-08-29 15:16:07.761363] E [glusterd-utils.c:3471:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/44fd086eace82657915d08b093e1de3e.socket error: Permission denied [2013-08-29 15:16:07.761696] I [glusterd-utils.c:3505:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully [2013-08-29 15:16:07.761826] I [glusterd-utils.c:3510:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully [2013-08-29 15:16:07.761947] I [glusterd-utils.c:3515:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully [2013-08-29 15:16:07.762066] I [glusterd-utils.c:3520:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully [2013-08-29 15:16:07.762186] I [glusterd-utils.c:3525:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully [2013-08-29 15:16:07.762326] I [glusterd-utils.c:3530:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully [2013-08-29 15:16:07.762481] I [mem-pool.c:539:mem_pool_destroy] 0-management: size=2236 max=0 total=0 [2013-08-29 15:16:07.762502] I [mem-pool.c:539:mem_pool_destroy] 0-management: size=124 max=0 total=0 [2013-08-29 15:17:39.635361] I [glusterd-handler.c:1067:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2013-08-29 15:18:20.357342] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /var/export/gluster/gv on port 49153 [2013-08-29 15:18:20.358101] I [rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2013-08-29 15:18:20.358170] I [socket.c:3515:socket_init] 0-management: SSL support is NOT enabled [2013-08-29 15:18:20.358180] I [socket.c:3530:socket_init] 0-management: using system polling thread [2013-08-29 15:18:20.859900] I [rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2013-08-29 15:18:20.859991] I [socket.c:3515:socket_init] 0-management: SSL support is NOT enabled [2013-08-29 15:18:20.860002] I [socket.c:3530:socket_init] 0-management: using system polling thread [2013-08-29 15:18:20.861389] I [socket.c:2240:socket_event_handler] 0-transport: disconnecting now [2013-08-29 18:37:06.816292] I [glusterd-handler.c:1012:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2013-08-29 18:37:17.022046] I [glusterd-handler.c:816:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req 192.168.232.101 24007 [2013-08-29 18:37:47.063223] I [glusterd-handler.c:2902:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: 192.168.232.101 (24007) [2013-08-29 18:37:47.171929] I [rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2013-08-29 18:37:47.172032] I [socket.c:3515:socket_init] 0-management: SSL support is NOT enabled [2013-08-29 18:37:47.172043] I [socket.c:3530:socket_init] 0-management: using system polling thread [2013-08-29 18:37:47.181165] I [glusterd-handler.c:2883:glusterd_friend_add] 0-management: connect returned 0 [2013-08-29 18:37:48.007856] I [glusterd-rpc-ops.c:241:__glusterd_probe_cbk] 0-glusterd: Received probe resp from uuid: 4bb5c714-0428-44fa-8ee6-8c3148a07d45, host: 192.168.232.101 [2013-08-29 18:37:48.111224] I [glusterd-rpc-ops.c:313:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2013-08-29 18:37:49.178825] I [glusterd-rpc-ops.c:363:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 4bb5c714-0428-44fa-8ee6-8c3148a07d45, host: 192.168.232.101, port: 0 [2013-08-29 18:37:49.542628] I [glusterd-handshake.c:553:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 3 [2013-08-29 18:37:49.753560] I [glusterd-handler.c:2317:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 4bb5c714-0428-44fa-8ee6-8c3148a07d45 [2013-08-29 18:37:49.753630] I [glusterd-handler.c:2369:__glusterd_handle_probe_query] 0-glusterd: Responded to 192.168.232.101, op_ret: 0, op_errno: 0, ret: 0 [2013-08-29 18:37:49.942875] I [glusterd-handler.c:2021:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 4bb5c714-0428-44fa-8ee6-8c3148a07d45 [2013-08-29 18:37:49.942994] I [glusterd-handler.c:3056:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.232.101 (0), ret: 0 [2013-08-29 18:37:50.006392] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 4bb5c714-0428-44fa-8ee6-8c3148a07d45, host: 192.168.232.101 [2013-08-29 18:37:50.098546] I [glusterd-handler.c:2183:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4bb5c714-0428-44fa-8ee6-8c3148a07d45 [2013-08-29 18:37:50.098599] I [glusterd-handler.c:2228:__glusterd_handle_friend_update] 0-: Received uuid: 42ff1e51-7c77-4c70-9e1b-3e6207935bee, hostname:192.168.232.179 [2013-08-29 18:37:50.098609] I [glusterd-handler.c:2237:__glusterd_handle_friend_update] 0-: Received my uuid as Friend [2013-08-29 18:37:50.287668] I [glusterd-rpc-ops.c:560:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4bb5c714-0428-44fa-8ee6-8c3148a07d45 [2013-08-29 18:37:56.255438] I [glusterd-handler.c:1012:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2013-08-29 18:38:04.913092] W [glusterd-op-sm.c:3079:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-08-29 18:38:04.914781] I [glusterd-handler.c:3492:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv [2013-08-29 18:39:40.638831] I [glusterd-brick-ops.c:370:__glusterd_handle_add_brick] 0-management: Received add brick req [2013-08-29 18:39:40.638903] I [glusterd-brick-ops.c:417:__glusterd_handle_add_brick] 0-management: replica-count is 2 [2013-08-29 18:39:40.638915] I [glusterd-brick-ops.c:193:gd_addbr_validate_replica_count] 0-management: Changing the type of volume gv from 'distribute' to 'replica' [2013-08-29 18:39:40.641794] I [glusterd-brick-ops.c:1044:glusterd_op_perform_add_bricks] 0-management: replica-count is set 2 [2013-08-29 18:39:40.641813] I [glusterd-brick-ops.c:1048:glusterd_op_perform_add_bricks] 0-management: type is set 2, need to change it [2013-08-29 18:39:42.395141] E [glusterd-utils.c:3471:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/44fd086eace82657915d08b093e1de3e.socket error: Permission denied [2013-08-29 18:39:42.419152] I [glusterd-utils.c:3505:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully [2013-08-29 18:39:42.419323] I [glusterd-utils.c:3510:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully [2013-08-29 18:39:42.419452] I [glusterd-utils.c:3515:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully [2013-08-29 18:39:42.419600] I [glusterd-utils.c:3520:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully [2013-08-29 18:39:42.419727] I [glusterd-utils.c:3525:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully [2013-08-29 18:39:42.419851] I [glusterd-utils.c:3530:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully [2013-08-29 18:39:42.423420] I [rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2013-08-29 18:39:42.423525] I [socket.c:3515:socket_init] 0-management: SSL support is NOT enabled [2013-08-29 18:39:42.423536] I [socket.c:3530:socket_init] 0-management: using system polling thread [2013-08-29 18:39:42.427017] I [rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2013-08-29 18:39:42.427118] I [socket.c:3515:socket_init] 0-management: SSL support is NOT enabled [2013-08-29 18:39:42.427128] I [socket.c:3530:socket_init] 0-management: using system polling thread [2013-08-29 18:39:42.427728] I [mem-pool.c:539:mem_pool_destroy] 0-management: size=2236 max=0 total=0 [2013-08-29 18:39:42.427762] I [mem-pool.c:539:mem_pool_destroy] 0-management: size=124 max=0 total=0 [2013-08-29 18:39:42.427788] I [socket.c:2240:socket_event_handler] 0-transport: disconnecting now [2013-08-29 18:39:42.427824] I [socket.c:2240:socket_event_handler] 0-transport: disconnecting now [2013-08-29 18:39:42.616379] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:39:45.889700] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:39:48.890531] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:39:51.891340] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:39:54.892139] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:39:57.892912] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:40:00.893721] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:40:03.894546] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:40:06.895326] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:40:09.896197] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:40:12.896937] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:40:15.897742] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:40:18.898555] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:40:21.899414] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:40:24.900324] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:40:27.901145] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:40:30.901923] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:40:33.902822] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:40:36.903625] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:40:39.904467] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:40:42.905249] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:40:45.906047] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:40:48.906826] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:40:51.907631] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:40:54.908472] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:40:57.909289] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:41:00.910101] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:41:03.910912] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:41:06.911718] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:41:09.912439] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:41:12.913236] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:41:15.914027] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:41:18.914810] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:41:21.915611] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:41:24.916459] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:41:27.917221] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:41:30.918020] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:41:33.918870] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:41:36.919698] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:41:39.920450] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:41:42.921250] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:41:45.922027] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:41:48.922817] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:41:51.923620] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:41:54.924461] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:41:57.925269] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:42:00.926091] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:42:03.926892] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:42:06.927714] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:42:09.928576] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:42:12.929352] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:42:15.930170] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:42:18.931019] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:42:21.931889] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:42:24.932685] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:42:27.933467] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:42:30.934272] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:42:33.935072] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:42:36.935866] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:42:39.936663] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:42:42.937464] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:42:45.938255] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:42:48.939058] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:42:51.939846] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:42:54.940635] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:42:57.941460] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:43:00.942219] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:43:03.943034] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:43:06.943869] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:43:09.944629] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:43:12.945428] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:43:15.946261] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:43:18.947061] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:43:21.947833] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:43:24.948649] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:43:27.949472] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:43:30.950291] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:43:33.951081] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-29 18:43:36.951911] E [rpcsvc.c:523:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2013-08-30 07:20:17.209927] W [glusterd-op-sm.c:3079:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-08-30 07:20:17.211539] I [glusterd-handler.c:3492:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv [2013-08-30 07:20:31.653830] I [glusterd-handler.c:1067:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2013-08-30 07:20:31.654563] I [glusterd-handler.c:1067:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req NOTE: On another message I will paste the logs of the other peer (containing the brick which has been added). BTW I notice lots of "non-privileged port" errors which confirm your idea. (Not sure if both patches you suggested have been merged into master as of now). More news as soon as I can. Thanks. Guido
