Hello, This is yet another problem we are experiencing with glusterfs 2.0.1 (both server and client). Every few hours the client gets disconnected in a way that processes trying to access the glusterfs mount volume die in a non-interruptible input-output state. We are forced to run `umount -l` on the partition and remount it with `mount` all over again. The logs in the side of the server are the following: [2009-05-27 01:18:25] C [server-protocol.c:6553:_volfile_update_checksum] foo-server: the volume file got modified between earlier access and now, this may lead to inconsistency between clien ts, advised to remount client [2009-05-27 01:18:25] N [server-protocol.c:7796:notify] foo-server: 192.168.168.210:1020 disconnected [2009-05-27 01:18:25] N [server-protocol.c:7796:notify] foo-server: 192.168.168.210:1021 disconnected [2009-05-27 01:18:26] N [server-protocol.c:7035:mop_setvolume] foo-server: accepted client from 192.168.168.210:1021 [2009-05-27 01:18:26] N [server-protocol.c:7035:mop_setvolume] foo-server: accepted client from 192.168.168.210:1020 [2009-05-27 01:52:50] N [server-protocol.c:7796:notify] foo-server: 192.168.168.212:1020 disconnected [2009-05-27 01:52:50] N [server-protocol.c:7796:notify] foo-server: 192.168.168.212:1021 disconnected [2009-05-27 01:52:50] N [server-protocol.c:7035:mop_setvolume] foo-server: accepted client from 192.168.168.212:1020 [2009-05-27 01:52:50] N [server-protocol.c:7035:mop_setvolume] foo-server: accepted client from 192.168.168.212:1021 [2009-05-27 04:11:36] N [server-protocol.c:7796:notify] foo-server: 192.168.168.210:1018 disconnected [2009-05-27 04:11:36] N [server-protocol.c:7796:notify] foo-server: 192.168.168.210:1019 disconnected [2009-05-27 04:11:37] N [server-protocol.c:7035:mop_setvolume] foo-server: accepted client from 192.168.168.210:1019 [2009-05-27 04:11:37] N [server-protocol.c:7035:mop_setvolume] foo-server: accepted client from 192.168.168.210:1018 [2009-05-27 04:42:49] N [server-protocol.c:7796:notify] foo-server: 192.168.168.212:1018 disconnected [2009-05-27 04:42:49] N [server-protocol.c:7796:notify] foo-server: 192.168.168.212:1019 disconnected [2009-05-27 04:42:50] N [server-protocol.c:7035:mop_setvolume] foo-server: accepted client from 192.168.168.212:1018 [2009-05-27 04:42:50] N [server-protocol.c:7035:mop_setvolume] foo-server: accepted client from 192.168.168.212:1019 [2009-05-27 07:25:31] N [server-protocol.c:7796:notify] foo-server: 192.168.168.210:1016 disconnected [2009-05-27 07:25:31] N [server-protocol.c:7796:notify] foo-server: 192.168.168.210:1017 disconnected [2009-05-27 07:25:31] N [server-protocol.c:7035:mop_setvolume] foo-server: accepted client from 192.168.168.210:1016 [2009-05-27 07:25:31] N [server-protocol.c:7035:mop_setvolume] foo-server: accepted client from 192.168.168.210:1017 [2009-05-27 07:39:25] N [server-protocol.c:7796:notify] foo-server: 192.168.168.210:1017 disconnected [2009-05-27 07:39:25] N [server-protocol.c:7796:notify] foo-server: 192.168.168.210:1016 disconnected [2009-05-27 07:39:25] N [server-helpers.c:723:server_connection_destroy] foo-server: destroyed connection of xxx010.bar.foo.net-28685-2009/05/27-07:25:30:922187-foo [2009-05-27 07:39:26] N [server-protocol.c:7796:notify] foo-server: 192.168.168.210:1015 disconnected [2009-05-27 07:39:26] N [server-protocol.c:7796:notify] foo-server: 192.168.168.210:1017 disconnected [2009-05-27 07:39:26] N [server-protocol.c:7035:mop_setvolume] foo-server: accepted client from 192.168.168.210:1017 [2009-05-27 07:39:26] N [server-protocol.c:7035:mop_setvolume] foo-server: accepted client from 192.168.168.210:1015 [2009-05-27 08:00:24] N [server-protocol.c:7796:notify] foo-server: 192.168.168.212:1016 disconnected [2009-05-27 08:00:24] N [server-protocol.c:7796:notify] foo-server: 192.168.168.212:1017 disconnected [2009-05-27 08:00:24] N [server-protocol.c:7035:mop_setvolume] foo-server: accepted client from 192.168.168.212:1016 [2009-05-27 08:00:24] N [server-protocol.c:7035:mop_setvolume] foo-server: accepted client from 192.168.168.212:1017 [2009-05-27 08:00:29] N [server-protocol.c:7796:notify] foo-server: 192.168.168.210:1015 disconnected [2009-05-27 08:00:29] N [server-protocol.c:7796:notify] foo-server: 192.168.168.210:1017 disconnected [2009-05-27 08:00:29] N [server-helpers.c:723:server_connection_destroy] foo-server: destroyed connection of xxx010.bar.foo.net-30040-2009/05/27-07:39:26:412074-foo [2009-05-27 08:00:30] N [server-protocol.c:7796:notify] foo-server: 192.168.168.210:1014 disconnected [2009-05-27 08:00:30] N [server-protocol.c:7796:notify] foo-server: 192.168.168.210:1016 disconnected [2009-05-27 08:00:30] N [server-protocol.c:7035:mop_setvolume] foo-server: accepted client from 192.168.168.210:1014 [2009-05-27 08:00:30] N [server-protocol.c:7035:mop_setvolume] foo-server: accepted client from 192.168.168.210:1016 [2009-05-27 11:37:59] N [server-protocol.c:7796:notify] foo-server: 192.168.168.210:1015 disconnected [2009-05-27 11:37:59] N [server-protocol.c:7796:notify] foo-server: 192.168.168.210:1017 disconnected [2009-05-27 11:38:00] N [server-protocol.c:7035:mop_setvolume] foo-server: accepted client from 192.168.168.210:1017 [2009-05-27 11:38:00] N [server-protocol.c:7035:mop_setvolume] foo-server: accepted client from 192.168.168.210:1015 [2009-05-27 11:38:12] N [server-protocol.c:7796:notify] foo-server: 192.168.168.212:1015 disconnected [2009-05-27 11:38:12] N [server-protocol.c:7796:notify] foo-server: 192.168.168.212:1014 disconnected [2009-05-27 11:38:12] N [server-protocol.c:7035:mop_setvolume] foo-server: accepted client from 192.168.168.212:1015 [2009-05-27 11:38:12] N [server-protocol.c:7035:mop_setvolume] foo-server: accepted client from 192.168.168.212:1014 In the client: ================================================================================ Version : glusterfs 2.0.1 built on May 26 2009 05:23:52 TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b Starting Time: 2009-05-26 12:09:27 Command line : /usr/sbin/glusterfs2 --log-level=NORMAL --volfile-server=file01.priv --volfile-server-port=6997 /mnt/file01 PID : 18548 System name : Linux Nodename : xxx012.bar.foo.net Kernel Release : 2.6.18-92.1.10.el5PAE Hardware Identifier: i686 Given volfile: +------------------------------------------------------------------------------+ 1: volume foo 2: type protocol/client 3: option transport-type tcp 4: option remote-host file01.priv # can be IP or hostname 5: option remote-port 6997 6: option remote-subvolume foo-remote 7: end-volume 8: 9: volume foo-readahead 10: type performance/read-ahead 11: option page-count 4 12: subvolumes foo 13: end-volume 14: 15: volume foo-writebehind 16: type performance/write-behind 17: option cache-size 512KB 18: subvolumes foo-readahead 19: end-volume 20: 21: volume foo-iothreads 22: type performance/io-threads 23: option autoscaling on 24: option min-threads 16 25: option max-threads 256 26: subvolumes foo-writebehind 27: end-volume 28: 29: volume foo-iocache 30: type performance/io-cache 31: option cache-size 64MB 32: option page-size 128KB 33: option cache-timeout 2 34: subvolumes foo-iothreads 35: end-volume 36: +------------------------------------------------------------------------------+ [2009-05-26 12:09:27] N [glusterfsd.c:1152:main] glusterfs: Successfully started [2009-05-26 12:09:27] N [client-protocol.c:5557:client_setvolume_cbk] foo: Connected to 192.168.168.231:6997, attached to remote volume 'foo-remote'. [2009-05-26 12:09:27] N [client-protocol.c:5557:client_setvolume_cbk] foo: Connected to 192.168.168.231:6997, attached to remote volume 'foo-remote'. [2009-05-26 12:09:38] W [fuse-bridge.c:1365:fuse_create_cbk] glusterfs-fuse: 21: /cust/a => -1 (Read-only file system) [2009-05-26 12:16:54] W [glusterfsd.c:827:cleanup_and_exit] glusterfs: shutting down [2009-05-26 12:16:54] N [fuse-bridge.c:2843:fini] fuse: Unmounting '/mnt/file01'. [2009-05-26 12:16:54] W [glusterfsd.c:827:cleanup_and_exit] glusterfs: shutting down [2009-05-26 12:20:08] W [socket.c:1319:socket_init] trans: disabling non-blocking IO [2009-05-26 12:20:08] W [socket.c:1319:socket_init] trans: disabling non-blocking IO [2009-05-26 12:20:08] W [client-protocol.c:6012:init] trans: WARNING: Failed to set 'ulimit -n 1M': Permission denied ================================================================================ Version : glusterfs 2.0.1 built on May 26 2009 05:23:52 TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b Starting Time: 2009-05-26 12:20:14 Command line : /usr/sbin/glusterfs2 --log-level=NORMAL --volfile-server=file01.priv --volfile-server-port=6997 /mnt/file01 PID : 3074 System name : Linux Nodename : xxx012.bar.foo.net Kernel Release : 2.6.18-92.1.10.el5PAE Hardware Identifier: i686 Given volfile: +------------------------------------------------------------------------------+ 1: volume foo 2: type protocol/client 3: option transport-type tcp 4: option remote-host file01.priv # can be IP or hostname 5: option remote-port 6997 6: option remote-subvolume foo-remote 7: end-volume 8: 9: volume foo-readahead 10: type performance/read-ahead 11: option page-count 4 12: subvolumes foo 13: end-volume 14: 15: volume foo-writebehind 16: type performance/write-behind 17: option cache-size 512KB 18: subvolumes foo-readahead 19: end-volume 20: 21: volume foo-iothreads 22: type performance/io-threads 23: option autoscaling on 24: option min-threads 16 25: option max-threads 256 26: subvolumes foo-writebehind 27: end-volume 28: 29: volume foo-iocache 30: type performance/io-cache 31: option cache-size 64MB 32: option page-size 128KB 33: option cache-timeout 2 34: subvolumes foo-iothreads 35: end-volume 36: +------------------------------------------------------------------------------+ [2009-05-26 12:20:15] W [client-protocol.c:6012:init] foo: WARNING: Failed to set 'ulimit -n 1M': Permission denied [2009-05-26 12:20:15] N [glusterfsd.c:1152:main] glusterfs: Successfully started [2009-05-26 12:20:15] N [client-protocol.c:5557:client_setvolume_cbk] foo: Connected to 192.168.168.231:6997, attached to remote volume 'foo-remote'. [2009-05-26 12:20:15] N [client-protocol.c:5557:client_setvolume_cbk] foo: Connected to 192.168.168.231:6997, attached to remote volume 'foo-remote'. [2009-05-27 01:52:50] W [socket.c:1319:socket_init] trans: disabling non-blocking IO [2009-05-27 01:52:50] W [socket.c:1319:socket_init] trans: disabling non-blocking IO ================================================================================ Version : glusterfs 2.0.1 built on May 26 2009 05:23:52 TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b Starting Time: 2009-05-27 01:52:50 Command line : /usr/sbin/glusterfs2 --log-level=NORMAL --volfile-server=file01.priv --volfile-server-port=6997 /mnt/file01 PID : 30582 System name : Linux Nodename : xxx012.bar.foo.net Kernel Release : 2.6.18-92.1.10.el5PAE Hardware Identifier: i686 Given volfile: +------------------------------------------------------------------------------+ 1: volume foo 2: type protocol/client 3: option transport-type tcp 4: option remote-host file01.priv # can be IP or hostname 5: option remote-port 6997 6: option remote-subvolume foo-remote 7: end-volume 8: 9: volume foo-readahead 10: type performance/read-ahead 11: option page-count 4 12: subvolumes foo 13: end-volume 14: 15: volume foo-writebehind 16: type performance/write-behind 17: option cache-size 512KB 18: subvolumes foo-readahead 19: end-volume 20: 21: volume foo-iothreads 22: type performance/io-threads 23: option autoscaling on 24: option min-threads 16 25: option max-threads 256 26: subvolumes foo-writebehind 27: end-volume 28: 29: volume foo-iocache 30: type performance/io-cache 31: option cache-size 64MB 32: option page-size 128KB 33: option cache-timeout 2 34: subvolumes foo-iothreads 35: end-volume +------------------------------------------------------------------------------+ [2009-05-27 01:52:50] N [glusterfsd.c:1152:main] glusterfs: Successfully started [2009-05-27 01:52:50] N [client-protocol.c:5557:client_setvolume_cbk] foo: Connected to 192.168.168.231:6997, attached to remote volume 'foo-remote'. [2009-05-27 01:52:50] N [client-protocol.c:5557:client_setvolume_cbk] foo: Connected to 192.168.168.231:6997, attached to remote volume 'foo-remote'. [2009-05-27 04:42:49] W [socket.c:1319:socket_init] trans: disabling non-blocking IO [2009-05-27 04:42:49] W [socket.c:1319:socket_init] trans: disabling non-blocking IO ================================================================================ Version : glusterfs 2.0.1 built on May 26 2009 05:23:52 TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b Starting Time: 2009-05-27 04:42:49 Command line : /usr/sbin/glusterfs2 --log-level=NORMAL --volfile-server=file01.priv --volfile-server-port=6997 /mnt/file01 PID : 13601 System name : Linux Nodename : xxx012.bar.foo.net Kernel Release : 2.6.18-92.1.10.el5PAE Hardware Identifier: i686 Given volfile: +------------------------------------------------------------------------------+ 1: volume foo 2: type protocol/client 3: option transport-type tcp 4: option remote-host file01.priv # can be IP or hostname 5: option remote-port 6997 6: option remote-subvolume foo-remote 7: end-volume 8: 9: volume foo-readahead 10: type performance/read-ahead 11: option page-count 4 12: subvolumes foo 13: end-volume 14: 15: volume foo-writebehind 16: type performance/write-behind 17: option cache-size 512KB 18: subvolumes foo-readahead 19: end-volume 20: 21: volume foo-iothreads 22: type performance/io-threads 23: option autoscaling on 24: option min-threads 16 25: option max-threads 256 26: subvolumes foo-writebehind 27: end-volume 28: 29: volume foo-iocache 30: type performance/io-cache 31: option cache-size 64MB 32: option page-size 128KB 33: option cache-timeout 2 34: subvolumes foo-iothreads 35: end-volume +------------------------------------------------------------------------------+ [2009-05-27 04:42:50] N [glusterfsd.c:1152:main] glusterfs: Successfully started [2009-05-27 04:42:50] N [client-protocol.c:5557:client_setvolume_cbk] foo: Connected to 192.168.168.231:6997, attached to remote volume 'foo-remote'. [2009-05-27 04:42:50] N [client-protocol.c:5557:client_setvolume_cbk] foo: Connected to 192.168.168.231:6997, attached to remote volume 'foo-remote'. [2009-05-27 08:00:24] W [socket.c:1319:socket_init] trans: disabling non-blocking IO [2009-05-27 08:00:24] W [socket.c:1319:socket_init] trans: disabling non-blocking IO ================================================================================ Version : glusterfs 2.0.1 built on May 26 2009 05:23:52 TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b Starting Time: 2009-05-27 08:00:24 Command line : /usr/sbin/glusterfs2 --log-level=NORMAL --volfile-server=file01.priv --volfile-server-port=6997 /mnt/file01 PID : 30261 System name : Linux Nodename : xxx012.bar.foo.net Kernel Release : 2.6.18-92.1.10.el5PAE Hardware Identifier: i686 Given volfile: +------------------------------------------------------------------------------+ 1: volume foo 2: type protocol/client 3: option transport-type tcp 4: option remote-host file01.priv # can be IP or hostname 5: option remote-port 6997 6: option remote-subvolume foo-remote 7: end-volume 8: 9: volume foo-readahead 10: type performance/read-ahead 11: option page-count 4 12: subvolumes foo 13: end-volume 14: 15: volume foo-writebehind 16: type performance/write-behind 17: option cache-size 512KB 18: subvolumes foo-readahead 19: end-volume 20: 21: volume foo-iothreads 22: type performance/io-threads 23: option autoscaling on 24: option min-threads 16 25: option max-threads 256 26: subvolumes foo-writebehind 27: end-volume 28: 29: volume foo-iocache 30: type performance/io-cache 31: option cache-size 64MB 32: option page-size 128KB 33: option cache-timeout 2 34: subvolumes foo-iothreads 35: end-volume +------------------------------------------------------------------------------+ [2009-05-27 08:00:24] N [glusterfsd.c:1152:main] glusterfs: Successfully started [2009-05-27 08:00:24] N [client-protocol.c:5557:client_setvolume_cbk] foo: Connected to 192.168.168.231:6997, attached to remote volume 'foo-remote'. [2009-05-27 08:00:24] N [client-protocol.c:5557:client_setvolume_cbk] foo: Connected to 192.168.168.231:6997, attached to remote volume 'foo-remote'. [2009-05-27 11:38:12] W [socket.c:1319:socket_init] trans: disabling non-blocking IO [2009-05-27 11:38:12] W [socket.c:1319:socket_init] trans: disabling non-blocking IO ================================================================================ Version : glusterfs 2.0.1 built on May 26 2009 05:23:52 TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b Starting Time: 2009-05-27 11:38:12 Command line : /usr/sbin/glusterfs2 --log-level=NORMAL --volfile-server=file01.priv --volfile-server-port=6997 /mnt/file01 PID : 16293 System name : Linux Nodename : xxx012.bar.foo.net Kernel Release : 2.6.18-92.1.10.el5PAE Hardware Identifier: i686 Given volfile: +------------------------------------------------------------------------------+ 1: volume foo 2: type protocol/client 3: option transport-type tcp 4: option remote-host file01.priv # can be IP or hostname 5: option remote-port 6997 6: option remote-subvolume foo-remote 7: end-volume 8: 9: volume foo-readahead 10: type performance/read-ahead 11: option page-count 4 12: subvolumes foo 13: end-volume 14: 15: volume foo-writebehind 16: type performance/write-behind 17: option cache-size 512KB 18: subvolumes foo-readahead 19: end-volume 20: 21: volume foo-iothreads 22: type performance/io-threads 23: option autoscaling on 24: option min-threads 16 25: option max-threads 256 26: subvolumes foo-writebehind 27: end-volume 28: 29: volume foo-iocache 30: type performance/io-cache 31: option cache-size 64MB 32: option page-size 128KB 33: option cache-timeout 2 34: subvolumes foo-iothreads 35: end-volume +------------------------------------------------------------------------------+ [2009-05-27 11:38:12] N [glusterfsd.c:1152:main] glusterfs: Successfully started [2009-05-27 11:38:12] N [client-protocol.c:5557:client_setvolume_cbk] foo: Connected to 192.168.168.231:6997, attached to remote volume 'foo-remote'. [2009-05-27 11:38:12] N [client-protocol.c:5557:client_setvolume_cbk] foo: Connected to 192.168.168.231:6997, attached to remote volume 'foo-remote'. Thanks, Ioannis
begin:vcard fn:Ioannis Aslanidis n:Aslanidis;Ioannis org:Flumotion Services S.A.;Infrastructure Department adr:Edifici Nord Planta 2;;World Trade Center;Barcelona;Barcelona;08039;Spain email;internet:iaslanidis@xxxxxxxxxxxxx title:System and Network Administrator tel;work:+34935086359 tel;cell:+34672204575 note;quoted-printable:PGP Key: 0xBEAC0800 (pgp.rediris.es)=0D=0A= Key fingerprint =3D 73FE B836 D116 1EF1 D580 C06E 16AF BCC3 BEAC 0800 url:http://www.flumotion.com version:2.1 end:vcard
Attachment:
signature.asc
Description: OpenPGP digital signature
