Hi, for testing purposes I setup dht with afr with 4 bricks using the latest 1.4rc7. Making a simple "rsync -av --delete-during /etc /mount/r1" and killing some node during the copy crashes some random glusterfs nodes. For the exact volume specs etc. please have a look at the attached logs. start r1 r2 r3 r4 rsync -av --delete-during /etc /mount/r1 kill r3 (sometimes crashes happen here...) wait some seconds start r3 (sometimes crashes happen here...) All data directories and mount points are empty before starting the tests. If it's an error with my volume descriptions, if you need more info etc. please let me know too. Corin
Version : glusterfs 1.4.0rc7 built on Jan 9 2009 15:02:56 TLA Revision : glusterfs--mainline--3.0--patch-814 Starting Time: 2009-01-09 19:20:53 Command line : ../sbin/glusterfs --volfile=r1.vol --pid-file=r1.pid --log-file=r1.log --volume-name=client /home/glusterfs/test/mount/r1 given volfile +----- 1: # volume specification file for server r1 2: 3: # ---- 4: 5: volume storage-posix 6: type storage/posix 7: option directory /usr/local/glusterfs/test/data/r1 8: end-volume 9: 10: volume storage-locks 11: type features/locks 12: subvolumes storage-posix 13: end-volume 14: 15: volume r1 16: type performance/io-threads 17: option thread-count 8 18: subvolumes storage-locks 19: end-volume 20: 21: # ---- 22: 23: volume server 24: type protocol/server 25: option transport-type tcp 26: option bind-address 127.0.0.1 27: option listen-port 6997 28: option auth.addr.r1.allow * 29: subvolumes r1 30: end-volume 31: 32: # ---- 33: 34: 35: volume r2 36: type protocol/client 37: option transport-type tcp 38: option remote-host 127.0.0.1 39: option remote-port 6998 40: option transport-timeout 5 41: option remote-subvolume r2 42: end-volume 43: 44: volume r3 45: type protocol/client 46: option transport-type tcp 47: option remote-host 127.0.0.1 48: option remote-port 6999 49: option transport-timeout 5 50: option remote-subvolume r3 51: end-volume 52: 53: volume r4 54: type protocol/client 55: option transport-type tcp 56: option remote-host 127.0.0.1 57: option remote-port 7000 58: option transport-timeout 5 59: option remote-subvolume r4 60: end-volume 61: 62: 63: # ---- 64: 65: volume afr_r1_r2 66: type cluster/afr 67: option read-subvolume r1 # prefer local volume 68: subvolumes r1 r2 69: end-volume 70: 71: volume afr_r3_r4 72: type cluster/afr 73: subvolumes r3 r4 74: end-volume 75: 76: 77: # ---- 78: 79: volume client-dht 80: type cluster/dht 81: subvolumes afr_r1_r2 afr_r3_r4 82: end-volume 83: 84: volume client 85: type performance/io-cache 86: option cache-size 512MB 87: option page-size 16KB 88: option cache-timeout 3 89: subvolumes client-dht 90: end-volume +----- 2009-01-09 19:20:53 W [xlator.c:382:validate_xlator_volume_options] server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction 2009-01-09 19:20:53 W [xlator.c:382:validate_xlator_volume_options] server: option 'bind-address' is deprecated, preferred is 'transport.socket.bind-address', continuing with correction 2009-01-09 19:20:54 W [dht.c:108:dht_lookup_dir_cbk] client-dht: lookup of / on afr_r3_r4 returned error (Transport endpoint is not connected) 2009-01-09 19:20:54 E [dht-layout.c:456:dht_layout_normalize] client-dht: found anomalies in /. holes=1 overlaps=0 2009-01-09 19:20:54 W [dht.c:132:dht_lookup_dir_cbk] client-dht: fixing assignment on / 2009-01-09 19:20:54 E [dht-selfheal.c:359:dht_selfheal_directory] client-dht: 1 subvolumes down -- not fixing 2009-01-09 19:20:54 E [socket.c:708:socket_connect_finish] r4: connection failed (Connection refused) 2009-01-09 19:20:54 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash 2009-01-09 19:20:54 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash 2009-01-09 19:20:54 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 3: LOOKUP() /.Trash => -1 (Invalid argument) 2009-01-09 19:20:54 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash-1000 2009-01-09 19:20:54 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash-1000 2009-01-09 19:20:54 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 4: LOOKUP() /.Trash-1000 => -1 (Invalid argument) 2009-01-09 19:20:54 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash 2009-01-09 19:20:54 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash 2009-01-09 19:20:54 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 5: LOOKUP() /.Trash => -1 (Invalid argument) 2009-01-09 19:20:54 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash-1000 2009-01-09 19:20:54 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash-1000 2009-01-09 19:20:54 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 6: LOOKUP() /.Trash-1000 => -1 (Invalid argument) 2009-01-09 19:20:55 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash 2009-01-09 19:20:55 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash 2009-01-09 19:20:55 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 7: LOOKUP() /.Trash => -1 (Invalid argument) 2009-01-09 19:20:55 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash-1000 2009-01-09 19:20:55 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash-1000 2009-01-09 19:20:55 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 8: LOOKUP() /.Trash-1000 => -1 (Invalid argument) 2009-01-09 19:20:55 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash 2009-01-09 19:20:55 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash 2009-01-09 19:20:55 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 9: LOOKUP() /.Trash => -1 (Invalid argument) 2009-01-09 19:20:55 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash-1000 2009-01-09 19:20:55 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash-1000 2009-01-09 19:20:55 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 10: LOOKUP() /.Trash-1000 => -1 (Invalid argument) 2009-01-09 19:20:55 W [dht.c:598:dht_lookup] client-dht: incomplete layout failure for path=/ 2009-01-09 19:20:55 W [fuse-bridge.c:297:need_fresh_lookup] fuse-bridge: revalidate of / failed (Resource temporarily unavailable) 2009-01-09 19:21:04 E [saved-frames.c:148:saved_frames_unwind] r3: forced unwinding frame type(1) op(STAT) 2009-01-09 19:21:04 E [saved-frames.c:148:saved_frames_unwind] r3: forced unwinding frame type(1) op(XATTROP) 2009-01-09 19:21:04 E [saved-frames.c:148:saved_frames_unwind] r3: forced unwinding frame type(3) op(RELEASE) pending frames: à¼ÒHÿ Signal received: 11 configuration details:argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 tv_nsec 1 package-string: glusterfs 1.4.0rc7 /lib/libc.so.6[0x7fc340180060] ---------
Version : glusterfs 1.4.0rc7 built on Jan 9 2009 15:02:56 TLA Revision : glusterfs--mainline--3.0--patch-814 Starting Time: 2009-01-09 19:20:53 Command line : ../sbin/glusterfs --volfile=r2.vol --pid-file=r2.pid --log-file=r2.log --volume-name=client /home/glusterfs/test/mount/r2 given volfile +----- 1: # volume specification file for server r2 2: 3: # ---- 4: 5: volume storage-posix 6: type storage/posix 7: option directory /usr/local/glusterfs/test/data/r2 8: end-volume 9: 10: volume storage-locks 11: type features/locks 12: subvolumes storage-posix 13: end-volume 14: 15: volume r2 16: type performance/io-threads 17: option thread-count 8 18: subvolumes storage-locks 19: end-volume 20: 21: # ---- 22: 23: volume server 24: type protocol/server 25: option transport-type tcp 26: option bind-address 127.0.0.1 27: option listen-port 6998 28: option auth.addr.r2.allow * 29: subvolumes r2 30: end-volume 31: 32: # ---- 33: 34: volume r1 35: type protocol/client 36: option transport-type tcp 37: option remote-host 127.0.0.1 38: option remote-port 6997 39: option transport-timeout 5 40: option remote-subvolume r1 41: end-volume 42: 43: 44: volume r3 45: type protocol/client 46: option transport-type tcp 47: option remote-host 127.0.0.1 48: option remote-port 6999 49: option transport-timeout 5 50: option remote-subvolume r3 51: end-volume 52: 53: volume r4 54: type protocol/client 55: option transport-type tcp 56: option remote-host 127.0.0.1 57: option remote-port 7000 58: option transport-timeout 5 59: option remote-subvolume r4 60: end-volume 61: 62: 63: # ---- 64: 65: volume afr_r1_r2 66: type cluster/afr 67: option read-subvolume r2 # prefer local volume 68: subvolumes r2 r1 69: end-volume 70: 71: volume afr_r3_r4 72: type cluster/afr 73: subvolumes r3 r4 74: end-volume 75: 76: 77: # ---- 78: 79: volume client-dht 80: type cluster/dht 81: subvolumes afr_r1_r2 afr_r3_r4 82: end-volume 83: 84: volume client 85: type performance/io-cache 86: option cache-size 512MB 87: option page-size 16KB 88: option cache-timeout 3 89: subvolumes client-dht 90: end-volume +----- 2009-01-09 19:20:53 W [xlator.c:382:validate_xlator_volume_options] server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction 2009-01-09 19:20:53 W [xlator.c:382:validate_xlator_volume_options] server: option 'bind-address' is deprecated, preferred is 'transport.socket.bind-address', continuing with correction 2009-01-09 19:20:54 W [dht.c:108:dht_lookup_dir_cbk] client-dht: lookup of / on afr_r3_r4 returned error (Transport endpoint is not connected) 2009-01-09 19:20:54 E [dht-layout.c:456:dht_layout_normalize] client-dht: found anomalies in /. holes=1 overlaps=0 2009-01-09 19:20:54 W [dht.c:132:dht_lookup_dir_cbk] client-dht: fixing assignment on / 2009-01-09 19:20:54 E [dht-selfheal.c:359:dht_selfheal_directory] client-dht: 1 subvolumes down -- not fixing 2009-01-09 19:20:54 E [socket.c:708:socket_connect_finish] r4: connection failed (Connection refused) 2009-01-09 19:20:54 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash 2009-01-09 19:20:54 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash 2009-01-09 19:20:54 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 3: LOOKUP() /.Trash => -1 (Invalid argument) 2009-01-09 19:20:54 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash-1000 2009-01-09 19:20:54 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash-1000 2009-01-09 19:20:54 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 4: LOOKUP() /.Trash-1000 => -1 (Invalid argument) 2009-01-09 19:20:54 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash 2009-01-09 19:20:54 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash 2009-01-09 19:20:54 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 5: LOOKUP() /.Trash => -1 (Invalid argument) 2009-01-09 19:20:54 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash-1000 2009-01-09 19:20:54 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash-1000 2009-01-09 19:20:54 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 6: LOOKUP() /.Trash-1000 => -1 (Invalid argument) 2009-01-09 19:20:55 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash 2009-01-09 19:20:55 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash 2009-01-09 19:20:55 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 7: LOOKUP() /.Trash => -1 (Invalid argument) 2009-01-09 19:20:55 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash-1000 2009-01-09 19:20:55 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash-1000 2009-01-09 19:20:55 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 8: LOOKUP() /.Trash-1000 => -1 (Invalid argument) 2009-01-09 19:20:55 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash 2009-01-09 19:20:55 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash 2009-01-09 19:20:55 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 9: LOOKUP() /.Trash => -1 (Invalid argument) 2009-01-09 19:20:55 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash-1000 2009-01-09 19:20:55 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash-1000 2009-01-09 19:20:55 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 10: LOOKUP() /.Trash-1000 => -1 (Invalid argument) 2009-01-09 19:20:55 W [dht.c:598:dht_lookup] client-dht: incomplete layout failure for path=/ 2009-01-09 19:20:55 W [fuse-bridge.c:297:need_fresh_lookup] fuse-bridge: revalidate of / failed (Resource temporarily unavailable) 2009-01-09 19:21:04 E [socket.c:708:socket_connect_finish] r3: connection failed (Connection refused) 2009-01-09 19:21:04 E [socket.c:708:socket_connect_finish] r1: connection failed (Connection refused)
Version : glusterfs 1.4.0rc7 built on Jan 9 2009 15:02:56 TLA Revision : glusterfs--mainline--3.0--patch-814 Starting Time: 2009-01-09 19:21:09 Command line : ../sbin/glusterfs --volfile=r3.vol --pid-file=r3.pid --log-file=r3.log --volume-name=client /home/glusterfs/test/mount/r3 given volfile +----- 1: # volume specification file for server r3 2: 3: # ---- 4: 5: volume storage-posix 6: type storage/posix 7: option directory /usr/local/glusterfs/test/data/r3 8: end-volume 9: 10: volume storage-locks 11: type features/locks 12: subvolumes storage-posix 13: end-volume 14: 15: volume r3 16: type performance/io-threads 17: option thread-count 8 18: subvolumes storage-locks 19: end-volume 20: 21: # ---- 22: 23: volume server 24: type protocol/server 25: option transport-type tcp 26: option bind-address 127.0.0.1 27: option listen-port 6999 28: option auth.addr.r3.allow * 29: subvolumes r3 30: end-volume 31: 32: # ---- 33: 34: volume r1 35: type protocol/client 36: option transport-type tcp 37: option remote-host 127.0.0.1 38: option remote-port 6997 39: option transport-timeout 5 40: option remote-subvolume r1 41: end-volume 42: 43: volume r2 44: type protocol/client 45: option transport-type tcp 46: option remote-host 127.0.0.1 47: option remote-port 6998 48: option transport-timeout 5 49: option remote-subvolume r2 50: end-volume 51: 52: 53: volume r4 54: type protocol/client 55: option transport-type tcp 56: option remote-host 127.0.0.1 57: option remote-port 7000 58: option transport-timeout 5 59: option remote-subvolume r4 60: end-volume 61: 62: 63: # ---- 64: 65: volume afr_r1_r2 66: type cluster/afr 67: subvolumes r1 r2 68: end-volume 69: 70: volume afr_r3_r4 71: type cluster/afr 72: option read-subvolume r3 # prefer local volume 73: subvolumes r3 r4 74: end-volume 75: 76: 77: # ---- 78: 79: volume client-dht 80: type cluster/dht 81: subvolumes afr_r1_r2 afr_r3_r4 82: end-volume 83: 84: volume client 85: type performance/io-cache 86: option cache-size 512MB 87: option page-size 16KB 88: option cache-timeout 3 89: subvolumes client-dht 90: end-volume +----- 2009-01-09 19:21:09 W [xlator.c:382:validate_xlator_volume_options] server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction 2009-01-09 19:21:09 W [xlator.c:382:validate_xlator_volume_options] server: option 'bind-address' is deprecated, preferred is 'transport.socket.bind-address', continuing with correction 2009-01-09 19:21:10 W [dht.c:108:dht_lookup_dir_cbk] client-dht: lookup of / on afr_r1_r2 returned error (Transport endpoint is not connected) 2009-01-09 19:21:10 E [dht-layout.c:456:dht_layout_normalize] client-dht: found anomalies in /. holes=1 overlaps=0 2009-01-09 19:21:10 W [dht.c:132:dht_lookup_dir_cbk] client-dht: fixing assignment on / 2009-01-09 19:21:10 E [dht-selfheal.c:359:dht_selfheal_directory] client-dht: 1 subvolumes down -- not fixing 2009-01-09 19:21:10 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash 2009-01-09 19:21:10 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash 2009-01-09 19:21:10 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 4: LOOKUP() /.Trash => -1 (Invalid argument) 2009-01-09 19:21:10 E [socket.c:708:socket_connect_finish] r1: connection failed (Connection refused) 2009-01-09 19:21:10 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash 2009-01-09 19:21:10 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash 2009-01-09 19:21:10 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 6: LOOKUP() /.Trash => -1 (Invalid argument) 2009-01-09 19:21:12 W [dht.c:598:dht_lookup] client-dht: incomplete layout failure for path=/ 2009-01-09 19:21:12 W [fuse-bridge.c:297:need_fresh_lookup] fuse-bridge: revalidate of / failed (Resource temporarily unavailable) 2009-01-09 19:21:13 W [afr-self-heal-entry.c:1159:afr_sh_entry_impunge_symlink] afr_r3_r4: creating symlink /alternatives/jstack -> /usr/lib/jvm/java-1.5.0-sun/bin/jstack on r3
Version : glusterfs 1.4.0rc7 built on Jan 9 2009 15:02:56 TLA Revision : glusterfs--mainline--3.0--patch-814 Starting Time: 2009-01-09 19:20:54 Command line : ../sbin/glusterfs --volfile=r4.vol --pid-file=r4.pid --log-file=r4.log --volume-name=client /home/glusterfs/test/mount/r4 given volfile +----- 1: # volume specification file for server r4 2: 3: # ---- 4: 5: volume storage-posix 6: type storage/posix 7: option directory /usr/local/glusterfs/test/data/r4 8: end-volume 9: 10: volume storage-locks 11: type features/locks 12: subvolumes storage-posix 13: end-volume 14: 15: volume r4 16: type performance/io-threads 17: option thread-count 8 18: subvolumes storage-locks 19: end-volume 20: 21: # ---- 22: 23: volume server 24: type protocol/server 25: option transport-type tcp 26: option bind-address 127.0.0.1 27: option listen-port 7000 28: option auth.addr.r4.allow * 29: subvolumes r4 30: end-volume 31: 32: # ---- 33: 34: volume r1 35: type protocol/client 36: option transport-type tcp 37: option remote-host 127.0.0.1 38: option remote-port 6997 39: option transport-timeout 5 40: option remote-subvolume r1 41: end-volume 42: 43: volume r2 44: type protocol/client 45: option transport-type tcp 46: option remote-host 127.0.0.1 47: option remote-port 6998 48: option transport-timeout 5 49: option remote-subvolume r2 50: end-volume 51: 52: volume r3 53: type protocol/client 54: option transport-type tcp 55: option remote-host 127.0.0.1 56: option remote-port 6999 57: option transport-timeout 5 58: option remote-subvolume r3 59: end-volume 60: 61: 62: 63: # ---- 64: 65: volume afr_r1_r2 66: type cluster/afr 67: subvolumes r1 r2 68: end-volume 69: 70: volume afr_r3_r4 71: type cluster/afr 72: option read-subvolume r4 # prefer local volume 73: subvolumes r4 r3 74: end-volume 75: 76: 77: # ---- 78: 79: volume client-dht 80: type cluster/dht 81: subvolumes afr_r1_r2 afr_r3_r4 82: end-volume 83: 84: volume client 85: type performance/io-cache 86: option cache-size 512MB 87: option page-size 16KB 88: option cache-timeout 3 89: subvolumes client-dht 90: end-volume +----- 2009-01-09 19:20:54 W [xlator.c:382:validate_xlator_volume_options] server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction 2009-01-09 19:20:54 W [xlator.c:382:validate_xlator_volume_options] server: option 'bind-address' is deprecated, preferred is 'transport.socket.bind-address', continuing with correction 2009-01-09 19:20:55 W [dht.c:108:dht_lookup_dir_cbk] client-dht: lookup of / on afr_r1_r2 returned error (Transport endpoint is not connected) 2009-01-09 19:20:55 E [dht-layout.c:456:dht_layout_normalize] client-dht: found anomalies in /. holes=1 overlaps=0 2009-01-09 19:20:55 W [dht.c:132:dht_lookup_dir_cbk] client-dht: fixing assignment on / 2009-01-09 19:20:55 E [dht-selfheal.c:359:dht_selfheal_directory] client-dht: 1 subvolumes down -- not fixing 2009-01-09 19:20:55 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash 2009-01-09 19:20:55 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash 2009-01-09 19:20:55 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 3: LOOKUP() /.Trash => -1 (Invalid argument) 2009-01-09 19:20:55 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash-1000 2009-01-09 19:20:55 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash-1000 2009-01-09 19:20:55 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 4: LOOKUP() /.Trash-1000 => -1 (Invalid argument) 2009-01-09 19:20:55 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash 2009-01-09 19:20:55 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash 2009-01-09 19:20:55 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 5: LOOKUP() /.Trash => -1 (Invalid argument) 2009-01-09 19:20:55 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash-1000 2009-01-09 19:20:55 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash-1000 2009-01-09 19:20:55 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 6: LOOKUP() /.Trash-1000 => -1 (Invalid argument) 2009-01-09 19:20:55 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash 2009-01-09 19:20:55 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash 2009-01-09 19:20:55 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 7: LOOKUP() /.Trash => -1 (Invalid argument) 2009-01-09 19:20:55 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash-1000 2009-01-09 19:20:55 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash-1000 2009-01-09 19:20:55 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 8: LOOKUP() /.Trash-1000 => -1 (Invalid argument) 2009-01-09 19:20:55 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash 2009-01-09 19:20:55 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash 2009-01-09 19:20:55 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 9: LOOKUP() /.Trash => -1 (Invalid argument) 2009-01-09 19:20:55 E [dht-helper.c:225:dht_subvol_get_hashed] client-dht: could not find subvolume for path=/.Trash-1000 2009-01-09 19:20:55 E [dht.c:623:dht_lookup] client-dht: no subvolume in layout for path=/.Trash-1000 2009-01-09 19:20:55 E [fuse-bridge.c:406:fuse_entry_cbk] glusterfs-fuse: 10: LOOKUP() /.Trash-1000 => -1 (Invalid argument) 2009-01-09 19:20:56 W [dht.c:598:dht_lookup] client-dht: incomplete layout failure for path=/ 2009-01-09 19:20:56 W [fuse-bridge.c:297:need_fresh_lookup] fuse-bridge: revalidate of / failed (Resource temporarily unavailable) 2009-01-09 19:21:05 E [socket.c:708:socket_connect_finish] r1: connection failed (Connection refused) 2009-01-09 19:21:05 E [socket.c:708:socket_connect_finish] r3: connection failed (Connection refused)
