Greetings, I'm working on implementing GlusterFS as a replacement for NFS, and have ran into a strange hiccup with client-side performance translators when benchmarking with dbench (http://samba.org/ftp/tridge/dbench/). Below is the output. Note that it runs normally for ~80 seconds, then seems to stop responding (latency goes up by ~1s every second). Eventually around 50 seconds after that, it gives up. This happens with a 2-server AFR setup or a single backend. With a simple setup (client/io-threads on the client, posix/posix-locks/io-threads/server on the server), it works. If I enable any of iocache, readahead, or writeback on the client side, dbench will crash if the concurrency is very high ('50' breaks it for me... possibly lower, haven't tested exhaustively). IOzone is perfectly happy with all 3 running. I've not tested with any other FUSE filesystem. Honestly I can't say where the problem might be... dbench, FUSE, or GlusterFS. I've not tested any version but 1.3.12a of GlusterFS, running the stock CentOS 5.2 fuse and the GlusterFS patched fuse. Any ideas? Lots of troubleshooting info below.... Note it's supposed to run for 120sec 'warmup' and 600sec 'execute' ... 12 minutes total. Problems generally occur well before the half-way mark. Thanks, Jake CentOS release 5.2 (Final) Linux testbox2.localdomain 2.6.18-92.1.13.el5 #1 SMP Wed Sep 24 19:33:52 EDT 2008 i686 i686 i386 GNU/Linux glusterfs 1.3.12a built on Oct 9 2008 17:04:22 Repository revision: glusterfs--mainline--2.5--patch-799 glusterfs on /testbox3 type fuse (rw,nosuid,nodev,allow_other,default_permissions,max_read=1048576) ================= Test run on an AFR setup with all 3 enabled: ================= [root@testbox1 dbench]# /usr/src/dbench-4.0/dbench -c /usr/src/dbench-4.0/client.txt 50 <snip> 50 940 14.05 MB/sec warmup 78 sec latency 373.806 ms 50 957 13.93 MB/sec warmup 79 sec latency 365.397 ms 50 973 13.82 MB/sec warmup 80 sec latency 590.412 ms 50 985 13.71 MB/sec warmup 81 sec latency 536.578 ms 50 989 13.57 MB/sec warmup 82 sec latency 1025.981 ms 50 994 13.45 MB/sec warmup 83 sec latency 1406.175 ms 50 994 13.29 MB/sec warmup 84 sec latency 1896.278 ms 50 994 13.14 MB/sec warmup 85 sec latency 2897.077 ms 50 994 12.98 MB/sec warmup 86 sec latency 3899.889 ms 50 994 12.83 MB/sec warmup 87 sec latency 4902.711 ms <snip> 50 994 0.00 MB/sec execute 8 sec latency 46970.466 ms 50 994 0.00 MB/sec execute 9 sec latency 47972.272 ms 50 994 0.00 MB/sec execute 10 sec latency 48974.077 ms 50 994 0.00 MB/sec execute 11 sec latency 49975.883 ms [1001] read failed on handle 10087 (No such file or directory) [978] read failed on handle 10081 (No such file or directory) [1002] read failed on handle 10087 (No such file or directory) [971] open ./clients/client27/~dmtmp/PWRPNT/NEWTIPS.PPT failed for handle 10080 (Transport endpoint is not connected) (972) ERROR: handle 10080 was not found [971] open ./clients/client28/~dmtmp/PWRPNT/NEWTIPS.PPT failed for handle 10080 (Transport endpoint is not connected) (972) ERROR: handle 10080 was not found [1008] write failed on handle 10087 (Transport endpoint is not connected) [971] open ./clients/client19/~dmtmp/PWRPNT/NEWTIPS.PPT failed for handle 10080 (Transport endpoint is not connected) (972) ERROR: handle 10080 was not found [1041] open ./clients/client49/~dmtmp/WORD/~$CHAP10.DOC failed for handle 10090 (Transport endpoint is not connected) (1042) ERROR: handle 10090 was not found [1001] read failed on handle 10087 (No such file or directory) [1039] read failed on handle 10089 (No such file or directory) [972] write failed on handle 10080 (File descriptor in bad state) [922] read failed on handle 10066 (No such file or directory) [979] write failed on handle 10081 (Transport endpoint is not connected) [1003] read failed on handle 10087 (No such file or directory) [938] read failed on handle 10070 (Transport endpoint is not connected) [1004] read failed on handle 10087 (No such file or directory) [1002] read failed on handle 10087 (No such file or directory) [1040] read failed on handle 10089 (No such file or directory) Child failed with status 1 [1005] write failed on handle 10087 (Transport endpoint is not connected) [1003] read failed on handle 10087 (No such file or directory) [root@testbox1 dbench]# ==================== Test run with just iocache, single-server single-client (no AFR): ==================== <snip> 50 6960 16.74 MB/sec execute 32 sec latency 128.816 ms 50 7015 16.98 MB/sec execute 33 sec latency 143.153 ms 50 7063 16.96 MB/sec execute 34 sec latency 193.604 ms 50 7063 16.48 MB/sec execute 35 sec latency 1060.934 ms 50 7063 16.03 MB/sec execute 36 sec latency 2061.731 ms 50 7063 15.60 MB/sec execute 37 sec latency 3062.524 ms 50 7063 15.20 MB/sec execute 38 sec latency 4063.325 ms <snip 40+ lines> 50 7063 6.91 MB/sec execute 85 sec latency 50137.294 ms 50 7063 6.83 MB/sec execute 86 sec latency 51139.100 ms [6791] write failed on handle 11244 (Transport endpoint is not connected) Child failed with status 1 [root@testbox1 dbench]# ============ Client log for just iocache run above (many lines following this chunk omitted): ============ 2008-10-14 15:26:23 W [fuse-bridge.c:398:fuse_entry_cbk] glusterfs-fuse: 2: (34) / => 1 Rehashing 0/0 2008-10-14 15:42:57 W [fuse-bridge.c:398:fuse_entry_cbk] glusterfs-fuse: 2: (34) / => 1 Rehashing 0/0 2008-10-14 15:46:31 W [client-protocol.c:4784:client_protocol_cleanup] remote1: cleaning up state in transport object 0x9e8e858 2008-10-14 15:46:31 E [client-protocol.c:4834:client_protocol_cleanup] remote1: forced unwinding frame type(1) op(35) reply=@0x9a6fede8 2008-10-14 15:46:31 E [client-protocol.c:4834:client_protocol_cleanup] remote1: forced unwinding frame type(1) op(39) reply=@0x9a6fede8 2008-10-14 15:46:31 E [client-protocol.c:3446:client_readdir_cbk] remote1: no proper reply from server, returning ENOTCONN 2008-10-14 15:46:31 E [fuse-bridge.c:1940:fuse_readdir_cbk] glusterfs-fuse: 895468: READDIR => -1 (107) 2008-10-14 15:46:31 E [client-protocol.c:4834:client_protocol_cleanup] remote1: forced unwinding frame type(1) op(39) reply=@0x9a6fede8 2008-10-14 15:46:31 E [client-protocol.c:3446:client_readdir_cbk] remote1: no proper reply from server, returning ENOTCONN 2008-10-14 15:46:31 E [fuse-bridge.c:1940:fuse_readdir_cbk] glusterfs-fuse: 895469: READDIR => -1 (107) 2008-10-14 15:46:31 E [client-protocol.c:4834:client_protocol_cleanup] remote1: forced unwinding frame type(1) op(39) reply=@0x9a6fede8 2008-10-14 15:46:31 E [client-protocol.c:3446:client_readdir_cbk] remote1: no proper reply from server, returning ENOTCONN =================================== Client config (enable any of readahead, writeback, or iocache for crash): =================================== volume remote1 type protocol/client option transport-type tcp/client # for TCP/IP transport option remote-host testbox3 # IP address of the remote brick # option remote-port 6996 # default server port is 6996 # option transport-timeout 30 # seconds to wait for a reply # from server for each request option remote-subvolume io-thr # name of the remote volume end-volume ### Add io-threads feature volume iot type performance/io-threads option thread-count 2 # deault is 1 subvolumes remote1 end-volume ### Add readahead feature #volume readahead # type performance/read-ahead # option page-size 512KB # 256KB is the default option # option page-count 16 # 2 is default option # subvolumes iot #end-volume #### Add IO-Cache feature #volume iocache # type performance/io-cache # option cache-size 1024MB # default is 32MB # option page-size 2MB # default is 128KB # option force-revalidate-timeout 5 # subvolumes readahead #end-volume #### Add writeback feature #volume writeback # type performance/write-behind # option flush-behind on # default value is 'off' # option aggregate-size 1MB # default value is 0 # subvolumes iocache #end-volume =========== Server config: =========== volume brick type storage/posix # POSIX FS translator option directory /storage # Export this directory end-volume volume posix-locks type features/posix-locks option mandatory on subvolumes brick end-volume volume io-thr type performance/io-threads option thread-count 4 # deault is 1 option cache-size 64MB # default is 64MB. This is per thread. subvolumes posix-locks end-volume ### Add network serving capability to above brick. volume server type protocol/server option transport-type tcp/server # For TCP/IP transport # option bind-address 192.168.1.10 # Default is to listen on all interfaces # option listen-port 6996 # Default is 6996 # option client-volume-filename /etc/glusterfs/glusterfs-client.vol subvolumes io-thr option auth.ip.io-thr.allow * # Allow access to "brick" volume end-volume
