I recently upgraded my dev cluster to 3.3. To do this I copied the data out of the old volume into a bare disk, wiped out everything about Gluster, installed the 3.3 packages, create a new volume (I wanted to change my brick layout), then copied the data back into the new volume. Previously everything worked fine but now my users are complaining of random errors when compiling software. I enabled debug logging for the clients and I see this: x36e3613ad3] (-->/usr/lib64/libglusterfs.so.0(mem_get0+0x1b) [0x36e364018b]))) 0-mem-pool: Mem pool is full. Callocing mem [2012-06-12 17:12:02.783526] D [mem-pool.c:457:mem_get] (-->/usr/lib64/libglusterfs.so.0(dict_unserialize+0x28d) [0x36e361413d] (-->/usr/lib64/libglusterfs.so.0(dict_set+0x163) [0 x36e3613ad3] (-->/usr/lib64/libglusterfs.so.0(mem_get0+0x1b) [0x36e364018b]))) 0-mem-pool: Mem pool is full. Callocing mem [2012-06-12 17:12:02.783584] D [mem-pool.c:457:mem_get] (-->/usr/lib64/libglusterfs.so.0(dict_unserialize+0x28d) [0x36e361413d] (-->/usr/lib64/libglusterfs.so.0(dict_set+0x163) [0 x36e3613ad3] (-->/usr/lib64/libglusterfs.so.0(mem_get0+0x1b) [0x36e364018b]))) 0-mem-pool: Mem pool is full. Callocing mem [2012-06-12 17:12:45.726083] D [client-handshake.c:184:client_start_ping] 0-vol_home-client-0: returning as transport is already disconnected OR there are no frames (0 || 0) [2012-06-12 17:12:45.726154] D [client-handshake.c:184:client_start_ping] 0-vol_home-client-3: returning as transport is already disconnected OR there are no frames (0 || 0) [2012-06-12 17:12:45.726171] D [client-handshake.c:184:client_start_ping] 0-vol_home-client-1: returning as transport is already disconnected OR there are no frames (0 || 0) *[2012-06-12 17:15:35.888437] E [rpc-clnt.c:208:call_bail] 0-vol_home-client-2: bailing out frame type(GlusterFS 3.1) op(RENAME(8)) xid = 0x2015421x sent = 2012-06-12 16:45:26.237621. timeout = 1800* [2012-06-12 17:15:35.888507] W [client3_1-fops.c:2385:client3_1_rename_cbk] 0-vol_home-client-2: remote operation failed: Transport endpoint is not connected [2012-06-12 17:15:35.888529] W [dht-rename.c:478:dht_rename_cbk] 0-vol_home-dht: /sam/senthil/genboree/SupportingPkgs/gcc-3.4.6/x86_64-unknown-linux-gnu/32/libjava/java/net/SocketException.class.tmp: rename on vol_home-client-2 failed (Transport endpoint is not connected) [2012-06-12 17:15:35.889803] W [fuse-bridge.c:1516:fuse_rename_cbk] 0-glusterfs-fuse: 2776710: /sam/senthil/genboree/SupportingPkgs/gcc-3.4.6/x86_64-unknown-linux-gnu/32/libjava/java/net/SocketException.class.tmp -> /sam/senthil/genboree/SupportingPkgs/gcc-3.4.6/x86_64-unknown-linux-gnu/32/libjava/java/net/SocketException.class => -1 (Transport endpoint is not connected) [2012-06-12 17:15:35.890002] D [mem-pool.c:457:mem_get] (-->/usr/lib64/libglusterfs.so.0(dict_new+0xb) [0x36e3613d6b] (-->/usr/lib64/libglusterfs.so.0(get_new_dict_full+0x27) [0x36e3613c67] (-->/usr/lib64/libglusterfs.so.0(mem_get0+0x1b) [0x36e364018b]))) 0-mem-pool: Mem pool is full. Callocing mem [2012-06-12 17:15:35.890167] D [mem-pool.c:457:mem_get] (-->/usr/lib64/glusterfs/3.3.0/xlator/performance/md-cache.so(mdc_load_reqs+0x3d) [0x2aaaac201a2d] (-->/usr/lib64/libglusterfs.so.0(dict_set+0x163) [0x36e3613ad3] (-->/usr/lib64/libglusterfs.so.0(mem_get0+0x1b) [0x36e364018b]))) 0-mem-pool: Mem pool is full. Callocing mem [2012-06-12 17:15:35.890258] D [mem-pool.c:457:mem_get] (-->/usr/lib64/glusterfs/3.3.0/xlator/performance/md-cache.so(mdc_load_reqs+0x3d) [0x2aaaac201a2d] (-->/usr/lib64/libglusterfs.so.0(dict_set+0x163) [0x36e3613ad3] (-->/usr/lib64/libglusterfs.so.0(mem_get0+0x1b) [0x36e364018b]))) 0-mem-pool: Mem pool is full. Callocing mem [2012-06-12 17:15:35.890311] D [mem-pool.c:457:mem_get] (-->/usr/lib64/glusterfs/3.3.0/xlator/performance/md-cache.so(mdc_load_reqs+0x3d) [0x2aaaac201a2d] (-->/usr/lib64/libglusterfs.so.0(dict_set+0x163) [0x36e3613ad3] (-->/usr/lib64/libglusterfs.so.0(mem_get0+0x1b) [0x36e364018b]))) 0-mem-pool: Mem pool is full. Callocing mem [2012-06-12 17:15:35.890363] D [mem-pool.c:457:mem_get] (-->/usr/lib64/glusterfs/3.3.0/xlator/performance/md-cache.so(mdc_load_reqs+0x3d) [0x2aaaac201a2d] (-->/usr/lib64/libglusterfs.so.0(dict_set+0x163) [0x36e3613ad3] (-->/usr/lib64/libglusterfs.so.0(mem_get0+0x1b) [0x36e364018b]))) 0-mem-pool: Mem pool is full. Callocing mem ** and so on, more of the same... If I enable debug logging on the bricks I see thousands of these lines every minute and I'm forced to disable the logging: [2012-06-12 15:32:45.760598] D [io-threads.c:268:iot_schedule] 0-vol_home-io-threads: LOOKUP scheduled as fast fop Here's my config: # gluster volume info Volume Name: vol_home Type: Distribute Volume ID: 07ec60be-ec0c-4579-a675-069bb34c12ab Status: Started Number of Bricks: 4 Transport-type: tcp Bricks: Brick1: storage0-dev.cssd.pitt.edu:/brick/0 Brick2: storage1-dev.cssd.pitt.edu:/brick/2 Brick3: storage0-dev.cssd.pitt.edu:/brick/1 Brick4: storage1-dev.cssd.pitt.edu:/brick/3 Options Reconfigured: diagnostics.brick-log-level: INFO diagnostics.client-log-level: INFO features.limit-usage: /home/cssd/jaw171:50GB,/cssd:200GB,/cssd/jaw171:75GB nfs.rpc-auth-allow: 10.54.50.*,127.* auth.allow: 10.54.50.*,127.* performance.io-cache: off cluster.min-free-disk: 5 performance.cache-size: 128000000 features.quota: on nfs.disable: on # rpm -qa | grep gluster glusterfs-fuse-3.3.0-1.el6.x86_64 glusterfs-server-3.3.0-1.el6.x86_64 glusterfs-3.3.0-1.el6.x86_64 Name resolution is fine on everything, everything can ping everything else by name, no firewalls are running anywhere, there's no disk errors on the storage nodes. Did the way I copied data out of one volume and back into another cause this (some xattr problem)? What else could be causing this problem? I'm looking to go production with GlusterFS on a 242 (soon to grow) node HPC cluster at the end of this month. Also, one of my co-workers improved upon an existing remote quota viewer written in Python. I'll post the code soon for those interested. -- Jeff White - Linux/Unix Systems Engineer University of Pittsburgh - CSSD -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gluster.org/pipermail/gluster-users/attachments/20120613/4756895e/attachment.htm>