This is very disturbing. It will be good to see a response to this thread! On Tue, Apr 5, 2011 at 3:36 PM, <luomails at gmail.com> wrote: > Service is down when programs are running > > System: Ubuntu 10.04 (Linux version 2.6.32-21-server (buildd at yellow) (gcc > version 4.4.3 (Ubuntu 4.4.3-4ubuntu5) ) #32-Ubuntu SMP Fri Apr 16 09:17:34 UTC > 2010) > Mount type: Gluster Native Client > > Our service was down when we were running several programs, one machine was > running that would generate thousands files(maybe more), connected with the > main sever(Samba) and saved data in gluster-vol. The machine would run at least > one week, couple days ago, it was down. By the way, we also opened other > sessions to do the data backup(sftp) and some computing. > > When I checked the gluster-vol, I got: > $df -ah > df: `/mnt/gluster-vol': Transport endpoint is not connected > > Then I checked the mount point: > $ ls -la > ls: cannot access gluster-vol: Transport endpoint is not connected > d????????? ?? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ?? gluster-vol > > I also checked the peers from the main server, it couldn't see the peers, when > I ssh to data storages that can see each other. > $ sudo gluster volume info all > No volumes present > $ sudo gluster peer status > No peers present > > Then I checked the status > $ sudo /etc/init.d/glusterd status > ?* glusterd service is not running. > > After I restarted the services, unmount the gluster-vol, and mount it. It > works. My questions is: Is there any restriction for the tasks loaded for > glusterfs 3.1.2? For example, how many files can be accessed at the same time > from gluster-vol? and how many programs can we running at the same time? > > Please find the attached log file for your reference at that time. > > Note: Someone has reported from this group "I have been testing 3.1.2 over the last few days. > My overall impression is that it resolved several bugs from 3.1.1, but the > latest version is still prone to crashing under moderate to heavy loads. " > (http://www.opensubscriber.com/message/gluster-users at gluster.org/15035860.html). > Is it same issue? > > Thanks for your help. We want to figure it out and prevent it in the future. > > --------Related content of log file-------- > [2011-03-31 20:57:05.462233] I [afr-common.c:716:afr_lookup_done] > gluster-vol-replicate-4: background ?meta-data data entry self-heal triggered. > path: /home/junzhu/Projects/Mito_Xing_03_23_11/data > [2011-03-31 20:57:05.465182] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > gluster-vol-replicate-7: background ?meta-data data entry self-heal completed > on /home/junzhu/Projects/Mito_Xing_03_23_11/data > [2011-03-31 20:57:05.465748] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > gluster-vol-replicate-8: background ?meta-data data entry self-heal completed > on /home/junzhu/Projects/Mito_Xing_03_23_11/data > [2011-03-31 20:57:05.465788] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > gluster-vol-replicate-6: background ?meta-data data entry self-heal completed > on /home/junzhu/Projects/Mito_Xing_03_23_11/data > [2011-03-31 20:57:05.466198] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > gluster-vol-replicate-0: background ?meta-data data entry self-heal completed > on /home/junzhu/Projects/Mito_Xing_03_23_11/data > [2011-03-31 20:57:05.466243] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > gluster-vol-replicate-2: background ?meta-data data entry self-heal completed > on /home/junzhu/Projects/Mito_Xing_03_23_11/data > [2011-03-31 20:57:05.496131] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > gluster-vol-replicate-4: background ?meta-data data entry self-heal completed > on /home/junzhu/Projects/Mito_Xing_03_23_11/data > [2011-03-31 20:57:05.529025] I [afr-common.c:662:afr_lookup_done] > gluster-vol-replicate-8: entries are missing in lookup of > /home/junzhu/Projects/Mito_Xing_03_23_11/data/read1.txt. > [2011-03-31 20:57:05.529097] I [afr-common.c:716:afr_lookup_done] > gluster-vol-replicate-8: background ?meta-data data entry self-heal triggered. > path: /home/junzhu/Projects/Mito_Xing_03_23_11/data/read1.txt > [2011-03-31 20:57:05.535098] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > gluster-vol-replicate-8: background ?meta-data data entry self-heal completed > on /home/junzhu/Projects/Mito_Xing_03_23_11/data/read1.txt > pending frames: > > frame : type(1) op(READ) > frame : type(1) op(READ) > frame : type(1) op(READ) > frame : type(1) op(READ) > frame : type(1) op(READ) > frame : type(1) op(READ) > frame : type(1) op(READ) > frame : type(1) op(READ) > frame : type(1) op(CREATE) > frame : type(1) op(CREATE) > frame : type(1) op(CREATE) > frame : type(1) op(CREATE) > frame : type(1) op(CREATE) > frame : type(1) op(READ) > frame : type(1) op(READ) > > patchset: v3.1.1-64-gf2a067c > signal received: 6 > time of crash: 2011-03-31 22:10:01 > configuration details: > argp 1 > backtrace 1 > dlfcn 1 > fdatasync 1 > libpthread 1 > llistxattr 1 > setfsid 1 > spinlock 1 > epoll.h 1 > xattr.h 1 > st_atim.tv_nsec 1 > package-string: glusterfs 3.1.2 > /lib/libc.so.6(+0x33af0)[0x7ff413899af0] > /lib/libc.so.6(gsignal+0x35)[0x7ff413899a75] > /lib/libc.so.6(abort+0x180)[0x7ff41389d5c0] > /lib/libc.so.6(__assert_fail+0xf1)[0x7ff413892941] > /lib/libpthread.so.0(pthread_mutex_lock+0x7b)[0x7ff413bf243b] > /usr/lib64/glusterfs/3.1.2/xlator/performance/io-cache.so(ioc_create_cbk+0xb0)[0x7ff4101d9540] > /usr/lib64/glusterfs/3.1.2/xlator/performance/read-ahead.so(ra_create_cbk+0x1ba)[0x7ff4103e4daa] > /usr/lib64/glusterfs/3.1.2/xlator/performance/write-behind.so(wb_create_cbk+0x10b)[0x7ff4105eee0b] > /usr/lib64/glusterfs/3.1.2/xlator/cluster/distribute.so(dht_create_cbk+0x2b8)[0x7ff410816b18] > /usr/lib64/glusterfs/3.1.2/xlator/cluster/replicate.so(afr_create_unwind+0x12b)[0x7ff410a2fc9b] > /usr/lib64/glusterfs/3.1.2/xlator/cluster/replicate.so(afr_create_wind_cbk+0x128)[0x7ff410a32d78] > /usr/lib64/glusterfs/3.1.2/xlator/protocol/client.so(client3_1_create_cbk+0x92f)[0x7ff410c939cf] > /usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)[0x7ff41422dc15] > /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xc9)[0x7ff41422de69] > /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x2d)[0x7ff41422902d] > /usr/lib64/glusterfs/3.1.2/rpc-transport/socket.so(socket_event_poll_in+0x34)[0x7ff411abd344] > /usr/lib64/glusterfs/3.1.2/rpc-transport/socket.so(socket_event_handler+0xb3)[0x7ff411abd413] > /usr/lib64/libglusterfs.so.0(+0x38592)[0x7ff41446d592] > /usr/sbin/glusterfs(main+0x247)[0x4055a7] > /lib/libc.so.6(__libc_start_main+0xfd)[0x7ff413884c4d] > /usr/sbin/glusterfs[0x4032b9] > --------- > [2011-04-01 08:43:10.105965] W [io-stats.c:1644:init] gluster-vol: dangling > volume. check volfile > [2011-04-01 08:43:10.106106] W [dict.c:1205:data_to_str] dict: @data=(nil) > [2011-04-01 08:43:10.106142] W [dict.c:1205:data_to_str] dict: @data=(nil) > Given volfile: > +------------------------------------------------------------------------------+ > ?1: volume gluster-vol-client-0 > ?2: ? ? type protocol/client > ?3: ? ? option remote-host dscbc-storage1 > ?4: ? ? option remote-subvolume /mnt/gluster1 > ?5: ? ? option transport-type tcp > ?6: end-volume > ?7: > ?8: volume gluster-vol-client-1 > ?9: ? ? type protocol/client > ?10: ? ? option remote-host dscbc-storage2 > ?11: ? ? option remote-subvolume /mnt/gluster1 > ?12: ? ? option transport-type tcp > ?13: end-volume > ?14: > ?15: volume gluster-vol-client-2 > ?16: ? ? type protocol/client > ?17: ? ? option remote-host dscbc-storage3 > ?18: ? ? option remote-subvolume /mnt/gluster1 > ?19: ? ? option transport-type tcp > ?20: end-volume > ?21: > ?22: volume gluster-vol-client-3 > ?23: ? ? type protocol/client > ?24: ? ? option remote-host dscbc-storage4 > ?25: ? ? option remote-subvolume /mnt/gluster1 > ?26: ? ? option transport-type tcp > ?27: end-volume > ?28: > ?29: volume gluster-vol-client-4 > ?30: ? ? type protocol/client > ?31: ? ? option remote-host dscbc-storage1 > ?32: ? ? option remote-subvolume /mnt/gluster2 > ?33: ? ? option transport-type tcp > ?34: end-volume > ?35: > ?36: volume gluster-vol-client-5 > ?37: ? ? type protocol/client > ?38: ? ? option remote-host dscbc-storage2 > ?39: ? ? option remote-subvolume /mnt/gluster2 > ?40: ? ? option transport-type tcp > ?41: end-volume > ?42: > ?43: volume gluster-vol-client-6 > ?44: ? ? type protocol/client > ?45: ? ? option remote-host dscbc-storage3 > ?46: ? ? option remote-subvolume /mnt/gluster2 > ?47: ? ? option transport-type tcp > ?48: end-volume > ?49: > ?50: volume gluster-vol-client-7 > ?51: ? ? type protocol/client > ?52: ? ? option remote-host dscbc-storage4 > ?53: ? ? option remote-subvolume /mnt/gluster2 > ?54: ? ? option transport-type tcp > ?55: end-volume > ?56: > ?57: volume gluster-vol-client-8 > ?58: ? ? type protocol/client > ?59: ? ? option remote-host dscbc-storage1 > ?60: ? ? option remote-subvolume /mnt/gluster3 > ?61: ? ? option transport-type tcp > ?62: end-volume > ?63: > ?64: volume gluster-vol-client-9 > ?65: ? ? type protocol/client > ?66: ? ? option remote-host dscbc-storage2 > ?67: ? ? option remote-subvolume /mnt/gluster3 > ?68: ? ? option transport-type tcp > ?69: end-volume > ?70: > ?71: volume gluster-vol-client-10 > ?72: ? ? type protocol/client > ?73: ? ? option remote-host dscbc-storage3 > ?74: ? ? option remote-subvolume /mnt/gluster3 > ?75: ? ? option transport-type tcp > ?76: end-volume > ?77: > ?78: volume gluster-vol-client-11 > ?79: ? ? type protocol/client > ?80: ? ? option remote-host dscbc-storage4 > ?81: ? ? option remote-subvolume /mnt/gluster3 > ?82: ? ? option transport-type tcp > ?83: end-volume > ?84: > ?85: volume gluster-vol-client-12 > ?86: ? ? type protocol/client > ?87: ? ? option remote-host dscbc-storage1 > ?88: ? ? option remote-subvolume /mnt/gluster4 > ?89: ? ? option transport-type tcp > ?90: end-volume > ?91: > ?92: volume gluster-vol-client-13 > ?93: ? ? type protocol/client > ?94: ? ? option remote-host dscbc-storage2 > ?95: ? ? option remote-subvolume /mnt/gluster4 > ?96: ? ? option transport-type tcp > ?97: end-volume > ?98: > ?99: volume gluster-vol-client-14 > 100: ? ? type protocol/client > 101: ? ? option remote-host dscbc-storage1 > 102: ? ? option remote-subvolume /mnt/gluster5 > 103: ? ? option transport-type tcp > 104: end-volume > 105: > 106: volume gluster-vol-client-15 > 107: ? ? type protocol/client > 108: ? ? option remote-host dscbc-storage2 > 109: ? ? option remote-subvolume /mnt/gluster5 > 110: ? ? option transport-type tcp > 111: end-volume > 112: > 113: volume gluster-vol-client-16 > 114: ? ? type protocol/client > 115: ? ? option remote-host dscbc-storage1 > 116: ? ? option remote-subvolume /mnt/gluster6 > 117: ? ? option transport-type tcp > 118: end-volume > 119: > 120: volume gluster-vol-client-17 > 121: ? ? type protocol/client > 122: ? ? option remote-host dscbc-storage2 > 123: ? ? option remote-subvolume /mnt/gluster6 > 124: ? ? option transport-type tcp > 125: end-volume > 126: > 127: volume gluster-vol-replicate-0 > 128: ? ? type cluster/replicate > 129: ? ? subvolumes gluster-vol-client-0 gluster-vol-client-1 > 130: end-volume > 131: > 132: volume gluster-vol-replicate-1 > 133: ? ? type cluster/replicate > 134: ? ? subvolumes gluster-vol-client-2 gluster-vol-client-3 > 135: end-volume > 136: > 137: volume gluster-vol-replicate-2 > 138: ? ? type cluster/replicate > 139: ? ? subvolumes gluster-vol-client-4 gluster-vol-client-5 > 140: end-volume > 141: > 142: volume gluster-vol-replicate-3 > 143: ? ? type cluster/replicate > 144: ? ? subvolumes gluster-vol-client-6 gluster-vol-client-7 > 145: end-volume > 146: > 147: volume gluster-vol-replicate-4 > 148: ? ? type cluster/replicate > 149: ? ? subvolumes gluster-vol-client-8 gluster-vol-client-9 > 150: end-volume > 151: > 152: volume gluster-vol-replicate-5 > 153: ? ? type cluster/replicate > 154: ? ? subvolumes gluster-vol-client-10 gluster-vol-client-11 > 155: end-volume > 156: > 157: volume gluster-vol-replicate-6 > 158: ? ? type cluster/replicate > 159: ? ? subvolumes gluster-vol-client-12 gluster-vol-client-13 > 160: end-volume > 161: > 162: volume gluster-vol-replicate-7 > 163: ? ? type cluster/replicate > 164: ? ? subvolumes gluster-vol-client-14 gluster-vol-client-15 > 165: end-volume > 166: > 167: volume gluster-vol-replicate-8 > 168: ? ? type cluster/replicate > 169: ? ? subvolumes gluster-vol-client-16 gluster-vol-client-17 > 170: end-volume > 171: > 172: volume gluster-vol-dht > 173: ? ? type cluster/distribute > 174: ? ? subvolumes gluster-vol-replicate-0 gluster-vol-replicate-1 > gluster-vol-replicate-2 gluster-vol-replicate-3 gluster-vol-replicate-4 > gluster-vol-replicate-5 gluster-vol-replicate-6 gluster-vol-replicate-7 > gluster-vol-replicate-8 > 175: end-volume > 176: > 177: volume gluster-vol-write-behind > 178: ? ? type performance/write-behind > 179: ? ? subvolumes gluster-vol-dht > 180: end-volume > 181: > 182: volume gluster-vol-read-ahead > 183: ? ? type performance/read-ahead > 184: ? ? subvolumes gluster-vol-write-behind > 185: end-volume > 186: > 187: volume gluster-vol-io-cache > 188: ? ? type performance/io-cache > 189: ? ? subvolumes gluster-vol-read-ahead > 190: end-volume > 191: > 192: volume gluster-vol-quick-read > 193: ? ? type performance/quick-read > 194: ? ? subvolumes gluster-vol-io-cache > 195: end-volume > 196: > 197: volume gluster-vol-stat-prefetch > 198: ? ? type performance/stat-prefetch > 199: ? ? subvolumes gluster-vol-quick-read > 200: end-volume > 201: > 202: volume gluster-vol > 203: ? ? type debug/io-stats > 204: ? ? subvolumes gluster-vol-stat-prefetch > 205: end-volume > > +------------------------------------------------------------------------------+ > [2011-04-01 08:43:14.48158] I > [client-handshake.c:1005:select_server_supported_programs] > gluster-vol-client-13: Using Program GlusterFS-3.1.0, Num (1298437), Version > (310) > [2011-04-01 08:43:14.48607] I > [client-handshake.c:1005:select_server_supported_programs] > gluster-vol-client-17: Using Program GlusterFS-3.1.0, Num (1298437), Version > (310) > [2011-04-01 08:43:14.49124] I > [client-handshake.c:1005:select_server_supported_programs] > gluster-vol-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version > (310) > [2011-04-01 08:43:14.49581] I [client-handshake.c:841:client_setvolume_cbk] > gluster-vol-client-13: Connected to 165.112.107.217:24012, attached to remote > volume '/mnt/gluster4'. > [2011-04-01 08:43:14.49694] I [afr-common.c:2572:afr_notify] > gluster-vol-replicate-6: Subvolume 'gluster-vol-client-13' came back up; going > online. > [2011-04-01 08:43:14.49937] I [client-handshake.c:841:client_setvolume_cbk] > gluster-vol-client-17: Connected to 165.112.107.217:24014, attached to remote > volume '/mnt/gluster6'. > [2011-04-01 08:43:14.49981] I [afr-common.c:2572:afr_notify] > gluster-vol-replicate-8: Subvolume 'gluster-vol-client-17' came back up; going > online. > [2011-04-01 08:43:14.50182] I [client-handshake.c:841:client_setvolume_cbk] > gluster-vol-client-0: Connected to 165.112.107.216:24009, attached to remote > volume '/mnt/gluster1'. > [2011-04-01 08:43:14.50221] I [afr-common.c:2572:afr_notify] > gluster-vol-replicate-0: Subvolume 'gluster-vol-client-0' came back up; going > online. > [2011-04-01 08:43:14.50834] I > [client-handshake.c:1005:select_server_supported_programs] > gluster-vol-client-3: Using Program GlusterFS-3.1.0, Num (1298437), Version > (310) > [2011-04-01 08:43:14.51365] I > [client-handshake.c:1005:select_server_supported_programs] > gluster-vol-client-8: Using Program GlusterFS-3.1.0, Num (1298437), Version > (310) > [2011-04-01 08:43:14.51719] I [client-handshake.c:841:client_setvolume_cbk] > gluster-vol-client-3: Connected to 165.112.107.214:24009, attached to remote > volume '/mnt/gluster1'. > [2011-04-01 08:43:14.51791] I [afr-common.c:2572:afr_notify] > gluster-vol-replicate-1: Subvolume 'gluster-vol-client-3' came back up; going > online. > [2011-04-01 08:43:14.51970] I > [client-handshake.c:1005:select_server_supported_programs] > gluster-vol-client-10: Using Program GlusterFS-3.1.0, Num (1298437), Version > (310) > [2011-04-01 08:43:14.52103] I [client-handshake.c:841:client_setvolume_cbk] > gluster-vol-client-8: Connected to 165.112.107.216:24011, attached to remote > volume '/mnt/gluster3'. > [2011-04-01 08:43:14.52140] I [afr-common.c:2572:afr_notify] > gluster-vol-replicate-4: Subvolume 'gluster-vol-client-8' came back up; going > online. > [2011-04-01 08:43:14.52264] I > [client-handshake.c:1005:select_server_supported_programs] > gluster-vol-client-14: Using Program GlusterFS-3.1.0, Num (1298437), Version > (310) > [2011-04-01 08:43:14.52840] I [client-handshake.c:841:client_setvolume_cbk] > gluster-vol-client-14: Connected to 165.112.107.216:24013, attached to remote > volume '/mnt/gluster5'. > [2011-04-01 08:43:14.52948] I [afr-common.c:2572:afr_notify] > gluster-vol-replicate-7: Subvolume 'gluster-vol-client-14' came back up; going > online. > [2011-04-01 08:43:14.53177] I [client-handshake.c:841:client_setvolume_cbk] > gluster-vol-client-10: Connected to 165.112.107.213:24011, attached to remote > volume '/mnt/gluster3'. > [2011-04-01 08:43:14.53218] I [afr-common.c:2572:afr_notify] > gluster-vol-replicate-5: Subvolume 'gluster-vol-client-10' came back up; going > online. > [2011-04-01 08:43:14.53449] I > [client-handshake.c:1005:select_server_supported_programs] > gluster-vol-client-4: Using Program GlusterFS-3.1.0, Num (1298437), Version > (310) > [2011-04-01 08:43:14.54084] I [client-handshake.c:841:client_setvolume_cbk] > gluster-vol-client-4: Connected to 165.112.107.216:24010, attached to remote > volume '/mnt/gluster2'. > [2011-04-01 08:43:14.54147] I [afr-common.c:2572:afr_notify] > gluster-vol-replicate-2: Subvolume 'gluster-vol-client-4' came back up; going > online. > [2011-04-01 08:43:14.54307] I > [client-handshake.c:1005:select_server_supported_programs] > gluster-vol-client-12: Using Program GlusterFS-3.1.0, Num (1298437), Version > (310) > [2011-04-01 08:43:14.54487] I > [client-handshake.c:1005:select_server_supported_programs] > gluster-vol-client-16: Using Program GlusterFS-3.1.0, Num (1298437), Version > (310) > [2011-04-01 08:43:14.54973] I [client-handshake.c:841:client_setvolume_cbk] > gluster-vol-client-12: Connected to 165.112.107.216:24012, attached to remote > volume '/mnt/gluster4'. > [2011-04-01 08:43:14.55087] I > [client-handshake.c:1005:select_server_supported_programs] > gluster-vol-client-11: Using Program GlusterFS-3.1.0, Num (1298437), Version > (310) > [2011-04-01 08:43:14.55223] I [client-handshake.c:841:client_setvolume_cbk] > gluster-vol-client-16: Connected to 165.112.107.216:24014, attached to remote > volume '/mnt/gluster6'. > [2011-04-01 08:43:14.55797] I > [client-handshake.c:1005:select_server_supported_programs] > gluster-vol-client-2: Using Program GlusterFS-3.1.0, Num (1298437), Version > (310) > [2011-04-01 08:43:14.56001] I [client-handshake.c:841:client_setvolume_cbk] > gluster-vol-client-11: Connected to 165.112.107.214:24011, attached to remote > volume '/mnt/gluster3'. > [2011-04-01 08:43:14.56299] I > [client-handshake.c:1005:select_server_supported_programs] > gluster-vol-client-6: Using Program GlusterFS-3.1.0, Num (1298437), Version > (310) > [2011-04-01 08:43:14.56455] I [client-handshake.c:841:client_setvolume_cbk] > gluster-vol-client-2: Connected to 165.112.107.213:24009, attached to remote > volume '/mnt/gluster1'. > [2011-04-01 08:43:14.56732] I > [client-handshake.c:1005:select_server_supported_programs] > gluster-vol-client-7: Using Program GlusterFS-3.1.0, Num (1298437), Version > (310) > [2011-04-01 08:43:14.56909] I [client-handshake.c:841:client_setvolume_cbk] > gluster-vol-client-6: Connected to 165.112.107.213:24010, attached to remote > volume '/mnt/gluster2'. > [2011-04-01 08:43:14.56943] I [afr-common.c:2572:afr_notify] > gluster-vol-replicate-3: Subvolume 'gluster-vol-client-6' came back up; going > online. > [2011-04-01 08:43:14.69629] I [fuse-bridge.c:2821:fuse_init] glusterfs-fuse: > FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13 > [2011-04-01 08:43:14.69745] I [client-handshake.c:841:client_setvolume_cbk] > gluster-vol-client-7: Connected to 165.112.107.214:24010, attached to remote > volume '/mnt/gluster2'. > [2011-04-01 08:43:14.69961] I > [client-handshake.c:1005:select_server_supported_programs] > gluster-vol-client-15: Using Program GlusterFS-3.1.0, Num (1298437), Version > (310) > [2011-04-01 08:43:14.70467] I [client-handshake.c:841:client_setvolume_cbk] > gluster-vol-client-15: Connected to 165.112.107.217:24013, attached to remote > volume '/mnt/gluster5'. > [2011-04-01 08:43:14.70627] I [afr-common.c:819:afr_fresh_lookup_cbk] > gluster-vol-replicate-0: added root inode > [2011-04-01 08:43:14.72159] I [afr-common.c:819:afr_fresh_lookup_cbk] > gluster-vol-replicate-3: added root inode > [2011-04-01 08:43:14.72267] I [afr-common.c:819:afr_fresh_lookup_cbk] > gluster-vol-replicate-2: added root inode > [2011-04-01 08:43:14.72350] I [afr-common.c:819:afr_fresh_lookup_cbk] > gluster-vol-replicate-4: added root inode > [2011-04-01 08:43:14.72440] I [afr-common.c:819:afr_fresh_lookup_cbk] > gluster-vol-replicate-1: added root inode > [2011-04-01 08:43:14.72551] I [afr-common.c:819:afr_fresh_lookup_cbk] > gluster-vol-replicate-5: added root inode > [2011-04-01 08:43:14.72654] I [afr-common.c:819:afr_fresh_lookup_cbk] > gluster-vol-replicate-7: added root inode > [2011-04-01 08:43:14.72712] I [afr-common.c:819:afr_fresh_lookup_cbk] > gluster-vol-replicate-6: added root inode > [2011-04-01 08:43:14.72981] I [afr-common.c:819:afr_fresh_lookup_cbk] > gluster-vol-replicate-8: added root inode > [2011-04-01 08:43:14.148489] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk] > gluster-vol-replicate-0: ?entry self-heal triggered. path: > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005, reason: > checksums of directory differ, forced merge option set > [2011-04-01 08:43:14.149294] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk] > gluster-vol-replicate-2: ?entry self-heal triggered. path: > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005, reason: > checksums of directory differ, forced merge option set > [2011-04-01 08:43:14.149411] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk] > gluster-vol-replicate-4: ?entry self-heal triggered. path: > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005, reason: > checksums of directory differ, forced merge option set > [2011-04-01 08:43:14.149529] E [afr-common.c:110:afr_set_split_brain] > gluster-vol-replicate-0: invalid argument: inode > [2011-04-01 08:43:14.149563] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > gluster-vol-replicate-0: background ?entry self-heal completed on > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005 > [2011-04-01 08:43:14.150346] E [afr-common.c:110:afr_set_split_brain] > gluster-vol-replicate-2: invalid argument: inode > [2011-04-01 08:43:14.150409] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > gluster-vol-replicate-2: background ?entry self-heal completed on > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005 > [2011-04-01 08:43:14.150474] E [afr-common.c:110:afr_set_split_brain] > gluster-vol-replicate-4: invalid argument: inode > [2011-04-01 08:43:14.150503] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > gluster-vol-replicate-4: background ?entry self-heal completed on > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005 > [2011-04-01 08:43:14.725449] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk] > gluster-vol-replicate-0: ?entry self-heal triggered. path: > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005/C104.1, > reason: checksums of directory differ, forced merge option set > [2011-04-01 08:43:14.725637] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk] > gluster-vol-replicate-2: ?entry self-heal triggered. path: > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005/C104.1, > reason: checksums of directory differ, forced merge option set > [2011-04-01 08:43:14.726038] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk] > gluster-vol-replicate-4: ?entry self-heal triggered. path: > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005/C104.1, > reason: checksums of directory differ, forced merge option set > [2011-04-01 08:43:14.726518] E [afr-common.c:110:afr_set_split_brain] > gluster-vol-replicate-0: invalid argument: inode > [2011-04-01 08:43:14.726582] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > gluster-vol-replicate-0: background ?entry self-heal completed on > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005/C104.1 > [2011-04-01 08:43:14.726641] E [afr-common.c:110:afr_set_split_brain] > gluster-vol-replicate-2: invalid argument: inode > [2011-04-01 08:43:14.726670] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > gluster-vol-replicate-2: background ?entry self-heal completed on > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005/C104.1 > [2011-04-01 08:43:14.727285] E [afr-common.c:110:afr_set_split_brain] > gluster-vol-replicate-4: invalid argument: inode > [2011-04-01 08:43:14.727351] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > gluster-vol-replicate-4: background ?entry self-heal completed on > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005/C104.1 > [2011-04-01 08:43:15.50797] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk] > gluster-vol-replicate-2: ?entry self-heal triggered. path: > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005, reason: > checksums of directory differ, forced merge option set > [2011-04-01 08:43:15.51193] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk] > gluster-vol-replicate-4: ?entry self-heal triggered. path: > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005, reason: > checksums of directory differ, forced merge option set > [2011-04-01 08:43:15.51713] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk] > gluster-vol-replicate-0: ?entry self-heal triggered. path: > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005, reason: > checksums of directory differ, forced merge option set > [2011-04-01 08:43:15.52627] E [afr-common.c:110:afr_set_split_brain] > gluster-vol-replicate-2: invalid argument: inode > [2011-04-01 08:43:15.52690] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > gluster-vol-replicate-2: background ?entry self-heal completed on > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005 > [2011-04-01 08:43:15.52753] E [afr-common.c:110:afr_set_split_brain] > gluster-vol-replicate-4: invalid argument: inode > [2011-04-01 08:43:15.52781] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > gluster-vol-replicate-4: background ?entry self-heal completed on > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005 > [2011-04-01 08:43:15.53061] E [afr-common.c:110:afr_set_split_brain] > gluster-vol-replicate-0: invalid argument: inode > [2011-04-01 08:43:15.53123] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > gluster-vol-replicate-0: background ?entry self-heal completed on > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005 > [2011-04-01 08:43:15.490907] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk] > gluster-vol-replicate-0: ?entry self-heal triggered. path: > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005/C104.1, > reason: checksums of directory differ, forced merge option set > [2011-04-01 08:43:15.491743] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk] > gluster-vol-replicate-2: ?entry self-heal triggered. path: > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005/C104.1, > reason: checksums of directory differ, forced merge option set > [2011-04-01 08:43:15.492202] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk] > gluster-vol-replicate-4: ?entry self-heal triggered. path: > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005/C104.1, > reason: checksums of directory differ, forced merge option set > [2011-04-01 08:43:15.492399] E [afr-common.c:110:afr_set_split_brain] > gluster-vol-replicate-0: invalid argument: inode > [2011-04-01 08:43:15.492434] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > gluster-vol-replicate-0: background ?entry self-heal completed on > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005/C104.1 > [2011-04-01 08:43:15.492789] E [afr-common.c:110:afr_set_split_brain] > gluster-vol-replicate-2: invalid argument: inode > [2011-04-01 08:43:15.492850] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > gluster-vol-replicate-2: background ?entry self-heal completed on > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005/C104.1 > [2011-04-01 08:43:15.493051] E [afr-common.c:110:afr_set_split_brain] > gluster-vol-replicate-4: invalid argument: inode > [2011-04-01 08:43:15.493105] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > gluster-vol-replicate-4: background ?entry self-heal completed on > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005/C104.1 > [2011-04-01 08:43:15.553051] I [afr-common.c:716:afr_lookup_done] > gluster-vol-replicate-0: background ?entry self-heal triggered. path: > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005 > [2011-04-01 08:43:15.553465] I [afr-common.c:716:afr_lookup_done] > gluster-vol-replicate-4: background ?entry self-heal triggered. path: > /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005 > [2011-04-01 08:43:15.553640] I [afr-common.c:716:afr_lookup_done] > gluster-vol-replicate-2: background ?entry self-heal triggered. path: / > > -- > This message was sent on behalf of luomails at gmail.com at openSubscriber.com > http://www.opensubscriber.com/message/gluster-users at gluster.org/15035860.html > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >
