I have a SL 6.1, newly upgraded 5-node 12-brick 3.3.2 cluster. ext4 is the base filesystem. The clients are the same release. During a "remove-brick" drain of a failing array one of the other arrays failed. It was replaced. The filesystem is now no longer read-only, but it is in a mess. What can I do to try to retrieve the situation? Thank you for your time, James Bellinger # ls /data/uwa ls: cannot access /data/uwa/naoko: Invalid argument ls: cannot access /data/uwa/IC79Oscillations: Invalid argument ls: cannot access /data/uwa/desiati: Invalid argument ls: cannot access /data/uwa/kopper: Invalid argument ls: cannot access /data/uwa/omurchadha: Invalid argument ls: cannot access /data/uwa/mfbaker: Invalid argument ls: cannot access /data/uwa/cweaver: Invalid argument ls: cannot access /data/uwa/hoshina: Invalid argument ls: cannot access /data/uwa/pfendner: Invalid argument briedel ckopper cweaver dima hskarlupka jauffenb jfeintzeig karle krasberg mahlers mmerck nwhitehorn pettus rasha rmaruyama swesterhoff chwendt cprice desiati hoshina IC79Oscillations jeisch jvansanten kopper lost+found mfbaker naoko omurchadha pfendner richards santander sybenzvi Yet some file are accessible: # wc /data/uwa/jvansanten/sim/sweep.sh 10 31 153 /data/uwa/jvansanten/sim/sweep.sh The configuration is the same as it was before, modulo the removal of the brick I was trying to drain and the replacement of the failed brick, and the automatic changes going from 3.2.3 to 3.3.2 by rpm install. Samples from the log files Client: /var/log/glusterfs/data-uwa/log [2013-09-20 20:44:44.673716] E [client-handshake.c:1298:client_dump_version_cbk] 0-scratch-client-2: server doesn't support the version [2013-09-20 20:44:44.673778] I [client.c:1883:client_rpc_notify] 0-scratch-client-2: disconnected [2013-09-20 20:44:44.679971] E [client-handshake.c:1298:client_dump_version_cbk] 0-scratch-client-3: server doesn't support the version [2013-09-20 20:44:44.680008] I [client.c:1883:client_rpc_notify] 0-scratch-client-3: disconnected Server: /var/log/glusterfs/brick/sda: 613:[2013-09-18 20:43:27.922169] I [server-handshake.c:571:server_setvolume] 0-scratch-server: accepted client from npx4.icecube.wisc.edu-30969-2013/09/18-20:43:23:857292-scratch-client-4-0 (version: 3.3.2) 664:[2013-09-19 13:03:06.957171] I [server-handshake.c:571:server_setvolume] 0-scratch-server: accepted client from npx4.icecube.wisc.edu-30969-2013/09/18-20:43:23:857292-scratch-client-3-1 (version: 3.3.2) 798:[2013-09-19 13:03:53.802320] I [server.c:703:server_rpc_notify] 0-scratch-server: disconnecting connectionfrom npx4.icecube.wisc.edu-30969-2013/09/18-20:43:23:857292-scratch-client-4-0 799:[2013-09-19 13:03:53.802356] I [server-helpers.c:741:server_connection_put] 0-scratch-server: Shutting down connection npx4.icecube.wisc.edu-30969-2013/09/18-20:43:23:857292-scratch-client-4-0 800:[2013-09-19 13:03:53.802386] I [server-helpers.c:629:server_connection_destroy] 0-scratch-server: destroyed connection of npx4.icecube.wisc.edu-30969-2013/09/18-20:43:23:857292-scratch-client-4-0 801:[2013-09-19 13:04:10.383589] I [server.c:703:server_rpc_notify] 0-scratch-server: disconnecting connectionfrom npx4.icecube.wisc.edu-30969-2013/09/18-20:43:23:857292-scratch-client-3-1 802:[2013-09-19 13:04:10.383672] I [server-helpers.c:741:server_connection_put] 0-scratch-server: Shutting down connection npx4.icecube.wisc.edu-30969-2013/09/18-20:43:23:857292-scratch-client-3-1 803:[2013-09-19 13:04:10.383708] I [server-helpers.c:629:server_connection_destroy] 0-scratch-server: destroyed connection of npx4.icecube.wisc.edu-30969-2013/09/18-20:43:23:857292-scratch-client-3-1 804:[2013-09-19 13:04:17.228206] I [server-handshake.c:571:server_setvolume] 0-scratch-server: accepted client from npx4.icecube.wisc.edu-13204-2013/09/19-13:04:13:172868-scratch-client-3-0 (version: 3.3.2) data-uwa.log: [2012-12-26 10:40:14.893300] E [common-utils.c:125:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known) [2012-12-26 10:40:14.893343] E [name.c:253:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host gfs-npx [2012-12-26 10:40:14.893364] E [glusterfsd-mgmt.c:740:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: Success [2012-12-26 10:40:14.893429] W [glusterfsd.c:727:cleanup_and_exit] (-->/lib64/libpthread.so.0() [0x30fd2077f1] (-->/opt/glusterfs/3.2.3/lib64/libglusterfs.so.0(gf_timer_proc+0xb9) [0x7f61d4931089] (-->/opt/glusterfs/3.2.3/sbin/glusterfs() [0x407dbe]))) 0-: received signum (1), shutting down (The DNS resolution complaint is at least a year old, and seems to have gone away when I hardwired the /etc/hosts file. It makes no difference.) etc-glusterfs-glusterd.vol.log: [2013-09-21 01:53:02.499976] I [socket.c:1798:socket_event_handler] 0-transport: disconnecting now
