Jon Heese
Systems
Engineer
INetU
Managed Hosting
P:
610.266.7441 x 261
F:
610.266.7434
www.inetu.net
** This message
contains confidential information,
which also may be privileged, and
is intended only for the person(s)
addressed above. Any unauthorized
use, distribution, copying or
disclosure of confidential and/or
privileged information is strictly
prohibited. If you have received
this communication in error,
please erase all copies of the
message and its attachments and
notify the sender immediately via
reply e-mail. **
From: Jonathan
Heese
Sent: Tuesday, March 17,
2015 12:36 PM
To: 'Ravishankar N';
gluster-users@xxxxxxxxxxx
Subject: RE:
I/O error on
replicated volume
Ravi,
The last lines
in the mount log before the massive
vomit of I/O errors are from 22
minutes prior, and seem innocuous to
me:
[2015-03-16
01:37:07.126340] E
[client-handshake.c:1760:client_query_portmap_cbk]
0-gluster_disk-client-0: failed to get
the port number for remote subvolume.
Please run 'gluster volume status' on
server to see if brick process is
running.
[2015-03-16
01:37:07.126587] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a])))
0-gluster_disk-client-0: disconnect
called (peer:10.10.10.1:24008)
[2015-03-16
01:37:07.126687] E
[client-handshake.c:1760:client_query_portmap_cbk]
0-gluster_disk-client-1: failed to get
the port number for remote subvolume.
Please run 'gluster volume status' on
server to see if brick process is
running.
[2015-03-16
01:37:07.126737] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a])))
0-gluster_disk-client-1: disconnect
called (peer:10.10.10.2:24008)
[2015-03-16
01:37:10.730165] I
[rpc-clnt.c:1729:rpc_clnt_reconfig]
0-gluster_disk-client-0: changing port
to 49152 (from 0)
[2015-03-16
01:37:10.730276] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a])))
0-gluster_disk-client-0: disconnect
called (peer:10.10.10.1:24008)
[2015-03-16
01:37:10.739500] I
[rpc-clnt.c:1729:rpc_clnt_reconfig]
0-gluster_disk-client-1: changing port
to 49152 (from 0)
[2015-03-16
01:37:10.739560] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a])))
0-gluster_disk-client-1: disconnect
called (peer:10.10.10.2:24008)
[2015-03-16
01:37:10.741883] I
[client-handshake.c:1677:select_server_supported_programs]
0-gluster_disk-client-0: Using Program
GlusterFS 3.3, Num (1298437), Version
(330)
[2015-03-16
01:37:10.744524] I
[client-handshake.c:1462:client_setvolume_cbk]
0-gluster_disk-client-0: Connected to
10.10.10.1:49152, attached to remote
volume '/bricks/brick1'.
[2015-03-16
01:37:10.744537] I
[client-handshake.c:1474:client_setvolume_cbk]
0-gluster_disk-client-0: Server and
Client lk-version numbers are not
same, reopening the fds
[2015-03-16
01:37:10.744566] I
[afr-common.c:4267:afr_notify]
0-gluster_disk-replicate-0: Subvolume
'gluster_disk-client-0' came back up;
going online.
[2015-03-16
01:37:10.744627] I
[client-handshake.c:450:client_set_lk_version_cbk]
0-gluster_disk-client-0: Server lk
version = 1
[2015-03-16
01:37:10.753037] I
[client-handshake.c:1677:select_server_supported_programs]
0-gluster_disk-client-1: Using Program
GlusterFS 3.3, Num (1298437), Version
(330)
[2015-03-16
01:37:10.755657] I
[client-handshake.c:1462:client_setvolume_cbk]
0-gluster_disk-client-1: Connected to
10.10.10.2:49152, attached to remote
volume '/bricks/brick1'.
[2015-03-16
01:37:10.755676] I
[client-handshake.c:1474:client_setvolume_cbk]
0-gluster_disk-client-1: Server and
Client lk-version numbers are not
same, reopening the fds
[2015-03-16
01:37:10.761945] I
[fuse-bridge.c:5016:fuse_graph_setup]
0-fuse: switched to graph 0
[2015-03-16
01:37:10.762144] I
[client-handshake.c:450:client_set_lk_version_cbk]
0-gluster_disk-client-1: Server lk
version = 1
[2015-03-16
01:37:10.762279] I
[fuse-bridge.c:3953:fuse_init]
0-glusterfs-fuse: FUSE inited with
protocol versions: glusterfs 7.22
kernel 7.14
[2015-03-16
01:59:26.098670] W
[fuse-bridge.c:2242:fuse_writev_cbk]
0-glusterfs-fuse: 292084: WRITE =>
-1 (Input/output error)
…
I’ve seen no
indication of split-brain on any files
at any point in this (ever since
downdating from 3.6.2 to 3.5.3, which
is when this particular issue
started):
[root@duke
gfapi-module-for-linux-target-driver-]#
gluster v heal gluster_disk info
Brick
duke.jonheese.local:/bricks/brick1/
Number of
entries: 0
Brick
duchess.jonheese.local:/bricks/brick1/
Number of
entries: 0
Thanks.
Jon Heese
Systems
Engineer
INetU
Managed Hosting
P:
610.266.7441 x 261
F:
610.266.7434
www.inetu.net
** This message
contains confidential information,
which also may be privileged, and
is intended only for the person(s)
addressed above. Any unauthorized
use, distribution, copying or
disclosure of confidential and/or
privileged information is strictly
prohibited. If you have received
this communication in error,
please erase all copies of the
message and its attachments and
notify the sender immediately via
reply e-mail. **
On 03/17/2015 02:14
AM, Jonathan Heese wrote:
Hello,
So I resolved my previous issue
with split-brains and the lack
of self-healing by dropping my
installed glusterfs* packages
from 3.6.2 to 3.5.3, but now
I've picked up a new issue,
which actually makes normal use
of the volume practically
impossible.
A little background for those
not already paying close
attention:
I have a 2 node 2 brick
replicating volume whose purpose
in life is to hold iSCSI target
files, primarily for use to
provide datastores to a VMware
ESXi cluster. The plan is to
put a handful of image files on
the Gluster volume, mount them
locally on both Gluster nodes,
and run tgtd on both, pointed to
the image files on the mounted
gluster volume. Then the ESXi
boxes will use multipath
(active/passive) iSCSI to
connect to the nodes, with
automatic failover in case of
planned or unplanned downtime of
the Gluster nodes.
In my most recent round of
testing with 3.5.3, I'm seeing a
massive failure to write data to
the volume after about 5-10
minutes, so I've simplified the
scenario a bit (to minimize the
variables) to: both Gluster
nodes up, only one node (duke)
mounted and running tgtd, and
just regular (single path) iSCSI
from a single ESXi server.
About 5-10 minutes into
migration a VM onto the test
datastore, /var/log/messages on
duke gets blasted with a ton of
messages exactly like this:
Mar 15
22:24:06 duke tgtd:
bs_rdwr_request(180) io error
0x1781e00 2a -1 512 22971904,
Input/output error
And
/var/log/glusterfs/mnt-gluster_disk.log
gets blased with a ton of messages
exactly like this:
[2015-03-16
02:24:07.572279] W
[fuse-bridge.c:2242:fuse_writev_cbk]
0-glusterfs-fuse: 635299: WRITE
=> -1 (Input/output error)
Are there any messages in the mount
log from AFR about split-brain just
before the above line appears?
Does `gluster v heal <VOLNAME>
info` show any files? Performing I/O
on files that are in split-brain fail
with EIO.
-Ravi
And the
write operation from VMware's side
fails as soon as these messages
start.
I don't
see any other errors (in the log
files I know of) indicating the
root cause of these i/o errors.
I'm sure that this is not enough
information to tell what's going
on, but can anyone help me figure
out what to look at next to figure
this out?
I've also
considered using Dan Lambright's
libgfapi gluster module for tgtd
(or something similar) to avoid
going through FUSE, but I'm not
sure whether that would be
irrelevant to this problem, since
I'm not 100% sure if it lies in
FUSE or elsewhere.
Thanks!
Jon Heese
Systems
Engineer
INetU
Managed Hosting
P:
610.266.7441 x 261
F:
610.266.7434
www.inetu.net
** This message
contains confidential
information, which also may be
privileged, and is intended
only for the person(s)
addressed above. Any
unauthorized use,
distribution, copying or
disclosure of confidential
and/or privileged information
is strictly prohibited. If you
have received this
communication in error, please
erase all copies of the
message and its attachments
and notify the sender
immediately via reply e-mail.
**
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
