Jon Heese
Systems
Engineer
INetU
Managed Hosting
P:
610.266.7441 x 261
F:
610.266.7434
www.inetu.net
**
This message contains confidential
information, which also may be privileged,
and is intended only for the person(s)
addressed above. Any unauthorized use,
distribution, copying or disclosure of
confidential and/or privileged information
is strictly prohibited. If you have received
this communication in error, please erase
all copies of the message and its
attachments and notify the sender
immediately via reply e-mail. **
From: Jonathan Heese
Sent: Tuesday, March 17, 2015 12:36
PM
To: 'Ravishankar N';
gluster-users@xxxxxxxxxxx
Subject: RE: I/O
error on replicated volume
Ravi,
The
last lines in the mount log before the massive
vomit of I/O errors are from 22 minutes prior,
and seem innocuous to me:
[2015-03-16
01:37:07.126340] E
[client-handshake.c:1760:client_query_portmap_cbk]
0-gluster_disk-client-0: failed to get the port
number for remote subvolume. Please run 'gluster
volume status' on server to see if brick process
is running.
[2015-03-16
01:37:07.126587] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a]))) 0-gluster_disk-client-0:
disconnect called (peer:10.10.10.1:24008)
[2015-03-16
01:37:07.126687] E
[client-handshake.c:1760:client_query_portmap_cbk]
0-gluster_disk-client-1: failed to get the port
number for remote subvolume. Please run 'gluster
volume status' on server to see if brick process
is running.
[2015-03-16
01:37:07.126737] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a]))) 0-gluster_disk-client-1:
disconnect called (peer:10.10.10.2:24008)
[2015-03-16
01:37:10.730165] I
[rpc-clnt.c:1729:rpc_clnt_reconfig]
0-gluster_disk-client-0: changing port to 49152
(from 0)
[2015-03-16
01:37:10.730276] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a]))) 0-gluster_disk-client-0:
disconnect called (peer:10.10.10.1:24008)
[2015-03-16
01:37:10.739500] I
[rpc-clnt.c:1729:rpc_clnt_reconfig]
0-gluster_disk-client-1: changing port to 49152
(from 0)
[2015-03-16
01:37:10.739560] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a]))) 0-gluster_disk-client-1:
disconnect called (peer:10.10.10.2:24008)
[2015-03-16
01:37:10.741883] I
[client-handshake.c:1677:select_server_supported_programs]
0-gluster_disk-client-0: Using Program GlusterFS
3.3, Num (1298437), Version (330)
[2015-03-16
01:37:10.744524] I
[client-handshake.c:1462:client_setvolume_cbk]
0-gluster_disk-client-0: Connected to
10.10.10.1:49152, attached to remote volume
'/bricks/brick1'.
[2015-03-16
01:37:10.744537] I
[client-handshake.c:1474:client_setvolume_cbk]
0-gluster_disk-client-0: Server and Client
lk-version numbers are not same, reopening the
fds
[2015-03-16
01:37:10.744566] I
[afr-common.c:4267:afr_notify]
0-gluster_disk-replicate-0: Subvolume
'gluster_disk-client-0' came back up; going
online.
[2015-03-16
01:37:10.744627] I
[client-handshake.c:450:client_set_lk_version_cbk]
0-gluster_disk-client-0: Server lk version = 1
[2015-03-16
01:37:10.753037] I
[client-handshake.c:1677:select_server_supported_programs]
0-gluster_disk-client-1: Using Program GlusterFS
3.3, Num (1298437), Version (330)
[2015-03-16
01:37:10.755657] I
[client-handshake.c:1462:client_setvolume_cbk]
0-gluster_disk-client-1: Connected to
10.10.10.2:49152, attached to remote volume
'/bricks/brick1'.
[2015-03-16
01:37:10.755676] I
[client-handshake.c:1474:client_setvolume_cbk]
0-gluster_disk-client-1: Server and Client
lk-version numbers are not same, reopening the
fds
[2015-03-16
01:37:10.761945] I
[fuse-bridge.c:5016:fuse_graph_setup] 0-fuse:
switched to graph 0
[2015-03-16
01:37:10.762144] I
[client-handshake.c:450:client_set_lk_version_cbk]
0-gluster_disk-client-1: Server lk version = 1
[2015-03-16
01:37:10.762279] I
[fuse-bridge.c:3953:fuse_init] 0-glusterfs-fuse:
FUSE inited with protocol versions: glusterfs
7.22 kernel 7.14
[2015-03-16
01:59:26.098670] W
[fuse-bridge.c:2242:fuse_writev_cbk]
0-glusterfs-fuse: 292084: WRITE => -1
(Input/output error)
…
I’ve
seen no indication of split-brain on any files
at any point in this (ever since downdating from
3.6.2 to 3.5.3, which is when this particular
issue started):
[root@duke
gfapi-module-for-linux-target-driver-]# gluster
v heal gluster_disk info
Brick
duke.jonheese.local:/bricks/brick1/
Number
of entries: 0
Brick
duchess.jonheese.local:/bricks/brick1/
Number
of entries: 0
Thanks.
Jon Heese
Systems
Engineer
INetU
Managed Hosting
P:
610.266.7441 x 261
F:
610.266.7434
www.inetu.net
**
This message contains confidential
information, which also may be privileged,
and is intended only for the person(s)
addressed above. Any unauthorized use,
distribution, copying or disclosure of
confidential and/or privileged information
is strictly prohibited. If you have received
this communication in error, please erase
all copies of the message and its
attachments and notify the sender
immediately via reply e-mail. **
On 03/17/2015 02:14 AM,
Jonathan Heese wrote:
Hello,
So I resolved my previous issue with
split-brains and the lack of self-healing
by dropping my installed glusterfs*
packages from 3.6.2 to 3.5.3, but now I've
picked up a new issue, which actually
makes normal use of the volume practically
impossible.
A little background for those not already
paying close attention:
I have a 2 node 2 brick replicating volume
whose purpose in life is to hold iSCSI
target files, primarily for use to provide
datastores to a VMware ESXi cluster. The
plan is to put a handful of image files on
the Gluster volume, mount them locally on
both Gluster nodes, and run tgtd on both,
pointed to the image files on the mounted
gluster volume. Then the ESXi boxes will
use multipath (active/passive) iSCSI to
connect to the nodes, with automatic
failover in case of planned or unplanned
downtime of the Gluster nodes.
In my most recent round of testing with
3.5.3, I'm seeing a massive failure to
write data to the volume after about 5-10
minutes, so I've simplified the scenario a
bit (to minimize the variables) to: both
Gluster nodes up, only one node (duke)
mounted and running tgtd, and just regular
(single path) iSCSI from a single ESXi
server.
About 5-10 minutes into migration a VM
onto the test datastore, /var/log/messages
on duke gets blasted with a ton of
messages exactly like this:
Mar
15 22:24:06 duke tgtd: bs_rdwr_request(180)
io error 0x1781e00 2a -1 512 22971904,
Input/output error
And
/var/log/glusterfs/mnt-gluster_disk.log gets
blased with a ton of messages exactly like
this:
[2015-03-16
02:24:07.572279] W
[fuse-bridge.c:2242:fuse_writev_cbk]
0-glusterfs-fuse: 635299: WRITE => -1
(Input/output error)
Are there any messages in the mount log from AFR
about split-brain just before the above line
appears?
Does `gluster v heal <VOLNAME> info` show
any files? Performing I/O on files that are in
split-brain fail with EIO.
-Ravi
And
the write operation from VMware's side fails
as soon as these messages start.
I
don't see any other errors (in the log files
I know of) indicating the root cause of
these i/o errors. I'm sure that this is not
enough information to tell what's going on,
but can anyone help me figure out what to
look at next to figure this out?
I've
also considered using Dan Lambright's
libgfapi gluster module for tgtd (or
something similar) to avoid going through
FUSE, but I'm not sure whether that would be
irrelevant to this problem, since I'm not
100% sure if it lies in FUSE or elsewhere.
Thanks!
Jon Heese
Systems
Engineer
INetU
Managed Hosting
P:
610.266.7441 x 261
F:
610.266.7434
www.inetu.net
**
This message contains confidential
information, which also may be
privileged, and is intended only for the
person(s) addressed above. Any
unauthorized use, distribution, copying
or disclosure of confidential and/or
privileged information is strictly
prohibited. If you have received this
communication in error, please erase all
copies of the message and its
attachments and notify the sender
immediately via reply e-mail. **
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
