Jon Heese
Systems
Engineer
INetU
Managed Hosting
P:
610.266.7441 x 261
F:
610.266.7434
www.inetu.net
** This message
contains confidential
information, which also may be
privileged, and is intended only
for the person(s) addressed
above. Any unauthorized use,
distribution, copying or
disclosure of confidential
and/or privileged information is
strictly prohibited. If you have
received this communication in
error, please erase all copies
of the message and its
attachments and notify the
sender immediately via reply
e-mail. **
From:
Jonathan Heese
Sent: Tuesday, March 17,
2015 12:36 PM
To: 'Ravishankar N'; gluster-users@xxxxxxxxxxx
Subject: RE:
I/O error on
replicated volume
Ravi,
The last lines
in the mount log before the massive
vomit of I/O errors are from 22
minutes prior, and seem innocuous to
me:
[2015-03-16
01:37:07.126340] E
[client-handshake.c:1760:client_query_portmap_cbk]
0-gluster_disk-client-0: failed to
get the port number for remote
subvolume. Please run 'gluster
volume status' on server to see if
brick process is running.
[2015-03-16
01:37:07.126587] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a])))
0-gluster_disk-client-0: disconnect
called (peer:10.10.10.1:24008)
[2015-03-16
01:37:07.126687] E
[client-handshake.c:1760:client_query_portmap_cbk]
0-gluster_disk-client-1: failed to
get the port number for remote
subvolume. Please run 'gluster
volume status' on server to see if
brick process is running.
[2015-03-16
01:37:07.126737] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a])))
0-gluster_disk-client-1: disconnect
called (peer:10.10.10.2:24008)
[2015-03-16
01:37:10.730165] I
[rpc-clnt.c:1729:rpc_clnt_reconfig]
0-gluster_disk-client-0: changing
port to 49152 (from 0)
[2015-03-16
01:37:10.730276] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a])))
0-gluster_disk-client-0: disconnect
called (peer:10.10.10.1:24008)
[2015-03-16
01:37:10.739500] I
[rpc-clnt.c:1729:rpc_clnt_reconfig]
0-gluster_disk-client-1: changing
port to 49152 (from 0)
[2015-03-16
01:37:10.739560] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a])))
0-gluster_disk-client-1: disconnect
called (peer:10.10.10.2:24008)
[2015-03-16
01:37:10.741883] I
[client-handshake.c:1677:select_server_supported_programs]
0-gluster_disk-client-0: Using
Program GlusterFS 3.3, Num
(1298437), Version (330)
[2015-03-16
01:37:10.744524] I
[client-handshake.c:1462:client_setvolume_cbk]
0-gluster_disk-client-0: Connected
to 10.10.10.1:49152, attached to
remote volume '/bricks/brick1'.
[2015-03-16
01:37:10.744537] I
[client-handshake.c:1474:client_setvolume_cbk]
0-gluster_disk-client-0: Server and
Client lk-version numbers are not
same, reopening the fds
[2015-03-16
01:37:10.744566] I
[afr-common.c:4267:afr_notify]
0-gluster_disk-replicate-0:
Subvolume 'gluster_disk-client-0'
came back up; going online.
[2015-03-16
01:37:10.744627] I
[client-handshake.c:450:client_set_lk_version_cbk]
0-gluster_disk-client-0: Server lk
version = 1
[2015-03-16
01:37:10.753037] I
[client-handshake.c:1677:select_server_supported_programs]
0-gluster_disk-client-1: Using
Program GlusterFS 3.3, Num
(1298437), Version (330)
[2015-03-16
01:37:10.755657] I
[client-handshake.c:1462:client_setvolume_cbk]
0-gluster_disk-client-1: Connected
to 10.10.10.2:49152, attached to
remote volume '/bricks/brick1'.
[2015-03-16
01:37:10.755676] I
[client-handshake.c:1474:client_setvolume_cbk]
0-gluster_disk-client-1: Server and
Client lk-version numbers are not
same, reopening the fds
[2015-03-16
01:37:10.761945] I
[fuse-bridge.c:5016:fuse_graph_setup]
0-fuse: switched to graph 0
[2015-03-16
01:37:10.762144] I
[client-handshake.c:450:client_set_lk_version_cbk]
0-gluster_disk-client-1: Server lk
version = 1
[2015-03-16
01:37:10.762279] I
[fuse-bridge.c:3953:fuse_init]
0-glusterfs-fuse: FUSE inited with
protocol versions: glusterfs 7.22
kernel 7.14
[2015-03-16
01:59:26.098670] W
[fuse-bridge.c:2242:fuse_writev_cbk]
0-glusterfs-fuse: 292084: WRITE
=> -1 (Input/output error)
…
I’ve seen no
indication of split-brain on any
files at any point in this (ever
since downdating from 3.6.2 to
3.5.3, which is when this particular
issue started):
[root@duke
gfapi-module-for-linux-target-driver-]#
gluster v heal gluster_disk info
Brick
duke.jonheese.local:/bricks/brick1/
Number of
entries: 0
Brick
duchess.jonheese.local:/bricks/brick1/
Number of
entries: 0
Thanks.
Jon Heese
Systems
Engineer
INetU
Managed Hosting
P:
610.266.7441 x 261
F:
610.266.7434
www.inetu.net
** This message
contains confidential
information, which also may be
privileged, and is intended only
for the person(s) addressed
above. Any unauthorized use,
distribution, copying or
disclosure of confidential
and/or privileged information is
strictly prohibited. If you have
received this communication in
error, please erase all copies
of the message and its
attachments and notify the
sender immediately via reply
e-mail. **
On 03/17/2015
02:14 AM, Jonathan Heese wrote:
Hello,
So I resolved my previous
issue with split-brains and
the lack of self-healing by
dropping my installed
glusterfs* packages from 3.6.2
to 3.5.3, but now I've picked
up a new issue, which actually
makes normal use of the volume
practically impossible.
A little background for those
not already paying close
attention:
I have a 2 node 2 brick
replicating volume whose
purpose in life is to hold
iSCSI target files, primarily
for use to provide datastores
to a VMware ESXi cluster. The
plan is to put a handful of
image files on the Gluster
volume, mount them locally on
both Gluster nodes, and run
tgtd on both, pointed to the
image files on the mounted
gluster volume. Then the ESXi
boxes will use multipath
(active/passive) iSCSI to
connect to the nodes, with
automatic failover in case of
planned or unplanned downtime
of the Gluster nodes.
In my most recent round of
testing with 3.5.3, I'm seeing
a massive failure to write
data to the volume after about
5-10 minutes, so I've
simplified the scenario a bit
(to minimize the variables)
to: both Gluster nodes up,
only one node (duke) mounted
and running tgtd, and just
regular (single path) iSCSI
from a single ESXi server.
About 5-10 minutes into
migration a VM onto the test
datastore, /var/log/messages
on duke gets blasted with a
ton of messages exactly like
this:
Mar 15
22:24:06 duke tgtd:
bs_rdwr_request(180) io error
0x1781e00 2a -1 512 22971904,
Input/output error
And
/var/log/glusterfs/mnt-gluster_disk.log
gets blased with a ton of
messages exactly like this:
[2015-03-16
02:24:07.572279] W
[fuse-bridge.c:2242:fuse_writev_cbk]
0-glusterfs-fuse: 635299: WRITE
=> -1 (Input/output error)
Are there any messages in the mount
log from AFR about split-brain just
before the above line appears?
Does `gluster v heal <VOLNAME>
info` show any files? Performing I/O
on files that are in split-brain
fail with EIO.
-Ravi
And the
write operation from VMware's
side fails as soon as these
messages start.
I don't
see any other errors (in the log
files I know of) indicating the
root cause of these i/o errors.
I'm sure that this is not enough
information to tell what's going
on, but can anyone help me
figure out what to look at next
to figure this out?
I've
also considered using Dan
Lambright's libgfapi gluster
module for tgtd (or something
similar) to avoid going through
FUSE, but I'm not sure whether
that would be irrelevant to this
problem, since I'm not 100% sure
if it lies in FUSE or elsewhere.
Thanks!
Jon Heese
Systems
Engineer
INetU
Managed Hosting
P:
610.266.7441 x 261
F:
610.266.7434
www.inetu.net
** This
message contains
confidential information,
which also may be
privileged, and is intended
only for the person(s)
addressed above. Any
unauthorized use,
distribution, copying or
disclosure of confidential
and/or privileged
information is strictly
prohibited. If you have
received this communication
in error, please erase all
copies of the message and
its attachments and notify
the sender immediately via
reply e-mail. **
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
