Jon Heese
Systems
Engineer
INetU
Managed Hosting
P:
610.266.7441 x 261
F:
610.266.7434
www.inetu.net
** This
message contains
confidential information,
which also may be
privileged, and is
intended only for the
person(s) addressed above.
Any unauthorized use,
distribution, copying or
disclosure of confidential
and/or privileged
information is strictly
prohibited. If you have
received this
communication in error,
please erase all copies of
the message and its
attachments and notify the
sender immediately via
reply e-mail. **
From:
Jonathan Heese
Sent: Tuesday,
March 17, 2015 12:36 PM
To: 'Ravishankar
N';
gluster-users@xxxxxxxxxxx
Subject: RE:
I/O error
on replicated volume
Ravi,
The last
lines in the mount log before
the massive vomit of I/O
errors are from 22 minutes
prior, and seem innocuous to
me:
[2015-03-16
01:37:07.126340] E
[client-handshake.c:1760:client_query_portmap_cbk]
0-gluster_disk-client-0:
failed to get the port number
for remote subvolume. Please
run 'gluster volume status' on
server to see if brick process
is running.
[2015-03-16
01:37:07.126587] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a])))
0-gluster_disk-client-0:
disconnect called
(peer:10.10.10.1:24008)
[2015-03-16
01:37:07.126687] E
[client-handshake.c:1760:client_query_portmap_cbk]
0-gluster_disk-client-1:
failed to get the port number
for remote subvolume. Please
run 'gluster volume status' on
server to see if brick process
is running.
[2015-03-16
01:37:07.126737] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a])))
0-gluster_disk-client-1:
disconnect called
(peer:10.10.10.2:24008)
[2015-03-16
01:37:10.730165] I
[rpc-clnt.c:1729:rpc_clnt_reconfig]
0-gluster_disk-client-0:
changing port to 49152 (from
0)
[2015-03-16
01:37:10.730276] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a])))
0-gluster_disk-client-0:
disconnect called
(peer:10.10.10.1:24008)
[2015-03-16
01:37:10.739500] I
[rpc-clnt.c:1729:rpc_clnt_reconfig]
0-gluster_disk-client-1:
changing port to 49152 (from
0)
[2015-03-16
01:37:10.739560] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a])))
0-gluster_disk-client-1:
disconnect called
(peer:10.10.10.2:24008)
[2015-03-16
01:37:10.741883] I
[client-handshake.c:1677:select_server_supported_programs]
0-gluster_disk-client-0: Using
Program GlusterFS 3.3, Num
(1298437), Version (330)
[2015-03-16
01:37:10.744524] I
[client-handshake.c:1462:client_setvolume_cbk]
0-gluster_disk-client-0:
Connected to 10.10.10.1:49152,
attached to remote volume
'/bricks/brick1'.
[2015-03-16
01:37:10.744537] I
[client-handshake.c:1474:client_setvolume_cbk]
0-gluster_disk-client-0:
Server and Client lk-version
numbers are not same,
reopening the fds
[2015-03-16
01:37:10.744566] I
[afr-common.c:4267:afr_notify]
0-gluster_disk-replicate-0:
Subvolume
'gluster_disk-client-0' came
back up; going online.
[2015-03-16
01:37:10.744627] I
[client-handshake.c:450:client_set_lk_version_cbk]
0-gluster_disk-client-0:
Server lk version = 1
[2015-03-16
01:37:10.753037] I
[client-handshake.c:1677:select_server_supported_programs]
0-gluster_disk-client-1: Using
Program GlusterFS 3.3, Num
(1298437), Version (330)
[2015-03-16
01:37:10.755657] I
[client-handshake.c:1462:client_setvolume_cbk]
0-gluster_disk-client-1:
Connected to 10.10.10.2:49152,
attached to remote volume
'/bricks/brick1'.
[2015-03-16
01:37:10.755676] I
[client-handshake.c:1474:client_setvolume_cbk]
0-gluster_disk-client-1:
Server and Client lk-version
numbers are not same,
reopening the fds
[2015-03-16
01:37:10.761945] I
[fuse-bridge.c:5016:fuse_graph_setup]
0-fuse: switched to graph 0
[2015-03-16
01:37:10.762144] I
[client-handshake.c:450:client_set_lk_version_cbk]
0-gluster_disk-client-1:
Server lk version = 1
[2015-03-16
01:37:10.762279] I
[fuse-bridge.c:3953:fuse_init]
0-glusterfs-fuse: FUSE inited
with protocol versions:
glusterfs 7.22 kernel 7.14
[2015-03-16
01:59:26.098670] W
[fuse-bridge.c:2242:fuse_writev_cbk]
0-glusterfs-fuse: 292084:
WRITE => -1 (Input/output
error)
…
I’ve
seen no indication of
split-brain on any files at
any point in this (ever since
downdating from 3.6.2 to
3.5.3, which is when this
particular issue started):
[root@duke
gfapi-module-for-linux-target-driver-]#
gluster v heal gluster_disk
info
Brick
duke.jonheese.local:/bricks/brick1/
Number
of entries: 0
Brick
duchess.jonheese.local:/bricks/brick1/
Number
of entries: 0
Thanks.
Jon Heese
Systems
Engineer
INetU
Managed Hosting
P:
610.266.7441 x 261
F:
610.266.7434
www.inetu.net
** This
message contains
confidential information,
which also may be
privileged, and is
intended only for the
person(s) addressed above.
Any unauthorized use,
distribution, copying or
disclosure of confidential
and/or privileged
information is strictly
prohibited. If you have
received this
communication in error,
please erase all copies of
the message and its
attachments and notify the
sender immediately via
reply e-mail. **
On
03/17/2015 02:14 AM, Jonathan
Heese wrote:
Hello,
So I resolved my
previous issue with
split-brains and the
lack of self-healing by
dropping my installed
glusterfs* packages from
3.6.2 to 3.5.3, but now
I've picked up a new
issue, which actually
makes normal use of the
volume practically
impossible.
A little background for
those not already paying
close attention:
I have a 2 node 2 brick
replicating volume whose
purpose in life is to
hold iSCSI target files,
primarily for use to
provide datastores to a
VMware ESXi cluster.
The plan is to put a
handful of image files
on the Gluster volume,
mount them locally on
both Gluster nodes, and
run tgtd on both,
pointed to the image
files on the mounted
gluster volume. Then the
ESXi boxes will use
multipath
(active/passive) iSCSI
to connect to the nodes,
with automatic failover
in case of planned or
unplanned downtime of
the Gluster nodes.
In my most recent round
of testing with 3.5.3,
I'm seeing a massive
failure to write data to
the volume after about
5-10 minutes, so I've
simplified the scenario
a bit (to minimize the
variables) to: both
Gluster nodes up, only
one node (duke) mounted
and running tgtd, and
just regular (single
path) iSCSI from a
single ESXi server.
About 5-10 minutes into
migration a VM onto the
test datastore,
/var/log/messages on
duke gets blasted with a
ton of messages exactly
like this:
Mar
15 22:24:06 duke tgtd:
bs_rdwr_request(180) io
error 0x1781e00 2a -1 512
22971904, Input/output
error
And
/var/log/glusterfs/mnt-gluster_disk.log
gets blased with a ton of
messages exactly like
this:
[2015-03-16
02:24:07.572279] W
[fuse-bridge.c:2242:fuse_writev_cbk]
0-glusterfs-fuse: 635299:
WRITE => -1
(Input/output error)
Are there any messages in the
mount log from AFR about
split-brain just before the
above line appears?
Does `gluster v heal
<VOLNAME> info` show any
files? Performing I/O on files
that are in split-brain fail
with EIO.
-Ravi
And
the write operation from
VMware's side fails as
soon as these messages
start.
I
don't see any other errors
(in the log files I know
of) indicating the root
cause of these i/o
errors. I'm sure that
this is not enough
information to tell what's
going on, but can anyone
help me figure out what to
look at next to figure
this out?
I've
also considered using Dan
Lambright's libgfapi
gluster module for tgtd
(or something similar) to
avoid going through FUSE,
but I'm not sure whether
that would be irrelevant
to this problem, since I'm
not 100% sure if it lies
in FUSE or elsewhere.
Thanks!
Jon
Heese
Systems
Engineer
INetU
Managed Hosting
P:
610.266.7441 x 261
F:
610.266.7434
www.inetu.net
** This
message contains
confidential
information, which
also may be
privileged, and is
intended only for the
person(s) addressed
above. Any
unauthorized use,
distribution, copying
or disclosure of
confidential and/or
privileged information
is strictly
prohibited. If you
have received this
communication in
error, please erase
all copies of the
message and its
attachments and notify
the sender immediately
via reply e-mail. **
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
