Jon
Heese
Systems
Engineer
INetU
Managed Hosting
P:
610.266.7441 x 261
F:
610.266.7434
www.inetu.net
**
This message
contains
confidential
information,
which also may
be privileged,
and is intended
only for the
person(s)
addressed above.
Any unauthorized
use,
distribution,
copying or
disclosure of
confidential
and/or
privileged
information is
strictly
prohibited. If
you have
received this
communication in
error, please
erase all copies
of the message
and its
attachments and
notify the
sender
immediately via
reply e-mail. **
From:
Jonathan Heese
Sent:
Tuesday, March
17, 2015 12:36
PM
To:
'Ravishankar N';
gluster-users@xxxxxxxxxxx
Subject:
RE:
I/O error on
replicated
volume
Ravi,
The last lines in the mount log before the massive
vomit of I/O errors
are from 22 minutes
prior, and seem
innocuous to me:
[2015-03-16 01:37:07.126340] E
[client-handshake.c:1760:client_query_portmap_cbk]
0-gluster_disk-client-0:
failed to get the
port number for
remote subvolume.
Please run 'gluster
volume status' on
server to see if
brick process is
running.
[2015-03-16 01:37:07.126587] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a])))
0-gluster_disk-client-0:
disconnect called
(peer:10.10.10.1:24008)
[2015-03-16 01:37:07.126687] E
[client-handshake.c:1760:client_query_portmap_cbk]
0-gluster_disk-client-1:
failed to get the
port number for
remote subvolume.
Please run 'gluster
volume status' on
server to see if
brick process is
running.
[2015-03-16 01:37:07.126737] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a])))
0-gluster_disk-client-1:
disconnect called
(peer:10.10.10.2:24008)
[2015-03-16 01:37:10.730165] I
[rpc-clnt.c:1729:rpc_clnt_reconfig]
0-gluster_disk-client-0:
changing port to
49152 (from 0)
[2015-03-16 01:37:10.730276] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a])))
0-gluster_disk-client-0:
disconnect called
(peer:10.10.10.1:24008)
[2015-03-16 01:37:10.739500] I
[rpc-clnt.c:1729:rpc_clnt_reconfig]
0-gluster_disk-client-1:
changing port to
49152 (from 0)
[2015-03-16 01:37:10.739560] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a])))
0-gluster_disk-client-1:
disconnect called
(peer:10.10.10.2:24008)
[2015-03-16 01:37:10.741883] I
[client-handshake.c:1677:select_server_supported_programs]
0-gluster_disk-client-0:
Using Program
GlusterFS 3.3, Num
(1298437), Version
(330)
[2015-03-16 01:37:10.744524] I
[client-handshake.c:1462:client_setvolume_cbk]
0-gluster_disk-client-0:
Connected to
10.10.10.1:49152,
attached to remote
volume
'/bricks/brick1'.
[2015-03-16 01:37:10.744537] I
[client-handshake.c:1474:client_setvolume_cbk]
0-gluster_disk-client-0:
Server and Client
lk-version numbers
are not same,
reopening the fds
[2015-03-16 01:37:10.744566] I
[afr-common.c:4267:afr_notify]
0-gluster_disk-replicate-0:
Subvolume
'gluster_disk-client-0'
came back up; going
online.
[2015-03-16 01:37:10.744627] I
[client-handshake.c:450:client_set_lk_version_cbk]
0-gluster_disk-client-0:
Server lk version =
1
[2015-03-16 01:37:10.753037] I
[client-handshake.c:1677:select_server_supported_programs]
0-gluster_disk-client-1:
Using Program
GlusterFS 3.3, Num
(1298437), Version
(330)
[2015-03-16 01:37:10.755657] I
[client-handshake.c:1462:client_setvolume_cbk]
0-gluster_disk-client-1:
Connected to
10.10.10.2:49152,
attached to remote
volume
'/bricks/brick1'.
[2015-03-16 01:37:10.755676] I
[client-handshake.c:1474:client_setvolume_cbk]
0-gluster_disk-client-1:
Server and Client
lk-version numbers
are not same,
reopening the fds
[2015-03-16 01:37:10.761945] I
[fuse-bridge.c:5016:fuse_graph_setup]
0-fuse: switched to
graph 0
[2015-03-16 01:37:10.762144] I
[client-handshake.c:450:client_set_lk_version_cbk]
0-gluster_disk-client-1:
Server lk version =
1
[2015-03-16 01:37:10.762279] I
[fuse-bridge.c:3953:fuse_init]
0-glusterfs-fuse:
FUSE inited with
protocol versions:
glusterfs 7.22
kernel 7.14
[2015-03-16 01:59:26.098670] W
[fuse-bridge.c:2242:fuse_writev_cbk]
0-glusterfs-fuse:
292084: WRITE =>
-1 (Input/output
error)
…
I’ve seen no indication of split-brain on any
files at any point
in this (ever since
downdating from
3.6.2 to 3.5.3,
which is when this
particular issue
started):
[root@duke gfapi-module-for-linux-target-driver-]#
gluster v heal
gluster_disk info
Brick duke.jonheese.local:/bricks/brick1/
Number of entries: 0
Brick duchess.jonheese.local:/bricks/brick1/
Number of entries: 0
Thanks.
Jon
Heese
Systems
Engineer
INetU
Managed Hosting
P:
610.266.7441 x 261
F:
610.266.7434
www.inetu.net
**
This message
contains
confidential
information,
which also may
be privileged,
and is intended
only for the
person(s)
addressed above.
Any unauthorized
use,
distribution,
copying or
disclosure of
confidential
and/or
privileged
information is
strictly
prohibited. If
you have
received this
communication in
error, please
erase all copies
of the message
and its
attachments and
notify the
sender
immediately via
reply e-mail. **
On
03/17/2015 02:14 AM,
Jonathan Heese
wrote:
Hello,
So I resolved
my previous
issue with
split-brains
and the lack
of
self-healing
by dropping my
installed
glusterfs*
packages from
3.6.2 to
3.5.3, but now
I've picked up
a new issue,
which actually
makes normal
use of the
volume
practically
impossible.
A little
background for
those not
already paying
close
attention:
I have a 2
node 2 brick
replicating
volume whose
purpose in
life is to
hold iSCSI
target files,
primarily for
use to provide
datastores to
a VMware ESXi
cluster. The
plan is to put
a handful of
image files on
the Gluster
volume, mount
them locally
on both
Gluster nodes,
and run tgtd
on both,
pointed to the
image files on
the mounted
gluster
volume. Then
the ESXi boxes
will use
multipath
(active/passive)
iSCSI to
connect to the
nodes, with
automatic
failover in
case of
planned or
unplanned
downtime of
the Gluster
nodes.
In my most
recent round
of testing
with 3.5.3,
I'm seeing a
massive
failure to
write data to
the volume
after about
5-10 minutes,
so I've
simplified the
scenario a bit
(to minimize
the variables)
to: both
Gluster nodes
up, only one
node (duke)
mounted and
running tgtd,
and just
regular
(single path)
iSCSI from a
single ESXi
server.
About 5-10
minutes into
migration a VM
onto the test
datastore,
/var/log/messages
on duke gets
blasted with a
ton of
messages
exactly like
this:
Mar 15 22:24:06 duke tgtd: bs_rdwr_request(180)
io error
0x1781e00 2a -1
512 22971904,
Input/output
error
And /var/log/glusterfs/mnt-gluster_disk.log
gets blased with
a ton of
messages exactly
like this:
[2015-03-16 02:24:07.572279] W
[fuse-bridge.c:2242:fuse_writev_cbk]
0-glusterfs-fuse:
635299: WRITE
=> -1
(Input/output
error)
Are there any
messages in the
mount log from AFR
about split-brain
just before the
above line appears?
Does `gluster v heal
<VOLNAME>
info` show any
files? Performing
I/O on files that
are in split-brain
fail with EIO.
-Ravi
And the write operation from VMware's side
fails as soon as
these messages
start.
I don't see any other errors (in the log files
I know of)
indicating the
root cause of
these i/o
errors. I'm
sure that this
is not enough
information to
tell what's
going on, but
can anyone help
me figure out
what to look at
next to figure
this out?
I've also considered using Dan Lambright's
libgfapi gluster
module for tgtd
(or something
similar) to
avoid going
through FUSE,
but I'm not sure
whether that
would be
irrelevant to
this problem,
since I'm not
100% sure if it
lies in FUSE or
elsewhere.
Thanks!
Jon
Heese
Systems Engineer
INetU Managed Hosting
P:
610.266.7441 x
261
F:
610.266.7434
www.inetu.net
**
This message
contains
confidential
information,
which also may
be privileged,
and is
intended only
for the
person(s)
addressed
above. Any
unauthorized
use,
distribution,
copying or
disclosure of
confidential
and/or
privileged
information is
strictly
prohibited. If
you have
received this
communication
in error,
please erase
all copies of
the message
and its
attachments
and notify the
sender
immediately
via reply
e-mail. **
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
