Jon
Heese
Systems
Engineer
INetU
Managed Hosting
P:
610.266.7441 x 261
F:
610.266.7434
www.inetu.net
**
This message
contains
confidential
information, which
also may be
privileged, and is
intended only for
the person(s)
addressed above. Any
unauthorized use,
distribution,
copying or
disclosure of
confidential and/or
privileged
information is
strictly prohibited.
If you have received
this communication
in error, please
erase all copies of
the message and its
attachments and
notify the sender
immediately via
reply e-mail. **
From:
Jonathan Heese
Sent:
Tuesday, March 17,
2015 12:36 PM
To:
'Ravishankar N';
gluster-users@xxxxxxxxxxx
Subject: RE:
I/O
error on replicated
volume
Ravi,
The
last lines in the mount
log before the massive
vomit of I/O errors are
from 22 minutes prior,
and seem innocuous to
me:
[2015-03-16
01:37:07.126340] E
[client-handshake.c:1760:client_query_portmap_cbk]
0-gluster_disk-client-0:
failed to get the port
number for remote
subvolume. Please run
'gluster volume status'
on server to see if
brick process is
running.
[2015-03-16
01:37:07.126587] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a])))
0-gluster_disk-client-0:
disconnect called
(peer:10.10.10.1:24008)
[2015-03-16
01:37:07.126687] E
[client-handshake.c:1760:client_query_portmap_cbk]
0-gluster_disk-client-1:
failed to get the port
number for remote
subvolume. Please run
'gluster volume status'
on server to see if
brick process is
running.
[2015-03-16
01:37:07.126737] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a])))
0-gluster_disk-client-1:
disconnect called
(peer:10.10.10.2:24008)
[2015-03-16
01:37:10.730165] I
[rpc-clnt.c:1729:rpc_clnt_reconfig]
0-gluster_disk-client-0:
changing port to 49152
(from 0)
[2015-03-16
01:37:10.730276] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a])))
0-gluster_disk-client-0:
disconnect called
(peer:10.10.10.1:24008)
[2015-03-16
01:37:10.739500] I
[rpc-clnt.c:1729:rpc_clnt_reconfig]
0-gluster_disk-client-1:
changing port to 49152
(from 0)
[2015-03-16
01:37:10.739560] W
[rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a])))
0-gluster_disk-client-1:
disconnect called
(peer:10.10.10.2:24008)
[2015-03-16
01:37:10.741883] I
[client-handshake.c:1677:select_server_supported_programs]
0-gluster_disk-client-0:
Using Program GlusterFS
3.3, Num (1298437),
Version (330)
[2015-03-16
01:37:10.744524] I
[client-handshake.c:1462:client_setvolume_cbk]
0-gluster_disk-client-0:
Connected to
10.10.10.1:49152,
attached to remote
volume '/bricks/brick1'.
[2015-03-16
01:37:10.744537] I
[client-handshake.c:1474:client_setvolume_cbk]
0-gluster_disk-client-0:
Server and Client
lk-version numbers are
not same, reopening the
fds
[2015-03-16
01:37:10.744566] I
[afr-common.c:4267:afr_notify]
0-gluster_disk-replicate-0:
Subvolume
'gluster_disk-client-0'
came back up; going
online.
[2015-03-16
01:37:10.744627] I
[client-handshake.c:450:client_set_lk_version_cbk]
0-gluster_disk-client-0:
Server lk version = 1
[2015-03-16
01:37:10.753037] I
[client-handshake.c:1677:select_server_supported_programs]
0-gluster_disk-client-1:
Using Program GlusterFS
3.3, Num (1298437),
Version (330)
[2015-03-16
01:37:10.755657] I
[client-handshake.c:1462:client_setvolume_cbk]
0-gluster_disk-client-1:
Connected to
10.10.10.2:49152,
attached to remote
volume '/bricks/brick1'.
[2015-03-16
01:37:10.755676] I
[client-handshake.c:1474:client_setvolume_cbk]
0-gluster_disk-client-1:
Server and Client
lk-version numbers are
not same, reopening the
fds
[2015-03-16
01:37:10.761945] I
[fuse-bridge.c:5016:fuse_graph_setup]
0-fuse: switched to
graph 0
[2015-03-16
01:37:10.762144] I
[client-handshake.c:450:client_set_lk_version_cbk]
0-gluster_disk-client-1:
Server lk version = 1
[2015-03-16
01:37:10.762279] I
[fuse-bridge.c:3953:fuse_init]
0-glusterfs-fuse: FUSE
inited with protocol
versions: glusterfs 7.22
kernel 7.14
[2015-03-16
01:59:26.098670] W
[fuse-bridge.c:2242:fuse_writev_cbk]
0-glusterfs-fuse:
292084: WRITE => -1
(Input/output error)
…
I’ve
seen no indication of
split-brain on any files
at any point in this
(ever since downdating
from 3.6.2 to 3.5.3,
which is when this
particular issue
started):
[root@duke
gfapi-module-for-linux-target-driver-]#
gluster v heal
gluster_disk info
Brick
duke.jonheese.local:/bricks/brick1/
Number
of entries: 0
Brick
duchess.jonheese.local:/bricks/brick1/
Number
of entries: 0
Thanks.
Jon
Heese
Systems
Engineer
INetU
Managed Hosting
P:
610.266.7441 x 261
F:
610.266.7434
www.inetu.net
**
This message
contains
confidential
information, which
also may be
privileged, and is
intended only for
the person(s)
addressed above. Any
unauthorized use,
distribution,
copying or
disclosure of
confidential and/or
privileged
information is
strictly prohibited.
If you have received
this communication
in error, please
erase all copies of
the message and its
attachments and
notify the sender
immediately via
reply e-mail. **
On
03/17/2015 02:14 AM,
Jonathan Heese wrote:
Hello,
So I resolved my
previous issue
with split-brains
and the lack of
self-healing by
dropping my
installed
glusterfs*
packages from
3.6.2 to 3.5.3,
but now I've
picked up a new
issue, which
actually makes
normal use of the
volume practically
impossible.
A little
background for
those not already
paying close
attention:
I have a 2 node 2
brick replicating
volume whose
purpose in life is
to hold iSCSI
target files,
primarily for use
to provide
datastores to a
VMware ESXi
cluster. The plan
is to put a
handful of image
files on the
Gluster volume,
mount them locally
on both Gluster
nodes, and run
tgtd on both,
pointed to the
image files on the
mounted gluster
volume. Then the
ESXi boxes will
use multipath
(active/passive)
iSCSI to connect
to the nodes, with
automatic failover
in case of planned
or unplanned
downtime of the
Gluster nodes.
In my most recent
round of testing
with 3.5.3, I'm
seeing a massive
failure to write
data to the volume
after about 5-10
minutes, so I've
simplified the
scenario a bit (to
minimize the
variables) to:
both Gluster nodes
up, only one node
(duke) mounted and
running tgtd, and
just regular
(single path)
iSCSI from a
single ESXi
server.
About 5-10 minutes
into migration a
VM onto the test
datastore,
/var/log/messages
on duke gets
blasted with a ton
of messages
exactly like this:
Mar
15 22:24:06 duke
tgtd:
bs_rdwr_request(180)
io error 0x1781e00
2a -1 512 22971904,
Input/output error
And
/var/log/glusterfs/mnt-gluster_disk.log
gets blased with a
ton of messages
exactly like this:
[2015-03-16
02:24:07.572279] W
[fuse-bridge.c:2242:fuse_writev_cbk]
0-glusterfs-fuse:
635299: WRITE =>
-1 (Input/output
error)
Are there any messages
in the mount log from
AFR about split-brain
just before the above
line appears?
Does `gluster v heal
<VOLNAME> info`
show any files?
Performing I/O on files
that are in split-brain
fail with EIO.
-Ravi
And
the write operation
from VMware's side
fails as soon as
these messages
start.
I
don't see any other
errors (in the log
files I know of)
indicating the root
cause of these i/o
errors. I'm sure
that this is not
enough information
to tell what's going
on, but can anyone
help me figure out
what to look at next
to figure this out?
I've
also considered
using Dan
Lambright's libgfapi
gluster module for
tgtd (or something
similar) to avoid
going through FUSE,
but I'm not sure
whether that would
be irrelevant to
this problem, since
I'm not 100% sure if
it lies in FUSE or
elsewhere.
Thanks!
Jon
Heese
Systems
Engineer
INetU
Managed Hosting
P:
610.266.7441 x 261
F:
610.266.7434
www.inetu.net
**
This message
contains
confidential
information,
which also may
be privileged,
and is intended
only for the
person(s)
addressed above.
Any unauthorized
use,
distribution,
copying or
disclosure of
confidential
and/or
privileged
information is
strictly
prohibited. If
you have
received this
communication in
error, please
erase all copies
of the message
and its
attachments and
notify the
sender
immediately via
reply e-mail. **
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
