Re: Openstack Havana Gluster /var/lib/instances issues

"Aydelott, Ryan M." <ryade@xxxxxxxxxxx> · Wed, 19 Mar 2014 19:10:47 +0000

So a little further down the road I have destroyed/rebuilt the gluster volume and restarted glusterd on each of the nodes. Shortly after the instances directory is created and the console.log file is placed, I receive the following fuse errors on the client:
 [client-rpc-fops.c:873:client3_3_writev_cbk] 0-novashared-client-4: remote operation failed: Invalid argument
[2014-03-19 19:06:35.326487] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-novashared-client-5: remote operation failed: Invalid argument
[2014-03-19 19:06:35.328750] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106504: WRITE => -1 (Invalid argument)
[2014-03-19 19:06:35.328855] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106505: FTRUNCATE() ERR => -1 (Bad file descriptor)
[2014-03-19 19:06:35.329506] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106507: WRITE => -1 (Bad file descriptor)
[2014-03-19 19:06:35.329775] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106508: FTRUNCATE() ERR => -1 (Bad file descriptor)
[2014-03-19 19:06:35.330495] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106510: WRITE => -1 (Bad file descriptor)
[2014-03-19 19:06:35.330758] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106511: FTRUNCATE() ERR => -1 (Bad file descriptor)
[2014-03-19 19:06:35.331538] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106513: WRITE => -1 (Bad file descriptor)
[2014-03-19 19:06:35.331792] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106514: FTRUNCATE() ERR => -1 (Bad file descriptor)

One one of the servers the normal messages apply:

[2014-03-13 21:47:33.735042] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0 (version: 3.4.2)
[2014-03-18 21:17:42.763044] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0
[2014-03-18 21:17:42.763076] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0
[2014-03-18 21:17:42.763101] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0
[2014-03-18 21:50:02.206635] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0 (version: 3.4.2)
[2014-03-18 21:50:23.037886] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0 (version: 3.4.2)
[2014-03-18 21:50:23.051419] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0
[2014-03-18 21:50:23.051446] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0
[2014-03-18 21:50:23.051466] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0
[2014-03-18 21:56:22.215866] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0
[2014-03-18 21:56:22.215898] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0
[2014-03-18 21:56:22.215923] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0
[2014-03-18 21:57:04.821397] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0 (version: 3.4.2)
[2014-03-19 16:48:32.135593] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0
[2014-03-19 16:48:32.135629] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0
[2014-03-19 16:48:32.135652] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0
[2014-03-19 16:48:47.712973] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0 (version: 3.4.2)
[2014-03-19 18:07:44.332240] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0
[2014-03-19 18:07:44.332272] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0
[2014-03-19 18:07:44.332296] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0
[2014-03-19 19:00:51.369205] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-21755-2014/03/19-19:00:51:286018-novashared-client-0-0 (version: 3.4.2)

These are all on 3.4.2 with Ubuntu 13.10. 

-ryan

On Mar 19, 2014, at 11:15 AM, Carlos Capriotti <capriotti.carlos@xxxxxxxxx> wrote:

This rings a bell...

I mapped a remote nfs share to one of my nodes and mapped the local nfs
share, to the "local" gluster volume as well and started to copy (with cp)
files from one mount to the other.

Somewhere down the line all stalled, the server was irresponsive, even
restarting glusterd solved nothing. Had to reboot the nodes.

CentOS 6.5 and Gluster 3.4.2.

Since then,  I haven't tried those copies, but, I also, I am now using the
native glusterfs client just in case.

On Wed, Mar 19, 2014 at 3:51 PM, Ray Powell <rpowell1@xxxxxxxxxxxx> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

I have not seen that with the fuse client.  I had it happen repeatedly
when we were using gluster/NFS for /var/lib/nova/instances and/or
/var/lib/nova/instances/_base.  The access to instances-XXXXXX/disk
(or its qcow2 backing image) would hang completely or be very slow
under heavy load.  The VM interprets the delay as a problem on
/dev/vda and remounts RO.

On 3/18/14, 5:04 PM, Aydelott, Ryan M. wrote:
I was wondering if anyone had seen similar issues before, when
booting a vm with a root volume on local disk things work fine.

When I try to do the same with /var/lib/nova/instances mounted via
gluster/fuse I get the following: http://paste.ubuntu.com/7116594/

I have of course insured perms were correct with chown -R nova:nova
/var/lib/nova/instances, additionally app armor is completely
unloaded/disabled.

Has anyone encountered this before? Gluster version is 3.4.2

-ryan

_______________________________________________ Gluster-users
mailing list Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users

- --
Raymond Powell
Systems Administrator
Laboratory for Advanced Computing
University of Chicago, KCBD 10160
900 East 57th Street
Chicago, IL 60637
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCgAGBQJTKa8HAAoJEBIjAczqLCW8fYMQAIMsxPVG0n+ujRJeCYvphUy9
lHe7bkTLiJIkMs51hzXCfsqT3tEkvixYbjaAfjuHTdg0q5sK4zu+5MUM4KqG7oZy
pYMzX2VrYnkTqQRFTOQbzntSvKlDjuSebuGpbI8sCxfgOR6/25tuzcJDCvn7sp4B
YxVFaLZx5Xd6U24BHT9MVVJgAFXCVoalj2XMy+WHGhM07VvEXHxf62oW2VymLHOm
eYiQGpjDWwb218/6NtJ2z47xr7HIQF1sW1Hhu/YRsnNzAYPh1FL3NwvnkFvNwGdg
huTFpQegV9bmg78A3eEnIEEjDm1haX/E1NOo2bHdRYF2+Fwz+rHgkQJxFfEHxOV/
HDylUIPeMue9qEgttRg345NuypoVrBN3sFGf72UOfgwwElr+Z0t83y6uquTcNfOD
lWIPa7d0vyLWfm6+mjR1hDHsLx6H/Y0MyAKFgYUIerdO8+TgAnLsv4FfBGlbm+Xa
sutQxYPxBeAlSnKXmwrP9keB1ttiMV8IptHbuM5cQna5jdPeouwpluquZsP30bpA
o/a7nPaogBx+X6BsM4aBjRLSjqtu+mywNh1AqdY1dIpvDtD9lXrO5L+Z2SLZUDeF
JvFWkp8Epdn0k7vn6WKT6RZvjSji9WgTIdbFrKEjfEqAz/+9UbrG2D/vYL2TlgAe
+UhAq6+74ie1MokNssB8
=un7j
-----END PGP SIGNATURE-----
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Attachment:
signature.asc

Description: Message signed with OpenPGP using GPGMail
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users