Re: Openstack Havana Gluster /var/lib/instances issues

SATHEESARAN <sasundar@xxxxxxxxxx> · Tue, 25 Mar 2014 20:03:45 +0530



    Resending it as I suppose it was not
      delievered :(

    
      Hi Ryan/Narayan,

        
        Apologies for top posting.

        After today's call, we had a chat with Vijay and he added a
        hint.

        
        You need to enable remote-dio for that volume that
        serves for virt-store.

        > gluster volume set <vol-name> remote-dio enable

        
        Hope this could solve your issue most probably.

        Could you test it with this option 'on' and let us know the
        results ?

        
        -- sas

        
        On 03/20/2014 12:40 AM, Aydelott, Ryan M. wrote:

      
        So a little further down the road I have destroyed/rebuilt the
        gluster volume and restarted glusterd on each of the nodes.
        Shortly after the instances directory is created and the
        console.log file is placed, I receive the following fuse errors
        on the client:
        

           [client-rpc-fops.c:873:client3_3_writev_cbk]
            0-novashared-client-4: remote operation failed: Invalid
            argument
          [2014-03-19 19:06:35.326487] W
            [client-rpc-fops.c:873:client3_3_writev_cbk]
            0-novashared-client-5: remote operation failed: Invalid
            argument
          [2014-03-19 19:06:35.328750] W
            [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse:
            106504: WRITE => -1 (Invalid argument)
          [2014-03-19 19:06:35.328855] W
            [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse:
            106505: FTRUNCATE() ERR => -1 (Bad file descriptor)
          [2014-03-19 19:06:35.329506] W
            [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse:
            106507: WRITE => -1 (Bad file descriptor)
          [2014-03-19 19:06:35.329775] W
            [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse:
            106508: FTRUNCATE() ERR => -1 (Bad file descriptor)
          [2014-03-19 19:06:35.330495] W
            [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse:
            106510: WRITE => -1 (Bad file descriptor)
          [2014-03-19 19:06:35.330758] W
            [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse:
            106511: FTRUNCATE() ERR => -1 (Bad file descriptor)
          [2014-03-19 19:06:35.331538] W
            [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse:
            106513: WRITE => -1 (Bad file descriptor)
          [2014-03-19 19:06:35.331792] W
            [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse:
            106514: FTRUNCATE() ERR => -1 (Bad file descriptor)
          

          One one of the servers the normal messages apply:
          

            [2014-03-13 21:47:33.735042] I
              [server-handshake.c:567:server_setvolume]
              0-novashared-server: accepted client from
              cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0
              (version: 3.4.2)
            [2014-03-18 21:17:42.763044] I
              [server.c:762:server_rpc_notify] 0-novashared-server:
              disconnecting connectionfrom
              cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0
            [2014-03-18 21:17:42.763076] I
              [server-helpers.c:729:server_connection_put]
              0-novashared-server: Shutting down connection
              cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0
            [2014-03-18 21:17:42.763101] I
              [server-helpers.c:617:server_connection_destroy]
              0-novashared-server: destroyed connection of
              cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0
            [2014-03-18 21:50:02.206635] I
              [server-handshake.c:567:server_setvolume]
              0-novashared-server: accepted client from
              cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0
              (version: 3.4.2)
            [2014-03-18 21:50:23.037886] I
              [server-handshake.c:567:server_setvolume]
              0-novashared-server: accepted client from
              cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0
              (version: 3.4.2)
            [2014-03-18 21:50:23.051419] I
              [server.c:762:server_rpc_notify] 0-novashared-server:
              disconnecting connectionfrom
              cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0
            [2014-03-18 21:50:23.051446] I
              [server-helpers.c:729:server_connection_put]
              0-novashared-server: Shutting down connection
              cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0
            [2014-03-18 21:50:23.051466] I
              [server-helpers.c:617:server_connection_destroy]
              0-novashared-server: destroyed connection of
              cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0
            [2014-03-18 21:56:22.215866] I
              [server.c:762:server_rpc_notify] 0-novashared-server:
              disconnecting connectionfrom
              cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0
            [2014-03-18 21:56:22.215898] I
              [server-helpers.c:729:server_connection_put]
              0-novashared-server: Shutting down connection
              cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0
            [2014-03-18 21:56:22.215923] I
              [server-helpers.c:617:server_connection_destroy]
              0-novashared-server: destroyed connection of
              cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0
            [2014-03-18 21:57:04.821397] I
              [server-handshake.c:567:server_setvolume]
              0-novashared-server: accepted client from
              cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0
              (version: 3.4.2)
            [2014-03-19 16:48:32.135593] I
              [server.c:762:server_rpc_notify] 0-novashared-server:
              disconnecting connectionfrom
              cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0
            [2014-03-19 16:48:32.135629] I
              [server-helpers.c:729:server_connection_put]
              0-novashared-server: Shutting down connection
              cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0
            [2014-03-19 16:48:32.135652] I
              [server-helpers.c:617:server_connection_destroy]
              0-novashared-server: destroyed connection of
              cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0
            [2014-03-19 16:48:47.712973] I
              [server-handshake.c:567:server_setvolume]
              0-novashared-server: accepted client from
              cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0
              (version: 3.4.2)
            [2014-03-19 18:07:44.332240] I
              [server.c:762:server_rpc_notify] 0-novashared-server:
              disconnecting connectionfrom
              cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0
            [2014-03-19 18:07:44.332272] I
              [server-helpers.c:729:server_connection_put]
              0-novashared-server: Shutting down connection
              cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0
            [2014-03-19 18:07:44.332296] I
              [server-helpers.c:617:server_connection_destroy]
              0-novashared-server: destroyed connection of
              cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0
            [2014-03-19 19:00:51.369205] I
              [server-handshake.c:567:server_setvolume]
              0-novashared-server: accepted client from
              cm6-p-21755-2014/03/19-19:00:51:286018-novashared-client-0-0
              (version: 3.4.2)
          
          
          These are all on 3.4.2 with Ubuntu 13.10. 
          

          -ryan
          

            On Mar 19, 2014, at 11:15 AM, Carlos Capriotti <capriotti.carlos@xxxxxxxxx>

              wrote:
            

              This
                rings a bell...

                
                I mapped a remote nfs share to one of my nodes and
                mapped the local nfs

                share, to the "local" gluster volume as well and started
                to copy (with cp)

                files from one mount to the other.

                
                Somewhere down the line all stalled, the server was
                irresponsive, even

                restarting glusterd solved nothing. Had to reboot the
                nodes.

                
                CentOS 6.5 and Gluster 3.4.2.

                
                Since then,  I haven't tried those copies, but, I also,
                I am now using the

                native glusterfs client just in case.

                
                On Wed, Mar 19, 2014 at 3:51 PM, Ray Powell <rpowell1@xxxxxxxxxxxx>

                wrote:

                
                -----BEGIN PGP SIGNED
                  MESSAGE-----

                  Hash: SHA512

                  
                  I have not seen that with the fuse client.  I had it
                  happen repeatedly

                  when we were using gluster/NFS for
                  /var/lib/nova/instances and/or

                  /var/lib/nova/instances/_base.  The access to
                  instances-XXXXXX/disk

                  (or its qcow2 backing image) would hang completely or
                  be very slow

                  under heavy load.  The VM interprets the delay as a
                  problem on

                  /dev/vda and remounts RO.

                  
                  On 3/18/14, 5:04 PM, Aydelott, Ryan M. wrote:

                  I was wondering if anyone had
                    seen similar issues before, when

                    booting a vm with a root volume on local disk things
                    work fine.

                    
                    When I try to do the same with
                    /var/lib/nova/instances mounted via

                    gluster/fuse I get the following: http://paste.ubuntu.com/7116594/

                    
                    I have of course insured perms were correct with
                    chown -R nova:nova

                    /var/lib/nova/instances, additionally app armor is
                    completely

                    unloaded/disabled.

                    
                    Has anyone encountered this before? Gluster version
                    is 3.4.2

                    
                    -ryan

                    
                    _______________________________________________
                    Gluster-users

                    mailing list Gluster-users@xxxxxxxxxxx

                    http://supercolony.gluster.org/mailman/listinfo/gluster-users

                    
                  - --

                  Raymond Powell

                  Systems Administrator

                  Laboratory for Advanced Computing

                  University of Chicago, KCBD 10160

                  900 East 57th Street

                  Chicago, IL 60637

                  -----BEGIN PGP SIGNATURE-----

                  Version: GnuPG/MacGPG2 v2.0.22 (Darwin)

                  Comment: GPGTools - http://gpgtools.org

                  Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

                  
iQIcBAEBCgAGBQJTKa8HAAoJEBIjAczqLCW8fYMQAIMsxPVG0n+ujRJeCYvphUy9

lHe7bkTLiJIkMs51hzXCfsqT3tEkvixYbjaAfjuHTdg0q5sK4zu+5MUM4KqG7oZy

pYMzX2VrYnkTqQRFTOQbzntSvKlDjuSebuGpbI8sCxfgOR6/25tuzcJDCvn7sp4B

YxVFaLZx5Xd6U24BHT9MVVJgAFXCVoalj2XMy+WHGhM07VvEXHxf62oW2VymLHOm

eYiQGpjDWwb218/6NtJ2z47xr7HIQF1sW1Hhu/YRsnNzAYPh1FL3NwvnkFvNwGdg

huTFpQegV9bmg78A3eEnIEEjDm1haX/E1NOo2bHdRYF2+Fwz+rHgkQJxFfEHxOV/

HDylUIPeMue9qEgttRg345NuypoVrBN3sFGf72UOfgwwElr+Z0t83y6uquTcNfOD

lWIPa7d0vyLWfm6+mjR1hDHsLx6H/Y0MyAKFgYUIerdO8+TgAnLsv4FfBGlbm+Xa

sutQxYPxBeAlSnKXmwrP9keB1ttiMV8IptHbuM5cQna5jdPeouwpluquZsP30bpA

o/a7nPaogBx+X6BsM4aBjRLSjqtu+mywNh1AqdY1dIpvDtD9lXrO5L+Z2SLZUDeF

JvFWkp8Epdn0k7vn6WKT6RZvjSji9WgTIdbFrKEjfEqAz/+9UbrG2D/vYL2TlgAe

                  +UhAq6+74ie1MokNssB8

                  =un7j

                  -----END PGP SIGNATURE-----

                  _______________________________________________

                  Gluster-users mailing list

                  Gluster-users@xxxxxxxxxxx

                  http://supercolony.gluster.org/mailman/listinfo/gluster-users

                  
                _______________________________________________

                Gluster-users mailing list

                Gluster-users@xxxxxxxxxxx

                http://supercolony.gluster.org/mailman/listinfo/gluster-users
            
          
        _______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users
      
      
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users