[2014-02-19 12:15:15.921993] E [afr-self-heal-common.c:197:afr_sh_print_split_brain_log] 0-devstatic-replicate-0: Unable to self-heal contents of '/employees/htdocs/software/contact_mstr/secure/app/views/.svn/props' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 2 ] [ 3 0 ] ]
Khoi
From: gluster-users-request@xxxxxxxxxxx
To: gluster-users@xxxxxxxxxxx
Date: 02/19/2014 05:59 AM
Subject: Gluster-users Digest, Vol 70, Issue 18
Sent by: gluster-users-bounces@xxxxxxxxxxx
Send Gluster-users mailing list submissions to
gluster-users@xxxxxxxxxxx
To subscribe or unsubscribe via the World Wide Web, visit
http://supercolony.gluster.org/mailman/listinfo/gluster-users
or, via email, send a message with subject or body 'help' to
gluster-users-request@xxxxxxxxxxx
You can reach the person managing the list at
gluster-users-owner@xxxxxxxxxxx
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Gluster-users digest..."
Today's Topics:
1. Complete machine lockup, v3.4.2 (Laurent Chouinard)
2. Want to Ask (Prian Dwiatma)
3. Re: [Bug 1057645] ownership of diskimage changes during
livemigration, livemigration with kvm/libvirt fails (Adam Huffman)
4. Re: [Bug 1057645] ownership of diskimage changes during
livemigration, livemigration with kvm/libvirt fails (Paul Boven)
5. Re: Complete machine lockup, v3.4.2 (James)
6. Re: Problem with duplicate files (Justin Dossey)
7. upgrading from gluster-3.2.6 to gluster-3.4.2 (Dmitry Kopelevich)
8. Problems to work with mounted directory in Gluster 3.2.7
(Targino Silveira)
9. Re: Node down and volumes unreachable (Marco Zanger)
10. Re: <host> not in 'Peer in Cluster' state (Jon Cope)
11. Re: [Bug 1057645] ownership of diskimage changes during
livemigration, livemigration with kvm/libvirt fails (bernhard glomm)
12. Re: [Bug 1057645] ownership of diskimage changes during
livemigration, livemigration with kvm/libvirt fails (Josh Boon)
13. 3.4 gluster volume heal-failed recovery (Steve Dainard)
14. Re: add-brick and fix-layout takes some VMs offline
(Nicholas Majeran)
15. Re: Problems to work with mounted directory in Gluster 3.2.7
(Franco Broi)
16. Re: Problems to work with mounted directory in Gluster 3.2.7
(Targino Silveira)
17. Agenda for Community meeting today (Vijay Bellur)
----------------------------------------------------------------------
Message: 1
Date: Tue, 18 Feb 2014 08:12:33 -0500
From: Laurent Chouinard <laurent.chouinard@xxxxxxxxxxx>
To: "gluster-users@xxxxxxxxxxx" <gluster-users@xxxxxxxxxxx>
Subject: Complete machine lockup, v3.4.2
Message-ID:
<8CFDF989A34FE54FA678C09C6D9F6D39B560BC342B@xxxxxxxxxxxxxxxxxxxxxxxxxx>
Content-Type: text/plain; charset="us-ascii"
Hi everyone,
We've been using 3.3.2 for a while, and recently started to migrate to 3.4.2. We run on platform CentOS 6.5 for 3.4.2 (while 3.3.2 were installed on CentOS 6.4)
Recently, we've have a very scary condition happen and we do not know exactly the cause of it.
We have a 3 nodes cluster with a replication factor of 3. Each node has one brick, which is made out of one RAID0 volume, comprised of multiple SSDs.
Following some read/write errors, nodes 2 and 3 have completely locked. Nothing could be done physically (nothing on the screen, nothing by SSH), physical power cycle had to be done. Node 1 was still accessible, but its fuse client rejected most if not all reads and writes.
Has anyone experienced something similar?
Before the system freeze, the last thing the kernel seemed to be doing is killing HTTPD threads (INFO: task httpd:7910 blocked for more than 120 seconds.) End-users talk to Apache in order to read/write from the Gluster volume, so it seems a simple case of "something wrong" with gluster which locks read/writes, and eventually the kernel kills them.
At this point, we're unsure where to look. Nothing very specific can be found in the logs, but perhaps if someone has pointers of what to look for, that could give us a new search track.
Thanks
Laurent Chouinard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/c03f9e09/attachment-0001.html>
------------------------------
Message: 2
Date: Tue, 18 Feb 2014 10:24:48 +0700
From: Prian Dwiatma <priandwiatma@xxxxxxxxx>
To: gluster-users@xxxxxxxxxxx
Subject: [Gluster-users] Want to Ask
Message-ID:
<CAAN9Jt2v2a2ftNZNUhBztZpLgqhs=drGiYoxYUhXxEbi9YwB2A@xxxxxxxxxxxxxx>
Content-Type: text/plain; charset="iso-8859-1"
hello....i'm Prian
I want to ask about glusterfs, is there any file types that affect Gluster
for data transmission?
thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/d9454deb/attachment-0001.html>
------------------------------
Message: 3
Date: Tue, 18 Feb 2014 13:59:10 +0000
From: Adam Huffman <adam.huffman@xxxxxxxxx>
To: gluster-users@xxxxxxxxxxx
Subject: Re: [Bug 1057645] ownership of diskimage
changes during livemigration, livemigration with kvm/libvirt fails
Message-ID:
<CAP5prOjzhC1fBqYtpjSnFsfT1xQkDWCGzkX2e_JErq+oW1R1pQ@xxxxxxxxxxxxxx>
Content-Type: text/plain; charset=ISO-8859-1
Hi Paul,
Could you keep the list updated? That bug has been marked private, so
I can't see it.
Best Wishes,
Adam
On Tue, Jan 28, 2014 at 9:29 AM, Paul Boven <boven@xxxxxxx> wrote:
> Hi Bernhard, everyone,
>
> The same problem has now been reproduced on RedHat, please see:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1058032
>
> With 3.4.0 and Ubuntu 13.04, live migrations worked fine. For me it broke
> when the packages were upgraded to 3.4.1.
>
> I've set AppArmor to 'complain' as part of the debugging, so that's not the
> issue.
>
> I'm still not convinced that the file ownership itself is the root cause of
> this issue, it could well be just a symptom. Libvirt/qemu is perfectly happy
> to start a VM when its image file is owned root:root, and change ownership
> to libvirt-qemu:kvm. So I see no reason why it couldn't do the same during a
> live migration.
>
> In my opinion the real issue is the failure at the fuse level, that makes
> file access to the image on the destination impossible, even for root.
>
> Regards, Paul Boven.
>
>
> On 01/27/2014 07:51 PM, BGM wrote:
>>
>> Hi Paul & all
>> I'm really keen on getting this solved,
>> right now it's a nasty show stopper.
>> I could try different gluster versions,
>> as long as I can get the .debs for it,
>> wouldn't want to start compiling
>> (although.... does a config option have changed on package build?)
>> you reported that 3.4.0 on ubuntu 13.04 was working, right?
>> code diff, config options for package build.
>> Another approach: can anyone verify or falsify
>> https://bugzilla.redhat.com/show_bug.cgi?id=1057645
>> on another distro than ubuntu/debian?
>> thinking of it... could it be an apparmor interference?
>> I had fun with apparmor and mysql on ubuntu 12.04 once...
>> will have a look at that tomorrow.
>> As mentioned before, a straight drbd/ocfs2 works (with only 1/4 speed
>> and the pain of maintenance) so AFAIK I have to blame the ownership change
>> on gluster, not on an issue with my general setup....
>> best regards
>> Bernhard
>
>
>
> --
> Paul Boven <boven@xxxxxxx> +31 (0)521-596547
> Unix/Linux/Networking specialist
> Joint Institute for VLBI in Europe - www.jive.nl
> VLBI - It's a fringe science
> _______________________________________________
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
------------------------------
Message: 4
Date: Tue, 18 Feb 2014 15:11:05 +0100
From: Paul Boven <boven@xxxxxxx>
To: gluster-users@xxxxxxxxxxx
Subject: Re: [Bug 1057645] ownership of diskimage
changes during livemigration, livemigration with kvm/libvirt fails
Message-ID: <530369F9.4060404@xxxxxxx>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Hi Adam,
This is rather odd - the Redhat specific clone of the bug we filed is
now marked private, even if I log in with my RH bugtracker account.
However, the original bug is still accessible:
https://bugzilla.redhat.com/show_bug.cgi?id=1057645
There's not been any progress as far as I know. We are using the
workaround (which stops libvirt/qemu from doing the chown) in
production. With the release of Ubuntu 14.04 LTS, I hope to be able to
use libgfapi on our setup.
Perhaps the fact that the RedHat specific bug is now private might mean
that they're actually doing something with it, but I wouldn't know.
Regards, Paul Boven.
On 02/18/2014 02:59 PM, Adam Huffman wrote:
> Hi Paul,
>
> Could you keep the list updated? That bug has been marked private, so
> I can't see it.
>
> Best Wishes,
> Adam
>
> On Tue, Jan 28, 2014 at 9:29 AM, Paul Boven <boven@xxxxxxx> wrote:
>> Hi Bernhard, everyone,
>>
>> The same problem has now been reproduced on RedHat, please see:
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1058032
>>
>> With 3.4.0 and Ubuntu 13.04, live migrations worked fine. For me it broke
>> when the packages were upgraded to 3.4.1.
>>
>> I've set AppArmor to 'complain' as part of the debugging, so that's not the
>> issue.
>>
>> I'm still not convinced that the file ownership itself is the root cause of
>> this issue, it could well be just a symptom. Libvirt/qemu is perfectly happy
>> to start a VM when its image file is owned root:root, and change ownership
>> to libvirt-qemu:kvm. So I see no reason why it couldn't do the same during a
>> live migration.
>>
>> In my opinion the real issue is the failure at the fuse level, that makes
>> file access to the image on the destination impossible, even for root.
>>
>> Regards, Paul Boven.
>>
>>
>> On 01/27/2014 07:51 PM, BGM wrote:
>>>
>>> Hi Paul & all
>>> I'm really keen on getting this solved,
>>> right now it's a nasty show stopper.
>>> I could try different gluster versions,
>>> as long as I can get the .debs for it,
>>> wouldn't want to start compiling
>>> (although.... does a config option have changed on package build?)
>>> you reported that 3.4.0 on ubuntu 13.04 was working, right?
>>> code diff, config options for package build.
>>> Another approach: can anyone verify or falsify
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1057645
>>> on another distro than ubuntu/debian?
>>> thinking of it... could it be an apparmor interference?
>>> I had fun with apparmor and mysql on ubuntu 12.04 once...
>>> will have a look at that tomorrow.
>>> As mentioned before, a straight drbd/ocfs2 works (with only 1/4 speed
>>> and the pain of maintenance) so AFAIK I have to blame the ownership change
>>> on gluster, not on an issue with my general setup....
>>> best regards
>>> Bernhard
>>
>>
>>
>> --
>> Paul Boven <boven@xxxxxxx> +31 (0)521-596547
>> Unix/Linux/Networking specialist
>> Joint Institute for VLBI in Europe - www.jive.nl
>> VLBI - It's a fringe science
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users@xxxxxxxxxxx
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
--
Paul Boven <boven@xxxxxxx> +31 (0)521-596547
Unix/Linux/Networking specialist
Joint Institute for VLBI in Europe - www.jive.nl
VLBI - It's a fringe science
------------------------------
Message: 5
Date: Tue, 18 Feb 2014 11:17:46 -0500
From: James <purpleidea@xxxxxxxxx>
To: Laurent Chouinard <laurent.chouinard@xxxxxxxxxxx>
Cc: "gluster-users@xxxxxxxxxxx" <gluster-users@xxxxxxxxxxx>
Subject: Re: Complete machine lockup, v3.4.2
Message-ID:
<CADCaTgrkYpU00E6Gk4t2S0oZaERq6mQAq8nPTscdiRf2XnWFWg@xxxxxxxxxxxxxx>
Content-Type: text/plain; charset=UTF-8
On Tue, Feb 18, 2014 at 8:12 AM, Laurent Chouinard
<laurent.chouinard@xxxxxxxxxxx> wrote:
> Before the system freeze, the last thing the kernel seemed to be doing is
> killing HTTPD threads (INFO: task httpd:7910 blocked for more than 120
> seconds.) End-users talk to Apache in order to read/write from the Gluster
> volume, so it seems a simple case of ?something wrong? with gluster which
> locks read/writes, and eventually the kernel kills them.
If the kernel was killing things, check that it wasn't the OOM killer.
If so, you might want to ensure you've got swap, enough memory, check
if anything is leaking, and finally if you have memory management
issues between services, cgroups might be the thing to use to control
this.
HTH,
James
------------------------------
Message: 6
Date: Tue, 18 Feb 2014 08:30:20 -0800
From: Justin Dossey <jbd@xxxxxxxxxxxxx>
To: tegner@xxxxxxxxx
Cc: gluster-users <gluster-users@xxxxxxxxxxx>
Subject: Re: Problem with duplicate files
Message-ID:
<CAPMPShzN6nprhm17DQzFbM4oBz09-VLntnK2XsVa3XPLB0sgrw@xxxxxxxxxxxxxx>
Content-Type: text/plain; charset="iso-8859-1"
I've seen issues with GlusterFS and rsync. They were mostly ameliorated by
adding --inplace to my rsync arguments.
On Mon, Feb 17, 2014 at 12:57 AM, <tegner@xxxxxxxxx> wrote:
> As a follow up. Lots of the following from log:
>
> [dht_lookup_everywhere_cbk] 0-glusterKumiko-dht: deleting stale linkfile
> 0.33702/T on glusterKumiko-client-2
>
> [client-rpc-fops.c:5724:client3_3_setattr]
> (-->/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_lookup_linkfile_create_cbk+0x262)
> [0x7fd78746a8f2]
> (-->/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_linkfile_attr_heal+0x3ce)
> [0x7fd7874541fe]
> (-->/usr/lib64/glusterfs/3.4.2/xlator/protocol/client.so(client_setattr+0x89)
> [0x7fd78769a649]))) 0-: Assertion failed: 0
>
> [2014-02-17 08:21:48.200685] E
> [dht-linkfile.c:213:dht_linkfile_setattr_cbk] 0-glusterKumiko-dht: setattr
> of uid/gid on 0.33702/T :<gfid:00000000-0000-0000-0000-000000000000> failed
> (Invalid argument)
>
> There were also quite a few cases were there were directories that were
> empty except for a bunch of "mode 1000" files.
>
> The system in question is used as a mirror of a file system which is used
> quite heavily, new files are created, old ones are deleted (and quite often
> recreated). I use rsync to update the system, like
>
> rsync -vau --delete live:/gluster_1/ mirror:gluster_2/
>
> Could the issues I'm experiencing be connected to the way I use rsync?
>
> Thanks,
>
> /jon
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
--
Justin Dossey
CTO, PodOmatic
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/71f28aab/attachment-0001.html>
------------------------------
Message: 7
Date: Tue, 18 Feb 2014 14:51:39 -0500
From: Dmitry Kopelevich <dkopelevich@xxxxxxxxxxx>
To: <gluster-users@xxxxxxxxxxx>
Subject: upgrading from gluster-3.2.6 to gluster-3.4.2
Message-ID: <5303B9CB.7000800@xxxxxxxxxxx>
Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"
I am attempting to upgrade my GlusterFS from 3.2.6 to 3.4.2 using the
instructions posted at
http://vbellur.wordpress.com/2012/05/31/upgrading-to-glusterfs-3-3.
These guidelines are for an upgrade to 3.3 but it is stated at
http://vbellur.wordpress.com/2013/07/15/upgrading-to-glusterfs-3-4that
they can also be used to upgrade to 3.4.0. So I was hoping that they
would also work with an upgrade to 3.4.2.
I'm running CentOS 5 and installed the following rpms on the gluster
servers:
glusterfs-libs-3.4.2-1.el5.x86_64.rpm
glusterfs-3.4.2-1.el5.x86_64.rpm
glusterfs-fuse-3.4.2-1.el5.x86_64.rpm
glusterfs-cli-3.4.2-1.el5.x86_64.rpm
glusterfs-server-3.4.2-1.el5.x86_64.rpm
glusterfs-rdma-3.4.2-1.el5.x86_64.rpm
glusterfs-geo-replication-3.4.2-1.el5.x86_64.rpm
According to the installation guidelines, installation from rpms should
automatically copy the files from /etc/glusterd to /var/lib/glusterd.
This didn't happen for me -- the directory /var/lib/glusterd contained
only empty subdirectories. But the content of /etc/glusterd directory
has moved to /etc/glusterd/glusterd.
So, I decided to manually copy files from /etc/glusterd/glusterd to
/var/lib/glusterd and follow step 5 of the installation guidelines
(which was supposed to be skipped when installing from rpms):
glusterd --xlator-option *.upgrade=on -N
This didn't work (error message: glusterd: No match)
Then I triedspecifying explicitly the name of my volume:
glusterd --xlator-option <volume>.upgrade=on -N
This lead to the following messages in file etc-glusterfs-glusterd.vol.log:
[2014-02-18 17:22:27.146449] I [glusterd.c:961:init] 0-management: Using
/var/lib/glusterd as working directory
[2014-02-18 17:22:27.149097] I [socket.c:3480:socket_init]
0-socket.management: SSL support is NOT enabled
[2014-02-18 17:22:27.149126] I [socket.c:3495:socket_init]
0-socket.management: using system polling thread
[2014-02-18 17:22:29.282665] I
[glusterd-store.c:1339:glusterd_restore_op_version] 0-glusterd:
retrieved op-version: 1
[2014-02-18 17:22:29.283478] E
[glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-0
[2014-02-18 17:22:29.283513] E
[glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-1
[2014-02-18 17:22:29.283534] E
[glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-2
...
and so on for all other bricks.
After that, files nfs.log, glustershd.log, and
etc-glusterfs-glusterd.vol.log get filled with a large number of warning
messages and nothing else seems to happen. The following messages appear
to be relevant:
- Files nfs.log, glustershd.log:
2014-02-18 15:58:01.889847] W [rdma.c:1079:gf_rdma_cm_event_handler]
0-data-volume-client-2: cma event RDMA_CM_EVENT_ADDR_ERROR, error -2
(me: peer:)
(the name of my volume is data-volume and its transport type is RDMA)
- File etc-glusterfs-glusterd.vol.log
[2014-02-18 17:22:33.322565] W [socket.c:514:__socket_rwv] 0-management:
readv failed (No data available)
Also, for some reason the time stamps in the log files are incorrect.
Any suggestions for fixing this would be greatly appreciated.
Thanks,
Dmitry
--
Dmitry Kopelevich
Associate Professor
Chemical Engineering Department
University of Florida
Gainesville, FL 32611
Phone: (352)-392-4422
Fax: (352)-392-9513
E-mail:dkopelevich@xxxxxxxxxxx
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/a7d9f296/attachment-0001.html>
------------------------------
Message: 8
Date: Tue, 18 Feb 2014 17:30:35 -0300
From: Targino Silveira <targinosilveira@xxxxxxxxx>
To: gluster-users@xxxxxxxxxxx
Subject: Problems to work with mounted directory in
Gluster 3.2.7
Message-ID:
<CA+PDx1i_6=baTLc=s_F2sA6vwT=YV3vT_pLPNh+kYUM_TSDcwA@xxxxxxxxxxxxxx>
Content-Type: text/plain; charset="iso-8859-1"
Hi all,
I am getting a problem and can't understand why, I have created a cluster
with gluster following the most simple way as explaind in QuickStart on the
glustger.org.
I have created 3 VM in KVM.
2 To host gluster server with one disk image with 1tb.
1 To host gluster client to mounting my volume.
I'm using Debian 7 and used apt-get to install Gluster 3.2.7, Server and
Client.
After all finished I could to mount as glusterfs with no problem "mount -t
glusterfs /mnt/cloudbackup_data/ vm-gluster-cloudbackup1:/vol1" but I can't
to work on the mounted directory, if I perform a "ls -lh" it's running
forver, and I can't do any other operation and VM is blocked.
If I try to mount as NFS "mount -t nfs vm-gluster-cloudbackup2:/vol1
/mnt/cloudbackup_data/ "I get a "Time out" I'm not so much expert in
gluster, but I can't see any reason for this problem, someone know
something like that?
Regards,
Targino Silveira
+55-85-8626-7297
www.twitter.com/targinosilveira
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/e27144fe/attachment-0001.html>
------------------------------
Message: 9
Date: Tue, 18 Feb 2014 20:44:18 +0000
From: Marco Zanger <mzanger@xxxxxxxxxxx>
To: Vijay Bellur <vbellur@xxxxxxxxxx>, "gluster-users@xxxxxxxxxxx"
<gluster-users@xxxxxxxxxxx>
Subject: Re: Node down and volumes unreachable
Message-ID:
<A2DF3CE8B1F5C8418E1721A57D20B527DB792827@xxxxxxxxxxxxxxxxxx>
Content-Type: text/plain; charset="iso-8859-1"
The Log of that particular volume says:
[2014-02-18 09:43:17.136182] W [socket.c:410:__socket_keepalive] 0-socket: failed to set keep idle on socket 8
[2014-02-18 09:43:17.136285] W [socket.c:1876:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported
[2014-02-18 09:43:18.343409] I [server-handshake.c:571:server_setvolume] 0-teoswitch_default_storage-server: accepted client from xxxxx55.domain.com-2075-2014/02/18-09:43:14:302234-teoswitch_default_storage-client-1-0 (version: 3.3.0)
[2014-02-18 09:43:21.356302] I [server-handshake.c:571:server_setvolume] 0-teoswitch_default_storage-server: accepted client from xxxxx54. domain.com-9651-2014/02/18-09:42:00:141779-teoswitch_default_storage-client-1-0 (version: 3.3.0)
[2014-02-18 10:38:26.488333] W [socket.c:195:__socket_rwv] 0-tcp.teoswitch_default_storage-server: readv failed (Connection timed out)
[2014-02-18 10:38:26.488431] I [server.c:685:server_rpc_notify] 0-teoswitch_default_storage-server: disconnecting connectionfrom xxxxx54.hexacta.com-9651-2014/02/18-09:42:00:141779-teoswitch_default_storage-client-1-0
[2014-02-18 10:38:26.488494] I [server-helpers.c:741:server_connection_put] 0-teoswitch_default_storage-server: Shutting down connection xxxxx54.hexacta.com-9651-2014/02/18-09:42:00:141779-teoswitch_default_storage-client-1-0
[2014-02-18 10:38:26.488541] I [server-helpers.c:629:server_connection_destroy] 0-teoswitch_default_storage-server: destroyed connection of xxxxx54.hexacta.com-9651-2014/02/18-09:42:00:141779-teoswitch_default_storage-client-1-0
When I try to access the folder I get.
[root@hxteo55 ~]# ll /<path> /1001/voicemail/
ls: /<path>/1001/voicemail/: Input/output error?
This is the volume info:
Volume Name: teoswitch_default_storage
Type: Distribute
Volume ID: 83c9d6f3-0288-4358-9fdc-b1d062cc8fca
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 12.12.123.54:/<path>/gluster/36779974/teoswitch_default_storage
Brick2: 12.12.123.55:/<path>/gluster/36779974/teoswitch_default_storage
Any ideas?
Marco Zanger
Phone 54 11 5299-5400 (int. 5501)
Clay 2954, C1426DLD, Buenos Aires, Argentina
Think Green - Please do not print this email unless you really need to
-----Original Message-----
From: Vijay Bellur [mailto:vbellur@xxxxxxxxxx]
Sent: martes, 18 de febrero de 2014 03:56 a.m.
To: Marco Zanger; gluster-users@xxxxxxxxxxx
Subject: Re: Node down and volumes unreachable
On 02/17/2014 11:19 PM, Marco Zanger wrote:
> Read/write operations hang for long period of time (too long). I've
> seen it in that state (waiting) for something like 5 minutes, which
> makes every application fail trying to read or write. These are the
> Errors I found in the logs in the server A which is still accessible
> (B was down)
>
> etc-glusterfs-glusterd.vol.log
>
> ...
> [2014-01-31 07:56:49.780247] W
> [socket.c:1512:__socket_proto_state_machine] 0-management: reading
> from socket failed. Error (Connection timed out), peer
> (<SERVER_B_IP>:24007)
> [2014-01-31 07:58:25.965783] E [socket.c:1715:socket_connect_finish]
> 0-management: connection to <SERVER_B_IP>:24007 failed (No route to
> host)
> [2014-01-31 08:59:33.923250] I
> [glusterd-handshake.c:397:glusterd_set_clnt_mgmt_program] 0-: Using
> Program glusterd mgmt, Num (1238433), Version (2)
> [2014-01-31 08:59:33.923289] I
> [glusterd-handshake.c:403:glusterd_set_clnt_mgmt_program] 0-: Using Program Peer mgmt, Num (1238437), Version (2) ...
>
>
> glustershd.log
>
> [2014-01-27 12:07:03.644849] W
> [socket.c:1512:__socket_proto_state_machine]
> 0-teoswitch_custom_music-client-1: reading from socket failed. Error
> (Connection timed out), peer (<SERVER_B_IP>:24010)
> [2014-01-27 12:07:03.644888] I [client.c:2090:client_rpc_notify]
> 0-teoswitch_custom_music-client-1: disconnected
> [2014-01-27 12:09:35.553628] E [socket.c:1715:socket_connect_finish]
> 0-teoswitch_greetings-client-1: connection to <SERVER_B_IP>:24011
> failed (Connection timed out)
> [2014-01-27 12:10:13.588148] E [socket.c:1715:socket_connect_finish]
> 0-license_path-client-1: connection to <SERVER_B_IP>:24013 failed
> (Connection timed out)
> [2014-01-27 12:10:15.593699] E [socket.c:1715:socket_connect_finish]
> 0-upload_path-client-1: connection to <SERVER_B_IP>:24009 failed
> (Connection timed out)
> [2014-01-27 12:10:21.601670] E [socket.c:1715:socket_connect_finish]
> 0-teoswitch_ivr_greetings-client-1: connection to <SERVER_B_IP>:24012
> failed (Connection timed out)
> [2014-01-27 12:10:23.607312] E [socket.c:1715:socket_connect_finish]
> 0-teoswitch_custom_music-client-1: connection to <SERVER_B_IP>:24010
> failed (Connection timed out)
> [2014-01-27 12:11:21.866604] E [afr-self-heald.c:418:_crawl_proceed]
> 0-teoswitch_ivr_greetings-replicate-0: Stopping crawl as < 2 children
> are up
> [2014-01-27 12:11:21.867874] E [afr-self-heald.c:418:_crawl_proceed]
> 0-teoswitch_greetings-replicate-0: Stopping crawl as < 2 children are
> up
> [2014-01-27 12:11:21.868134] E [afr-self-heald.c:418:_crawl_proceed]
> 0-teoswitch_custom_music-replicate-0: Stopping crawl as < 2 children
> are up
> [2014-01-27 12:11:21.869417] E [afr-self-heald.c:418:_crawl_proceed]
> 0-license_path-replicate-0: Stopping crawl as < 2 children are up
> [2014-01-27 12:11:21.869659] E [afr-self-heald.c:418:_crawl_proceed]
> 0-upload_path-replicate-0: Stopping crawl as < 2 children are up
> [2014-01-27 12:12:53.948154] I
> [client-handshake.c:1636:select_server_supported_programs]
> 0-teoswitch_greetings-client-1: Using Program GlusterFS 3.3.0, Num
> (1298437), Version (330)
> [2014-01-27 12:12:53.952894] I
> [client-handshake.c:1433:client_setvolume_cbk]
> 0-teoswitch_greetings-client-1: Connected to <SERVER_B_IP>:24011,
> attached to remote volume
>
> nfs.log there are lots of errors but the one that insist most Is this:
>
> [2014-01-27 12:12:27.136033] E [socket.c:1715:socket_connect_finish]
> 0-teoswitch_custom_music-client-1: connection to <SERVER_B_IP>:24010
> failed (Connection timed out)
>
> Any ideas? From the logs I see nothing but confirm the fact that A cannot reach B which makes sense since B is down. But A is not, and it's volume should still be accesible. Right?
Nothing very obvious from these logs.
Can you share relevant portions of the client log file? Usually the name of the mount point would be a part of the client log file.
-Vijay
------------------------------
Message: 10
Date: Tue, 18 Feb 2014 16:23:19 -0500 (EST)
From: Jon Cope <jcope@xxxxxxxxxx>
To: Kaushal M <kshlmster@xxxxxxxxx>
Cc: gluster-users@xxxxxxxxxxx
Subject: Re: <host> not in 'Peer in Cluster' state
Message-ID:
<515830110.1228610.1392758599748.JavaMail.zimbra@xxxxxxxxxx>
Content-Type: text/plain; charset=utf-8
Hi Kaushal,
Thanks for the input. I gave it a go and it produced identical results. I was, however, pointed towards an article that's led me to a solution, linked below. As I understand it, by assigning each node an elastic IP, you cement the public DNS (now containing the elasticIP), preventing AWS from changing it during reboot. Querying the public DNS from inside EC2 returns the private IP addresses, while a query from outside EC2 returns the elastic IP. Gluster seems happy with this, so I am too.
Regards,
Jon
http://alestic.com/2009/06/ec2-elastic-ip-internal
----- Original Message -----
From: "Kaushal M" <kshlmster@xxxxxxxxx>
To: "Jon Cope" <jcope@xxxxxxxxxx>
Cc: gluster-users@xxxxxxxxxxx
Sent: Saturday, February 15, 2014 5:40:32 AM
Subject: Re: <host> not in 'Peer in Cluster' state
Peer status having node1's elastic ip suggests that you probed the
other peers from node1. This would mean that the other peers don't
know of node1's hostname. Even though you've edited the hosts file on
the peers, a reverse resolution on node1s ip wouldn't return the
hostnames you've set. Gluster uses reverse resolution to match
hostnames when it doesn't have a straight match in the peer list.
To recover from this. just probe node1 from another peer. Do '#
gluster peer probe node1.ec2' from another peer. This will update
gluster's peerlist to contain the name node1.ec2. After this other
operations will continue successfully.
~kaushal
On Sat, Feb 15, 2014 at 5:23 AM, Jon Cope <jcope@xxxxxxxxxx> wrote:
> Hi all,
>
> I'm attempting to create a 4 nodes cluster over EC2. I'm fairly new to this and so may not be seeing something obvious.
>
> - Established passworldless SSH between nodes.
> - edited /etc/sysconfig/network HOSTNAME=node#.ec2 to satisfy FQDN
> - mounted xfs /dev/xvdh /mnt/brick1
> - stopped iptables
>
>
> The error I'm getting occurs when invoking the following, where <volume> is the volume name:
>
> # gluster volume create <volume> replica 2 node1.ec2:/mnt/brick1 node2.ec2:/mnt/brick1 node3.ec2:/mnt/brick1 node4.ec2:/mnt/brick1
> # volume create: <volume>: failed: Host node1.ec2 is not in 'Peer in Cluster' state
>
> Checking peer status of node1.ec2 from node{2..4}.ec2 produces the following. Note that node1.ec2's elastic IP appears instead of the FQDN; not sure if that's relevant or not.
>
> [root@node2 ~]# gluster peer status
> Number of Peers: 3
>
> Hostname: node4.ec2
> Uuid: ab2bcdd8-2c0b-439d-b685-3be457988abc
> State: Peer in Cluster (Connected)
>
> Hostname: node3.ec2
> Uuid: 4f128213-3549-494a-af04-822b5e2f2b96
> State: Peer in Cluster (Connected)
>
> Hostname: ###.##.##.### #node1.ec2 elastic IP
> Uuid: 09d81803-e5e1-43b1-9faf-e94f730acc3e
> State: Peer in Cluster (Connected)
>
> The error as it appears in vim etc-glusterfs-glusterd.vol.log:
>
> [2014-02-14 23:28:44.634663] E [glusterd-utils.c:5351:glusterd_new_brick_validate] 0-management: Host node1.ec2 is not in 'Peer in Cluster' state
> [2014-02-14 23:28:44.634699] E [glusterd-volume-ops.c:795:glusterd_op_stage_create_volume] 0-management: Host node1.ec2 is not in 'Peer in Cluster' state
> [2014-02-14 23:28:44.634718] E [glusterd-syncop.c:890:gd_stage_op_phase] 0-management: Staging of operation 'Volume Create' failed on localhost : Host node1.ec2 is not in 'Peer in Cluster' state
>
> Can someone suggest possible cause of this error or point me in a viable direction?
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
------------------------------
Message: 11
Date: Tue, 18 Feb 2014 22:23:53 +0100
From: bernhard glomm <bernhard.glomm@xxxxxxxxxxx>
To: Paul Boven <boven@xxxxxxx>, gluster-devel@xxxxxxxxxx
Cc: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>
Subject: Re: [Bug 1057645] ownership of diskimage
changes during livemigration, livemigration with kvm/libvirt fails
Message-ID: <7396719A-ADEE-4F49-93BA-D924B469A0EE@xxxxxxxxxxx>
Content-Type: text/plain; charset="windows-1252"
Hi Paul, and all
> With the release of Ubuntu 14.04 LTS, I hope to be able to use libgfapi on our setup.
Well I hope that too, but I'm not sure if that will happen (soon).
I , recompiled qemu as described here:
http://www.gluster.org/community/documentation/index.php/Building_QEMU_with_gfapi_for_Debian_based_systems
with little luck since finally the ipxe-qemu component (which I need/want)
and I didn't had the time to dig deeper into that.
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1224517
still says "won't fix it" :-(
Josh Boon (hi Josh) suggested to create a PPA for an ubuntu qemu with gfapi support
which might be a temporary solution?
But from my "happy end - user" perspective (as a "non-dev-but-op")
that looks like a lot of parallel extra work
on maintaining that "fork"in the long run.
I hope the gluster devels either can get it into debian(and/or ubuntu)
or even qemu (as a default option?) directly
> Perhaps the fact that the RedHat specific bug is now private might mean that they're actually doing something with it, but I wouldn't know.
we'll stay on that topic
Regards,
Bernahrd
> Regards, Paul Boven.
> On 02/18/2014 02:59 PM, Adam Huffman wrote:
>> Hi Paul,
>>
>> Could you keep the list updated? That bug has been marked private, so
>> I can't see it.
>>
>> Best Wishes,
>> Adam
>>
>> On Tue, Jan 28, 2014 at 9:29 AM, Paul Boven <boven@xxxxxxx> wrote:
>>> Hi Bernhard, everyone,
>>>
>>> The same problem has now been reproduced on RedHat, please see:
>>>
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1058032
>>>
>>> With 3.4.0 and Ubuntu 13.04, live migrations worked fine. For me it broke
>>> when the packages were upgraded to 3.4.1.
>>>
>>> I've set AppArmor to 'complain' as part of the debugging, so that's not the
>>> issue.
>>>
>>> I'm still not convinced that the file ownership itself is the root cause of
>>> this issue, it could well be just a symptom. Libvirt/qemu is perfectly happy
>>> to start a VM when its image file is owned root:root, and change ownership
>>> to libvirt-qemu:kvm. So I see no reason why it couldn't do the same during a
>>> live migration.
>>>
>>> In my opinion the real issue is the failure at the fuse level, that makes
>>> file access to the image on the destination impossible, even for root.
>>>
>>> Regards, Paul Boven.
>>>
>>>
>>> On 01/27/2014 07:51 PM, BGM wrote:
>>>>
>>>> Hi Paul & all
>>>> I'm really keen on getting this solved,
>>>> right now it's a nasty show stopper.
>>>> I could try different gluster versions,
>>>> as long as I can get the .debs for it,
>>>> wouldn't want to start compiling
>>>> (although.... does a config option have changed on package build?)
>>>> you reported that 3.4.0 on ubuntu 13.04 was working, right?
>>>> code diff, config options for package build.
>>>> Another approach: can anyone verify or falsify
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1057645
>>>> on another distro than ubuntu/debian?
>>>> thinking of it... could it be an apparmor interference?
>>>> I had fun with apparmor and mysql on ubuntu 12.04 once...
>>>> will have a look at that tomorrow.
>>>> As mentioned before, a straight drbd/ocfs2 works (with only 1/4 speed
>>>> and the pain of maintenance) so AFAIK I have to blame the ownership change
>>>> on gluster, not on an issue with my general setup....
>>>> best regards
>>>> Bernhard
>>>
>>>
>>>
>>> --
>>> Paul Boven <boven@xxxxxxx> +31 (0)521-596547
>>> Unix/Linux/Networking specialist
>>> Joint Institute for VLBI in Europe - www.jive.nl
>>> VLBI - It's a fringe science
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users@xxxxxxxxxxx
>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users@xxxxxxxxxxx
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>
>
> --
> Paul Boven <boven@xxxxxxx> +31 (0)521-596547
> Unix/Linux/Networking specialist
> Joint Institute for VLBI in Europe - www.jive.nl
> VLBI - It's a fringe science
> _______________________________________________
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/ffabe77c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/ffabe77c/attachment-0001.sig>
------------------------------
Message: 12
Date: Tue, 18 Feb 2014 15:31:24 -0600 (CST)
From: Josh Boon <gluster@xxxxxxxxxxxx>
To: bernhard glomm <bernhard.glomm@xxxxxxxxxxx>
Cc: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>,
gluster-devel@xxxxxxxxxx
Subject: Re: [Bug 1057645] ownership of diskimage
changes during livemigration, livemigration with kvm/libvirt fails
Message-ID: <1078179789.1672169.1392759084410.JavaMail.zimbra@xxxxxxx>
Content-Type: text/plain; charset="utf-8"
I've yet to crack into 14.04 and see what that beast looks like as we don't have it in use here yet. I'll add that to my todo list. I can be bribed if someone needs it sooner :)
As a separate task, I'll look into the ipxe problem. I remember it was file conflict between the packages so I may have to do some hacks in the debian build rules for one of the packages.
The PPA is an option but yes maintenance would be ongoing and best effort as that's all I can afford currently.
----- Original Message -----
From: "bernhard glomm" <bernhard.glomm@xxxxxxxxxxx>
To: "Paul Boven" <boven@xxxxxxx>, gluster-devel@xxxxxxxxxx
Cc: "Josh Boon" <gluster@xxxxxxxxxxxx>, "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>
Sent: Tuesday, February 18, 2014 4:23:53 PM
Subject: Re: [Bug 1057645] ownership of diskimage changes during livemigration, livemigration with kvm/libvirt fails
Hi Paul, and all
With the release of Ubuntu 14.04 LTS, I hope to be able to use libgfapi on our setup.
Well I hope that too, but I'm not sure if that will happen (soon).
I , recompiled qemu as described here:
http://www.gluster.org/community/documentation/index.php/Building_QEMU_with_gfapi_for_Debian_based_systems
with little luck since finally the ipxe-qemu component (which I need/want)
and I didn't had the time to dig deeper into that.
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1224517
still says "won't fix it" :-(
Josh Boon (hi Josh) suggested to create a PPA for an ubuntu qemu with gfapi support
which might be a temporary solution ?
But from my "happy end - user" perspective (as a "non-dev-but-op")
that looks like a lot of parallel extra work
on maintaining that "fork"in the long run.
I hope the gluster devels either can get it into debian(and/or ubuntu)
or even qemu (as a default option?) directly
<blockquote>
Perhaps the fact that the RedHat specific bug is now private might mean that they're actually doing something with it, but I wouldn't know.
</blockquote>
we'll stay on that topic
Regards,
Bernahrd
<blockquote>
Regards, Paul Boven.
</blockquote>
<blockquote>
On 02/18/2014 02:59 PM, Adam Huffman wrote:
<blockquote>
Hi Paul,
Could you keep the list updated? That bug has been marked private, so
I can't see it.
Best Wishes,
Adam
On Tue, Jan 28, 2014 at 9:29 AM, Paul Boven < boven@xxxxxxx > wrote:
<blockquote>
Hi Bernhard, everyone,
The same problem has now been reproduced on RedHat, please see:
https://bugzilla.redhat.com/show_bug.cgi?id=1058032
With 3.4.0 and Ubuntu 13.04, live migrations worked fine. For me it broke
when the packages were upgraded to 3.4.1.
I've set AppArmor to 'complain' as part of the debugging, so that's not the
issue.
I'm still not convinced that the file ownership itself is the root cause of
this issue, it could well be just a symptom. Libvirt/qemu is perfectly happy
to start a VM when its image file is owned root:root, and change ownership
to libvirt-qemu:kvm. So I see no reason why it couldn't do the same during a
live migration.
In my opinion the real issue is the failure at the fuse level, that makes
file access to the image on the destination impossible, even for root.
Regards, Paul Boven.
On 01/27/2014 07:51 PM, BGM wrote:
<blockquote>
Hi Paul & all
I'm really keen on getting this solved,
right now it's a nasty show stopper.
I could try different gluster versions,
as long as I can get the .debs for it,
wouldn't want to start compiling
(although.... does a config option have changed on package build?)
you reported that 3.4.0 on ubuntu 13.04 was working, right?
code diff, config options for package build.
Another approach: can anyone verify or falsify
https://bugzilla.redhat.com/show_bug.cgi?id=1057645
on another distro than ubuntu/debian?
thinking of it... could it be an apparmor interference?
I had fun with apparmor and mysql on ubuntu 12.04 once...
will have a look at that tomorrow.
As mentioned before, a straight drbd/ocfs2 works (with only 1/4 speed
and the pain of maintenance) so AFAIK I have to blame the ownership change
on gluster, not on an issue with my general setup....
best regards
Bernhard
</blockquote>
--
Paul Boven <boven@xxxxxxx> +31 (0)521-596547
Unix/Linux/Networking specialist
Joint Institute for VLBI in Europe - www.jive.nl
VLBI - It's a fringe science
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users
</blockquote>
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users
</blockquote>
--
Paul Boven < boven@xxxxxxx > +31 (0)521-596547
Unix/Linux/Networking specialist
Joint Institute for VLBI in Europe - www.jive.nl
VLBI - It's a fringe science
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users
</blockquote>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/2839804c/attachment-0001.html>
------------------------------
Message: 13
Date: Tue, 18 Feb 2014 16:57:14 -0500
From: Steve Dainard <sdainard@xxxxxxxxxxxxx>
To: gluster-users@xxxxxxxxxxx
Subject: 3.4 gluster volume heal-failed recovery
Message-ID:
<CAHnsdUtwRTccoAy-ghiqUHx-KbL5jVmOuGFxmC0cvtwktJnR=g@xxxxxxxxxxxxxx>
Content-Type: text/plain; charset="iso-8859-1"
Is there a proper method to recover from heal-failed entries?
rep1 is a 2 brick, quorum enabled volume used for virt storage for Ovirt,
accessed via NFS:
gluster> volume info rep1
Volume Name: rep1
Type: Replicate
Volume ID: 5876746d-5977-4fa9-a829-69f44b3364ec
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.0.10.2:/mnt/storage/lv-storage-domain/rep1
Brick2: 10.0.10.3:/mnt/storage/lv-storage-domain/rep1
Options Reconfigured:
cluster.server-quorum-type: server
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
nfs.trusted-write: on
nfs.trusted-sync: off
storage.owner-gid: 36
storage.owner-uid: 36
server.allow-insecure: on
cluster.quorum-type: auto
gluster> volume heal rep1 info heal-failed
Gathering Heal info on volume rep1 has been successful
Brick 10.0.10.2:/mnt/storage/lv-storage-domain/rep1
Number of entries: 6
at path on brick
-----------------------------------
2014-02-14 17:25:50 <gfid:2d619e4d-6823-4fb3-ad9d-f33459833f85>
2014-02-14 17:25:50 <gfid:0708b030-3445-4015-a6dd-c28446e81e67>
2014-02-14 17:25:50 <gfid:ce29e947-bebd-4b6a-a619-010931b5f9a8>
2014-02-14 17:25:50 <gfid:2d619e4d-6823-4fb3-ad9d-f33459833f85>
2014-02-14 17:25:50 <gfid:0708b030-3445-4015-a6dd-c28446e81e67>
2014-02-14 17:25:50 <gfid:ce29e947-bebd-4b6a-a619-010931b5f9a8>
Brick 10.0.10.3:/mnt/storage/lv-storage-domain/rep1
Number of entries: 0
Seeing as the hosts are both in quorum I was surprised to see this, but
these id's have been repeating in my logs.
Thanks,
*Steve Dainard *
IT Infrastructure Manager
Miovision <http://miovision.com/> | *Rethink Traffic*
*Blog <http://miovision.com/blog> | **LinkedIn
<https://www.linkedin.com/company/miovision-technologies> | Twitter
<https://twitter.com/miovision> | Facebook
<https://www.facebook.com/miovision>*
------------------------------
Miovision Technologies Inc. | 148 Manitou Drive, Suite 101, Kitchener, ON,
Canada | N2C 1L3
This e-mail may contain information that is privileged or confidential. If
you are not the intended recipient, please delete the e-mail and any
attachments and notify us immediately.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/4385a702/attachment-0001.html>
------------------------------
Message: 14
Date: Tue, 18 Feb 2014 16:19:42 -0600
From: Nicholas Majeran <nmajeran@xxxxxxxxxxxxxxxxx>
To: gluster-users@xxxxxxxxxxx
Subject: Re: add-brick and fix-layout takes some VMs
offline
Message-ID: <5303DC7E.7040902@xxxxxxxxxxxxxxxxx>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
On 02/13/2014 09:22 AM, Nicholas Majeran wrote:
> Hi there,
>
> We have a distributed-replicated volume hosting KVM guests running
> Gluster 3.4.1.
> We've grown from 1 x 2 -> 2 x 2 -> 3 x 2,but each time we've added
> nodes or run a fix layout,
> some of our guests go offline (or worse with error=continue they
> silently error).
> After the last addition we didn't even run fix-layout as the guests
> are becoming increasingly important.
>
> Those guests are currently are using a combination of FUSE and libgfapi.
> Is there a setting or group of settings we should use to ameliorate
> this problem?
> Is FUSE or libgfapi more forgiving when add-brick or fix-layout is run?
>
> Any info is greatly appreciated.
> Thanks.
>
>
Hi,
Does anyone have any ideas on the above?
Thanks.
------------------------------
Message: 15
Date: Wed, 19 Feb 2014 08:30:25 +0800
From: Franco Broi <franco.broi@xxxxxxxxxx>
To: Targino Silveira <targinosilveira@xxxxxxxxx>
Cc: gluster-users@xxxxxxxxxxx
Subject: Re: Problems to work with mounted directory
in Gluster 3.2.7
Message-ID: <1392769825.2437.83.camel@tc1>
Content-Type: text/plain; charset="UTF-8"
You need to check the log files for obvious problems. On the servers
these should be in /var/log/glusterfs/ and you can turn on logging for
the fuse client like this:
# mount -olog-level=debug,log-file=/var/log/glusterfs/glusterfs.log -t
glusterfs ....
On Tue, 2014-02-18 at 17:30 -0300, Targino Silveira wrote:
> Hi all,
>
>
> I am getting a problem and can't understand why, I have created a
> cluster with gluster following the most simple way as explaind in
> QuickStart on the glustger.org.
>
>
> I have created 3 VM in KVM.
>
>
> 2 To host gluster server with one disk image with 1tb.
>
>
> 1 To host gluster client to mounting my volume.
>
>
> I'm using Debian 7 and used apt-get to install Gluster 3.2.7, Server
> and Client.
>
>
> After all finished I could to mount as glusterfs with no problem
> "mount -t glusterfs /mnt/cloudbackup_data/
> vm-gluster-cloudbackup1:/vol1" but I can't to work on the mounted
> directory, if I perform a "ls -lh" it's running forver, and I can't do
> any other operation and VM is blocked.
>
>
> If I try to mount as NFS "mount -t nfs
> vm-gluster-cloudbackup2:/vol1 /mnt/cloudbackup_data/ "I get a "Time
> out" I'm not so much expert in gluster, but I can't see any reason for
> this problem, someone know something like that?
>
>
> Regards,
>
>
>
> Targino Silveira
> +55-85-8626-7297
> www.twitter.com/targinosilveira
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
------------------------------
Message: 16
Date: Wed, 19 Feb 2014 00:08:50 -0300
From: Targino Silveira <targinosilveira@xxxxxxxxx>
To: bernhard glomm <bernhard.glomm@xxxxxxxxxxx>,
gluster-users@xxxxxxxxxxx
Subject: Re: Problems to work with mounted directory
in Gluster 3.2.7
Message-ID:
<CA+PDx1iwT5z22b7ytiwXvDxnc=H6gRZS7YgzyfoH=sxkR3deyw@xxxxxxxxxxxxxx>
Content-Type: text/plain; charset="iso-8859-1"
Hi Bernhard,
2014-02-18 19:45 GMT-03:00 bernhard glomm <bernhard.glomm@xxxxxxxxxxx>:
> hi,
> not very clear to me,
> you run VMs as gluster servers?
>
Right, I am using VM to run glusters servers.
> so 2 of your VMs using each a 1tb brick for what a ...
>
I am making a server for store some old data via FTP, so I have a two
bricks with 1tb each one, in a volume replica 2.
> could you give a
> node1:~ # gluster volume info <volname>
> ...
> node1:~ # gluster volume status <volname>
> ...
>
#First Node
root@vm-gluster-cloudbackup1:~# gluster volume info
Volume Name: vol1
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: vm-gluster-cloudbackup1:/mnt/data1/vol1
Brick2: vm-gluster-cloudbackup2:/mnt/data1/vol1
#Second node
root@vm-gluster-cloudbackup2:~# gluster volume info
Volume Name: vol1
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: vm-gluster-cloudbackup1:/mnt/data1/vol1
Brick2: vm-gluster-cloudbackup2:/mnt/data1/vol1
The volumes are started normaly with no problem, I am trying to get some
thing on the log files, I know that speed may be slow, but it's not a
problem to me at this moment.
Regards,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140219/35a04de2/attachment-0001.html>
------------------------------
Message: 17
Date: Wed, 19 Feb 2014 16:12:44 +0530
From: Vijay Bellur <vbellur@xxxxxxxxxx>
To: "'gluster-devel@xxxxxxxxxx'" <gluster-devel@xxxxxxxxxx>,
gluster-users Discussion List <Gluster-users@xxxxxxxxxxx>
Subject: Agenda for Community meeting today
Message-ID: <53048AA4.2070005@xxxxxxxxxx>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Agenda for the weekly community meeting has been updated at:
http://titanpad.com/gluster-community-meetings
Please update the agenda if you have items for discussion.
Cheers,
Vijay
------------------------------
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users
End of Gluster-users Digest, Vol 70, Issue 18
*********************************************
**
This email and any attachments may contain information that is confidential and/or privileged for the sole use of the intended recipient. Any use, review, disclosure, copying, distribution or reliance by others, and any forwarding of this email or its contents, without the express permission of the sender is strictly prohibited by law. If you are not the intended recipient, please contact the sender immediately, delete the e-mail and destroy all copies.
**
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
