Re: GFS2 hangs after one node going down

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Il 25/03/2013 17:49, Digimer ha scritto:
> On 03/25/2013 08:44 AM, Maurizio Giungato wrote:
>> Il 22/03/2013 16:27, Digimer ha scritto:
>>> On 03/22/2013 11:21 AM, Maurizio Giungato wrote:
>>>> Il 22/03/2013 00:34, Digimer ha scritto:
>>>>> On 03/21/2013 02:09 PM, Maurizio Giungato wrote:
>>>>>> Il 21/03/2013 18:48, Maurizio Giungato ha scritto:
>>>>>>> Il 21/03/2013 18:14, Digimer ha scritto:
>>>>>>>> On 03/21/2013 01:11 PM, Maurizio Giungato wrote:
>>>>>>>>> Hi guys,
>>>>>>>>>
>>>>>>>>> my goal is to create a reliable virtualization environment using
>>>>>>>>> CentOS
>>>>>>>>> 6.4 and KVM, I've three nodes and a clustered GFS2.
>>>>>>>>>
>>>>>>>>> The enviroment is up and working, but I'm worry for the
>>>>>>>>> reliability, if
>>>>>>>>> I turn the network interface down on one node to simulate a crash
>>>>>>>>> (for
>>>>>>>>> example on the node "node6.blade"):
>>>>>>>>>
>>>>>>>>> 1) GFS2 hangs (processes go in D state) until node6.blade get
>>>>>>>>> fenced
>>>>>>>>> 2) not only node6.blade get fenced, but also node5.blade!
>>>>>>>>>
>>>>>>>>> Help me to save my last neurons!
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Maurizio
>>>>>>>>
>>>>>>>> DLM, the distributed lock manager provided by the cluster, is
>>>>>>>> designed to block when a known goes into an unknown state. It does
>>>>>>>> not unblock until that node is confirmed to be fenced. This is by
>>>>>>>> design. GFS2, rgmanager and clustered LVM all use DLM, so they 
>>>>>>>> will
>>>>>>>> all block as well.
>>>>>>>>
>>>>>>>> As for why two nodes get fenced, you will need to share more about
>>>>>>>> your configuration.
>>>>>>>>
>>>>>>> My configuration is very simple I attached cluster.conf and hosts
>>>>>>> files.
>>>>>>> This is the row I added in /etc/fstab:
>>>>>>> /dev/mapper/KVM_IMAGES-VL_KVM_IMAGES /var/lib/libvirt/images gfs2
>>>>>>> defaults,noatime,nodiratime 0 0
>>>>>>>
>>>>>>> I set also fallback_to_local_locking = 0 in lvm.conf (but nothing
>>>>>>> change)
>>>>>>>
>>>>>>> PS: I had two virtualization enviroments working like a charm on
>>>>>>> OCFS2, but since Centos 6.x I'm not able to install it, there is 
>>>>>>> same
>>>>>>> way to achieve the same results with GFS2 (with GFS2 sometime 
>>>>>>> I've a
>>>>>>> crash after only a "service network restart" [I've many interfaces
>>>>>>> then this operation takes more than 10 seconds], with OCFS2 I've
>>>>>>> never
>>>>>>> had this problem.
>>>>>>>
>>>>>>> Thanks
>>>>>> I attached my logs from /var/log/cluster/*
>>>>>
>>>>> The configuration itself seems ok, though I think you can safely take
>>>>> qdisk out to simplify things. That's neither here nor there though.
>>>>>
>>>>> This concerns me:
>>>>>
>>>>> Mar 21 19:00:14 fenced fence lama6.blade dev 0.0 agent
>>>>> fence_bladecenter result: error from agent
>>>>> Mar 21 19:00:14 fenced fence lama6.blade failed
>>>>>
>>>>> How are you triggering the failure(s)? The failed fence would
>>>>> certainly help explain the delays. As I mentioned earlier, DLM is
>>>>> designed to block when a node is in an unknowned state (failed but 
>>>>> not
>>>>> yet successfully fenced).
>>>>>
>>>>> As an aside; I do my HA VMs using clustered LVM LVs as the backing
>>>>> storage behind the VMs. GFS2 is an excellent file system, but it is
>>>>> expensive. Putting your VMs directly on the LV takes them out of the
>>>>> equation
>>>>
>>>> I used 'service network stop' to simulate the failure, the node get
>>>> fenced through fence_bladecenter (BladeCenter HW)
>>>>
>>>> Anyway, I took qdisk out and put GFS2 aside and now I've my VM on LVM
>>>> LVs, I'm trying for many hours to reproduce the issue
>>>>
>>>> - only the node where I execute 'service network stop' get fenced
>>>> - using fallback_to_local_locking = 0 in lvm.conf LVM LVs remain
>>>> writable also while fencing take place
>>>>
>>>> All seems to work like a charm now.
>>>>
>>>> I'd like to understand what was happening. I'll try for same day 
>>>> before
>>>> trusting it.
>>>>
>>>> Thank you so much.
>>>> Maurizio
>>>>
>>>
>>> Testing testing testing. It's good that you plan to test before
>>> trusting. I wish everyone had that philosophy!
>>>
>>> The clustered locking for LVM comes into play for
>>> activating/inactivating, creating, deleting, resizing and so on. It
>>> does not affect what happens in an LV. That's why an LV remains
>>> writeable when a fence is pending. However, I feel this is safe
>>> because rgmanager won't recover a VM on another node until the lost
>>> node is fenced.
>>>
>>> Cheers
>>
>> Thank you very much! The cluster continue working like a charm. Failure
>> after failure I mean :)
>>
>> We are not using rgmanager fault management because doesn't have a check
>> about the memory availability on the destination node, then we prefer to
>> manage this situation with custom script we wrote.
>>
>> last questions:
>> - have you any advice to improve the tollerance against network 
>> failures?
>> - to avoid having a gfs2 only for VM's xml, I've thought to keep them on
>> each node synced with rsync. Any alternatives?
>> - If I want to have only the clustered LVM without no other functions,
>> can you advice about a minimal configuration? (for example I think that
>> rgmanager is not necessary)
>>
>> Thank you in advance
>
> For network redundancy, I use two switches and bonded (mode=1) links 
> with one link going to either switch. This way, losing a NIC or a 
> switch won't break the cluster. Details here:
>
> https://alteeve.ca/w/2-Node_Red_Hat_KVM_Cluster_Tutorial#Network
>
> Using rsync to keep the XML files in sync is fine, if you really don't 
> want to use GFS2.
>
> You do not need rgmanager for clvmd to work. All you need is the base 
> cluster.conf (and working fencing, as you've seen).
>
> If you are over-provisioning VMs and need to worry about memory on 
> target systems, then you might want to take a look at pacemaker. It's 
> in tech-preview currently and will replace rgmanager in rhel7 (well, 
> expected to, nothing is guaranteed 'til release day). Pacemaker is 
> designed, as I understand it, to handle conditions like yours. 
> Further, it is *much* better tested than anything you roll yourself. 
> You can use clvmd with pacemaker by tieing cman into pacemaker.
>
> digime
Perfect, I've the same network configuration, on the other cluster I've 
four switches and I could create two bonds, one is dedicated to corosync 
then I was afraid that a single bond was little ;)

Thank you again



_______________________________________________
CentOS-virt mailing list
CentOS-virt@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos-virt




[Index of Archives]     [CentOS Users]     [Linux Media]     [Asterisk]     [DCCP]     [Netdev]     [X.org]     [Xfree86]     [Linux USB]

  Powered by Linux