Thanks, I'll do that. Is it possible/likely that this lower op-version is also causing the issue I posted on gluster-users earlier: 0-rpcsvc: rpc actor failed to complete successfully
Any pointers on that would be greatly appreciated, since we've had multiple occurences of this since Sunday, three today only.
Thanks,
On 10 June 2015 at 14:42, Atin Mukherjee <amukherj@xxxxxxxxxx> wrote:
On 06/10/2015 05:32 PM, Tiemen Ruiten wrote:
> Hello Atin,
>
> We are running 3.7.0 on our storage nodes and suffer from the same issue.
> Is it safe to perform the same command or should we first upgrade to 3.7.1?
You should upgrade to 3.7.1
~Atin>
> On 10 June 2015 at 13:45, Atin Mukherjee <amukherj@xxxxxxxxxx> wrote:
>
>>
>>
>> On 06/10/2015 02:58 PM, Sergio Traldi wrote:
>>> On 06/10/2015 10:27 AM, Krishnan Parthasarathi wrote:
>>>>> Hi all,
>>>>> I two servers with 3.7.1 and have the same problem of this issue:
>>>>> http://comments.gmane.org/gmane.comp.file-systems.gluster.user/20693
>>>>>
>>>>> My servers packages:
>>>>> # rpm -qa | grep gluster | sort
>>>>> glusterfs-3.7.1-1.el6.x86_64
>>>>> glusterfs-api-3.7.1-1.el6.x86_64
>>>>> glusterfs-cli-3.7.1-1.el6.x86_64
>>>>> glusterfs-client-xlators-3.7.1-1.el6.x86_64
>>>>> glusterfs-fuse-3.7.1-1.el6.x86_64
>>>>> glusterfs-geo-replication-3.7.1-1.el6.x86_64
>>>>> glusterfs-libs-3.7.1-1.el6.x86_64
>>>>> glusterfs-server-3.7.1-1.el6.x86_64
>>>>>
>>>>> Command:
>>>>> # gluster volume status
>>>>> Another transaction is in progress. Please try again after sometime.
>> The problem is although you are running 3.7.1 binaries the cluster
>> op-version is set to 30501, because of glusterd still goes for acquiring
>> cluster lock instead of volume wise lock for every request. Command log
>> history indicates glusterD is getting multiple volume's status requests
>> and because of it fails to acquire cluster lock. Could you bump up your
>> cluster's op-version by the following command and recheck?
>>
>> gluster volume set all cluster.op-version 30701
>>
>> ~Atin
>>>>>
>>>>>
>>>>> In /var/log/gluster/etc-glusterfs-glusterd.vol.log I found:
>>>>>
>>>>> [2015-06-09 16:12:38.949842] E [glusterd-utils.c:164:glusterd_lock]
>>>>> 0-management: Unable to get lock for uuid:
>>>>> 99a41a2a-2ce5-461c-aec0-510bd5b37bf2, lock held by:
>>>>> 04a7d2bb-bdd9-4e0d-b460-87ad4adbe12c
>>>>> [2015-06-09 16:12:38.949864] E
>>>>> [glusterd-syncop.c:1766:gd_sync_task_begin]
>>>>> 0-management: Unable to acquire lock
>>>>>
>>>>> I check the files:
>>>>> From server 1:
>>>>> # cat /var/lib/glusterd/peers/04a7d2bb-bdd9-4e0d-b460-87ad4adbe12c
>>>>> uuid=04a7d2bb-bdd9-4e0d-b460-87ad4adbe12c
>>>>> state=3
>>>>> hostname1=192.168.61.101
>>>>>
>>>>> From server 2:
>>>>> # cat /var/lib/glusterd/peers/99a41a2a-2ce5-461c-aec0-510bd5b37bf2
>>>>> uuid=99a41a2a-2ce5-461c-aec0-510bd5b37bf2
>>>>> state=3
>>>>> hostname1=192.168.61.100
>>>> Could you attach the complete glusterd log file and cmd-history.log
>>>> file under /var/log/glusterfs directory? Could you provide a more
>>>> detailed listing of things you did before hitting this issue?
>>> Hi Krishnan,
>>> thanks to a quick answer.
>>> In attach you can found the two log you request:
>>> cmd_history.log
>>> etc-glusterfs-glusterd.vol.log
>>>
>>> We use the gluster volume as openstack nova, glance, cinder backend.
>>>
>>> The volume is configured using 2 bricks mounted by an iscsi device:
>>> [root@cld-stg-01 glusterfs]# gluster volume info volume-nova-prod
>>> Volume Name: volume-nova-prod
>>> Type: Distribute
>>> Volume ID: 4bbef4c8-0441-4e81-a2c5-559401adadc0
>>> Status: Started
>>> Number of Bricks: 2
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: 192.168.61.100:/brickOpenstack/nova-prod/mpathb
>>> Brick2: 192.168.61.101:/brickOpenstack/nova-prod/mpathb
>>> Options Reconfigured:
>>> storage.owner-gid: 162
>>> storage.owner-uid: 162
>>>
>>> Last week we update openstack from havana to icehouse and we rename the
>>> storage hosts but we didn't change the IP.
>>> All volume have been created using ip addresses.
>>>
>>> So last week we stop all services (openstack gluster and also iscsi).
>>> We change the name in DNS of private ip of the 2 nics.
>>> We reboot the storage servers
>>> We start agian iscsi, multipath, glusterd process.
>>> We have to stop and start the volumes, but after that everything works
>>> fine.
>>> Now we don't observe any other problems except this.
>>>
>>> We have a nagios probe which check the volume status each 5 minutes to
>>> ensure all gluster process is working fine and so we find this problem I
>>> post.
>>>
>>> Cheer
>>> Sergio
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users@xxxxxxxxxxx
>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>> --
>> ~Atin
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users@xxxxxxxxxxx
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
>
>
>
--
Tiemen Ruiten
Systems Engineer
R&D Media
Systems Engineer
R&D Media
_______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-devel
