Re: gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set

[Steve Dainard <sdainard@xxxxxxxx>]

And I'll mention again, those Debian clients are not running 3.6. As shown in previous email, they are both running:

# apt-cache policy glusterfs-client

glusterfs-client:

  Installed: 3.7.6-2

  Candidate: 3.7.6-2

  Version table:

 *** 3.7.6-2 0

        500 http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/ wheezy/main amd64 Packages

# glusterfs --version

glusterfs 3.7.6 built on Feb  4 2016 06:25:19

Repository revision: git://git.gluster.com/glusterfs.git

Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>

GlusterFS comes with ABSOLUTELY NO WARRANTY.

It is licensed to you under your choice of the GNU Lesser

General Public License, version 3 or any later version (LGPLv3

or later), or the GNU General Public License, version 2 (GPLv2),

in all cases as published by the Free Software Foundation.

On Fri, Mar 4, 2016 at 8:53 AM, Steve Dainard <sdainard@xxxxxxxx> wrote:

Except that even when I unmount gluster from both Debian clients, I'm still getting the same error, and a dump of those clients after unmounting still shows:

glusterd.client1.identifier=10.0.231.10:1023

glusterd.client1.volname=storage

glusterd.client1.max-op-version=30603

glusterd.client1.min-op-version=1

glusterd.client4.identifier=10.0.231.11:1022

glusterd.client4.volname=storage

glusterd.client4.max-op-version=30603

glusterd.client4.min-op-version=1

On Thu, Mar 3, 2016 at 5:28 PM, Atin Mukherjee <atin.mukherjee83@xxxxxxxxx> wrote:

-Atin
Sent from one plus one
On 04-Mar-2016 3:35 am, "Steve Dainard" <sdainard@xxxxxxxx> wrote:
>
> FYI Gluster storage node hostnames are gluster0[1-6].
>
> Full dump attached. I see a few clients not on 30706. Most notably the two debian 7 servers (using packages from gluster.org) seem to be running lower op versions than the centos7 machines (every other client in 10.0.231.0/24 subnet).
>
> glusterd.client1.identifier=10.0.231.10:1023 <-- debian 7, glusterfs 3.7.6 built on Feb  4 2016 06:25:19
> glusterd.client1.volname=storage
> glusterd.client1.max-op-version=30603
This is running with 3.6.3 and that's why volume set fails. Unmount this client and upgrade and remount it back.

> glusterd.client1.min-op-version=1
>
> glusterd.client2.identifier=10.0.231.51:65515 <-- gluster02 (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)
> glusterd.client2.volname=
> glusterd.client2.max-op-version=0
> glusterd.client2.min-op-version=0
>
> glusterd.client3.identifier=10.0.231.54:65521 <-- gluster05 (no actual mounts)
> glusterd.client3.volname=
> glusterd.client3.max-op-version=0
> glusterd.client3.min-op-version=0
>
> glusterd.client4.identifier=10.0.231.11:1022 <--- debian 7, glusterfs 3.7.6 built on Feb  4 2016 06:25:19
> glusterd.client4.volname=storage
> glusterd.client4.max-op-version=30603
> glusterd.client4.min-op-version=1
>
> glusterd.client5.identifier=10.0.231.55:65530 <-- gluster06 (no actual mounts)
> glusterd.client5.volname=
> glusterd.client5.max-op-version=0
> glusterd.client5.min-op-version=0
>
> glusterd.client6.identifier=10.0.231.53:65516 <-- gluster04 (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)
> glusterd.client6.volname=
> glusterd.client6.max-op-version=0
> glusterd.client6.min-op-version=0
>
> glusterd.client7.identifier=10.0.231.50:65529
> glusterd.client7.volname=export-domain-storage.10.0.231.50.mnt-lv-export-domain-storage-export-domain-storage
> glusterd.client7.max-op-version=30706
> glusterd.client7.min-op-version=1
>
> ...
>
> Debian package info:
> apt-cache policy glusterfs-client
> glusterfs-client:
>   Installed: 3.7.6-2
>   Candidate: 3.7.6-2
>   Version table:
>  *** 3.7.6-2 0
>         500 http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/ wheezy/main amd64 Packages
>
>
> Thanks
>
> On Wed, Mar 2, 2016 at 10:29 PM, Gaurav Garg <ggarg@xxxxxxxxxx> wrote:
>>
>> Hi Steve,
>>
>> As atin pointed out to take statedump by running #kill -SIGUSR1 $(pidof glusterd)  command. it will create .dump file in /var/run/gluster/ directory. client-op-version information will be present in dump file.
>>
>> Thanks,
>> ~Gaurav
>>
>> ----- Original Message -----
>> From: "Steve Dainard" <sdainard@xxxxxxxx>
>> To: "Gaurav Garg" <ggarg@xxxxxxxxxx>
>> Cc: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>
>> Sent: Thursday, March 3, 2016 12:07:25 AM
>> Subject: Re: gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set
>>
>> From the the client side logs I can see version info on mount:
>>
>> Final graph:
>> +------------------------------------------------------------------------------+
>>   1: volume storage-client-0
>>   2:     type protocol/client
>>   3:     option clnt-lk-version 1
>>   4:     option volfile-checksum 0
>>   5:     option volfile-key /storage
>>   6:     option client-version 3.7.6
>>   7:     option process-uuid
>> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0
>>   8:     option fops-version 1298437
>>   9:     option ping-timeout 42
>>  10:     option remote-host 10.0.231.50
>>  11:     option remote-subvolume /mnt/raid6-storage/storage
>>  12:     option transport-type socket
>>  13:     option send-gids true
>>  14: end-volume
>>  15:
>>  16: volume storage-client-1
>>  17:     type protocol/client
>>  18:     option clnt-lk-version 1
>>  19:     option volfile-checksum 0
>>  20:     option volfile-key /storage
>>  21:     option client-version 3.7.6
>>  22:     option process-uuid
>> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0
>>  23:     option fops-version 1298437
>>  24:     option ping-timeout 42
>>  25:     option remote-host 10.0.231.51
>>  26:     option remote-subvolume /mnt/raid6-storage/storage
>>  27:     option transport-type socket
>>  28:     option send-gids true
>>  29: end-volume
>>  30:
>>  31: volume storage-client-2
>>  32:     type protocol/client
>>  33:     option clnt-lk-version 1
>>  34:     option volfile-checksum 0
>>  35:     option volfile-key /storage
>>  36:     option client-version 3.7.6
>>  37:     option process-uuid
>> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0
>>  38:     option fops-version 1298437
>>  39:     option ping-timeout 42
>>  40:     option remote-host 10.0.231.52
>>  41:     option remote-subvolume /mnt/raid6-storage/storage
>>  42:     option transport-type socket
>>  43:     option send-gids true
>>  44: end-volume
>>  45:
>>  46: volume storage-client-3
>>  47:     type protocol/client
>>  48:     option clnt-lk-version 1
>>  49:     option volfile-checksum 0
>>  50:     option volfile-key /storage
>>  51:     option client-version 3.7.6
>>  52:     option process-uuid
>> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-3-0-0
>>  53:     option fops-version 1298437
>>  54:     option ping-timeout 42
>>  55:     option remote-host 10.0.231.53
>>  56:     option remote-subvolume /mnt/raid6-storage/storage
>>  57:     option transport-type socket
>>  58:     option send-gids true
>>  59: end-volume
>>  60:
>>  61: volume storage-client-4
>>  62:     type protocol/client
>>  63:     option ping-timeout 42
>>  64:     option remote-host 10.0.231.54
>>  65:     option remote-subvolume /mnt/raid6-storage/storage
>>  66:     option transport-type socket
>>  67:     option send-gids true
>>  68: end-volume
>>  69:
>>  70: volume storage-client-5
>>  71:     type protocol/client
>>  72:     option ping-timeout 42
>>  73:     option remote-host 10.0.231.55
>>  74:     option remote-subvolume /mnt/raid6-storage/storage
>>  75:     option transport-type socket
>>  76:     option send-gids true
>>  77: end-volume
>>  78:
>>  79: volume storage-dht
>>  80:     type cluster/distribute
>>  81:     subvolumes storage-client-0 storage-client-1 storage-client-2
>> storage-client-3 storage-client-4 storage-client-5
>>  82: end-volume
>>
>>
>> But not the client op-version, how can I retrieve this info?
>>
>> Thanks
>>
>> On Tue, Mar 1, 2016 at 10:19 PM, Gaurav Garg <ggarg@xxxxxxxxxx> wrote:
>>
>> > Hi Steve,
>> >
>> > Which version you have upgraded client, could you tell us client
>> > op-version after upgrade ?
>> >
>> >
>> > have you upgraded all of your clients ?
>> >
>> >
>> > Thanks,
>> > Gaurav
>> >
>> >
>> > ----- Original Message -----
>> > From: "Steve Dainard" <sdainard@xxxxxxxx>
>> > To: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>
>> > Sent: Wednesday, March 2, 2016 1:10:27 AM
>> > Subject: gluster 3.7.6 volume set: failed: One or more
>> > connected clients cannot support the feature being set
>> >
>> > Gluster 3.7.6
>> > 'storage' is a distributed volume
>> >
>> > # gluster volume set storage rebal-throttle lazy
>> > volume set: failed: One or more connected clients cannot support the
>> > feature being set. These clients need to be upgraded or disconnected before
>> > running this command again
>> >
>> > I found a client connected using version 3.6.7 so I upgraded &
>> > umount/mount the gluster volume on the client but I'm still getting this
>> > error.
>> >
>> > I've run grep "accepted client from" /var/log/glusterfs/bricks/* | grep -v
>> > 3.7.6 and I get a few returns from the client above, all dated last week.
>> >
>> > I've run 'gluster volume status storage clients' and checked the connected
>> > clients manually, they're all running 3.7.6.
>> >
>> > /var/log/gluster/etc-glusterfs-glusterd.vol.log:
>> > [2016-03-01 19:23:20.180821] E [MSGID: 106022]
>> > [glusterd-utils.c:10154:glusterd_check_client_op_version_support]
>> > 0-management: One or more c
>> > lients don't support the required op-version
>> > [2016-03-01 19:23:20.180853] E [MSGID: 106301]
>> > [glusterd-syncop.c:1274:gd_stage_op_phase] 0-management: Staging of
>> > operation 'Volume Set' fa
>> > iled on localhost : One or more connected clients cannot support the
>> > feature being set. These clients need to be upgraded or disconnected be
>> > fore running this command again
>> >
>> > Also tried setting the diagnostics.brick-log-level logging level and got
>> > the same error.
>> >
>> > /var/lib/glusterd/vols/storage/info:
>> > type=0
>> > count=6
>> > status=1
>> > sub_count=0
>> > stripe_count=1
>> > replica_count=1
>> > disperse_count=0
>> > redundancy_count=0
>> > version=26
>> > transport-type=0
>> > volume-id=26d355cb-c486-481f-ac16-e25390e73775
>> > username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c
>> > password=
>> > op-version=3
>> > client-op-version=3
>> > quota-version=1
>> > parent_volname=N/A
>> > restored_from_snap=00000000-0000-0000-0000-000000000000
>> > snap-max-hard-limit=256
>> > features.quota-deem-statfs=on
>> > features.inode-quota=on
>> > diagnostics.brick-log-level=WARNING
>> > features.quota=on
>> > performance.readdir-ahead=on
>> > performance.cache-size=1GB
>> > performance.stat-prefetch=on
>> > brick-0=10.0.231.50:-mnt-raid6-storage-storage
>> > brick-1=10.0.231.51:-mnt-raid6-storage-storage
>> > brick-2=10.0.231.52:-mnt-raid6-storage-storage
>> > brick-3=10.0.231.53:-mnt-raid6-storage-storage
>> > brick-4=10.0.231.54:-mnt-raid6-storage-storage
>> > brick-5=10.0.231.55:-mnt-raid6-storage-storage
>> >
>> >
>> > _______________________________________________
>> > Gluster-users mailing list
>> > Gluster-users@xxxxxxxxxxx
>> > http://www.gluster.org/mailman/listinfo/gluster-users
>> >
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx
> http://www.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users