Yep, unmount was successful. I didn't see any useful logging to say otherwise, but I have reduced logging on my clients because the logs grow way too quickly otherwise.
I had a maintenance window this morning, and worked around this by unmounting and rebooting the two debian hosts. I was then able to set the lazy rebalance without restarting any gluster services so I'm ticking along now.
Maybe the client state info is cached for some period of time after the client disconnects.
On Fri, Mar 4, 2016 at 11:20 PM, Atin Mukherjee <atin.mukherjee83@xxxxxxxxx> wrote:
This is really weird, if you unmount the clients then statedump shouldn't be showing the same. Was unmount successful? Do you see any related error log entry in mount & glusterd log?
-Atin
Sent from one plus oneOn 04-Mar-2016 10:23 pm, "Steve Dainard" <sdainard@xxxxxxxx> wrote:Except that even when I unmount gluster from both Debian clients, I'm still getting the same error, and a dump of those clients after unmounting still shows:glusterd.client1.identifier=10.0.231.10:1023glusterd.client1.volname=storageglusterd.client1.max-op-version=30603glusterd.client1.min-op-version=1glusterd.client4.identifier=10.0.231.11:1022glusterd.client4.volname=storageglusterd.client4.max-op-version=30603glusterd.client4.min-op-version=1On Thu, Mar 3, 2016 at 5:28 PM, Atin Mukherjee <atin.mukherjee83@xxxxxxxxx> wrote:-Atin
Sent from one plus one
On 04-Mar-2016 3:35 am, "Steve Dainard" <sdainard@xxxxxxxx> wrote:
>
> FYI Gluster storage node hostnames are gluster0[1-6].
>
> Full dump attached. I see a few clients not on 30706. Most notably the two debian 7 servers (using packages from gluster.org) seem to be running lower op versions than the centos7 machines (every other client in 10.0.231.0/24 subnet).
>
> glusterd.client1.identifier=10.0.231.10:1023 <-- debian 7, glusterfs 3.7.6 built on Feb 4 2016 06:25:19
> glusterd.client1.volname=storage
> glusterd.client1.max-op-version=30603
This is running with 3.6.3 and that's why volume set fails. Unmount this client and upgrade and remount it back.
> glusterd.client1.min-op-version=1
>
> glusterd.client2.identifier=10.0.231.51:65515 <-- gluster02 (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)
> glusterd.client2.volname=
> glusterd.client2.max-op-version=0
> glusterd.client2.min-op-version=0
>
> glusterd.client3.identifier=10.0.231.54:65521 <-- gluster05 (no actual mounts)
> glusterd.client3.volname=
> glusterd.client3.max-op-version=0
> glusterd.client3.min-op-version=0
>
> glusterd.client4.identifier=10.0.231.11:1022 <--- debian 7, glusterfs 3.7.6 built on Feb 4 2016 06:25:19
> glusterd.client4.volname=storage
> glusterd.client4.max-op-version=30603
> glusterd.client4.min-op-version=1
>
> glusterd.client5.identifier=10.0.231.55:65530 <-- gluster06 (no actual mounts)
> glusterd.client5.volname=
> glusterd.client5.max-op-version=0
> glusterd.client5.min-op-version=0
>
> glusterd.client6.identifier=10.0.231.53:65516 <-- gluster04 (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)
> glusterd.client6.volname=
> glusterd.client6.max-op-version=0
> glusterd.client6.min-op-version=0
>
> glusterd.client7.identifier=10.0.231.50:65529
> glusterd.client7.volname=export-domain-storage.10.0.231.50.mnt-lv-export-domain-storage-export-domain-storage
> glusterd.client7.max-op-version=30706
> glusterd.client7.min-op-version=1
>
> ...
>
> Debian package info:
> apt-cache policy glusterfs-client
> glusterfs-client:
> Installed: 3.7.6-2
> Candidate: 3.7.6-2
> Version table:
> *** 3.7.6-2 0
> 500 http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/ wheezy/main amd64 Packages
>
>
> Thanks
>
> On Wed, Mar 2, 2016 at 10:29 PM, Gaurav Garg <ggarg@xxxxxxxxxx> wrote:
>>
>> Hi Steve,
>>
>> As atin pointed out to take statedump by running #kill -SIGUSR1 $(pidof glusterd) command. it will create .dump file in /var/run/gluster/ directory. client-op-version information will be present in dump file.
>>
>> Thanks,
>> ~Gaurav
>>
>> ----- Original Message -----
>> From: "Steve Dainard" <sdainard@xxxxxxxx>
>> To: "Gaurav Garg" <ggarg@xxxxxxxxxx>
>> Cc: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>
>> Sent: Thursday, March 3, 2016 12:07:25 AM
>> Subject: Re: gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set
>>
>> From the the client side logs I can see version info on mount:
>>
>> Final graph:
>> +------------------------------------------------------------------------------+
>> 1: volume storage-client-0
>> 2: type protocol/client
>> 3: option clnt-lk-version 1
>> 4: option volfile-checksum 0
>> 5: option volfile-key /storage
>> 6: option client-version 3.7.6
>> 7: option process-uuid
>> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0
>> 8: option fops-version 1298437
>> 9: option ping-timeout 42
>> 10: option remote-host 10.0.231.50
>> 11: option remote-subvolume /mnt/raid6-storage/storage
>> 12: option transport-type socket
>> 13: option send-gids true
>> 14: end-volume
>> 15:
>> 16: volume storage-client-1
>> 17: type protocol/client
>> 18: option clnt-lk-version 1
>> 19: option volfile-checksum 0
>> 20: option volfile-key /storage
>> 21: option client-version 3.7.6
>> 22: option process-uuid
>> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0
>> 23: option fops-version 1298437
>> 24: option ping-timeout 42
>> 25: option remote-host 10.0.231.51
>> 26: option remote-subvolume /mnt/raid6-storage/storage
>> 27: option transport-type socket
>> 28: option send-gids true
>> 29: end-volume
>> 30:
>> 31: volume storage-client-2
>> 32: type protocol/client
>> 33: option clnt-lk-version 1
>> 34: option volfile-checksum 0
>> 35: option volfile-key /storage
>> 36: option client-version 3.7.6
>> 37: option process-uuid
>> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0
>> 38: option fops-version 1298437
>> 39: option ping-timeout 42
>> 40: option remote-host 10.0.231.52
>> 41: option remote-subvolume /mnt/raid6-storage/storage
>> 42: option transport-type socket
>> 43: option send-gids true
>> 44: end-volume
>> 45:
>> 46: volume storage-client-3
>> 47: type protocol/client
>> 48: option clnt-lk-version 1
>> 49: option volfile-checksum 0
>> 50: option volfile-key /storage
>> 51: option client-version 3.7.6
>> 52: option process-uuid
>> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-3-0-0
>> 53: option fops-version 1298437
>> 54: option ping-timeout 42
>> 55: option remote-host 10.0.231.53
>> 56: option remote-subvolume /mnt/raid6-storage/storage
>> 57: option transport-type socket
>> 58: option send-gids true
>> 59: end-volume
>> 60:
>> 61: volume storage-client-4
>> 62: type protocol/client
>> 63: option ping-timeout 42
>> 64: option remote-host 10.0.231.54
>> 65: option remote-subvolume /mnt/raid6-storage/storage
>> 66: option transport-type socket
>> 67: option send-gids true
>> 68: end-volume
>> 69:
>> 70: volume storage-client-5
>> 71: type protocol/client
>> 72: option ping-timeout 42
>> 73: option remote-host 10.0.231.55
>> 74: option remote-subvolume /mnt/raid6-storage/storage
>> 75: option transport-type socket
>> 76: option send-gids true
>> 77: end-volume
>> 78:
>> 79: volume storage-dht
>> 80: type cluster/distribute
>> 81: subvolumes storage-client-0 storage-client-1 storage-client-2
>> storage-client-3 storage-client-4 storage-client-5
>> 82: end-volume
>>
>>
>> But not the client op-version, how can I retrieve this info?
>>
>> Thanks
>>
>> On Tue, Mar 1, 2016 at 10:19 PM, Gaurav Garg <ggarg@xxxxxxxxxx> wrote:
>>
>> > Hi Steve,
>> >
>> > Which version you have upgraded client, could you tell us client
>> > op-version after upgrade ?
>> >
>> >
>> > have you upgraded all of your clients ?
>> >
>> >
>> > Thanks,
>> > Gaurav
>> >
>> >
>> > ----- Original Message -----
>> > From: "Steve Dainard" <sdainard@xxxxxxxx>
>> > To: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>
>> > Sent: Wednesday, March 2, 2016 1:10:27 AM
>> > Subject: gluster 3.7.6 volume set: failed: One or more
>> > connected clients cannot support the feature being set
>> >
>> > Gluster 3.7.6
>> > 'storage' is a distributed volume
>> >
>> > # gluster volume set storage rebal-throttle lazy
>> > volume set: failed: One or more connected clients cannot support the
>> > feature being set. These clients need to be upgraded or disconnected before
>> > running this command again
>> >
>> > I found a client connected using version 3.6.7 so I upgraded &
>> > umount/mount the gluster volume on the client but I'm still getting this
>> > error.
>> >
>> > I've run grep "accepted client from" /var/log/glusterfs/bricks/* | grep -v
>> > 3.7.6 and I get a few returns from the client above, all dated last week.
>> >
>> > I've run 'gluster volume status storage clients' and checked the connected
>> > clients manually, they're all running 3.7.6.
>> >
>> > /var/log/gluster/etc-glusterfs-glusterd.vol.log:
>> > [2016-03-01 19:23:20.180821] E [MSGID: 106022]
>> > [glusterd-utils.c:10154:glusterd_check_client_op_version_support]
>> > 0-management: One or more c
>> > lients don't support the required op-version
>> > [2016-03-01 19:23:20.180853] E [MSGID: 106301]
>> > [glusterd-syncop.c:1274:gd_stage_op_phase] 0-management: Staging of
>> > operation 'Volume Set' fa
>> > iled on localhost : One or more connected clients cannot support the
>> > feature being set. These clients need to be upgraded or disconnected be
>> > fore running this command again
>> >
>> > Also tried setting the diagnostics.brick-log-level logging level and got
>> > the same error.
>> >
>> > /var/lib/glusterd/vols/storage/info:
>> > type=0
>> > count=6
>> > status=1
>> > sub_count=0
>> > stripe_count=1
>> > replica_count=1
>> > disperse_count=0
>> > redundancy_count=0
>> > version=26
>> > transport-type=0
>> > volume-id=26d355cb-c486-481f-ac16-e25390e73775
>> > username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c
>> > password=
>> > op-version=3
>> > client-op-version=3
>> > quota-version=1
>> > parent_volname=N/A
>> > restored_from_snap=00000000-0000-0000-0000-000000000000
>> > snap-max-hard-limit=256
>> > features.quota-deem-statfs=on
>> > features.inode-quota=on
>> > diagnostics.brick-log-level=WARNING
>> > features.quota=on
>> > performance.readdir-ahead=on
>> > performance.cache-size=1GB
>> > performance.stat-prefetch=on
>> > brick-0=10.0.231.50:-mnt-raid6-storage-storage
>> > brick-1=10.0.231.51:-mnt-raid6-storage-storage
>> > brick-2=10.0.231.52:-mnt-raid6-storage-storage
>> > brick-3=10.0.231.53:-mnt-raid6-storage-storage
>> > brick-4=10.0.231.54:-mnt-raid6-storage-storage
>> > brick-5=10.0.231.55:-mnt-raid6-storage-storage
>> >
>> >
>> > _______________________________________________
>> > Gluster-users mailing list
>> > Gluster-users@xxxxxxxxxxx
>> > http://www.gluster.org/mailman/listinfo/gluster-users
>> >
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx
> http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
