Hi Klaus, Can you please provide us the brick log, so that we can get more details on the issue. If you have any core-dumps, then can you include the stack-trace of that as well ? Thanks & Regards, Sachin Pandit. ----- Original Message ----- From: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx> To: "Klaus Bekk (IKP)" <klaus.bekk@xxxxxxx>, gluster-users@xxxxxxxxxxx, "Vijaikumar Mallikarjuna" <vmallika@xxxxxxxxxx>, "Sachin Pandit" <spandit@xxxxxxxxxx> Cc: "Doris Wochele (IKP)" <doris.wochele@xxxxxxx> Sent: Wednesday, January 28, 2015 12:13:41 PM Subject: Re: Fwd: to mailing list gluster.org Vijaikumar, Sachin will work on this quota issue. Pranith On 01/22/2015 12:46 AM, Pranith Kumar Karampuri wrote: > + quota devs > > Pranith > On 01/07/2015 07:37 PM, Bekk, Klaus (IKP) wrote: >> >>> Hi all, >>> >>> we are running 4 server with Ubuntu 14.04 as glusterfs server with >>> 3.5.2-ubuntu1~trusty1. >>> >> >>> When starting quotas on any of the volumes the brick will be not >>> available after some time. So we have to disable quota on this >>> volume and stop and start this volume. >>> >>> We have the following configuration >>> >>> root@ikpsrv01:/var/log/glusterfs# gluster volume info >>> >>> Volume Name: gltestn >>> >>> Type: Distribute >>> >>> Volume ID: e50bd30d-dbc8-4279-ac9d-2e557328a643 >>> >>> Status: Started >>> >>> Number of Bricks: 1 >>> >>> Transport-type: tcp >>> >>> Bricks: >>> >>> Brick1: ikpsrv01:/gltest2/gl >>> >>> Volume Name: glwww1 >>> >>> Type: Replicate >>> >>> Volume ID: 811e4c2a-4520-4e61-bb84-955afbdb9dff >>> >>> Status: Started >>> >>> Number of Bricks: 1 x 2 = 2 >>> >>> Transport-type: tcp >>> >>> Bricks: >>> >>> Brick1: ikpsrv03:/glwwwn/gl >>> >>> Brick2: ikpsrv01:/glwwwn2/gl >>> >>> Options Reconfigured: >>> >>> features.quota: on >>> >>> Volume Name: corsika >>> >>> Type: Distribute >>> >>> Volume ID: d6e07f3e-4b35-4736-9b71-afc3224f0f29 >>> >>> Status: Started >>> >>> Number of Bricks: 4 >>> >>> Transport-type: tcp >>> >>> Bricks: >>> >>> Brick1: ikpsrv03:/glcors3-1/gl >>> >>> Brick2: ikpsrv03:/glcors3-2/gl >>> >>> Brick3: ikpsrv02:/glcors2-1/gl >>> >>> Brick4: ikpsrv02:/glcors2-2/gl >>> >>> Volume Name: gldata >>> >>> Type: Distribute >>> >>> Volume ID: 82485d03-9ef4-4151-a2cc-881072e871a1 >>> >>> Status: Started >>> >>> Number of Bricks: 2 >>> >>> Transport-type: tcp >>> >>> Bricks: >>> >>> Brick1: ikpsrv02:/gldata2-1/gl >>> >>> Brick2: ikpsrv01:/gldata1-2/gl >>> >>> Options Reconfigured: >>> >>> features.quota: on >>> >>> Volume Name: gltest >>> >>> Type: Distribute >>> >>> Volume ID: 9e69afcc-d421-41c9-824b-7d9a31240072 >>> >>> Status: Started >>> >>> Number of Bricks: 2 >>> >>> Transport-type: tcp >>> >>> Bricks: >>> >>> Brick1: ikpsrv01:/gltest3/gl >>> >>> Brick2: ikpsrv01:/gltest4/gl >>> >>> Options Reconfigured: >>> >>> features.quota: off >>> >>> Volume Name: glusersold >>> >>> Type: Distribute >>> >>> Volume ID: 91a79e1a-4ff3-4191-b0ae-73357105774c >>> >>> Status: Started >>> >>> Number of Bricks: 1 >>> >>> Transport-type: tcp >>> >>> Bricks: >>> >>> Brick1: ikpsrv02:/glusers2-2/gl >>> >>> Volume Name: glusers >>> >>> Type: Distribute >>> >>> Volume ID: f48b1941-9c19-4311-bd9e-497660fbcc80 >>> >>> Status: Started >>> >>> Number of Bricks: 1 >>> >>> Transport-type: tcp >>> >>> Bricks: >>> >>> Brick1: ikpsrv02:/glusers2-1/gl >>> >>> Options Reconfigured: >>> >>> diagnostics.client-log-level: CRITICAL >>> >>> diagnostics.brick-log-level: CRITICAL >>> >>> features.quota: off >>> >>> When starting quotas on any of the volumes the brick will be not >>> available after some time. So we have to disable quota on this >>> volume and stop and start this volume. >>> >>> The command >>> >>> Gluster volume quota gltestn enable >>> >>> gives the following output in quotad.log >>> >>> [2015-01-07 12:01:55.522229] W [glusterfsd.c:1095:cleanup_and_exit] >>> (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fb455bd9fbd] >>> (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7fb455ead182] >>> (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xd5) >>> [0x7fb4569a7265]))) 0-: received signum (15), shutting down >>> >>> [2015-01-07 12:01:56.534931] I [glusterfsd.c:1959:main] >>> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version >>> 3.5.2 (/usr/sbin/glusterfs -s localhost --volfile-id gluster/quotad >>> -p /var/lib/glusterd/quotad/run/quotad.pid -l >>> /var/log/glusterfs/quotad.log -S >>> /var/run/fa869d5552bf340ec7506747e21b2841.socket --xlator-option >>> *replicate*.data-self-heal=off --xlator-option >>> *replicate*.metadata-self-heal=off --xlator-option >>> *replicate*.entry-self-heal=off) >>> >>> [2015-01-07 12:01:56.537061] I [socket.c:3561:socket_init] >>> 0-socket.glusterfsd: SSL support is NOT enabled >>> >>> [2015-01-07 12:01:56.537134] I [socket.c:3576:socket_init] >>> 0-socket.glusterfsd: using system polling thread >>> >>> [2015-01-07 12:01:56.537355] I [socket.c:3561:socket_init] >>> 0-glusterfs: SSL support is NOT enabled >>> >>> [2015-01-07 12:01:56.537383] I [socket.c:3576:socket_init] >>> 0-glusterfs: using system polling thread >>> >>> [2015-01-07 12:01:56.541480] I [graph.c:254:gf_add_cmdline_options] >>> 0-glwww1-replicate-0: adding option 'entry-self-heal' for volume >>> 'glwww1-replicate-0' with value 'off' >>> >>> [2015-01-07 12:01:56.541516] I [graph.c:254:gf_add_cmdline_options] >>> 0-glwww1-replicate-0: adding option 'metadata-self-heal' for volume >>> 'glwww1-replicate-0' with value 'off' >>> >>> [2015-01-07 12:01:56.541536] I [graph.c:254:gf_add_cmdline_options] >>> 0-glwww1-replicate-0: adding option 'data-self-heal' for volume >>> 'glwww1-replicate-0' with value 'off' >>> >>> [2015-01-07 12:01:56.543122] I [socket.c:3561:socket_init] >>> 0-socket.quotad: SSL support is NOT enabled >>> >>> [2015-01-07 12:01:56.543157] I [socket.c:3576:socket_init] >>> 0-socket.quotad: using system polling thread >>> >>> [2015-01-07 12:01:56.543404] I [dht-shared.c:311:dht_init_regex] >>> 0-glwww1: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >>> >>> [2015-01-07 12:01:56.547124] I [socket.c:3561:socket_init] >>> 0-glwww1-client-1: SSL support is NOT enabled >>> >>> [2015-01-07 12:01:56.547158] I [socket.c:3576:socket_init] >>> 0-glwww1-client-1: using system polling thread >>> >>> [2015-01-07 12:01:56.547728] I [socket.c:3561:socket_init] >>> 0-glwww1-client-0: SSL support is NOT enabled >>> >>> [2015-01-07 12:01:56.547754] I [socket.c:3576:socket_init] >>> 0-glwww1-client-0: using system polling thread >>> >>> [2015-01-07 12:01:56.547820] I [dht-shared.c:311:dht_init_regex] >>> 0-gltestn: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >>> >>> [2015-01-07 12:01:56.549471] I [socket.c:3561:socket_init] >>> 0-gltestn-client-0: SSL support is NOT enabled >>> >>> [2015-01-07 12:01:56.549501] I [socket.c:3576:socket_init] >>> 0-gltestn-client-0: using system polling thread >>> >>> [2015-01-07 12:01:56.549565] I [dht-shared.c:311:dht_init_regex] >>> 0-gldata: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >>> >>> [2015-01-07 12:01:56.551155] I [socket.c:3561:socket_init] >>> 0-gldata-client-1: SSL support is NOT enabled >>> >>> [2015-01-07 12:01:56.551184] I [socket.c:3576:socket_init] >>> 0-gldata-client-1: using system polling thread >>> >>> [2015-01-07 12:01:56.551744] I [socket.c:3561:socket_init] >>> 0-gldata-client-0: SSL support is NOT enabled >>> >>> [2015-01-07 12:01:56.551770] I [socket.c:3576:socket_init] >>> 0-gldata-client-0: using system polling thread >>> >>> [2015-01-07 12:01:56.551790] W [graph.c:329:_log_if_unknown_option] >>> 0-quotad: option 'rpc-auth.auth-glusterfs' is not recognized >>> >>> [2015-01-07 12:01:56.551819] W [graph.c:329:_log_if_unknown_option] >>> 0-quotad: option 'rpc-auth.auth-unix' is not recognized >>> >>> [2015-01-07 12:01:56.551838] W [graph.c:329:_log_if_unknown_option] >>> 0-quotad: option 'rpc-auth.auth-null' is not recognized >>> >>> [2015-01-07 12:01:56.551854] W [graph.c:329:_log_if_unknown_option] >>> 0-quotad: option 'glwww1.volume-id' is not recognized >>> >>> [2015-01-07 12:01:56.551870] W [graph.c:329:_log_if_unknown_option] >>> 0-quotad: option 'gltestn.volume-id' is not recognized >>> >>> [2015-01-07 12:01:56.551885] W [graph.c:329:_log_if_unknown_option] >>> 0-quotad: option 'gldata.volume-id' is not recognized >>> >>> [2015-01-07 12:01:56.551922] I [client.c:2294:notify] >>> 0-glwww1-client-0: parent translators are ready, attempting connect >>> on transport >>> >>> [2015-01-07 12:01:56.552579] I [client.c:2294:notify] >>> 0-glwww1-client-1: parent translators are ready, attempting connect >>> on transport >>> >>> [2015-01-07 12:01:56.553179] I [client.c:2294:notify] >>> 0-gltestn-client-0: parent translators are ready, attempting connect >>> on transport >>> >>> [2015-01-07 12:01:56.553757] I [client.c:2294:notify] >>> 0-gldata-client-0: parent translators are ready, attempting connect >>> on transport >>> >>> [2015-01-07 12:01:56.554372] I [client.c:2294:notify] >>> 0-gldata-client-1: parent translators are ready, attempting connect >>> on transport >>> >>> Final graph: >>> >>> +------------------------------------------------------------------------------+ >>> >>> 1: volume glwww1-client-0 >>> >>> 2: type protocol/client >>> >>> 3: option remote-host ikpsrv03 >>> >>> 4: option remote-subvolume /glwwwn/gl >>> >>> 5: option transport-type socket >>> >>> 6: option username 7c5931b4-d41e-40f7-978e-4b5a40f49481 >>> >>> 7: option password 6ab1661b-4752-42a4-80cc-ca0bd0167a9f >>> >>> 8: end-volume >>> >>> 9: >>> >>> 10: volume glwww1-client-1 >>> >>> 11: type protocol/client >>> >>> 12: option remote-host ikpsrv01 >>> >>> 13: option remote-subvolume /glwwwn2/gl >>> >>> 14: option transport-type socket >>> >>> 15: option username 7c5931b4-d41e-40f7-978e-4b5a40f49481 >>> >>> 16: option password 6ab1661b-4752-42a4-80cc-ca0bd0167a9f >>> >>> 17: end-volume >>> >>> 18: >>> >>> 19: volume glwww1-replicate-0 >>> >>> 20: type cluster/replicate >>> >>> 21: option data-self-heal off >>> >>> 22: option metadata-self-heal off >>> >>> 23: option entry-self-heal off >>> >>> 24: subvolumes glwww1-client-0 glwww1-client-1 >>> >>> 25: end-volume >>> >>> 26: >>> >>> 27: volume glwww1 >>> >>> 28: type cluster/distribute >>> >>> 29: subvolumes glwww1-replicate-0 >>> >>> 30: end-volume >>> >>> 31: >>> >>> 32: volume gltestn-client-0 >>> >>> 33: type protocol/client >>> >>> 34: option remote-host ikpsrv01 >>> >>> 35: option remote-subvolume /gltest2/gl >>> >>> 36: option transport-type socket >>> >>> 37: option username 76c8bf85-a5c5-445d-8e04-e596b6eb173b >>> >>> 38: option password ceec81fd-5afd-4d6b-a413-00b2688f4c62 >>> >>> 39: end-volume >>> >>> 40: >>> >>> 41: volume gltestn >>> >>> 42: type cluster/distribute >>> >>> 43: subvolumes gltestn-client-0 >>> >>> 44: end-volume >>> >>> 45: >>> >>> 46: volume gldata-client-0 >>> >>> 47: type protocol/client >>> >>> 48: option remote-host ikpsrv02 >>> >>> 49: option remote-subvolume /gldata2-1/gl >>> >>> 50: option transport-type socket >>> >>> 51: option username e9dcdbec-1110-4ab2-b667-63625126b246 >>> >>> 52: option password 05a9e37d-67a0-476b-88df-93d612a4e08e >>> >>> 53: end-volume >>> >>> 54: >>> >>> 55: volume gldata-client-1 >>> >>> 56: type protocol/client >>> >>> 57: option remote-host ikpsrv01 >>> >>> 58: option remote-subvolume /gldata1-2/gl >>> >>> 59: option transport-type socket >>> >>> 60: option username e9dcdbec-1110-4ab2-b667-63625126b246 >>> >>> 61: option password 05a9e37d-67a0-476b-88df-93d612a4e08e >>> >>> 62: end-volume >>> >>> 63: >>> >>> 64: volume gldata >>> >>> 65: type cluster/distribute >>> >>> 66: subvolumes gldata-client-0 gldata-client-1 >>> >>> 67: end-volume >>> >>> 68: >>> >>> 69: volume quotad >>> >>> 70: type features/quotad >>> >>> 71: option rpc-auth.auth-glusterfs on >>> >>> 72: option rpc-auth.auth-unix on >>> >>> 73: option rpc-auth.auth-null on >>> >>> 74: option transport.socket.listen-path /tmp/quotad.socket >>> >>> 75: option transport-type socket >>> >>> 76: option transport.address-family unix >>> >>> 77: option glwww1.volume-id glwww1 >>> >>> 78: option gltestn.volume-id gltestn >>> >>> 79: option gldata.volume-id gldata >>> >>> 80: subvolumes glwww1 gltestn gldata >>> >>> 81: end-volume >>> >>> 82: >>> >>> +------------------------------------------------------------------------------+ >>> >>> [2015-01-07 12:01:56.555618] I [rpc-clnt.c:1729:rpc_clnt_reconfig] >>> 0-gldata-client-0: changing port to 49160 (from 0) >>> >>> [2015-01-07 12:01:56.556529] I [rpc-clnt.c:1729:rpc_clnt_reconfig] >>> 0-glwww1-client-1: changing port to 49165 (from 0) >>> >>> [2015-01-07 12:01:56.556597] I [rpc-clnt.c:1729:rpc_clnt_reconfig] >>> 0-gltestn-client-0: changing port to 49166 (from 0) >>> >>> [2015-01-07 12:01:56.556643] I [rpc-clnt.c:1729:rpc_clnt_reconfig] >>> 0-gldata-client-1: changing port to 49178 (from 0) >>> >>> [2015-01-07 12:01:56.558382] W >>> [client-callback.c:31:client_cbk_fetchspec] 0-glwww1-client-0: this >>> function should not be called >>> >>> [2015-01-07 12:01:56.558484] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] >>> 0-mgmt: Volume file changed >>> >>> [2015-01-07 12:01:56.558601] I >>> [client-handshake.c:1677:select_server_supported_programs] >>> 0-gldata-client-0: Using Program GlusterFS 3.3, Num (1298437), >>> Version (330) >>> >>> [2015-01-07 12:01:56.559594] I >>> [client-handshake.c:1677:select_server_supported_programs] >>> 0-gltestn-client-0: Using Program GlusterFS 3.3, Num (1298437), >>> Version (330) >>> >>> [2015-01-07 12:01:56.559686] I >>> [client-handshake.c:1677:select_server_supported_programs] >>> 0-glwww1-client-1: Using Program GlusterFS 3.3, Num (1298437), >>> Version (330) >>> >>> [2015-01-07 12:01:56.559768] I >>> [client-handshake.c:1677:select_server_supported_programs] >>> 0-gldata-client-1: Using Program GlusterFS 3.3, Num (1298437), >>> Version (330) >>> >>> [2015-01-07 12:01:56.559852] I >>> [client-handshake.c:1462:client_setvolume_cbk] 0-gldata-client-0: >>> Connected to 141.52.67.86:49160, attached to remote volume >>> '/gldata2-1/gl'. >>> >>> [2015-01-07 12:01:56.559873] I >>> [client-handshake.c:1474:client_setvolume_cbk] 0-gldata-client-0: >>> Server and Client lk-version numbers are not same, reopening the fds >>> >>> [2015-01-07 12:01:56.559996] I >>> [client-handshake.c:1462:client_setvolume_cbk] 0-gltestn-client-0: >>> Connected to 141.52.67.85:49166, attached to remote volume >>> '/gltest2/gl'. >>> >>> [2015-01-07 12:01:56.560017] I >>> [client-handshake.c:1474:client_setvolume_cbk] 0-gltestn-client-0: >>> Server and Client lk-version numbers are not same, reopening the fds >>> >>> [2015-01-07 12:01:56.560089] I >>> [client-handshake.c:450:client_set_lk_version_cbk] >>> 0-gldata-client-0: Server lk version = 1 >>> >>> [2015-01-07 12:01:56.560139] I >>> [client-handshake.c:1462:client_setvolume_cbk] 0-glwww1-client-1: >>> Connected to 141.52.67.85:49165, attached to remote volume >>> '/glwwwn2/gl'. >>> >>> [2015-01-07 12:01:56.560158] I >>> [client-handshake.c:1474:client_setvolume_cbk] 0-glwww1-client-1: >>> Server and Client lk-version numbers are not same, reopening the fds >>> >>> [2015-01-07 12:01:56.560204] I [afr-common.c:4131:afr_notify] >>> 0-glwww1-replicate-0: Subvolume 'glwww1-client-1' came back up; >>> going online. >>> >>> [2015-01-07 12:01:56.560262] I >>> [client-handshake.c:1462:client_setvolume_cbk] 0-gldata-client-1: >>> Connected to 141.52.67.85:49178, attached to remote volume >>> '/gldata1-2/gl'. >>> >>> [2015-01-07 12:01:56.560301] I >>> [client-handshake.c:1474:client_setvolume_cbk] 0-gldata-client-1: >>> Server and Client lk-version numbers are not same, reopening the fds >>> >>> [2015-01-07 12:01:56.560385] I >>> [client-handshake.c:450:client_set_lk_version_cbk] >>> 0-gltestn-client-0: Server lk version = 1 >>> >>> [2015-01-07 12:01:56.560467] I >>> [client-handshake.c:450:client_set_lk_version_cbk] >>> 0-glwww1-client-1: Server lk version = 1 >>> >>> [2015-01-07 12:01:56.560546] I >>> [client-handshake.c:450:client_set_lk_version_cbk] >>> 0-gldata-client-1: Server lk version = 1 >>> >>> [2015-01-07 12:01:57.255911] I >>> [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk] 0-glusterfs: No change in >>> volfile, continuing >>> >>> [2015-01-07 12:01:57.557002] I [rpc-clnt.c:1729:rpc_clnt_reconfig] >>> 0-glwww1-client-0: changing port to 49153 (from 0) >>> >>> [2015-01-07 12:01:57.557924] I >>> [client-handshake.c:1677:select_server_supported_programs] >>> 0-glwww1-client-0: Using Program GlusterFS 3.3, Num (1298437), >>> Version (330) >>> >>> [2015-01-07 12:01:57.558360] I >>> [client-handshake.c:1462:client_setvolume_cbk] 0-glwww1-client-0: >>> Connected to 141.52.67.96:49153, attached to remote volume '/glwwwn/gl'. >>> >>> [2015-01-07 12:01:57.558405] I >>> [client-handshake.c:1474:client_setvolume_cbk] 0-glwww1-client-0: >>> Server and Client lk-version numbers are not same, reopening the fds >>> >>> [2015-01-07 12:01:57.558706] I >>> [client-handshake.c:450:client_set_lk_version_cbk] >>> 0-glwww1-client-0: Server lk version = 1 >>> >>> what means >>> >>> 2015-01-07 12:01:56.559594] I >>> [client-handshake.c:1677:select_server_supported_programs] >>> 0-gltestn-client-0: Using Program GlusterFS 3.3, Num (1298437), >>> Version (330) >>> >>> In the log output. >>> >>> Our glusterfs clients all have 3.5.2 glusterfs version. >>> >>> Can someone help? >>> >>> kind regards >>> >>> Klaus Bekk >>> >>> ---------------------------------------------------------- >>> >>> Karlsruher Institut of Technologie (KIT) >>> >>> Institut für Kernphysik (IKP) >>> >>> Dr. Klaus Bekk >>> >>> Hermann-von-Helmholtz-Platz 1 >>> >>> Campus Nord Gebäude 425 >>> >>> 76344 Eggenstein-Leopoldshafen >>> >>> Telefon: +49 721 608-23382 >>> >>> Fax: +49 721 608-23321 >>> >>> E-Mail: Klaus.Bekk@xxxxxxx <mailto:Klaus.Bekk@xxxxxxx> >>> >>> http://www.kit.edu/ >>> >>> KIT -- University of the State of Baden-Wuerttemberg and >>> >>> National Research Center of the Helmholtz Association >>> >>> ---------------------------------------------------------- >>> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users@xxxxxxxxxxx >> http://www.gluster.org/mailman/listinfo/gluster-users > _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
