Hi, A part of the log won't be enough to debug the issue. Need the whole log messages till date. You can send it as attachments. Yes the quota.conf is a binary file. And I need the volume status output too. On Tue, Feb 13, 2018 at 1:56 PM, mabi <mabi@xxxxxxxxxxxxx> wrote: > Hi Hari, > Sorry for not providing you more details from the start. Here below you will > find all the relevant log entries and info. Regarding the quota.conf file I > have found one for my volume but it is a binary file. Is it supposed to be > binary or text? > Regards, > M. > > *** gluster volume info myvolume *** > > Volume Name: myvolume > Type: Replicate > Volume ID: e7a40a1b-45c9-4d3c-bb19-0c59b4eceec5 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x (2 + 1) = 3 > Transport-type: tcp > Bricks: > Brick1: gfs1a.domain.local:/data/myvolume/brick > Brick2: gfs1b.domain.local:/data/myvolume/brick > Brick3: gfs1c.domain.local:/srv/glusterfs/myvolume/brick (arbiter) > Options Reconfigured: > server.event-threads: 4 > client.event-threads: 4 > performance.readdir-ahead: on > nfs.disable: on > features.quota: on > features.inode-quota: on > features.quota-deem-statfs: on > transport.address-family: inet > > > *** /var/log/glusterfs/glusterd.log *** > > [2018-02-13 08:16:09.929568] E [MSGID: 101042] [compat.c:569:gf_umount_lazy] > 0-management: Lazy unmount of /var/run/gluster/myvolume_quota_list/ > [2018-02-13 08:16:28.596527] I [MSGID: 106499] > [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: > Received status volume req for volume myvolume > [2018-02-13 08:16:28.601097] I [MSGID: 106419] > [glusterd-utils.c:6110:glusterd_add_inode_size_to_dict] 0-management: the > brick on data/myvolume (zfs) uses dynamic inode sizes > > > *** /var/log/glusterfs/cmd_history.log *** > > [2018-02-13 08:16:28.605478] : volume status myvolume detail : SUCCESS > > > *** /var/log/glusterfs/quota-mount-myvolume.log *** > > [2018-02-13 08:16:09.934117] I [MSGID: 100030] [glusterfsd.c:2503:main] > 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.10.7 > (args: /usr/sbin/glusterfs --volfile-server localhost --volfile-id myvolume > -l /var/log/glusterfs/quota-mount-myvolume.log -p > /var/run/gluster/myvolume_quota_list.pid --client-pid -5 > /var/run/gluster/myvolume_quota_list/) > [2018-02-13 08:16:09.940432] I [MSGID: 101190] > [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 1 > [2018-02-13 08:16:09.940491] E [socket.c:2327:socket_connect_finish] > 0-glusterfs: connection to ::1:24007 failed (Connection refused); > disconnecting socket > [2018-02-13 08:16:09.940519] I [glusterfsd-mgmt.c:2134:mgmt_rpc_notify] > 0-glusterfsd-mgmt: disconnected from remote-host: localhost > [2018-02-13 08:16:13.943827] I [afr.c:94:fix_quorum_options] > 0-myvolume-replicate-0: reindeer: incoming qtype = none > [2018-02-13 08:16:13.943857] I [afr.c:116:fix_quorum_options] > 0-myvolume-replicate-0: reindeer: quorum_count = 2147483647 > [2018-02-13 08:16:13.945194] I [MSGID: 101190] > [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 2 > [2018-02-13 08:16:13.945237] I [MSGID: 101190] > [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 4 > [2018-02-13 08:16:13.945247] I [MSGID: 101190] > [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 3 > [2018-02-13 08:16:13.945941] W [MSGID: 101174] > [graph.c:361:_log_if_unknown_option] 0-myvolume-readdir-ahead: option > 'parallel-readdir' is not recognized > [2018-02-13 08:16:13.946342] I [MSGID: 114020] [client.c:2352:notify] > 0-myvolume-client-0: parent translators are ready, attempting connect on > transport > [2018-02-13 08:16:13.946789] I [MSGID: 114020] [client.c:2352:notify] > 0-myvolume-client-1: parent translators are ready, attempting connect on > transport > [2018-02-13 08:16:13.947151] I [rpc-clnt.c:2000:rpc_clnt_reconfig] > 0-myvolume-client-0: changing port to 49153 (from 0) > [2018-02-13 08:16:13.947846] I [MSGID: 114057] > [client-handshake.c:1451:select_server_supported_programs] > 0-myvolume-client-0: Using Program GlusterFS 3.3, Num (1298437), Version > (330) > [2018-02-13 08:16:13.948073] I [MSGID: 114020] [client.c:2352:notify] > 0-myvolume-client-2: parent translators are ready, attempting connect on > transport > [2018-02-13 08:16:13.948579] I [rpc-clnt.c:2000:rpc_clnt_reconfig] > 0-myvolume-client-1: changing port to 49154 (from 0) > [2018-02-13 08:16:13.948699] I [MSGID: 114046] > [client-handshake.c:1216:client_setvolume_cbk] 0-myvolume-client-0: > Connected to myvolume-client-0, attached to remote volume > '/data/myvolume/brick'. > [2018-02-13 08:16:13.948747] I [MSGID: 114047] > [client-handshake.c:1227:client_setvolume_cbk] 0-myvolume-client-0: Server > and Client lk-version numbers are not same, reopening the fds > [2018-02-13 08:16:13.948842] I [MSGID: 108005] > [afr-common.c:4813:afr_notify] 0-myvolume-replicate-0: Subvolume > 'myvolume-client-0' came back up; going online. > Final graph: > +------------------------------------------------------------------------------+ > 1: volume myvolume-client-0 > 2: type protocol/client > 3: option opversion 31004 > 4: option clnt-lk-version 1 > 5: option volfile-checksum 0 > 6: option volfile-key myvolume > 7: option client-version 3.10.7 > 8: option process-uuid > gfs1a-14348-2018/02/13-08:16:09:933625-myvolume-client-0-0-0 > 9: option fops-version 1298437 > 10: option ping-timeout 42 > 11: option remote-host gfs1a.domain.local > 12: option remote-subvolume /data/myvolume/brick > 13: option transport-type socket > 14: option transport.address-family inet > 15: option username bea3e634-e174-4bb3-a1d6-25b09d03b536 > 16: option password 3a6f98bd-795e-4ec4-adfe-42c61ccfa0a6 > 17: option event-threads 4 > 18: option transport.tcp-user-timeout 0 > 19: option transport.socket.keepalive-time 20 > 20: option transport.socket.keepalive-interval 2 > 21: option transport.socket.keepalive-count 9 > [2018-02-13 08:16:13.949147] I [MSGID: 114035] > [client-handshake.c:202:client_set_lk_version_cbk] 0-myvolume-client-0: > Server lk version = 1 > 22: option send-gids true > 23: end-volume > 24: > 25: volume myvolume-client-1 > 26: type protocol/client > 27: option ping-timeout 42 > 28: option remote-host gfs1b.domain.local > 29: option remote-subvolume /data/myvolume/brick > 30: option transport-type socket > 31: option transport.address-family inet > 32: option username bea3e634-e174-4bb3-a1d6-25b09d03b536 > 33: option password 3a6f98bd-795e-4ec4-adfe-42c61ccfa0a6 > 34: option event-threads 4 > 35: option transport.tcp-user-timeout 0 > 36: option transport.socket.keepalive-time 20 > 37: option transport.socket.keepalive-interval 2 > 38: option transport.socket.keepalive-count 9 > 39: option send-gids true > 40: end-volume > 41: > 42: volume myvolume-client-2 > 43: type protocol/client > 44: option ping-timeout 42 > 45: option remote-host gfs1c.domain.local > 46: option remote-subvolume /srv/glusterfs/myvolume/brick > 47: option transport-type socket > 48: option transport.address-family inet > 49: option username bea3e634-e174-4bb3-a1d6-25b09d03b536 > 50: option password 3a6f98bd-795e-4ec4-adfe-42c61ccfa0a6 > 51: option event-threads 4 > 52: option transport.tcp-user-timeout 0 > 53: option transport.socket.keepalive-time 20 > 54: option transport.socket.keepalive-interval 2 > 55: option transport.socket.keepalive-count 9 > 56: option send-gids true > 57: end-volume > 58: > 59: volume myvolume-replicate-0 > 60: type cluster/replicate > 61: option afr-pending-xattr > myvolume-client-0,myvolume-client-1,myvolume-client-2 > 62: option arbiter-count 1 > 63: option use-compound-fops off > 64: subvolumes myvolume-client-0 myvolume-client-1 myvolume-client-2 > 65: end-volume > [2018-02-13 08:16:13.949442] I [rpc-clnt.c:2000:rpc_clnt_reconfig] > 0-myvolume-client-2: changing port to 49153 (from 0) > 66: > 67: volume myvolume-dht > 68: type cluster/distribute > 69: option lock-migration off > 70: subvolumes myvolume-replicate-0 > 71: end-volume > 72: > 73: volume myvolume-write-behind > 74: type performance/write-behind > 75: subvolumes myvolume-dht > 76: end-volume > 77: > 78: volume myvolume-read-ahead > 79: type performance/read-ahead > 80: subvolumes myvolume-write-behind > 81: end-volume > 82: > 83: volume myvolume-readdir-ahead > 84: type performance/readdir-ahead > 85: option parallel-readdir off > 86: option rda-request-size 131072 > 87: option rda-cache-limit 10MB > 88: subvolumes myvolume-read-ahead > 89: end-volume > 90: > 91: volume myvolume-io-cache > 92: type performance/io-cache > 93: subvolumes myvolume-readdir-ahead > 94: end-volume > 95: > 96: volume myvolume-quick-read > 97: type performance/quick-read > 98: subvolumes myvolume-io-cache > 99: end-volume > 100: > 101: volume myvolume-open-behind > 102: type performance/open-behind > 103: subvolumes myvolume-quick-read > 104: end-volume > 105: > 106: volume myvolume-md-cache > 107: type performance/md-cache > 108: subvolumes myvolume-open-behind > 109: end-volume > 110: > 111: volume myvolume > 112: type debug/io-stats > 113: option log-level INFO > 114: option latency-measurement off > 115: option count-fop-hits off > 116: subvolumes myvolume-md-cache > 117: end-volume > 118: > 119: volume meta-autoload > 120: type meta > 121: subvolumes myvolume > 122: end-volume > 123: > +------------------------------------------------------------------------------+ > [2018-02-13 08:16:13.949813] I [MSGID: 114057] > [client-handshake.c:1451:select_server_supported_programs] > 0-myvolume-client-1: Using Program GlusterFS 3.3, Num (1298437), Version > (330) > [2018-02-13 08:16:13.950686] I [MSGID: 114057] > [client-handshake.c:1451:select_server_supported_programs] > 0-myvolume-client-2: Using Program GlusterFS 3.3, Num (1298437), Version > (330) > [2018-02-13 08:16:13.950698] I [MSGID: 114046] > [client-handshake.c:1216:client_setvolume_cbk] 0-myvolume-client-1: > Connected to myvolume-client-1, attached to remote volume > '/data/myvolume/brick'. > [2018-02-13 08:16:13.950759] I [MSGID: 114047] > [client-handshake.c:1227:client_setvolume_cbk] 0-myvolume-client-1: Server > and Client lk-version numbers are not same, reopening the fds > [2018-02-13 08:16:13.951009] I [MSGID: 114035] > [client-handshake.c:202:client_set_lk_version_cbk] 0-myvolume-client-1: > Server lk version = 1 > [2018-02-13 08:16:13.951320] I [MSGID: 114046] > [client-handshake.c:1216:client_setvolume_cbk] 0-myvolume-client-2: > Connected to myvolume-client-2, attached to remote volume > '/srv/glusterfs/myvolume/brick'. > [2018-02-13 08:16:13.951348] I [MSGID: 114047] > [client-handshake.c:1227:client_setvolume_cbk] 0-myvolume-client-2: Server > and Client lk-version numbers are not same, reopening the fds > [2018-02-13 08:16:13.952626] I [MSGID: 114035] > [client-handshake.c:202:client_set_lk_version_cbk] 0-myvolume-client-2: > Server lk version = 1 > [2018-02-13 08:16:13.952704] I [fuse-bridge.c:4138:fuse_init] > 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel > 7.23 > [2018-02-13 08:16:13.952742] I [fuse-bridge.c:4823:fuse_graph_sync] 0-fuse: > switched to graph 0 > [2018-02-13 08:16:13.954337] I [MSGID: 108031] > [afr-common.c:2357:afr_local_discovery_cbk] 0-myvolume-replicate-0: > selecting local read_child myvolume-client-0 > [2018-02-13 08:16:14.020681] I [fuse-bridge.c:5081:fuse_thread_proc] 0-fuse: > unmounting /var/run/gluster/myvolume_quota_list/ > [2018-02-13 08:16:14.020981] W [glusterfsd.c:1360:cleanup_and_exit] > (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064) [0x7f92aaeef064] > -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55d8ddb85f75] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x55d8ddb85d97] ) 0-: > received signum (15), shutting down > [2018-02-13 08:16:14.021017] I [fuse-bridge.c:5804:fini] 0-fuse: Unmounting > '/var/run/gluster/myvolume_quota_list/'. > > > *** /var/log/glusterfs/quotad.log *** > > [2018-02-13 08:16:14.023519] W [MSGID: 108027] > [afr-common.c:2718:afr_discover_done] 0-myvolume-replicate-0: no read > subvols for (null) > [2018-02-13 08:16:14.064410] W [dict.c:599:dict_unref] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) > [0x7f63153e7f3d] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) > [0x7f63153e8d72] > -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) > [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] > [2018-02-13 08:16:14.074683] W [dict.c:599:dict_unref] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) > [0x7f63153e7f3d] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) > [0x7f63153e8d72] > -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) > [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] > [2018-02-13 08:16:14.082040] W [dict.c:599:dict_unref] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) > [0x7f63153e7f3d] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) > [0x7f63153e8d72] > -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) > [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] > [2018-02-13 08:16:14.086950] W [dict.c:599:dict_unref] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) > [0x7f63153e7f3d] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) > [0x7f63153e8d72] > -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) > [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] > [2018-02-13 08:16:14.087939] W [dict.c:599:dict_unref] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) > [0x7f63153e7f3d] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) > [0x7f63153e8d72] > -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) > [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] > [2018-02-13 08:16:14.089775] W [dict.c:599:dict_unref] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) > [0x7f63153e7f3d] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) > [0x7f63153e8d72] > -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) > [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] > [2018-02-13 08:16:14.092937] W [dict.c:599:dict_unref] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) > [0x7f63153e7f3d] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) > [0x7f63153e8d72] > -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) > [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] > The message "W [MSGID: 108027] [afr-common.c:2718:afr_discover_done] > 0-myvolume-replicate-0: no read subvols for (null)" repeated 34 times > between [2018-02-13 08:16:14.023519] and [2018-02-13 08:17:26.560943] > > > *** /var/log/glusterfs/cli.log *** > > [2018-02-13 08:16:14.064404] E > [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get > quota limits for 3df709ee-641d-46a2-bd61-889583e3033c > [2018-02-13 08:16:14.074693] E > [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get > quota limits for a27818fe-0248-40fe-bb23-d43d61010478 > [2018-02-13 08:16:14.082067] E > [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get > quota limits for daf97388-bcec-4cc0-a8ef-5b93f05b30f6 > [2018-02-13 08:16:14.086929] E > [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get > quota limits for 3c768b36-2625-4509-87ef-fe5214cb9b01 > [2018-02-13 08:16:14.087905] E > [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get > quota limits for f8cf47d4-4f54-43c5-ab0d-75b45b4677a3 > [2018-02-13 08:16:14.089788] E > [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get > quota limits for b4c81a39-2152-45c5-95d3-b796d88226fe > [2018-02-13 08:16:14.092919] E > [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get > quota limits for 16ac4cde-a5d4-451f-adcc-422a542fea24 > [2018-02-13 08:16:14.092980] I [input.c:31:cli_batch] 0-: Exiting with: 0 > > > *** /var/log/glusterfs/bricks/data-myvolume-brick.log *** > > [2018-02-13 08:16:13.948065] I [addr.c:182:gf_auth] 0-/data/myvolume/brick: > allowed = "*", received addr = "127.0.0.1" > [2018-02-13 08:16:13.948105] I [login.c:76:gf_auth] 0-auth/login: allowed > user names: bea3e634-e174-4bb3-a1d6-25b09d03b536 > [2018-02-13 08:16:13.948125] I [MSGID: 115029] > [server-handshake.c:695:server_setvolume] 0-myvolume-server: accepted client > from gfs1a-14348-2018/02/13-08:16:09:933625-myvolume-client-0-0-0 (version: > 3.10.7) > [2018-02-13 08:16:14.022257] I [MSGID: 115036] > [server.c:559:server_rpc_notify] 0-myvolume-server: disconnecting connection > from gfs1a-14348-2018/02/13-08:16:09:933625-myvolume-client-0-0-0 > [2018-02-13 08:16:14.022465] I [MSGID: 101055] > [client_t.c:436:gf_client_unref] 0-myvolume-server: Shutting down connection > gfs1a-14348-2018/02/13-08:16:09:933625-myvolume-client-0-0-0 > > > > -------- Original Message -------- > On February 13, 2018 12:47 AM, Hari Gowtham <hgowtham@xxxxxxxxxx> wrote: > > Hi, > > Can you provide more information like, the volume configuration, quota.conf > file and the log files. > > On Sat, Feb 10, 2018 at 1:05 AM, mabi <mabi@xxxxxxxxxxxxx> wrote: >> Hello, >> >> I am running GlusterFS 3.10.7 and just noticed by doing a "gluster volume >> quota <volname> list" that my quotas on that volume are broken. The command >> returns no output and no errors but by looking in /var/log/glusterfs.cli I >> found the following errors: >> >> [2018-02-09 19:31:24.242324] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for 3df709ee-641d-46a2-bd61-889583e3033c >> [2018-02-09 19:31:24.249790] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for a27818fe-0248-40fe-bb23-d43d61010478 >> [2018-02-09 19:31:24.252378] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for daf97388-bcec-4cc0-a8ef-5b93f05b30f6 >> [2018-02-09 19:31:24.256775] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for 3c768b36-2625-4509-87ef-fe5214cb9b01 >> [2018-02-09 19:31:24.257434] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for f8cf47d4-4f54-43c5-ab0d-75b45b4677a3 >> [2018-02-09 19:31:24.259126] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for b4c81a39-2152-45c5-95d3-b796d88226fe >> [2018-02-09 19:31:24.261664] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for 16ac4cde-a5d4-451f-adcc-422a542fea24 >> [2018-02-09 19:31:24.261719] I [input.c:31:cli_batch] 0-: Exiting with: 0 >> >> How can I fix my quota on that volume again? I had around 30 quotas set on >> different directories of that volume. >> >> Thanks in advance. >> >> Regards, >> M. >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users@xxxxxxxxxxx >> http://lists.gluster.org/mailman/listinfo/gluster-users > > > > -- > Regards, > Hari Gowtham. > > -- Regards, Hari Gowtham. _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
