Re: Quota trouble

[Avra Sengupta <asengupt@xxxxxxxxxx>]

Hi Kondo,

Can u also mention the version of gluster you are using.

+Adding gluster-users

Regards,
Avra
On 04/21/2015 02:27 PM, Avra Sengupta wrote:

Hi Kondo,

I went through the gluster13 logs you had sent. Seems like something on that machine is holding the lock and is not releasing it. There are ways in which the system might end up in this scenario. I will try and explain the same with an example.

Let's say I have gluster 11, gluster12, and gluster 13 in my cluster. I initiate a command from gluster11. Now the first thing that command does is, it holds a lock on all the nodes in the cluster on behalf of gluster11. Once the command does what's intended, it's last act before ending is to unlock all the nodes in the cluster. Now, only the node that has issued the lock, can issue the unlock.

In your case what has happened is some command after successfully acquired the lock on gluster13. Now the node which initiated the command, went down or glusterd on that node went down before it could complete the command and it never got to send the unlock to gluster13.

There's a workaround to it. You can restart glusterd on gluster13 and it should work fine.

Regards,
Avra

On 04/20/2015 06:55 PM, kenji kondo wrote:

Hello Vijay,

Maybe this is very rare case. But is there any idea?

Thanks,

Kondo

2015-04-15 9:47 GMT+09:00 Vijaikumar M <vmallika@xxxxxxxxxx>:

Adding Avra...

Thanks,
Vijay

-------- Forwarded Message --------

Subject:	Re: Quota trouble
Date:	Wed, 15 Apr 2015 00:27:26 +0900
From:	kenji kondo <kkay.jp@xxxxxxxxx>
To:	Vijaikumar M <vmallika@xxxxxxxxxx>

Hi Vijay,

Thanks for your comments.

The lock error occurs at one server it's called "gluster13".

In the gluster13, I tried to create new volume and start quota. But it failed as below,

In both host gluster10 and gluster13, ran below

$ sudo mkdir /export11/testbrick1

$ sudo mkdir /export11/testbrick2

 

In gluster13, ran below

$ sudo /usr/sbin/gluster volume create testvol2 gluster13:/export11/testbrick1 gluster13:/export11/testbrick2

volume create: testvol2: failed: Locking failed on gluster13. Please check log file for details.

 

$ sudo /usr/sbin/gluster volume create testvol2 gluster10:/export11/testbrick1 gluster10:/export11/testbrick2

volume create: testvol2: failed: Locking failed on gluster13. Please check log file for details.

 

But I recived error messages above.

On the other hand, in gluster10, it was success.

 

Again, in gluster13, I tried to run quota, but it failed as below.

$ sudo /usr/sbin/gluster volume quota testvol2 enable

quota command failed : Locking failed on gluster13. Please check log file for details.

Could you find attached?

We can find error messages in the log of gluster13.

Best regards,

Kondo

2015-04-13 19:38 GMT+09:00 Vijaikumar M <vmallika@xxxxxxxxxx>:

Hi Kondo,

The lock error you mentioned is because, another operation is still running on the volume and hence not able to acquire the lock.
This is bug of not displaying proper error message, we are working on fixing this issue.

I was not able to find any clue on why quotad is not running.

I wanted to check, if we can manually start quotad something like below:

# /usr/local/sbin/glusterfs -s localhost --volfile-id gluster/quotad -p /var/lib/glusterd/quotad/run/quotad.pid -l /var/log/glusterfs/quotad.log -S /var/run/gluster/myquotad.socket --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off

or

create a new temporary volume, and enable quota on this volume. (quotad will be same for all the volume which has quota enabled)


Thanks,
Vijay

On Sunday 12 April 2015 07:05 PM, kenji kondo wrote:

Hi Vijay,

Thank you for your suggestion. But I'm sorry, it's difficult to access from outside because my glusterfs system is closed.

I will give up if there is no clue information in attached log.

Best regards,

Kondo

2015-04-09 15:40 GMT+09:00 Vijaikumar M <vmallika@xxxxxxxxxx>:

On Thursday 09 April 2015 11:58 AM, Vijaikumar M wrote:

On Wednesday 08 April 2015 09:57 PM, kenji kondo wrote:

Hi Vijay,

I checked the all of the setting.

The all are 'features.quota=on' when I set quota enable and the all are 'features.quota=off' when I set quota disable.

But I could find new issue.

When I checked a volume status for all server, in one of the servers I received the error message as below.

$ sudo /usr/sbin/gluster volume status testvol

Locking failed on gluster13. Please check log file for details.

In etc-glusterfs-glusterd.vol.log of problem server, I found error messages as below.

 

[2015-04-08 08:40:04.782644] I [mem-pool.c:545:mem_pool_destroy] 0-management: size=588 max=0 total=0

[2015-04-08 08:40:04.782685] I [mem-pool.c:545:mem_pool_destroy] 0-management: size=124 max=0 total=0

[2015-04-08 08:40:04.782848] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/14b05cd492843e6e288e290c2d63093c.socket failed (Invalid arguments)

[2015-04-08 08:40:04.805407] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: nfs has disconnected from glusterd.

[2015-04-08 08:43:02.439001] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume req for volume testvol

[2015-04-08 08:43:02.460581] E [glusterd-utils.c:148:glusterd_lock] 0-management: Unable to get lock for uuid: 03a32bce-ec63-4dc3-a287-4901a55dd8c9, lock held by: 03a32bce-ec63-4dc3-a287-4901a55dd8c9

[2015-04-08 08:43:02.460632] E [glusterd-op-sm.c:6584:glusterd_op_sm] 0-management: handler returned: -1

[2015-04-08 08:43:02.460654] E [glusterd-syncop.c:105:gd_collate_errors] 0-: Locking failed on gluster13. Please check log file for details.

[2015-04-08 08:43:02.461409] E [glusterd-syncop.c:1602:gd_sync_task_begin] 0-management: Locking Peers Failed.

[2015-04-08 08:43:43.698168] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume req for volume testvol

[2015-04-08 08:43:43.698813] E [glusterd-utils.c:148:glusterd_lock] 0-management: Unable to get lock for uuid: 03a32bce-ec63-4dc3-a287-4901a55dd8c9, lock held by: 03a32bce-ec63-4dc3-a287-4901a55dd8c9

[2015-04-08 08:43:43.698898] E [glusterd-op-sm.c:6584:glusterd_op_sm] 0-management: handler returned: -1

[2015-04-08 08:43:43.698994] E [glusterd-syncop.c:105:gd_collate_errors] 0-: Locking failed on gluster13. Please check log file for details.

[2015-04-08 08:43:43.702126] E [glusterd-syncop.c:1602:gd_sync_task_begin] 0-management: Locking Peers Failed.

[2015-04-08 08:44:01.277139] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume req for volume testvol

[2015-04-08 08:44:01.277560] E [glusterd-utils.c:148:glusterd_lock] 0-management: Unable to get lock for uuid: 03a32bce-ec63-4dc3-a287-4901a55dd8c9, lock held by: 03a32bce-ec63-4dc3-a287-4901a55dd8c9

[2015-04-08 08:44:01.277639] E [glusterd-op-sm.c:6584:glusterd_op_sm] 0-management: handler returned: -1

[2015-04-08 08:44:01.277676] E [glusterd-syncop.c:105:gd_collate_errors] 0-: Locking failed on gluster13. Please check log file for details.

[2015-04-08 08:44:01.281514] E [glusterd-syncop.c:1602:gd_sync_task_begin] 0-management: Locking Peers Failed.

[2015-04-08 08:45:42.599796] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume req for volume testvol

[2015-04-08 08:45:42.600343] E [glusterd-utils.c:148:glusterd_lock] 0-management: Unable to get lock for uuid: 03a32bce-ec63-4dc3-a287-4901a55dd8c9, lock held by: 03a32bce-ec63-4dc3-a287-4901a55dd8c9

[2015-04-08 08:45:42.600417] E [glusterd-op-sm.c:6584:glusterd_op_sm] 0-management: handler returned: -1

[2015-04-08 08:45:42.600482] E [glusterd-syncop.c:105:gd_collate_errors] 0-: Locking failed on gluster13. Please check log file for details.

[2015-04-08 08:45:42.601039] E [glusterd-syncop.c:1602:gd_sync_task_begin] 0-management: Locking Peers Failed.

Does this situation relate to my quota problems?

This is a glusterd different issue. Can we get the glusterd logs from gluster13?
Can get access to these machines, so that we can debug live?

Thanks,
Vijay

Regarding quota issue, quota feature is enabled successfully. I am wondering why quotad is not started.
If we get the access to the machine, it will be easier to debug the issue.

Thanks,
Vijay

Best regards,

Kondo

 

2015-04-08 15:14 GMT+09:00 Vijaikumar M <vmallika@xxxxxxxxxx>:

Hi Kondo,

I suspect, in one of the node quota feature is not set for some reason and hence quotad is not starting.

On all the nodes can you check if below option is set to 'on'

# grep quota /var/lib/glusterd/vols/<volname>/info
features.quota=on


Also can I get brick logs from all the nodes?

Also can you create a temporary volume and enable the quota here and see if see quota works fine with this volume?


Thanks,
Vijay

On Tuesday 07 April 2015 08:34 PM, kenji kondo wrote:

Hi Vijay,

Could you find attached?

I got logs of server and client.

As same as before, I could not create a file after quota usage-limit setting.

Best regards,

Kondo

2015-04-07 18:34 GMT+09:00 Vijaikumar M <vmallika@xxxxxxxxxx>:

Hi Konda,

Can we get all the log files?

# gluster volume quota <volname> disable
# gluster volume quota <volname> enable


Now copy all the logs files.

Thanks,
Vijay

On Tuesday 07 April 2015 12:39 PM, K.Kondo wrote:

Thank you very much ! Vijay

I want to use a quota because each volume became too big.

Best regard

Kondo

2015/04/07 15:18、Vijaikumar M <vmallika@xxxxxxxxxx> のメッセージ:

Hi Kondo,

I couldn’t find clue from the logs. I will discuss about this issue with my colleagues today.


Thanks,
Vijay

On Monday 06 April 2015 10:56 PM, kenji kondo wrote:

Hello Vijay,

Is there something idea for this?

Best regards,

Kondo

2015-03-31 22:46 GMT+09:00 kenji kondo <kkay.jp@xxxxxxxxx>:

Hi Vijay,

I'm sorry for late reply.

I could get the debug mode log as attached.

In this test, unfortunately the quota did not work as same as before.

Could you find the cause of my problem?

Best regards,

Kondo

2015-03-25 17:20 GMT+09:00 Vijaikumar M <vmallika@xxxxxxxxxx>:

Hi Kondo,

For some reason quota enable was not successful. We may have re-try enabling quota.


Thanks,
Vijay

On Tuesday 24 March 2015 07:08 PM, kenji kondo wrote:

Hi Vijay,

Thanks for your checking. Unfortunately, currently I can't stop the service because many users are using.

But, I want to know this cause of this trouble, so I will plan to stop. Please wait to get the log.

Best regards,

Kondo

2015-03-24 17:01 GMT+09:00 Vijaikumar M <vmallika@xxxxxxxxxx>:

Hi Kondo,

I couldn't find much clue in the glusterd logs, other than the error message you mentioned below.
Can you try disabling and enabling the quota again and see if this start quotad?

Try below command:
# gluster volume quota <volname> disable

wait for all quota process to terminate
#ps -ef | quota

# service glusterd stop
# glusterd -LDEBUG
# gluster volume quota <volname> enable

Now verify if quotad is running


Thanks,
Vijay

On Monday 23 March 2015 06:24 PM, kenji kondo wrote:

Hi Vijay,

As you pointed out, the quotad is not running in the all of server.

I checked the volume status and got following log.

Quota Daemon on gluster25 N/A N N/A

So, I attached requested log 'etc-glusterfs-glusterd.vol.log'.

The error messages can be found in the log.

[2015-03-19 11:51:07.457697] E [glusterd-quota.c:1467:glusterd_op_stage_quota] 0-management: Quota is disabled, please enable quota

If you want more some information to solve this problems, please ask me.

Best regards,

Kondo

2015-03-23 16:04 GMT+09:00 Vijaikumar M <vmallika@xxxxxxxxxx>:

Hi Kondo,

Can you please verify if quotad is running?


root@rh1:~ # gluster volume status
Status of volume: vol1
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick rh1:/var/opt/gluster/bricks/b1/dir    49152     0          Y       1858
NFS Server on localhost                     2049      0          Y       1879
Quota Daemon on localhost                   N/A       N/A        Y       1914
 
Task Status of Volume vol1
------------------------------------------------------------------------------
There are no active volume tasks



root@rh1:~ # ps -ef | grep quotad
root      1914     1  0 12:29 ?        00:00:00 /usr/local/sbin/glusterfs -s localhost --volfile-id gluster/quotad -p /var/lib/glusterd/quotad/run/quotad.pid -l /var/log/glusterfs/quotad.log -S /var/run/gluster/bb6ab82f70f555fd5c0e188fa4e09584.socket --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off
root      1970  1511  0 12:31 pts/1    00:00:00 grep quotad



root@rh1:~ # gluster volume info
Volume Name: vol1
Type: Distribute
Volume ID: a55519ec-65d1-4741-9ad3-f94020fc9b21
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: rh1:/var/opt/gluster/bricks/b1/dir
Options Reconfigured:
features.quota: on



If quotad is not running, can you please provide glusterd logs 'usr-local-etc-glusterfs-glusterd.vol.log'. I will check is there are any issues starting quotad.


Thanks,
Vihay

On Monday 23 March 2015 11:54 AM, K.Kondo wrote:

Hi Vijay,
I could not find the"quotad.log" in directory /var/log/glusterfs in both servers and client. But other test server has the log.

Do you know why there is no the file?

Thanks,

Kondo

2015/03/23 13:41、Vijaikumar M <vmallika@xxxxxxxxxx> のメッセージ:

Hi Kondo,

log file 'quotad.log' is missing in the attachment. Can you provide this log file as well?

Thanks,
Vijay

On Monday 23 March 2015 09:50 AM, kenji kondo wrote:

Hi Vijay,

Could you find the attached?

Best regards,

Kondo

2015-03-23 12:53 GMT+09:00 Vijaikumar M <vmallika@xxxxxxxxxx>:

Hi Kondo,

Can you please provide below mentioned gluterfs logs?
    client logs (name of this log will be prefixed with mount-point dirname)
    brick logs
    quotad logs


Thanks,
Vijay

On Friday 20 March 2015 06:31 PM, kenji kondo wrote:

Hi, Vijay and Peter

Thanks for your reply.

I create new volume "testvol" with two bricks and set quota to simplify this problem.

I got the glusterfs log as following after try to create a directory and file.

BTW, my glusterd was upgraded from older version, although I don't know related to it.

Best regards,

Kondo

[2015-03-20 03:42:52.931016] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.29 (args: /usr/sbin/glusterfs --volfile-server=gluster10 --volfile-id=testvol testvol)

[2015-03-20 03:42:52.944850] I [dht-shared.c:337:dht_init_regex] 0-testvol-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$

[2015-03-20 03:42:52.946256] I [client.c:2280:notify] 0-testvol-client-0: parent translators are ready, attempting connect on transport

[2015-03-20 03:42:52.950674] I [client.c:2280:notify] 0-testvol-client-1: parent translators are ready, attempting connect on transport

Final graph:

+------------------------------------------------------------------------------+

  1: volume testvol-client-0

  2:     type protocol/client

  3:     option ping-timeout 42

  4:     option remote-host gluster24

  5:     option remote-subvolume /export25/brick

  6:     option transport-type socket

  7:     option send-gids true

  8: end-volume

  9:  

 10: volume testvol-client-1

 11:     type protocol/client

 12:     option ping-timeout 42

 13:     option remote-host gluster25

 14:     option remote-subvolume /export25/brick

 15:     option transport-type socket

 16:     option send-gids true

 17: end-volume

 18:  

 19: volume testvol-dht

 20:     type cluster/distribute

 21:     subvolumes testvol-client-0 testvol-client-1

 22: end-volume

 23:  

 24: volume testvol-write-behind

 25:     type performance/write-behind

 26:     subvolumes testvol-dht

 27: end-volume

 28:  

 29: volume testvol-read-ahead

 30:     type performance/read-ahead

 31:     subvolumes testvol-write-behind

 32: end-volume

 33:  

 34: volume testvol-io-cache

 35:     type performance/io-cache

 36:     subvolumes testvol-read-ahead

 37: end-volume

 38:  

 39: volume testvol-quick-read

 40:     type performance/quick-read

 41:     subvolumes testvol-io-cache

 42: end-volume

 43:  

 44: volume testvol-md-cache

 45:     type performance/md-cache

 46:     subvolumes testvol-quick-read

 47: end-volume

 48:  

 49: volume testvol

 50:     type debug/io-stats

 51:     option latency-measurement off

 52:     option count-fop-hits off

 53:     subvolumes testvol-md-cache

 54: end-volume

 55:  

 56: volume meta-autoload

 57:     type meta

 58:     subvolumes testvol

 59: end-volume

 60:  

+------------------------------------------------------------------------------+

[2015-03-20 03:42:52.955337] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-testvol-client-0: changing port to 49155 (from 0)

[2015-03-20 03:42:52.957549] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-testvol-client-1: changing port to 49155 (from 0)

[2015-03-20 03:42:52.959889] I [client-handshake.c:1415:select_server_supported_programs] 0-testvol-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2015-03-20 03:42:52.960090] I [client-handshake.c:1415:select_server_supported_programs] 0-testvol-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2015-03-20 03:42:52.960376] I [client-handshake.c:1200:client_setvolume_cbk] 0-testvol-client-0: Connected to testvol-client-0, attached to remote volume '/export25/brick'.

[2015-03-20 03:42:52.960405] I [client-handshake.c:1212:client_setvolume_cbk] 0-testvol-client-0: Server and Client lk-version numbers are not same, reopening the fds

[2015-03-20 03:42:52.960471] I [client-handshake.c:1200:client_setvolume_cbk] 0-testvol-client-1: Connected to testvol-client-1, attached to remote volume '/export25/brick'.

[2015-03-20 03:42:52.960478] I [client-handshake.c:1212:client_setvolume_cbk] 0-testvol-client-1: Server and Client lk-version numbers are not same, reopening the fds

[2015-03-20 03:42:52.962288] I [fuse-bridge.c:5042:fuse_graph_setup] 0-fuse: switched to graph 0

[2015-03-20 03:42:52.962351] I [client-handshake.c:188:client_set_lk_version_cbk] 0-testvol-client-1: Server lk version = 1

[2015-03-20 03:42:52.962362] I [client-handshake.c:188:client_set_lk_version_cbk] 0-testvol-client-0: Server lk version = 1

[2015-03-20 03:42:52.962424] I [fuse-bridge.c:3971:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.14

[2015-03-20 03:47:13.352234] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed

[2015-03-20 03:47:15.518667] I [dht-shared.c:337:dht_init_regex] 2-testvol-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$

[2015-03-20 03:47:15.520034] W [graph.c:344:_log_if_unknown_option] 2-testvol-quota: option 'timeout' is not recognized

[2015-03-20 03:47:15.520091] I [client.c:2280:notify] 2-testvol-client-0: parent translators are ready, attempting connect on transport

[2015-03-20 03:47:15.524546] I [client.c:2280:notify] 2-testvol-client-1: parent translators are ready, attempting connect on transport

Final graph:

+------------------------------------------------------------------------------+

  1: volume testvol-client-0

  2:     type protocol/client

  3:     option ping-timeout 42

  4:     option remote-host gluster24

  5:     option remote-subvolume /export25/brick

  6:     option transport-type socket

  7:     option send-gids true

  8: end-volume

  9:  

 10: volume testvol-client-1

 11:     type protocol/client

 12:     option ping-timeout 42

 13:     option remote-host gluster25

 14:     option remote-subvolume /export25/brick

 15:     option transport-type socket

 16:     option send-gids true

 17: end-volume

 18:  

 19: volume testvol-dht

 20:     type cluster/distribute

 21:     subvolumes testvol-client-0 testvol-client-1

 22: end-volume

 23:  

 24: volume testvol-quota

 25:     type features/quota

 26:     option timeout 0

 27:     option deem-statfs off

 28:     subvolumes testvol-dht

 29: end-volume

 30:  

 31: volume testvol-write-behind

 32:     type performance/write-behind

 33:     subvolumes testvol-quota

 34: end-volume

 35:  

 36: volume testvol-read-ahead

 37:     type performance/read-ahead

 38:     subvolumes testvol-write-behind

 39: end-volume

 40:  

 41: volume testvol-io-cache

 42:     type performance/io-cache

 43:     subvolumes testvol-read-ahead

 44: end-volume

 45:  

 46: volume testvol-quick-read

 47:     type performance/quick-read

 48:     subvolumes testvol-io-cache

 49: end-volume

 50:  

 51: volume testvol-md-cache

 52:     type performance/md-cache

 53:     subvolumes testvol-quick-read

 54: end-volume

 55:  

 56: volume testvol

 57:     type debug/io-stats

 58:     option latency-measurement off

 59:     option count-fop-hits off

 60:     subvolumes testvol-md-cache

 61: end-volume

 62:  

 63: volume meta-autoload

 64:     type meta

 65:     subvolumes testvol

 66: end-volume

 67:  

+------------------------------------------------------------------------------+

[2015-03-20 03:47:15.530005] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 2-testvol-client-1: changing port to 49155 (from 0)

[2015-03-20 03:47:15.530047] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 2-testvol-client-0: changing port to 49155 (from 0)

[2015-03-20 03:47:15.539062] I [client-handshake.c:1415:select_server_supported_programs] 2-testvol-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2015-03-20 03:47:15.539299] I [client-handshake.c:1415:select_server_supported_programs] 2-testvol-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2015-03-20 03:47:15.539462] I [client-handshake.c:1200:client_setvolume_cbk] 2-testvol-client-1: Connected to testvol-client-1, attached to remote volume '/export25/brick'.

[2015-03-20 03:47:15.539485] I [client-handshake.c:1212:client_setvolume_cbk] 2-testvol-client-1: Server and Client lk-version numbers are not same, reopening the fds

[2015-03-20 03:47:15.539729] I [client-handshake.c:1200:client_setvolume_cbk] 2-testvol-client-0: Connected to testvol-client-0, attached to remote volume '/export25/brick'.

[2015-03-20 03:47:15.539751] I [client-handshake.c:1212:client_setvolume_cbk] 2-testvol-client-0: Server and Client lk-version numbers are not same, reopening the fds

[2015-03-20 03:47:15.542878] I [fuse-bridge.c:5042:fuse_graph_setup] 0-fuse: switched to graph 2

[2015-03-20 03:47:15.542959] I [client-handshake.c:188:client_set_lk_version_cbk] 2-testvol-client-1: Server lk version = 1

[2015-03-20 03:47:15.542987] I [client-handshake.c:188:client_set_lk_version_cbk] 2-testvol-client-0: Server lk version = 1

[2015-03-20 03:48:04.586291] I [client.c:2289:notify] 0-testvol-client-0: current graph is no longer active, destroying rpc_client 

[2015-03-20 03:48:04.586360] I [client.c:2289:notify] 0-testvol-client-1: current graph is no longer active, destroying rpc_client 

[2015-03-20 03:48:04.586378] I [client.c:2215:client_rpc_notify] 0-testvol-client-0: disconnected from testvol-client-0. Client process will keep trying to connect to glusterd until brick's port is available

[2015-03-20 03:48:04.586430] I [client.c:2215:client_rpc_notify] 0-testvol-client-1: disconnected from testvol-client-1. Client process will keep trying to connect to glusterd until brick's port is available

[2015-03-20 03:48:04.589552] W [client-rpc-fops.c:306:client3_3_mkdir_cbk] 2-testvol-client-0: remote operation failed: Transport endpoint is not connected. Path: /test/a

[2015-03-20 03:48:04.589608] W [fuse-bridge.c:481:fuse_entry_cbk] 0-glusterfs-fuse: 78: MKDIR() /test/a => -1 (Transport endpoint is not connected)

[2015-03-20 03:48:11.073349] W [client-rpc-fops.c:2212:client3_3_create_cbk] 2-testvol-client-1: remote operation failed: Transport endpoint is not connected. Path: /test/f

[2015-03-20 03:48:11.073419] W [fuse-bridge.c:1937:fuse_create_cbk] 0-glusterfs-fuse: 82: /test/f => -1 (Transport endpoint is not connected)

2015-03-20 11:27 GMT+09:00 Vijaikumar M <vmallika@xxxxxxxxxx>:

Hi Kondo,

Can you please provide all the glusterfs log files?

Thanks,
Vijay

On Friday 20 March 2015 07:33 AM, K.Kondo wrote:

Hello, experts

I had a trouble about quota.

I set quota to one distributed volume "vol12" as bellow.

gluster> volume quota vol12 enable

volume quota : success

gluster> volume quota vol12 limit-usage /test 10GB

volume quota : success

 

But I couldn't create a file and directory with below error message. 

 

In a client host,

$cd test (mounted using fuse)

$mkdir a

mkdir: cannot create directory `a': Transport endpoint is not connected

 

Additionally, I couldn't check quota status using gluster command.

 

gluster> volume quota vol12 list

                  Path                   Hard-limit Soft-limit   Used  Available  Soft-limit exceeded? Hard-limit exceeded?

---------------------------------------------------------------------------------------------------------------------------

 Here, this command stops, so I have to do Ctrl-C.

Gluster version is 3.6.1 and 3.6.0.29 for server and client respectively.

Any idea for this?

Best regards,

K. Kondo

 

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users