Re: mkdir produces stale file handles

[Strahil <hunter86_bg@xxxxxxxxx>]

That seems a gluster control.

Still, for me the issue is quite obvious - you are at 100% (or almost)  storage and you should rebalance your VMs.

Can you do a storage migration from the storage using gluster volumes at 100% to another storage whose gluster volumes are not so full (for example those that are 88% full)?

Best Regards,
Strahil Nikolov

On Sep 19, 2019 11:43, Stefan Solbrig <stefan.solbrig@xxxxx> wrote:

Dear all,

I have a situation where "mkdir" on a client produces stale file handles.

This happened after upgrading from 3.12 to 6.5

I believe I found the reason for it:

6.5 (but not 3.12) checks if there is space left on the device before doing a "mkdir", but calculates the "fullness" in percent.   In my situations I have bricks that seem 100% full although there is plenty space left on the device (several GBytes, see listing below).  In this situation, a "mkdir" is not performed on bricks that are 100% full, but the "mkdir" succeeds from a user perspective.  Then, doing a "ls" on the recently created directory leads to the message "stale file handle".

I believe the call sequence is more or less this:

server-rpc-fops.c:539:server_mkdir_cbk

server-rpc-fops.c:2666:server_mkdir_resume

server-rpc-fops.c:5242:server3_3_mkdir

posix-entry-ops.c:625:posix_mkdir

posix-helpers.c:2271

My questions are:

* is it meant to operate in this way?

* is there a built-in way to fix the inconsistent directories? 

(I tried creating the missing directories on the bricks by hand, which seemed to fix the issue, but I'm not sure if this will introduce other problems.)

The obvious (good) fix would be to redistribute the data such that the 100% full bricks will have enough free space. However, if a user writes a really large file, the problem can re-occur any time... 

best wishes,

Stefan

PS:

File system listing.  Each file system is served as a brick, in a distribute-only system.

Filesystem                                       Size  Used Avail Use% Mounted on

/dev/mapper/vgosb03pool06vd03-lvosb03pool06vd03   30T   27T  3.8T  88% /gl/lvosb03pool06vd03

/dev/mapper/vgosb03pool06vd02-lvosb03pool06vd02   30T   27T  3.8T  88% /gl/lvosb03pool06vd02

/dev/mapper/vgosb03pool06vd01-lvosb03pool06vd01   30T   27T  3.7T  88% /gl/lvosb03pool06vd01

/dev/mapper/vgosb03pool01vd01-lvosb03pool01vd01   30T   30T  7.8G 100% /gl/lvosb03pool01vd01

/dev/mapper/vgosb03pool01vd02-lvosb03pool01vd02   30T   30T   41G 100% /gl/lvosb03pool01vd02

/dev/mapper/vgosb03pool01vd03-lvosb03pool01vd03   30T   29T  1.5T  96% /gl/lvosb03pool01vd03

/dev/mapper/vgosb03pool01vd04-lvosb03pool01vd04   30T   30T   17G 100% /gl/lvosb03pool01vd04

/dev/mapper/vgosb03pool02vd01-lvosb03pool02vd01   30T   30T   57G 100% /gl/lvosb03pool02vd01

/dev/mapper/vgosb03pool02vd02-lvosb03pool02vd02   30T   30T   29G 100% /gl/lvosb03pool02vd02

/dev/mapper/vgosb03pool02vd03-lvosb03pool02vd03   30T   30T   26G 100% /gl/lvosb03pool02vd03

/dev/mapper/vgosb03pool02vd04-lvosb03pool02vd04   31T   31T  9.7G 100% /gl/lvosb03pool02vd04

/dev/mapper/vgosb03pool03vd01-lvosb03pool03vd01   30T   30T   93G 100% /gl/lvosb03pool03vd01

/dev/mapper/vgosb03pool03vd02-lvosb03pool03vd02   30T   30T   23G 100% /gl/lvosb03pool03vd02

/dev/mapper/vgosb03pool03vd03-lvosb03pool03vd03   30T   30T  163G 100% /gl/lvosb03pool03vd03

/dev/mapper/vgosb03pool03vd04-lvosb03pool03vd04   31T   30T  384G  99% /gl/lvosb03pool03vd04

/dev/mapper/vgosb03pool04vd01-lvosb03pool04vd01   30T   29T  1.1T  97% /gl/lvosb03pool04vd01

/dev/mapper/vgosb03pool04vd02-lvosb03pool04vd02   30T   27T  3.9T  88% /gl/lvosb03pool04vd02

/dev/mapper/vgosb03pool04vd03-lvosb03pool04vd03   30T   29T  1.9T  94% /gl/lvosb03pool04vd03

/dev/mapper/vgosb03pool04vd04-lvosb03pool04vd04   31T   29T  1.9T  94% /gl/lvosb03pool04vd04

/dev/mapper/vgosb03pool05vd01-lvosb03pool05vd01   30T   28T  2.3T  93% /gl/lvosb03pool05vd01

/dev/mapper/vgosb03pool05vd02-lvosb03pool05vd02   30T   27T  3.9T  88% /gl/lvosb03pool05vd02

/dev/mapper/vgosb03pool05vd03-lvosb03pool05vd03   30T   27T  3.9T  88% /gl/lvosb03pool05vd03

/dev/mapper/vgosb03pool05vd04-lvosb03pool05vd04   31T   27T  3.9T  88% /gl/lvosb03pool05vd04

-- 
Dr. Stefan Solbrig
Universität Regensburg, Fakultät für Physik,
93040 Regensburg, Germany
Tel +49-941-943-2097

________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users