If you are creating a fresh volume, then it is your responsibility to have clean bricks.
I don't think gluster will give you a guarantee that it will be cleaned up.So you have to investigate if you have any previous data on that brick.
What I meant was that you have to find out the number of files you see on your mount point and the corresponding number of gfid are same.
If you create a file on mount point test-file, it will have a gfid xx-yy-jjjjjjjj and in .glusterfs the path would be xx/yy/xx-yy-jjjjjjjj
so one test-file = one xx/yy/xx-yy-jjjjj
In this way you have to find out if you have anything in .glusterfs which should not be. I would have started from the biggest entry I see in .glusterfs
like "154M /opt/lvmdir/c2/brick/.glusterfs/08"
From: "ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx>
To: "Ashish Pandey" <aspandey@xxxxxxxxxx>
Cc: "gluster-users" <gluster-users@xxxxxxxxxxx>, "Gluster Devel" <gluster-devel@xxxxxxxxxxx>
Sent: Friday, April 7, 2017 2:28:46 PM
Subject: Re: [Gluster-devel] Glusterfs meta data space consumption issue
Regards,
Abhishek
--
Regards
Abhishek Paliwal
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
To: "Ashish Pandey" <aspandey@xxxxxxxxxx>
Cc: "gluster-users" <gluster-users@xxxxxxxxxxx>, "Gluster Devel" <gluster-devel@xxxxxxxxxxx>
Sent: Friday, April 7, 2017 2:28:46 PM
Subject: Re: [Gluster-devel] Glusterfs meta data space consumption issue
Hi Ashish,
I don't think so that count of files on mount point and .glusterfs/ will remain same. Because I have created one file on the gluster mount poing but on .glusterfs/ it increased by 3 in numbers. Reason behind that is it creates .glusterfs/xx/xx/xxxxx....... which is two parent dir and one truefid file.On Fri, Apr 7, 2017 at 1:31 PM, Ashish Pandey <aspandey@xxxxxxxxxx> wrote:
Are you sure that the bricks which you used for this volume was not having any previous data?Find out the total number of files and directories on your mount point (recursively) and then see the number of entries on .glusterfs/From: "ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx>
To: "Gluster Devel" <gluster-devel@xxxxxxxxxxx>, "gluster-users" <gluster-users@xxxxxxxxxxx>
Sent: Friday, April 7, 2017 12:15:22 PM
Subject: Re: [Gluster-devel] Glusterfs meta data space consumption issueIs there any update ??On Thu, Apr 6, 2017 at 12:45 PM, ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx> wrote:Hi,We are currently experiencing a serious issue w.r.t volume space usage by glusterfs.In the below outputs, we can see that the size of the real data in /c (glusterfs volume) is nearly 1GB but the “.glusterfs” directory inside the brick (i.e., “/opt/lvmdir/c2/brick”) is consuming around 3.4 GBCan you tell us why the volume space is fully used by glusterfs even though the real data size is around 1GB itself ?# gluster peer statusNumber of Peers: 0### gluster volume statusStatus of volume: c_glusterfsGluster process TCP Port RDMA Port Online Pid------------------------------------------------------------------------------Brick 10.32.0.48:/opt/lvmdir/c2/brick 49152 0 Y 1507Task Status of Volume c_glusterfs------------------------------------------------------------------------------There are no active volume tasks# gluster volume infoVolume Name: c_glusterfsType: DistributeVolume ID: d83b1b8c-bc37-4615-bf4b-529f56968eccStatus: StartedNumber of Bricks: 1Transport-type: tcpBricks:Brick1: 10.32.0.48:/opt/lvmdir/c2/brickOptions Reconfigured:nfs.disable: onnetwork.ping-timeout: 4performance.readdir-ahead: on## ls -a /c/. .. .trashcan RNC_Exceptions configuration java license loadmodules loadmodules_norepl loadmodules_tftp logfiles lost+found node_id pm_data pmd public_html rnc security systemfiles tmp toplog.txt up usr xb# du -sh /c/.trashcan/8.0K /c/.trashcan/# du -sh /c/*11K /c/RNC_Exceptions5.5K /c/configuration62M /c/java138K /c/license609M /c/loadmodules90M /c/loadmodules_norepl246M /c/loadmodules_tftp4.1M /c/logfiles4.0K /c/lost+found5.0K /c/node_id8.0K /c/pm_data4.5K /c/pmd9.1M /c/public_html113K /c/rnc16K /c/security1.3M /c/systemfiles228K /c/tmp75K /c/toplog.txt1.5M /c/up4.0K /c/usr4.0K /c/xb# du -sh /c/1022M /c/# df -h /c/Filesystem Size Used Avail Use% Mounted on10.32.0.48:c_glusterfs 3.6G 3.4G 0 100% /mnt/c#### du -sh /opt/lvmdir/c2/brick/3.4G /opt/lvmdir/c2/brick/# du -sh /opt/lvmdir/c2/brick/*112K /opt/lvmdir/c2/brick/RNC_Exceptions36K /opt/lvmdir/c2/brick/configuration63M /opt/lvmdir/c2/brick/java176K /opt/lvmdir/c2/brick/license610M /opt/lvmdir/c2/brick/loadmodules95M /opt/lvmdir/c2/brick/loadmodules_norepl246M /opt/lvmdir/c2/brick/loadmodules_tftp4.2M /opt/lvmdir/c2/brick/logfiles8.0K /opt/lvmdir/c2/brick/lost+found24K /opt/lvmdir/c2/brick/node_id16K /opt/lvmdir/c2/brick/pm_data16K /opt/lvmdir/c2/brick/pmd9.2M /opt/lvmdir/c2/brick/public_html268K /opt/lvmdir/c2/brick/rnc80K /opt/lvmdir/c2/brick/security1.4M /opt/lvmdir/c2/brick/systemfiles252K /opt/lvmdir/c2/brick/tmp80K /opt/lvmdir/c2/brick/toplog.txt1.5M /opt/lvmdir/c2/brick/up8.0K /opt/lvmdir/c2/brick/usr8.0K /opt/lvmdir/c2/brick/xb# du -sh /opt/lvmdir/c2/brick/.glusterfs/3.4G /opt/lvmdir/c2/brick/.glusterfs/Below are the statics of the below command "du -sh /opt/lvmdir/c2/brick/.glusterfs/*"# du -sh /opt/lvmdir/c2/brick/.glusterfs/*14M /opt/lvmdir/c2/brick/.glusterfs/008.3M /opt/lvmdir/c2/brick/.glusterfs/0123M /opt/lvmdir/c2/brick/.glusterfs/0217M /opt/lvmdir/c2/brick/.glusterfs/037.1M /opt/lvmdir/c2/brick/.glusterfs/04336K /opt/lvmdir/c2/brick/.glusterfs/053.5M /opt/lvmdir/c2/brick/.glusterfs/061.7M /opt/lvmdir/c2/brick/.glusterfs/07154M /opt/lvmdir/c2/brick/.glusterfs/0814M /opt/lvmdir/c2/brick/.glusterfs/099.5M /opt/lvmdir/c2/brick/.glusterfs/0a5.5M /opt/lvmdir/c2/brick/.glusterfs/0b11M /opt/lvmdir/c2/brick/.glusterfs/0c764K /opt/lvmdir/c2/brick/.glusterfs/0d69M /opt/lvmdir/c2/brick/.glusterfs/0e3.7M /opt/lvmdir/c2/brick/.glusterfs/0f14M /opt/lvmdir/c2/brick/.glusterfs/101.8M /opt/lvmdir/c2/brick/.glusterfs/115.0M /opt/lvmdir/c2/brick/.glusterfs/1218M /opt/lvmdir/c2/brick/.glusterfs/137.8M /opt/lvmdir/c2/brick/.glusterfs/14151M /opt/lvmdir/c2/brick/.glusterfs/1515M /opt/lvmdir/c2/brick/.glusterfs/169.0M /opt/lvmdir/c2/brick/.glusterfs/1711M /opt/lvmdir/c2/brick/.glusterfs/183.1M /opt/lvmdir/c2/brick/.glusterfs/198.1M /opt/lvmdir/c2/brick/.glusterfs/1a82M /opt/lvmdir/c2/brick/.glusterfs/1b95M /opt/lvmdir/c2/brick/.glusterfs/1c2.1M /opt/lvmdir/c2/brick/.glusterfs/1d4.5M /opt/lvmdir/c2/brick/.glusterfs/1e9.7M /opt/lvmdir/c2/brick/.glusterfs/1f13M /opt/lvmdir/c2/brick/.glusterfs/2054M /opt/lvmdir/c2/brick/.glusterfs/219.4M /opt/lvmdir/c2/brick/.glusterfs/222.3M /opt/lvmdir/c2/brick/.glusterfs/2314M /opt/lvmdir/c2/brick/.glusterfs/2412M /opt/lvmdir/c2/brick/.glusterfs/2556M /opt/lvmdir/c2/brick/.glusterfs/265.6M /opt/lvmdir/c2/brick/.glusterfs/272.5M /opt/lvmdir/c2/brick/.glusterfs/2855M /opt/lvmdir/c2/brick/.glusterfs/2947M /opt/lvmdir/c2/brick/.glusterfs/2a6.1M /opt/lvmdir/c2/brick/.glusterfs/2b2.2M /opt/lvmdir/c2/brick/.glusterfs/2c4.7M /opt/lvmdir/c2/brick/.glusterfs/2d52M /opt/lvmdir/c2/brick/.glusterfs/2e16M /opt/lvmdir/c2/brick/.glusterfs/2f2.3M /opt/lvmdir/c2/brick/.glusterfs/307.6M /opt/lvmdir/c2/brick/.glusterfs/3112M /opt/lvmdir/c2/brick/.glusterfs/322.3M /opt/lvmdir/c2/brick/.glusterfs/334.5M /opt/lvmdir/c2/brick/.glusterfs/341.5M /opt/lvmdir/c2/brick/.glusterfs/3524M /opt/lvmdir/c2/brick/.glusterfs/366.8M /opt/lvmdir/c2/brick/.glusterfs/3710M /opt/lvmdir/c2/brick/.glusterfs/385.0M /opt/lvmdir/c2/brick/.glusterfs/3921M /opt/lvmdir/c2/brick/.glusterfs/3a9.6M /opt/lvmdir/c2/brick/.glusterfs/3b19M /opt/lvmdir/c2/brick/.glusterfs/3c8.0M /opt/lvmdir/c2/brick/.glusterfs/3d9.3M /opt/lvmdir/c2/brick/.glusterfs/3e9.9M /opt/lvmdir/c2/brick/.glusterfs/3f952K /opt/lvmdir/c2/brick/.glusterfs/4010M /opt/lvmdir/c2/brick/.glusterfs/4117M /opt/lvmdir/c2/brick/.glusterfs/4220M /opt/lvmdir/c2/brick/.glusterfs/431.7M /opt/lvmdir/c2/brick/.glusterfs/447.1M /opt/lvmdir/c2/brick/.glusterfs/4529M /opt/lvmdir/c2/brick/.glusterfs/464.5M /opt/lvmdir/c2/brick/.glusterfs/4717M /opt/lvmdir/c2/brick/.glusterfs/489.0M /opt/lvmdir/c2/brick/.glusterfs/494.4M /opt/lvmdir/c2/brick/.glusterfs/4a9.7M /opt/lvmdir/c2/brick/.glusterfs/4b3.1M /opt/lvmdir/c2/brick/.glusterfs/4c1.9M /opt/lvmdir/c2/brick/.glusterfs/4d9.7M /opt/lvmdir/c2/brick/.glusterfs/4e648K /opt/lvmdir/c2/brick/.glusterfs/4f8.0M /opt/lvmdir/c2/brick/.glusterfs/5014M /opt/lvmdir/c2/brick/.glusterfs/511.9M /opt/lvmdir/c2/brick/.glusterfs/521.6M /opt/lvmdir/c2/brick/.glusterfs/534.4M /opt/lvmdir/c2/brick/.glusterfs/543.3M /opt/lvmdir/c2/brick/.glusterfs/556.1M /opt/lvmdir/c2/brick/.glusterfs/5618M /opt/lvmdir/c2/brick/.glusterfs/572.8M /opt/lvmdir/c2/brick/.glusterfs/586.5M /opt/lvmdir/c2/brick/.glusterfs/592.2M /opt/lvmdir/c2/brick/.glusterfs/5a380K /opt/lvmdir/c2/brick/.glusterfs/5b12M /opt/lvmdir/c2/brick/.glusterfs/5c2.4M /opt/lvmdir/c2/brick/.glusterfs/5d22M /opt/lvmdir/c2/brick/.glusterfs/5e5.6M /opt/lvmdir/c2/brick/.glusterfs/5f15M /opt/lvmdir/c2/brick/.glusterfs/6015M /opt/lvmdir/c2/brick/.glusterfs/619.3M /opt/lvmdir/c2/brick/.glusterfs/621.1M /opt/lvmdir/c2/brick/.glusterfs/6357M /opt/lvmdir/c2/brick/.glusterfs/647.5M /opt/lvmdir/c2/brick/.glusterfs/6518M /opt/lvmdir/c2/brick/.glusterfs/664.6M /opt/lvmdir/c2/brick/.glusterfs/671.1M /opt/lvmdir/c2/brick/.glusterfs/689.3M /opt/lvmdir/c2/brick/.glusterfs/6912M /opt/lvmdir/c2/brick/.glusterfs/6a18M /opt/lvmdir/c2/brick/.glusterfs/6b8.5M /opt/lvmdir/c2/brick/.glusterfs/6c1.7M /opt/lvmdir/c2/brick/.glusterfs/6d7.8M /opt/lvmdir/c2/brick/.glusterfs/6e5.4M /opt/lvmdir/c2/brick/.glusterfs/6f13M /opt/lvmdir/c2/brick/.glusterfs/703.7M /opt/lvmdir/c2/brick/.glusterfs/716.4M /opt/lvmdir/c2/brick/.glusterfs/72596K /opt/lvmdir/c2/brick/.glusterfs/7313M /opt/lvmdir/c2/brick/.glusterfs/7410M /opt/lvmdir/c2/brick/.glusterfs/756.0M /opt/lvmdir/c2/brick/.glusterfs/76448K /opt/lvmdir/c2/brick/.glusterfs/77102M /opt/lvmdir/c2/brick/.glusterfs/783.3M /opt/lvmdir/c2/brick/.glusterfs/794.5M /opt/lvmdir/c2/brick/.glusterfs/7a5.2M /opt/lvmdir/c2/brick/.glusterfs/7b3.0M /opt/lvmdir/c2/brick/.glusterfs/7c6.7M /opt/lvmdir/c2/brick/.glusterfs/7d8.0M /opt/lvmdir/c2/brick/.glusterfs/7e10M /opt/lvmdir/c2/brick/.glusterfs/7f7.4M /opt/lvmdir/c2/brick/.glusterfs/807.0M /opt/lvmdir/c2/brick/.glusterfs/814.3M /opt/lvmdir/c2/brick/.glusterfs/825.7M /opt/lvmdir/c2/brick/.glusterfs/8312M /opt/lvmdir/c2/brick/.glusterfs/844.9M /opt/lvmdir/c2/brick/.glusterfs/857.9M /opt/lvmdir/c2/brick/.glusterfs/8614M /opt/lvmdir/c2/brick/.glusterfs/877.7M /opt/lvmdir/c2/brick/.glusterfs/8855M /opt/lvmdir/c2/brick/.glusterfs/8919M /opt/lvmdir/c2/brick/.glusterfs/8a7.4M /opt/lvmdir/c2/brick/.glusterfs/8b26M /opt/lvmdir/c2/brick/.glusterfs/8c2.0M /opt/lvmdir/c2/brick/.glusterfs/8d5.4M /opt/lvmdir/c2/brick/.glusterfs/8e13M /opt/lvmdir/c2/brick/.glusterfs/8f3.6M /opt/lvmdir/c2/brick/.glusterfs/9060M /opt/lvmdir/c2/brick/.glusterfs/9116M /opt/lvmdir/c2/brick/.glusterfs/9230M /opt/lvmdir/c2/brick/.glusterfs/932.6M /opt/lvmdir/c2/brick/.glusterfs/9452M /opt/lvmdir/c2/brick/.glusterfs/954.9M /opt/lvmdir/c2/brick/.glusterfs/961.7M /opt/lvmdir/c2/brick/.glusterfs/979.2M /opt/lvmdir/c2/brick/.glusterfs/986.2M /opt/lvmdir/c2/brick/.glusterfs/995.5M /opt/lvmdir/c2/brick/.glusterfs/9a4.2M /opt/lvmdir/c2/brick/.glusterfs/9b47M /opt/lvmdir/c2/brick/.glusterfs/9c2.2M /opt/lvmdir/c2/brick/.glusterfs/9d32M /opt/lvmdir/c2/brick/.glusterfs/9e13M /opt/lvmdir/c2/brick/.glusterfs/9f11M /opt/lvmdir/c2/brick/.glusterfs/a05.6M /opt/lvmdir/c2/brick/.glusterfs/a117M /opt/lvmdir/c2/brick/.glusterfs/a23.3M /opt/lvmdir/c2/brick/.glusterfs/a318M /opt/lvmdir/c2/brick/.glusterfs/a421M /opt/lvmdir/c2/brick/.glusterfs/a513M /opt/lvmdir/c2/brick/.glusterfs/a618M /opt/lvmdir/c2/brick/.glusterfs/a75.5M /opt/lvmdir/c2/brick/.glusterfs/a84.5M /opt/lvmdir/c2/brick/.glusterfs/a94.4M /opt/lvmdir/c2/brick/.glusterfs/aa21M /opt/lvmdir/c2/brick/.glusterfs/ab3.7M /opt/lvmdir/c2/brick/.glusterfs/ac13M /opt/lvmdir/c2/brick/.glusterfs/ad1.7M /opt/lvmdir/c2/brick/.glusterfs/ae9.1M /opt/lvmdir/c2/brick/.glusterfs/af50M /opt/lvmdir/c2/brick/.glusterfs/b03.9M /opt/lvmdir/c2/brick/.glusterfs/b18.8M /opt/lvmdir/c2/brick/.glusterfs/b264M /opt/lvmdir/c2/brick/.glusterfs/b313M /opt/lvmdir/c2/brick/.glusterfs/b47.7M /opt/lvmdir/c2/brick/.glusterfs/b53.0M /opt/lvmdir/c2/brick/.glusterfs/b67.8M /opt/lvmdir/c2/brick/.glusterfs/b72.8M /opt/lvmdir/c2/brick/.glusterfs/b811M /opt/lvmdir/c2/brick/.glusterfs/b94.6M /opt/lvmdir/c2/brick/.glusterfs/ba40M /opt/lvmdir/c2/brick/.glusterfs/bb17M /opt/lvmdir/c2/brick/.glusterfs/bc716K /opt/lvmdir/c2/brick/.glusterfs/bd2.2M /opt/lvmdir/c2/brick/.glusterfs/be14M /opt/lvmdir/c2/brick/.glusterfs/bf3.2M /opt/lvmdir/c2/brick/.glusterfs/c011M /opt/lvmdir/c2/brick/.glusterfs/c118M /opt/lvmdir/c2/brick/.glusterfs/c28.7M /opt/lvmdir/c2/brick/.glusterfs/c31.1M /opt/lvmdir/c2/brick/.glusterfs/c44.3M /opt/lvmdir/c2/brick/.glusterfs/c55.0M /opt/lvmdir/c2/brick/.glusterfs/c644M /opt/lvmdir/c2/brick/.glusterfs/c71.0M /opt/lvmdir/c2/brick/.glusterfs/c84.5M /opt/lvmdir/c2/brick/.glusterfs/c92.9M /opt/lvmdir/c2/brick/.glusterfs/ca8.4M /opt/lvmdir/c2/brick/.glusterfs/cb3.1M /opt/lvmdir/c2/brick/.glusterfs/cc14M /opt/lvmdir/c2/brick/.glusterfs/cd15M /opt/lvmdir/c2/brick/.glusterfs/ce4.7M /opt/lvmdir/c2/brick/.glusterfs/cf12K /opt/lvmdir/c2/brick/.glusterfs/changelogs36M /opt/lvmdir/c2/brick/.glusterfs/d08.8M /opt/lvmdir/c2/brick/.glusterfs/d16.8M /opt/lvmdir/c2/brick/.glusterfs/d24.4M /opt/lvmdir/c2/brick/.glusterfs/d32.3M /opt/lvmdir/c2/brick/.glusterfs/d41.1M /opt/lvmdir/c2/brick/.glusterfs/d516M /opt/lvmdir/c2/brick/.glusterfs/d648M /opt/lvmdir/c2/brick/.glusterfs/d76.8M /opt/lvmdir/c2/brick/.glusterfs/d820M /opt/lvmdir/c2/brick/.glusterfs/d95.7M /opt/lvmdir/c2/brick/.glusterfs/da3.9M /opt/lvmdir/c2/brick/.glusterfs/db788K /opt/lvmdir/c2/brick/.glusterfs/dc3.1M /opt/lvmdir/c2/brick/.glusterfs/dd3.5M /opt/lvmdir/c2/brick/.glusterfs/de45M /opt/lvmdir/c2/brick/.glusterfs/df5.2M /opt/lvmdir/c2/brick/.glusterfs/e04.2M /opt/lvmdir/c2/brick/.glusterfs/e19.2M /opt/lvmdir/c2/brick/.glusterfs/e22.4M /opt/lvmdir/c2/brick/.glusterfs/e311M /opt/lvmdir/c2/brick/.glusterfs/e461M /opt/lvmdir/c2/brick/.glusterfs/e512M /opt/lvmdir/c2/brick/.glusterfs/e61.1M /opt/lvmdir/c2/brick/.glusterfs/e75.9M /opt/lvmdir/c2/brick/.glusterfs/e85.3M /opt/lvmdir/c2/brick/.glusterfs/e91.6M /opt/lvmdir/c2/brick/.glusterfs/ea968K /opt/lvmdir/c2/brick/.glusterfs/eb9.5M /opt/lvmdir/c2/brick/.glusterfs/ec13M /opt/lvmdir/c2/brick/.glusterfs/ed15M /opt/lvmdir/c2/brick/.glusterfs/ee13M /opt/lvmdir/c2/brick/.glusterfs/ef4.1M /opt/lvmdir/c2/brick/.glusterfs/f013M /opt/lvmdir/c2/brick/.glusterfs/f132M /opt/lvmdir/c2/brick/.glusterfs/f25.8M /opt/lvmdir/c2/brick/.glusterfs/f35.2M /opt/lvmdir/c2/brick/.glusterfs/f45.9M /opt/lvmdir/c2/brick/.glusterfs/f514M /opt/lvmdir/c2/brick/.glusterfs/f62.3M /opt/lvmdir/c2/brick/.glusterfs/f72.5M /opt/lvmdir/c2/brick/.glusterfs/f812M /opt/lvmdir/c2/brick/.glusterfs/f9432K /opt/lvmdir/c2/brick/.glusterfs/fa15M /opt/lvmdir/c2/brick/.glusterfs/fb37M /opt/lvmdir/c2/brick/.glusterfs/fc3.1M /opt/lvmdir/c2/brick/.glusterfs/fd1.9M /opt/lvmdir/c2/brick/.glusterfs/fe4.1M /opt/lvmdir/c2/brick/.glusterfs/ff4.0K /opt/lvmdir/c2/brick/.glusterfs/health_check8.0K /opt/lvmdir/c2/brick/.glusterfs/indices4.0K /opt/lvmdir/c2/brick/.glusterfs/landfill#In which it is clear that all the backup/metadata files created w.r.t original files in gluster mount point taking to much spaces.Please let us know the reason of this problem.--
Regards
Abhishek Paliwal
--Regards
Abhishek Paliwal
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-devel
--
Abhishek Paliwal
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
