Re: Glusterfs meta data space consumption issue

ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx> · Fri, 7 Apr 2017 14:04:18 +0530

HI Ashish,

Even if there is a old data then it should be clear by gluster it self right? or you want to do it manually?

Regards,
Abhishek

On Fri, Apr 7, 2017 at 1:31 PM, Ashish Pandey <aspandey@xxxxxxxxxx> wrote:

Are you sure that the bricks which you used for this volume was not having any previous data?
Find out the total number of files and directories on your mount point (recursively) and then see the number of entries on .glusterfs/

From: "ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx>
To: "Gluster Devel" <gluster-devel@xxxxxxxxxxx>, "gluster-users" <gluster-users@xxxxxxxxxxx>
Sent: Friday, April 7, 2017 12:15:22 PM
Subject: Re:  Glusterfs meta data space consumption issue

Is there any update ??

On Thu, Apr 6, 2017 at 12:45 PM, ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx> wrote:
Hi,

We are currently experiencing a serious issue w.r.t volume space usage by glusterfs.

In the below outputs, we can see that the size of the real data in /c (glusterfs volume) is nearly 1GB but the “.glusterfs” directory inside the brick (i.e., “/opt/lvmdir/c2/brick”) is consuming around 3.4 GB

Can you tell us why the volume space is fully used by glusterfs even though the real data size is around 1GB itself ?

# gluster peer status
Number of Peers: 0
# 
# 
# gluster volume status
Status of volume: c_glusterfs
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.32.0.48:/opt/lvmdir/c2/brick       49152     0          Y       1507 

Task Status of Volume c_glusterfs
------------------------------------------------------------------------------
There are no active volume tasks

# gluster volume info

Volume Name: c_glusterfs
Type: Distribute
Volume ID: d83b1b8c-bc37-4615-bf4b-529f56968ecc
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: 10.32.0.48:/opt/lvmdir/c2/brick
Options Reconfigured:
nfs.disable: on
network.ping-timeout: 4
performance.readdir-ahead: on
#
# ls -a /c/ 
.  ..  .trashcan  RNC_Exceptions  configuration  java  license  loadmodules  loadmodules_norepl  loadmodules_tftp  logfiles  lost+found  node_id  pm_data  pmd  public_html  rnc  security  systemfiles  tmp  toplog.txt  up  usr  xb
# du -sh /c/.trashcan/
8.0K    /c/.trashcan/
# du -sh /c/*         
11K     /c/RNC_Exceptions
5.5K    /c/configuration
62M     /c/java
138K    /c/license
609M    /c/loadmodules
90M     /c/loadmodules_norepl
246M    /c/loadmodules_tftp
4.1M    /c/logfiles
4.0K    /c/lost+found
5.0K    /c/node_id
8.0K    /c/pm_data
4.5K    /c/pmd
9.1M    /c/public_html
113K    /c/rnc
16K     /c/security
1.3M    /c/systemfiles
228K    /c/tmp
75K     /c/toplog.txt
1.5M    /c/up
4.0K    /c/usr
4.0K    /c/xb
# du -sh /c/ 
1022M   /c/
# df -h /c/
Filesystem              Size  Used Avail Use% Mounted on
10.32.0.48:c_glusterfs  3.6G  3.4G     0 100% /mnt/c
# 
# 
# 
# du -sh /opt/lvmdir/c2/brick/
3.4G    /opt/lvmdir/c2/brick/
# du -sh /opt/lvmdir/c2/brick/*
112K    /opt/lvmdir/c2/brick/RNC_Exceptions
36K     /opt/lvmdir/c2/brick/configuration
63M     /opt/lvmdir/c2/brick/java
176K    /opt/lvmdir/c2/brick/license
610M    /opt/lvmdir/c2/brick/loadmodules
95M     /opt/lvmdir/c2/brick/loadmodules_norepl
246M    /opt/lvmdir/c2/brick/loadmodules_tftp
4.2M    /opt/lvmdir/c2/brick/logfiles
8.0K    /opt/lvmdir/c2/brick/lost+found
24K     /opt/lvmdir/c2/brick/node_id
16K     /opt/lvmdir/c2/brick/pm_data
16K     /opt/lvmdir/c2/brick/pmd
9.2M    /opt/lvmdir/c2/brick/public_html
268K    /opt/lvmdir/c2/brick/rnc
80K     /opt/lvmdir/c2/brick/security
1.4M    /opt/lvmdir/c2/brick/systemfiles
252K    /opt/lvmdir/c2/brick/tmp
80K     /opt/lvmdir/c2/brick/toplog.txt
1.5M    /opt/lvmdir/c2/brick/up
8.0K    /opt/lvmdir/c2/brick/usr
8.0K    /opt/lvmdir/c2/brick/xb
# du -sh /opt/lvmdir/c2/brick/.glusterfs/
3.4G    /opt/lvmdir/c2/brick/.glusterfs/

Below are the statics of the below command "du -sh /opt/lvmdir/c2/brick/.glusterfs/*"

# du -sh /opt/lvmdir/c2/brick/.glusterfs/*
14M     /opt/lvmdir/c2/brick/.glusterfs/00
8.3M    /opt/lvmdir/c2/brick/.glusterfs/01
23M     /opt/lvmdir/c2/brick/.glusterfs/02
17M     /opt/lvmdir/c2/brick/.glusterfs/03
7.1M    /opt/lvmdir/c2/brick/.glusterfs/04
336K    /opt/lvmdir/c2/brick/.glusterfs/05
3.5M    /opt/lvmdir/c2/brick/.glusterfs/06
1.7M    /opt/lvmdir/c2/brick/.glusterfs/07
154M    /opt/lvmdir/c2/brick/.glusterfs/08
14M     /opt/lvmdir/c2/brick/.glusterfs/09
9.5M    /opt/lvmdir/c2/brick/.glusterfs/0a
5.5M    /opt/lvmdir/c2/brick/.glusterfs/0b
11M     /opt/lvmdir/c2/brick/.glusterfs/0c
764K    /opt/lvmdir/c2/brick/.glusterfs/0d
69M     /opt/lvmdir/c2/brick/.glusterfs/0e
3.7M    /opt/lvmdir/c2/brick/.glusterfs/0f
14M     /opt/lvmdir/c2/brick/.glusterfs/10
1.8M    /opt/lvmdir/c2/brick/.glusterfs/11
5.0M    /opt/lvmdir/c2/brick/.glusterfs/12
18M     /opt/lvmdir/c2/brick/.glusterfs/13
7.8M    /opt/lvmdir/c2/brick/.glusterfs/14
151M    /opt/lvmdir/c2/brick/.glusterfs/15
15M     /opt/lvmdir/c2/brick/.glusterfs/16
9.0M    /opt/lvmdir/c2/brick/.glusterfs/17
11M     /opt/lvmdir/c2/brick/.glusterfs/18
3.1M    /opt/lvmdir/c2/brick/.glusterfs/19
8.1M    /opt/lvmdir/c2/brick/.glusterfs/1a
82M     /opt/lvmdir/c2/brick/.glusterfs/1b
95M     /opt/lvmdir/c2/brick/.glusterfs/1c
2.1M    /opt/lvmdir/c2/brick/.glusterfs/1d
4.5M    /opt/lvmdir/c2/brick/.glusterfs/1e
9.7M    /opt/lvmdir/c2/brick/.glusterfs/1f
13M     /opt/lvmdir/c2/brick/.glusterfs/20
54M     /opt/lvmdir/c2/brick/.glusterfs/21
9.4M    /opt/lvmdir/c2/brick/.glusterfs/22
2.3M    /opt/lvmdir/c2/brick/.glusterfs/23
14M     /opt/lvmdir/c2/brick/.glusterfs/24
12M     /opt/lvmdir/c2/brick/.glusterfs/25
56M     /opt/lvmdir/c2/brick/.glusterfs/26
5.6M    /opt/lvmdir/c2/brick/.glusterfs/27
2.5M    /opt/lvmdir/c2/brick/.glusterfs/28
55M     /opt/lvmdir/c2/brick/.glusterfs/29
47M     /opt/lvmdir/c2/brick/.glusterfs/2a
6.1M    /opt/lvmdir/c2/brick/.glusterfs/2b
2.2M    /opt/lvmdir/c2/brick/.glusterfs/2c
4.7M    /opt/lvmdir/c2/brick/.glusterfs/2d
52M     /opt/lvmdir/c2/brick/.glusterfs/2e
16M     /opt/lvmdir/c2/brick/.glusterfs/2f
2.3M    /opt/lvmdir/c2/brick/.glusterfs/30
7.6M    /opt/lvmdir/c2/brick/.glusterfs/31
12M     /opt/lvmdir/c2/brick/.glusterfs/32
2.3M    /opt/lvmdir/c2/brick/.glusterfs/33
4.5M    /opt/lvmdir/c2/brick/.glusterfs/34
1.5M    /opt/lvmdir/c2/brick/.glusterfs/35
24M     /opt/lvmdir/c2/brick/.glusterfs/36
6.8M    /opt/lvmdir/c2/brick/.glusterfs/37
10M     /opt/lvmdir/c2/brick/.glusterfs/38
5.0M    /opt/lvmdir/c2/brick/.glusterfs/39
21M     /opt/lvmdir/c2/brick/.glusterfs/3a
9.6M    /opt/lvmdir/c2/brick/.glusterfs/3b
19M     /opt/lvmdir/c2/brick/.glusterfs/3c
8.0M    /opt/lvmdir/c2/brick/.glusterfs/3d
9.3M    /opt/lvmdir/c2/brick/.glusterfs/3e
9.9M    /opt/lvmdir/c2/brick/.glusterfs/3f
952K    /opt/lvmdir/c2/brick/.glusterfs/40
10M     /opt/lvmdir/c2/brick/.glusterfs/41
17M     /opt/lvmdir/c2/brick/.glusterfs/42
20M     /opt/lvmdir/c2/brick/.glusterfs/43
1.7M    /opt/lvmdir/c2/brick/.glusterfs/44
7.1M    /opt/lvmdir/c2/brick/.glusterfs/45
29M     /opt/lvmdir/c2/brick/.glusterfs/46
4.5M    /opt/lvmdir/c2/brick/.glusterfs/47
17M     /opt/lvmdir/c2/brick/.glusterfs/48
9.0M    /opt/lvmdir/c2/brick/.glusterfs/49
4.4M    /opt/lvmdir/c2/brick/.glusterfs/4a
9.7M    /opt/lvmdir/c2/brick/.glusterfs/4b
3.1M    /opt/lvmdir/c2/brick/.glusterfs/4c
1.9M    /opt/lvmdir/c2/brick/.glusterfs/4d
9.7M    /opt/lvmdir/c2/brick/.glusterfs/4e
648K    /opt/lvmdir/c2/brick/.glusterfs/4f
8.0M    /opt/lvmdir/c2/brick/.glusterfs/50
14M     /opt/lvmdir/c2/brick/.glusterfs/51
1.9M    /opt/lvmdir/c2/brick/.glusterfs/52
1.6M    /opt/lvmdir/c2/brick/.glusterfs/53
4.4M    /opt/lvmdir/c2/brick/.glusterfs/54
3.3M    /opt/lvmdir/c2/brick/.glusterfs/55
6.1M    /opt/lvmdir/c2/brick/.glusterfs/56
18M     /opt/lvmdir/c2/brick/.glusterfs/57
2.8M    /opt/lvmdir/c2/brick/.glusterfs/58
6.5M    /opt/lvmdir/c2/brick/.glusterfs/59
2.2M    /opt/lvmdir/c2/brick/.glusterfs/5a
380K    /opt/lvmdir/c2/brick/.glusterfs/5b
12M     /opt/lvmdir/c2/brick/.glusterfs/5c
2.4M    /opt/lvmdir/c2/brick/.glusterfs/5d
22M     /opt/lvmdir/c2/brick/.glusterfs/5e
5.6M    /opt/lvmdir/c2/brick/.glusterfs/5f
15M     /opt/lvmdir/c2/brick/.glusterfs/60
15M     /opt/lvmdir/c2/brick/.glusterfs/61
9.3M    /opt/lvmdir/c2/brick/.glusterfs/62
1.1M    /opt/lvmdir/c2/brick/.glusterfs/63
57M     /opt/lvmdir/c2/brick/.glusterfs/64
7.5M    /opt/lvmdir/c2/brick/.glusterfs/65
18M     /opt/lvmdir/c2/brick/.glusterfs/66
4.6M    /opt/lvmdir/c2/brick/.glusterfs/67
1.1M    /opt/lvmdir/c2/brick/.glusterfs/68
9.3M    /opt/lvmdir/c2/brick/.glusterfs/69
12M     /opt/lvmdir/c2/brick/.glusterfs/6a
18M     /opt/lvmdir/c2/brick/.glusterfs/6b
8.5M    /opt/lvmdir/c2/brick/.glusterfs/6c
1.7M    /opt/lvmdir/c2/brick/.glusterfs/6d
7.8M    /opt/lvmdir/c2/brick/.glusterfs/6e
5.4M    /opt/lvmdir/c2/brick/.glusterfs/6f
13M     /opt/lvmdir/c2/brick/.glusterfs/70
3.7M    /opt/lvmdir/c2/brick/.glusterfs/71
6.4M    /opt/lvmdir/c2/brick/.glusterfs/72
596K    /opt/lvmdir/c2/brick/.glusterfs/73
13M     /opt/lvmdir/c2/brick/.glusterfs/74
10M     /opt/lvmdir/c2/brick/.glusterfs/75
6.0M    /opt/lvmdir/c2/brick/.glusterfs/76
448K    /opt/lvmdir/c2/brick/.glusterfs/77
102M    /opt/lvmdir/c2/brick/.glusterfs/78
3.3M    /opt/lvmdir/c2/brick/.glusterfs/79
4.5M    /opt/lvmdir/c2/brick/.glusterfs/7a
5.2M    /opt/lvmdir/c2/brick/.glusterfs/7b
3.0M    /opt/lvmdir/c2/brick/.glusterfs/7c
6.7M    /opt/lvmdir/c2/brick/.glusterfs/7d
8.0M    /opt/lvmdir/c2/brick/.glusterfs/7e
10M     /opt/lvmdir/c2/brick/.glusterfs/7f
7.4M    /opt/lvmdir/c2/brick/.glusterfs/80
7.0M    /opt/lvmdir/c2/brick/.glusterfs/81
4.3M    /opt/lvmdir/c2/brick/.glusterfs/82
5.7M    /opt/lvmdir/c2/brick/.glusterfs/83
12M     /opt/lvmdir/c2/brick/.glusterfs/84
4.9M    /opt/lvmdir/c2/brick/.glusterfs/85
7.9M    /opt/lvmdir/c2/brick/.glusterfs/86
14M     /opt/lvmdir/c2/brick/.glusterfs/87
7.7M    /opt/lvmdir/c2/brick/.glusterfs/88
55M     /opt/lvmdir/c2/brick/.glusterfs/89
19M     /opt/lvmdir/c2/brick/.glusterfs/8a
7.4M    /opt/lvmdir/c2/brick/.glusterfs/8b
26M     /opt/lvmdir/c2/brick/.glusterfs/8c
2.0M    /opt/lvmdir/c2/brick/.glusterfs/8d
5.4M    /opt/lvmdir/c2/brick/.glusterfs/8e
13M     /opt/lvmdir/c2/brick/.glusterfs/8f
3.6M    /opt/lvmdir/c2/brick/.glusterfs/90
60M     /opt/lvmdir/c2/brick/.glusterfs/91
16M     /opt/lvmdir/c2/brick/.glusterfs/92
30M     /opt/lvmdir/c2/brick/.glusterfs/93
2.6M    /opt/lvmdir/c2/brick/.glusterfs/94
52M     /opt/lvmdir/c2/brick/.glusterfs/95
4.9M    /opt/lvmdir/c2/brick/.glusterfs/96
1.7M    /opt/lvmdir/c2/brick/.glusterfs/97
9.2M    /opt/lvmdir/c2/brick/.glusterfs/98
6.2M    /opt/lvmdir/c2/brick/.glusterfs/99
5.5M    /opt/lvmdir/c2/brick/.glusterfs/9a
4.2M    /opt/lvmdir/c2/brick/.glusterfs/9b
47M     /opt/lvmdir/c2/brick/.glusterfs/9c
2.2M    /opt/lvmdir/c2/brick/.glusterfs/9d
32M     /opt/lvmdir/c2/brick/.glusterfs/9e
13M     /opt/lvmdir/c2/brick/.glusterfs/9f
11M     /opt/lvmdir/c2/brick/.glusterfs/a0
5.6M    /opt/lvmdir/c2/brick/.glusterfs/a1
17M     /opt/lvmdir/c2/brick/.glusterfs/a2
3.3M    /opt/lvmdir/c2/brick/.glusterfs/a3
18M     /opt/lvmdir/c2/brick/.glusterfs/a4
21M     /opt/lvmdir/c2/brick/.glusterfs/a5
13M     /opt/lvmdir/c2/brick/.glusterfs/a6
18M     /opt/lvmdir/c2/brick/.glusterfs/a7
5.5M    /opt/lvmdir/c2/brick/.glusterfs/a8
4.5M    /opt/lvmdir/c2/brick/.glusterfs/a9
4.4M    /opt/lvmdir/c2/brick/.glusterfs/aa
21M     /opt/lvmdir/c2/brick/.glusterfs/ab
3.7M    /opt/lvmdir/c2/brick/.glusterfs/ac
13M     /opt/lvmdir/c2/brick/.glusterfs/ad
1.7M    /opt/lvmdir/c2/brick/.glusterfs/ae
9.1M    /opt/lvmdir/c2/brick/.glusterfs/af
50M     /opt/lvmdir/c2/brick/.glusterfs/b0
3.9M    /opt/lvmdir/c2/brick/.glusterfs/b1
8.8M    /opt/lvmdir/c2/brick/.glusterfs/b2
64M     /opt/lvmdir/c2/brick/.glusterfs/b3
13M     /opt/lvmdir/c2/brick/.glusterfs/b4
7.7M    /opt/lvmdir/c2/brick/.glusterfs/b5
3.0M    /opt/lvmdir/c2/brick/.glusterfs/b6
7.8M    /opt/lvmdir/c2/brick/.glusterfs/b7
2.8M    /opt/lvmdir/c2/brick/.glusterfs/b8
11M     /opt/lvmdir/c2/brick/.glusterfs/b9
4.6M    /opt/lvmdir/c2/brick/.glusterfs/ba
40M     /opt/lvmdir/c2/brick/.glusterfs/bb
17M     /opt/lvmdir/c2/brick/.glusterfs/bc
716K    /opt/lvmdir/c2/brick/.glusterfs/bd
2.2M    /opt/lvmdir/c2/brick/.glusterfs/be
14M     /opt/lvmdir/c2/brick/.glusterfs/bf
3.2M    /opt/lvmdir/c2/brick/.glusterfs/c0
11M     /opt/lvmdir/c2/brick/.glusterfs/c1
18M     /opt/lvmdir/c2/brick/.glusterfs/c2
8.7M    /opt/lvmdir/c2/brick/.glusterfs/c3
1.1M    /opt/lvmdir/c2/brick/.glusterfs/c4
4.3M    /opt/lvmdir/c2/brick/.glusterfs/c5
5.0M    /opt/lvmdir/c2/brick/.glusterfs/c6
44M     /opt/lvmdir/c2/brick/.glusterfs/c7
1.0M    /opt/lvmdir/c2/brick/.glusterfs/c8
4.5M    /opt/lvmdir/c2/brick/.glusterfs/c9
2.9M    /opt/lvmdir/c2/brick/.glusterfs/ca
8.4M    /opt/lvmdir/c2/brick/.glusterfs/cb
3.1M    /opt/lvmdir/c2/brick/.glusterfs/cc
14M     /opt/lvmdir/c2/brick/.glusterfs/cd
15M     /opt/lvmdir/c2/brick/.glusterfs/ce
4.7M    /opt/lvmdir/c2/brick/.glusterfs/cf
12K     /opt/lvmdir/c2/brick/.glusterfs/changelogs
36M     /opt/lvmdir/c2/brick/.glusterfs/d0
8.8M    /opt/lvmdir/c2/brick/.glusterfs/d1
6.8M    /opt/lvmdir/c2/brick/.glusterfs/d2
4.4M    /opt/lvmdir/c2/brick/.glusterfs/d3
2.3M    /opt/lvmdir/c2/brick/.glusterfs/d4
1.1M    /opt/lvmdir/c2/brick/.glusterfs/d5
16M     /opt/lvmdir/c2/brick/.glusterfs/d6
48M     /opt/lvmdir/c2/brick/.glusterfs/d7
6.8M    /opt/lvmdir/c2/brick/.glusterfs/d8
20M     /opt/lvmdir/c2/brick/.glusterfs/d9
5.7M    /opt/lvmdir/c2/brick/.glusterfs/da
3.9M    /opt/lvmdir/c2/brick/.glusterfs/db
788K    /opt/lvmdir/c2/brick/.glusterfs/dc
3.1M    /opt/lvmdir/c2/brick/.glusterfs/dd
3.5M    /opt/lvmdir/c2/brick/.glusterfs/de
45M     /opt/lvmdir/c2/brick/.glusterfs/df
5.2M    /opt/lvmdir/c2/brick/.glusterfs/e0
4.2M    /opt/lvmdir/c2/brick/.glusterfs/e1
9.2M    /opt/lvmdir/c2/brick/.glusterfs/e2
2.4M    /opt/lvmdir/c2/brick/.glusterfs/e3
11M     /opt/lvmdir/c2/brick/.glusterfs/e4
61M     /opt/lvmdir/c2/brick/.glusterfs/e5
12M     /opt/lvmdir/c2/brick/.glusterfs/e6
1.1M    /opt/lvmdir/c2/brick/.glusterfs/e7
5.9M    /opt/lvmdir/c2/brick/.glusterfs/e8
5.3M    /opt/lvmdir/c2/brick/.glusterfs/e9
1.6M    /opt/lvmdir/c2/brick/.glusterfs/ea
968K    /opt/lvmdir/c2/brick/.glusterfs/eb
9.5M    /opt/lvmdir/c2/brick/.glusterfs/ec
13M     /opt/lvmdir/c2/brick/.glusterfs/ed
15M     /opt/lvmdir/c2/brick/.glusterfs/ee
13M     /opt/lvmdir/c2/brick/.glusterfs/ef
4.1M    /opt/lvmdir/c2/brick/.glusterfs/f0
13M     /opt/lvmdir/c2/brick/.glusterfs/f1
32M     /opt/lvmdir/c2/brick/.glusterfs/f2
5.8M    /opt/lvmdir/c2/brick/.glusterfs/f3
5.2M    /opt/lvmdir/c2/brick/.glusterfs/f4
5.9M    /opt/lvmdir/c2/brick/.glusterfs/f5
14M     /opt/lvmdir/c2/brick/.glusterfs/f6
2.3M    /opt/lvmdir/c2/brick/.glusterfs/f7
2.5M    /opt/lvmdir/c2/brick/.glusterfs/f8
12M     /opt/lvmdir/c2/brick/.glusterfs/f9
432K    /opt/lvmdir/c2/brick/.glusterfs/fa
15M     /opt/lvmdir/c2/brick/.glusterfs/fb
37M     /opt/lvmdir/c2/brick/.glusterfs/fc
3.1M    /opt/lvmdir/c2/brick/.glusterfs/fd
1.9M    /opt/lvmdir/c2/brick/.glusterfs/fe
4.1M    /opt/lvmdir/c2/brick/.glusterfs/ff
4.0K    /opt/lvmdir/c2/brick/.glusterfs/health_check
8.0K    /opt/lvmdir/c2/brick/.glusterfs/indices
4.0K    /opt/lvmdir/c2/brick/.glusterfs/landfill
#

In which it is clear that all the backup/metadata files created w.r.t original files in gluster mount point taking to much spaces.

Please let us know the reason of this problem.
-- 

Regards
 Abhishek Paliwal

-- 

Regards
 Abhishek Paliwal

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-devel

-- 

Regards

Abhishek Paliwal

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-devel