Re: Glusterfs meta data space consumption issue

ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx> · Fri, 7 Apr 2017 12:15:22 +0530

Is there any update ??

On Thu, Apr 6, 2017 at 12:45 PM, ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx> wrote:
Hi,
 We are currently experiencing a serious issue w.r.t volume space usage by glusterfs.

In
 the below outputs, we can see that the size of the real data in /c 
(glusterfs volume) is nearly 1GB but the “.glusterfs” directory inside 
the
brick (i.e., “/opt/lvmdir/c2/brick”) is consuming around 3.4 GB

Can you tell us why the volume space is fully used by glusterfs even though the real data size is around 1GB itself ?

# gluster peer status
Number of Peers: 0
# 
# 
# gluster volume status
Status of volume: c_glusterfs
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.32.0.48:/opt/lvmdir/c2/brick       49152     0          Y       1507 

Task Status of Volume c_glusterfs
------------------------------------------------------------------------------
There are no active volume tasks

# gluster volume info

Volume Name: c_glusterfs
Type: Distribute
Volume ID: d83b1b8c-bc37-4615-bf4b-529f56968ecc
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: 10.32.0.48:/opt/lvmdir/c2/brick
Options Reconfigured:
nfs.disable: on
network.ping-timeout: 4
performance.readdir-ahead: on
#
# ls -a /c/ 
. 
 ..  .trashcan  RNC_Exceptions  configuration  java  license  
loadmodules  loadmodules_norepl  loadmodules_tftp  logfiles  lost+found 
 node_id 
pm_data  pmd  public_html  rnc  security  systemfiles  tmp  toplog.txt  
up  usr  xb
# du -sh /c/.trashcan/
8.0K    /c/.trashcan/
# du -sh /c/*         
11K     /c/RNC_Exceptions
5.5K    /c/configuration
62M     /c/java
138K    /c/license
609M    /c/loadmodules
90M     /c/loadmodules_norepl
246M    /c/loadmodules_tftp
4.1M    /c/logfiles
4.0K    /c/lost+found
5.0K    /c/node_id
8.0K    /c/pm_data
4.5K    /c/pmd
9.1M    /c/public_html
113K    /c/rnc
16K     /c/security
1.3M    /c/systemfiles
228K    /c/tmp
75K     /c/toplog.txt
1.5M    /c/up
4.0K    /c/usr
4.0K    /c/xb
# du -sh /c/ 
1022M   /c/
# df -h /c/
Filesystem              Size  Used Avail Use% Mounted on
10.32.0.48:c_glusterfs  3.6G  3.4G     0 100% /mnt/c
# 
# 
# 
# du -sh /opt/lvmdir/c2/brick/
3.4G    /opt/lvmdir/c2/brick/
# du -sh /opt/lvmdir/c2/brick/*
112K    /opt/lvmdir/c2/brick/RNC_Exceptions
36K     /opt/lvmdir/c2/brick/configuration
63M     /opt/lvmdir/c2/brick/java
176K    /opt/lvmdir/c2/brick/license
610M    /opt/lvmdir/c2/brick/loadmodules
95M     /opt/lvmdir/c2/brick/loadmodules_norepl
246M    /opt/lvmdir/c2/brick/loadmodules_tftp
4.2M    /opt/lvmdir/c2/brick/logfiles
8.0K    /opt/lvmdir/c2/brick/lost+found
24K     /opt/lvmdir/c2/brick/node_id
16K     /opt/lvmdir/c2/brick/pm_data
16K     /opt/lvmdir/c2/brick/pmd
9.2M    /opt/lvmdir/c2/brick/public_html
268K    /opt/lvmdir/c2/brick/rnc
80K     /opt/lvmdir/c2/brick/security
1.4M    /opt/lvmdir/c2/brick/systemfiles
252K    /opt/lvmdir/c2/brick/tmp
80K     /opt/lvmdir/c2/brick/toplog.txt
1.5M    /opt/lvmdir/c2/brick/up
8.0K    /opt/lvmdir/c2/brick/usr
8.0K    /opt/lvmdir/c2/brick/xb
# du -sh /opt/lvmdir/c2/brick/.glusterfs/
3.4G    /opt/lvmdir/c2/brick/.glusterfs/
Below are the statics of the below command "du -sh /opt/lvmdir/c2/brick/.glusterfs/*"

# du -sh /opt/lvmdir/c2/brick/.glusterfs/*
14M     /opt/lvmdir/c2/brick/.glusterfs/00
8.3M    /opt/lvmdir/c2/brick/.glusterfs/01
23M     /opt/lvmdir/c2/brick/.glusterfs/02
17M     /opt/lvmdir/c2/brick/.glusterfs/03
7.1M    /opt/lvmdir/c2/brick/.glusterfs/04
336K    /opt/lvmdir/c2/brick/.glusterfs/05
3.5M    /opt/lvmdir/c2/brick/.glusterfs/06
1.7M    /opt/lvmdir/c2/brick/.glusterfs/07
154M    /opt/lvmdir/c2/brick/.glusterfs/08
14M     /opt/lvmdir/c2/brick/.glusterfs/09
9.5M    /opt/lvmdir/c2/brick/.glusterfs/0a
5.5M    /opt/lvmdir/c2/brick/.glusterfs/0b
11M     /opt/lvmdir/c2/brick/.glusterfs/0c
764K    /opt/lvmdir/c2/brick/.glusterfs/0d
69M     /opt/lvmdir/c2/brick/.glusterfs/0e
3.7M    /opt/lvmdir/c2/brick/.glusterfs/0f
14M     /opt/lvmdir/c2/brick/.glusterfs/10
1.8M    /opt/lvmdir/c2/brick/.glusterfs/11
5.0M    /opt/lvmdir/c2/brick/.glusterfs/12
18M     /opt/lvmdir/c2/brick/.glusterfs/13
7.8M    /opt/lvmdir/c2/brick/.glusterfs/14
151M    /opt/lvmdir/c2/brick/.glusterfs/15
15M     /opt/lvmdir/c2/brick/.glusterfs/16
9.0M    /opt/lvmdir/c2/brick/.glusterfs/17
11M     /opt/lvmdir/c2/brick/.glusterfs/18
3.1M    /opt/lvmdir/c2/brick/.glusterfs/19
8.1M    /opt/lvmdir/c2/brick/.glusterfs/1a
82M     /opt/lvmdir/c2/brick/.glusterfs/1b
95M     /opt/lvmdir/c2/brick/.glusterfs/1c
2.1M    /opt/lvmdir/c2/brick/.glusterfs/1d
4.5M    /opt/lvmdir/c2/brick/.glusterfs/1e
9.7M    /opt/lvmdir/c2/brick/.glusterfs/1f
13M     /opt/lvmdir/c2/brick/.glusterfs/20
54M     /opt/lvmdir/c2/brick/.glusterfs/21
9.4M    /opt/lvmdir/c2/brick/.glusterfs/22
2.3M    /opt/lvmdir/c2/brick/.glusterfs/23
14M     /opt/lvmdir/c2/brick/.glusterfs/24
12M     /opt/lvmdir/c2/brick/.glusterfs/25
56M     /opt/lvmdir/c2/brick/.glusterfs/26
5.6M    /opt/lvmdir/c2/brick/.glusterfs/27
2.5M    /opt/lvmdir/c2/brick/.glusterfs/28
55M     /opt/lvmdir/c2/brick/.glusterfs/29
47M     /opt/lvmdir/c2/brick/.glusterfs/2a
6.1M    /opt/lvmdir/c2/brick/.glusterfs/2b
2.2M    /opt/lvmdir/c2/brick/.glusterfs/2c
4.7M    /opt/lvmdir/c2/brick/.glusterfs/2d
52M     /opt/lvmdir/c2/brick/.glusterfs/2e
16M     /opt/lvmdir/c2/brick/.glusterfs/2f
2.3M    /opt/lvmdir/c2/brick/.glusterfs/30
7.6M    /opt/lvmdir/c2/brick/.glusterfs/31
12M     /opt/lvmdir/c2/brick/.glusterfs/32
2.3M    /opt/lvmdir/c2/brick/.glusterfs/33
4.5M    /opt/lvmdir/c2/brick/.glusterfs/34
1.5M    /opt/lvmdir/c2/brick/.glusterfs/35
24M     /opt/lvmdir/c2/brick/.glusterfs/36
6.8M    /opt/lvmdir/c2/brick/.glusterfs/37
10M     /opt/lvmdir/c2/brick/.glusterfs/38
5.0M    /opt/lvmdir/c2/brick/.glusterfs/39
21M     /opt/lvmdir/c2/brick/.glusterfs/3a
9.6M    /opt/lvmdir/c2/brick/.glusterfs/3b
19M     /opt/lvmdir/c2/brick/.glusterfs/3c
8.0M    /opt/lvmdir/c2/brick/.glusterfs/3d
9.3M    /opt/lvmdir/c2/brick/.glusterfs/3e
9.9M    /opt/lvmdir/c2/brick/.glusterfs/3f
952K    /opt/lvmdir/c2/brick/.glusterfs/40
10M     /opt/lvmdir/c2/brick/.glusterfs/41
17M     /opt/lvmdir/c2/brick/.glusterfs/42
20M     /opt/lvmdir/c2/brick/.glusterfs/43
1.7M    /opt/lvmdir/c2/brick/.glusterfs/44
7.1M    /opt/lvmdir/c2/brick/.glusterfs/45
29M     /opt/lvmdir/c2/brick/.glusterfs/46
4.5M    /opt/lvmdir/c2/brick/.glusterfs/47
17M     /opt/lvmdir/c2/brick/.glusterfs/48
9.0M    /opt/lvmdir/c2/brick/.glusterfs/49
4.4M    /opt/lvmdir/c2/brick/.glusterfs/4a
9.7M    /opt/lvmdir/c2/brick/.glusterfs/4b
3.1M    /opt/lvmdir/c2/brick/.glusterfs/4c
1.9M    /opt/lvmdir/c2/brick/.glusterfs/4d
9.7M    /opt/lvmdir/c2/brick/.glusterfs/4e
648K    /opt/lvmdir/c2/brick/.glusterfs/4f
8.0M    /opt/lvmdir/c2/brick/.glusterfs/50
14M     /opt/lvmdir/c2/brick/.glusterfs/51
1.9M    /opt/lvmdir/c2/brick/.glusterfs/52
1.6M    /opt/lvmdir/c2/brick/.glusterfs/53
4.4M    /opt/lvmdir/c2/brick/.glusterfs/54
3.3M    /opt/lvmdir/c2/brick/.glusterfs/55
6.1M    /opt/lvmdir/c2/brick/.glusterfs/56
18M     /opt/lvmdir/c2/brick/.glusterfs/57
2.8M    /opt/lvmdir/c2/brick/.glusterfs/58
6.5M    /opt/lvmdir/c2/brick/.glusterfs/59
2.2M    /opt/lvmdir/c2/brick/.glusterfs/5a
380K    /opt/lvmdir/c2/brick/.glusterfs/5b
12M     /opt/lvmdir/c2/brick/.glusterfs/5c
2.4M    /opt/lvmdir/c2/brick/.glusterfs/5d
22M     /opt/lvmdir/c2/brick/.glusterfs/5e
5.6M    /opt/lvmdir/c2/brick/.glusterfs/5f
15M     /opt/lvmdir/c2/brick/.glusterfs/60
15M     /opt/lvmdir/c2/brick/.glusterfs/61
9.3M    /opt/lvmdir/c2/brick/.glusterfs/62
1.1M    /opt/lvmdir/c2/brick/.glusterfs/63
57M     /opt/lvmdir/c2/brick/.glusterfs/64
7.5M    /opt/lvmdir/c2/brick/.glusterfs/65
18M     /opt/lvmdir/c2/brick/.glusterfs/66
4.6M    /opt/lvmdir/c2/brick/.glusterfs/67
1.1M    /opt/lvmdir/c2/brick/.glusterfs/68
9.3M    /opt/lvmdir/c2/brick/.glusterfs/69
12M     /opt/lvmdir/c2/brick/.glusterfs/6a
18M     /opt/lvmdir/c2/brick/.glusterfs/6b
8.5M    /opt/lvmdir/c2/brick/.glusterfs/6c
1.7M    /opt/lvmdir/c2/brick/.glusterfs/6d
7.8M    /opt/lvmdir/c2/brick/.glusterfs/6e
5.4M    /opt/lvmdir/c2/brick/.glusterfs/6f
13M     /opt/lvmdir/c2/brick/.glusterfs/70
3.7M    /opt/lvmdir/c2/brick/.glusterfs/71
6.4M    /opt/lvmdir/c2/brick/.glusterfs/72
596K    /opt/lvmdir/c2/brick/.glusterfs/73
13M     /opt/lvmdir/c2/brick/.glusterfs/74
10M     /opt/lvmdir/c2/brick/.glusterfs/75
6.0M    /opt/lvmdir/c2/brick/.glusterfs/76
448K    /opt/lvmdir/c2/brick/.glusterfs/77
102M    /opt/lvmdir/c2/brick/.glusterfs/78
3.3M    /opt/lvmdir/c2/brick/.glusterfs/79
4.5M    /opt/lvmdir/c2/brick/.glusterfs/7a
5.2M    /opt/lvmdir/c2/brick/.glusterfs/7b
3.0M    /opt/lvmdir/c2/brick/.glusterfs/7c
6.7M    /opt/lvmdir/c2/brick/.glusterfs/7d
8.0M    /opt/lvmdir/c2/brick/.glusterfs/7e
10M     /opt/lvmdir/c2/brick/.glusterfs/7f
7.4M    /opt/lvmdir/c2/brick/.glusterfs/80
7.0M    /opt/lvmdir/c2/brick/.glusterfs/81
4.3M    /opt/lvmdir/c2/brick/.glusterfs/82
5.7M    /opt/lvmdir/c2/brick/.glusterfs/83
12M     /opt/lvmdir/c2/brick/.glusterfs/84
4.9M    /opt/lvmdir/c2/brick/.glusterfs/85
7.9M    /opt/lvmdir/c2/brick/.glusterfs/86
14M     /opt/lvmdir/c2/brick/.glusterfs/87
7.7M    /opt/lvmdir/c2/brick/.glusterfs/88
55M     /opt/lvmdir/c2/brick/.glusterfs/89
19M     /opt/lvmdir/c2/brick/.glusterfs/8a
7.4M    /opt/lvmdir/c2/brick/.glusterfs/8b
26M     /opt/lvmdir/c2/brick/.glusterfs/8c
2.0M    /opt/lvmdir/c2/brick/.glusterfs/8d
5.4M    /opt/lvmdir/c2/brick/.glusterfs/8e
13M     /opt/lvmdir/c2/brick/.glusterfs/8f
3.6M    /opt/lvmdir/c2/brick/.glusterfs/90
60M     /opt/lvmdir/c2/brick/.glusterfs/91
16M     /opt/lvmdir/c2/brick/.glusterfs/92
30M     /opt/lvmdir/c2/brick/.glusterfs/93
2.6M    /opt/lvmdir/c2/brick/.glusterfs/94
52M     /opt/lvmdir/c2/brick/.glusterfs/95
4.9M    /opt/lvmdir/c2/brick/.glusterfs/96
1.7M    /opt/lvmdir/c2/brick/.glusterfs/97
9.2M    /opt/lvmdir/c2/brick/.glusterfs/98
6.2M    /opt/lvmdir/c2/brick/.glusterfs/99
5.5M    /opt/lvmdir/c2/brick/.glusterfs/9a
4.2M    /opt/lvmdir/c2/brick/.glusterfs/9b
47M     /opt/lvmdir/c2/brick/.glusterfs/9c
2.2M    /opt/lvmdir/c2/brick/.glusterfs/9d
32M     /opt/lvmdir/c2/brick/.glusterfs/9e
13M     /opt/lvmdir/c2/brick/.glusterfs/9f
11M     /opt/lvmdir/c2/brick/.glusterfs/a0
5.6M    /opt/lvmdir/c2/brick/.glusterfs/a1
17M     /opt/lvmdir/c2/brick/.glusterfs/a2
3.3M    /opt/lvmdir/c2/brick/.glusterfs/a3
18M     /opt/lvmdir/c2/brick/.glusterfs/a4
21M     /opt/lvmdir/c2/brick/.glusterfs/a5
13M     /opt/lvmdir/c2/brick/.glusterfs/a6
18M     /opt/lvmdir/c2/brick/.glusterfs/a7
5.5M    /opt/lvmdir/c2/brick/.glusterfs/a8
4.5M    /opt/lvmdir/c2/brick/.glusterfs/a9
4.4M    /opt/lvmdir/c2/brick/.glusterfs/aa
21M     /opt/lvmdir/c2/brick/.glusterfs/ab
3.7M    /opt/lvmdir/c2/brick/.glusterfs/ac
13M     /opt/lvmdir/c2/brick/.glusterfs/ad
1.7M    /opt/lvmdir/c2/brick/.glusterfs/ae
9.1M    /opt/lvmdir/c2/brick/.glusterfs/af
50M     /opt/lvmdir/c2/brick/.glusterfs/b0
3.9M    /opt/lvmdir/c2/brick/.glusterfs/b1
8.8M    /opt/lvmdir/c2/brick/.glusterfs/b2
64M     /opt/lvmdir/c2/brick/.glusterfs/b3
13M     /opt/lvmdir/c2/brick/.glusterfs/b4
7.7M    /opt/lvmdir/c2/brick/.glusterfs/b5
3.0M    /opt/lvmdir/c2/brick/.glusterfs/b6
7.8M    /opt/lvmdir/c2/brick/.glusterfs/b7
2.8M    /opt/lvmdir/c2/brick/.glusterfs/b8
11M     /opt/lvmdir/c2/brick/.glusterfs/b9
4.6M    /opt/lvmdir/c2/brick/.glusterfs/ba
40M     /opt/lvmdir/c2/brick/.glusterfs/bb
17M     /opt/lvmdir/c2/brick/.glusterfs/bc
716K    /opt/lvmdir/c2/brick/.glusterfs/bd
2.2M    /opt/lvmdir/c2/brick/.glusterfs/be
14M     /opt/lvmdir/c2/brick/.glusterfs/bf
3.2M    /opt/lvmdir/c2/brick/.glusterfs/c0
11M     /opt/lvmdir/c2/brick/.glusterfs/c1
18M     /opt/lvmdir/c2/brick/.glusterfs/c2
8.7M    /opt/lvmdir/c2/brick/.glusterfs/c3
1.1M    /opt/lvmdir/c2/brick/.glusterfs/c4
4.3M    /opt/lvmdir/c2/brick/.glusterfs/c5
5.0M    /opt/lvmdir/c2/brick/.glusterfs/c6
44M     /opt/lvmdir/c2/brick/.glusterfs/c7
1.0M    /opt/lvmdir/c2/brick/.glusterfs/c8
4.5M    /opt/lvmdir/c2/brick/.glusterfs/c9
2.9M    /opt/lvmdir/c2/brick/.glusterfs/ca
8.4M    /opt/lvmdir/c2/brick/.glusterfs/cb
3.1M    /opt/lvmdir/c2/brick/.glusterfs/cc
14M     /opt/lvmdir/c2/brick/.glusterfs/cd
15M     /opt/lvmdir/c2/brick/.glusterfs/ce
4.7M    /opt/lvmdir/c2/brick/.glusterfs/cf
12K     /opt/lvmdir/c2/brick/.glusterfs/changelogs
36M     /opt/lvmdir/c2/brick/.glusterfs/d0
8.8M    /opt/lvmdir/c2/brick/.glusterfs/d1
6.8M    /opt/lvmdir/c2/brick/.glusterfs/d2
4.4M    /opt/lvmdir/c2/brick/.glusterfs/d3
2.3M    /opt/lvmdir/c2/brick/.glusterfs/d4
1.1M    /opt/lvmdir/c2/brick/.glusterfs/d5
16M     /opt/lvmdir/c2/brick/.glusterfs/d6
48M     /opt/lvmdir/c2/brick/.glusterfs/d7
6.8M    /opt/lvmdir/c2/brick/.glusterfs/d8
20M     /opt/lvmdir/c2/brick/.glusterfs/d9
5.7M    /opt/lvmdir/c2/brick/.glusterfs/da
3.9M    /opt/lvmdir/c2/brick/.glusterfs/db
788K    /opt/lvmdir/c2/brick/.glusterfs/dc
3.1M    /opt/lvmdir/c2/brick/.glusterfs/dd
3.5M    /opt/lvmdir/c2/brick/.glusterfs/de
45M     /opt/lvmdir/c2/brick/.glusterfs/df
5.2M    /opt/lvmdir/c2/brick/.glusterfs/e0
4.2M    /opt/lvmdir/c2/brick/.glusterfs/e1
9.2M    /opt/lvmdir/c2/brick/.glusterfs/e2
2.4M    /opt/lvmdir/c2/brick/.glusterfs/e3
11M     /opt/lvmdir/c2/brick/.glusterfs/e4
61M     /opt/lvmdir/c2/brick/.glusterfs/e5
12M     /opt/lvmdir/c2/brick/.glusterfs/e6
1.1M    /opt/lvmdir/c2/brick/.glusterfs/e7
5.9M    /opt/lvmdir/c2/brick/.glusterfs/e8
5.3M    /opt/lvmdir/c2/brick/.glusterfs/e9
1.6M    /opt/lvmdir/c2/brick/.glusterfs/ea
968K    /opt/lvmdir/c2/brick/.glusterfs/eb
9.5M    /opt/lvmdir/c2/brick/.glusterfs/ec
13M     /opt/lvmdir/c2/brick/.glusterfs/ed
15M     /opt/lvmdir/c2/brick/.glusterfs/ee
13M     /opt/lvmdir/c2/brick/.glusterfs/ef
4.1M    /opt/lvmdir/c2/brick/.glusterfs/f0
13M     /opt/lvmdir/c2/brick/.glusterfs/f1
32M     /opt/lvmdir/c2/brick/.glusterfs/f2
5.8M    /opt/lvmdir/c2/brick/.glusterfs/f3
5.2M    /opt/lvmdir/c2/brick/.glusterfs/f4
5.9M    /opt/lvmdir/c2/brick/.glusterfs/f5
14M     /opt/lvmdir/c2/brick/.glusterfs/f6
2.3M    /opt/lvmdir/c2/brick/.glusterfs/f7
2.5M    /opt/lvmdir/c2/brick/.glusterfs/f8
12M     /opt/lvmdir/c2/brick/.glusterfs/f9
432K    /opt/lvmdir/c2/brick/.glusterfs/fa
15M     /opt/lvmdir/c2/brick/.glusterfs/fb
37M     /opt/lvmdir/c2/brick/.glusterfs/fc
3.1M    /opt/lvmdir/c2/brick/.glusterfs/fd
1.9M    /opt/lvmdir/c2/brick/.glusterfs/fe
4.1M    /opt/lvmdir/c2/brick/.glusterfs/ff
4.0K    /opt/lvmdir/c2/brick/.glusterfs/health_check
8.0K    /opt/lvmdir/c2/brick/.glusterfs/indices
4.0K    /opt/lvmdir/c2/brick/.glusterfs/landfill
#

In which it is clear that all the backup/metadata files created w.r.t original files in gluster mount point taking to much spaces.

Please let us know the reason of this problem.
-- 

Regards

Abhishek Paliwal

-- 

Regards

Abhishek Paliwal

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users