Hi Igor,
Many thanks for your reply. Here are the details about the
cluster:
1. Ceph version - 13.2.5-1xenial (installed from Ceph
repository for ubuntu 16.04)
2. main devices for radosgw pool - hdd. we do use a few
ssds for the other pool, but it is not used by radosgw
3. we use BlueStore
4. Average rgw object size - I have no idea how to check
that. Couldn't find a simple answer from google either. Could
you please let me know how to check that?
5. Ceph osd df tree:
6. Other useful info on the cluster:
# ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE
VAR PGS TYPE NAME
-1 112.17979 - 113 TiB 90 TiB 23 TiB
79.25 1.00 - root uk
-5 112.17979 - 113 TiB 90 TiB 23 TiB
79.25 1.00 - datacenter ldex
-11 112.17979 - 113 TiB 90 TiB 23 TiB
79.25 1.00 - room ldex-dc3
-13 112.17979 - 113 TiB 90 TiB 23 TiB
79.25 1.00 - row row-a
-4 112.17979 - 113 TiB 90 TiB 23 TiB
79.25 1.00 - rack ldex-rack-a5
-2 28.04495 - 28 TiB 22 TiB 6.2 TiB
77.96 0.98 - host arh-ibstorage1-ib
0 hdd 2.73000 0.79999 2.8 TiB 2.3 TiB 519 GiB
81.61 1.03 145 osd.0
1 hdd 2.73000 1.00000 2.8 TiB 1.9 TiB 847 GiB
70.00 0.88 130 osd.1
2 hdd 2.73000 1.00000 2.8 TiB 2.2 TiB 561 GiB
80.12 1.01 152 osd.2
3 hdd 2.73000 1.00000 2.8 TiB 2.3 TiB 469 GiB
83.41 1.05 160 osd.3
4 hdd 2.73000 1.00000 2.8 TiB 1.8 TiB 983 GiB
65.18 0.82 141 osd.4
32 hdd 5.45999 1.00000 5.5 TiB 4.4 TiB 1.1 TiB
80.68 1.02 306 osd.32
35 hdd 2.73000 1.00000 2.8 TiB 1.7 TiB 1.0 TiB
62.89 0.79 126 osd.35
36 hdd 2.73000 1.00000 2.8 TiB 2.3 TiB 464 GiB
83.58 1.05 175 osd.36
37 hdd 2.73000 0.89999 2.8 TiB 2.5 TiB 301 GiB
89.34 1.13 160 osd.37
5 ssd 0.74500 1.00000 745 GiB 642 GiB 103 GiB
86.15 1.09 65 osd.5
-3 28.04495 - 28 TiB 24 TiB 4.5 TiB
84.03 1.06 - host arh-ibstorage2-ib
9 hdd 2.73000 0.95000 2.8 TiB 2.4 TiB 405 GiB
85.65 1.08 158 osd.9
10 hdd 2.73000 0.89999 2.8 TiB 2.4 TiB 352 GiB
87.52 1.10 169 osd.10
11 hdd 2.73000 1.00000 2.8 TiB 2.0 TiB 783 GiB
72.28 0.91 160 osd.11
12 hdd 2.73000 0.84999 2.8 TiB 2.4 TiB 359 GiB
87.27 1.10 153 osd.12
13 hdd 2.73000 1.00000 2.8 TiB 2.4 TiB 348 GiB
87.69 1.11 169 osd.13
14 hdd 2.73000 1.00000 2.8 TiB 2.5 TiB 283 GiB
89.97 1.14 170 osd.14
15 hdd 2.73000 1.00000 2.8 TiB 2.2 TiB 560 GiB
80.18 1.01 155 osd.15
16 hdd 2.73000 0.95000 2.8 TiB 2.4 TiB 332 GiB
88.26 1.11 178 osd.16
26 hdd 5.45999 1.00000 5.5 TiB 4.4 TiB 1.0 TiB
81.04 1.02 324 osd.26
7 ssd 0.74500 1.00000 745 GiB 607 GiB 138 GiB
81.48 1.03 62 osd.7
-15 28.04495 - 28 TiB 22 TiB 6.4 TiB
77.40 0.98 - host
arh-ibstorage3-ib
18 hdd 2.73000 0.95000 2.8 TiB 2.5 TiB 312 GiB
88.96 1.12 156 osd.18
19 hdd 2.73000 1.00000 2.8 TiB 2.0 TiB 771 GiB
72.68 0.92 162 osd.19
20 hdd 2.73000 1.00000 2.8 TiB 2.0 TiB 733 GiB
74.04 0.93 149 osd.20
21 hdd 2.73000 1.00000 2.8 TiB 2.2 TiB 533
GiB 81.12 1.02 155 osd.21
22 hdd 2.73000 1.00000 2.8 TiB 2.1 TiB 692
GiB 75.48 0.95 144 osd.22
23 hdd 2.73000 1.00000 2.8 TiB 1.6 TiB 1.1
TiB 58.43 0.74 130 osd.23
24 hdd 2.73000 1.00000 2.8 TiB 2.2 TiB 579
GiB 79.51 1.00 146 osd.24
25 hdd 2.73000 1.00000 2.8 TiB 1.9 TiB 886
GiB 68.63 0.87 147 osd.25
31 hdd 5.45999 1.00000 5.5 TiB 4.7 TiB 758
GiB 86.50 1.09 326 osd.31
6 ssd 0.74500 0.89999 744 GiB 640 GiB 104
GiB 86.01 1.09 61 osd.6
-17 28.04494 - 28 TiB 22 TiB 6.3
TiB 77.61 0.98 - host
arh-ibstorage4-ib
8 hdd 2.73000 1.00000 2.8 TiB 1.9 TiB 909
GiB 67.80 0.86 141 osd.8
17 hdd 2.73000 1.00000 2.8 TiB 1.9 TiB 904
GiB 67.99 0.86 144 osd.17
27 hdd 2.73000 1.00000 2.8 TiB 2.1 TiB 654
GiB 76.84 0.97 152 osd.27
28 hdd 2.73000 1.00000 2.8 TiB 2.3 TiB 481
GiB 82.98 1.05 153 osd.28
29 hdd 2.73000 1.00000 2.8 TiB 1.9 TiB 829
GiB 70.65 0.89 137 osd.29
30 hdd 2.73000 1.00000 2.8 TiB 2.0 TiB 762
GiB 73.03 0.92 142 osd.30
33 hdd 2.73000 1.00000 2.8 TiB 2.3 TiB 501
GiB 82.25 1.04 166 osd.33
34 hdd 5.45998 1.00000 5.5 TiB 4.5 TiB 968
GiB 82.77 1.04 325 osd.34
39 hdd 2.73000 0.95000 2.8 TiB 2.4 TiB 402
GiB 85.77 1.08 162 osd.39
38 ssd 0.74500 1.00000 745 GiB 671 GiB 74
GiB 90.02 1.14 68 osd.38
TOTAL 113 TiB 90 TiB 23
TiB 79.25
MIN/MAX VAR: 0.74/1.14 STDDEV: 8.14
# for i in $(radosgw-admin
bucket list | jq -r '.[]'); do radosgw-admin bucket stats
--bucket=$i | jq '.usage | ."rgw.main" | .size_kb' ; done |
awk '{ SUM += $1} END { print SUM/1024/1024/1024 }'
6.59098
# ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
113 TiB 23 TiB 90 TiB 79.25
POOLS:
NAME ID USED
%USED MAX AVAIL OBJECTS
Primary-ubuntu-1 5 27 TiB
87.56 3.9 TiB 7302534
.users.uid 15 6.8 KiB
0 3.9 TiB 39
.users 16 335 B
0 3.9 TiB 20
.users.swift 17 14 B
0 3.9 TiB 1
.rgw.buckets 19 15 TiB 79.88
3.9 TiB 8787763
.users.email 22 0 B
0 3.9 TiB 0
.log 24 109 MiB
0 3.9 TiB 102301
.rgw.buckets.extra 37 0 B
0 2.6 TiB 0
.rgw.root 44 2.9 KiB
0 2.6 TiB 16
.rgw.meta 45 1.7 MiB
0 2.6 TiB 6249
.rgw.control 46 0 B
0 2.6 TiB 8
.rgw.gc 47 0 B
0 2.6 TiB 32
.usage 52 0 B
0 2.6 TiB 0
.intent-log 53 0 B
0 2.6 TiB 0
default.rgw.buckets.non-ec 54 0 B
0 2.6 TiB 0
.rgw.buckets.index 55 0 B
0 2.6 TiB 11485
.rgw 56 491 KiB
0 2.6 TiB 1686
Primary-ubuntu-1-ssd 57 1.2 TiB
92.39 105 GiB 379516
I am not too sure if the issue relates to the BlueStore
overhead as I would probably have seen the discrepancy in my
Primary-ubuntu-1 pool as well. However, the data usage on
Primary-ubuntu-1 pool seems to be consistent with my
expectations (precise numbers to be verified soon). The
issues seems to be only with the .rgw-buckets pool where the
"ceph df " output shows 15TB of usage and the sum of all
buckets in that pool shows just over 6.5TB.
Cheers
Andrei
Hi Andrei,
The most obvious reason is space usage overhead caused by
BlueStore allocation granularity, e.g. if
bluestore_min_alloc_size is 64K and average object size
is 16K one will waste 48K per object in average. This is
rather a speculation so far as we lack key the information
about your cluster:
- Ceph version
- What are the main devices for OSD: hdd or ssd.
- BlueStore or FileStore.
- average RGW object size.
You might also want to collect and share performance
counter dumps (ceph daemon osd.N perf dump) and "
" reports from a couple of your OSDs.
Thanks,
Igor
On 7/2/2019 11:43 AM, Andrei
Mikhailovsky wrote:
Bump!
Hi
Could someone please explain / show how to
troubleshoot the space usage in Ceph and how to
reclaim the unused space?
I have a small cluster with 40 osds, replica
of 2, mainly used as a backend for cloud stack
as well as the S3 gateway. The used space
doesn't make any sense to me, especially the rgw
pool, so I am seeking help.
Here is what I found from the client:
Ceph -s shows the
usage: 89 TiB used, 24 TiB / 113 TiB avail
Ceph df shows:
Primary-ubuntu-1 5 27 TiB
90.11 3.0 TiB 7201098
Primary-ubuntu-1-ssd 57 1.2 TiB
89.62 143 GiB 359260
.rgw.buckets
19 15 TiB 83.73
3.0 TiB 8742222
the usage of
the Primary-ubuntu-1 and Primary-ubuntu-1-ssd
is in line with my expectations.
However, the .rgw.buckets pool seems to be using
way too much. The usage of all rgw buckets shows
6.5TB usage (looking at the size_kb values from
the "radosgw-admin bucket stats"). I am trying
to figure out why .rgw.buckets is using 15TB of
space instead of the 6.5TB as shown from the
bucket usage.
Thanks
Andrei
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
