Got it! I can calculate individual clone usage using “rbd du”, but does anything exist to show total clone usage across the pool? Otherwise it looks like phantom space is just missing.
Thanks,
--
Kenneth Van Alstyne
Systems Architect
M: 228.547.8045
15052 Conference Center Dr, Chantilly, VA 20151
perspecta
On Aug 13, 2019, at 11:05 PM, Konstantin Shalygin < k0ste@xxxxxxxx> wrote:
Hey guys, this is probably a really silly question, but I’m trying to reconcile where all of my space has gone in one cluster that I am responsible for.
The cluster is made up of 36 2TB SSDs across 3 nodes (12 OSDs per node), all using FileStore on XFS. We are running Ceph Luminous 12.2.8 on this particular cluster. The only pool where data is heavily stored is the “rbd” pool, of which 7.09TiB is consumed. With a replication of “3”, I would expect that the raw used to be close to 21TiB, but it’s actually closer to 35TiB. Some additional details are below. Any thoughts?
[cluster] root at dashboard:~# ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
62.8TiB 27.8TiB 35.1TiB 55.81
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
rbd 0 7.09TiB 53.76 6.10TiB 3056783
data 3 29.4GiB 0.47 6.10TiB 7918
metadata 4 57.2MiB 0 6.10TiB 95
.rgw.root 5 1.09KiB 0 6.10TiB 4
default.rgw.control 6 0B 0 6.10TiB 8
default.rgw.meta 7 0B 0 6.10TiB 0
default.rgw.log 8 0B 0 6.10TiB 207
default.rgw.buckets.index 9 0B 0 6.10TiB 0
default.rgw.buckets.data 10 0B 0 6.10TiB 0
default.rgw.buckets.non-ec 11 0B 0 6.10TiB 0
[cluster] root at dashboard:~# ceph --version
ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)
[cluster] root at dashboard:~# ceph osd dump | grep 'replicated size'
pool 0 'rbd' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 682 pgp_num 682 last_change 414873 flags hashpspool min_write_recency_for_promote 1 stripe_width 0 application rbd
pool 3 'data' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 682 pgp_num 682 last_change 409614 flags hashpspool crash_replay_interval 45 min_write_recency_for_promote 1 stripe_width 0 application cephfs
pool 4 'metadata' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 682 pgp_num 682 last_change 409617 flags hashpspool min_write_recency_for_promote 1 stripe_width 0 application cephfs
pool 5 '.rgw.root' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 409 pgp_num 409 last_change 409710 lfor 0/336229 flags hashpspool stripe_width 0 application rgw
pool 6 'default.rgw.control' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 409 pgp_num 409 last_change 409711 lfor 0/336232 flags hashpspool stripe_width 0 application rgw
pool 7 'default.rgw.meta' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 409 pgp_num 409 last_change 409713 lfor 0/336235 flags hashpspool stripe_width 0 application rgw
pool 8 'default.rgw.log' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 409 pgp_num 409 last_change 409712 lfor 0/336238 flags hashpspool stripe_width 0 application rgw
pool 9 'default.rgw.buckets.index' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 409 pgp_num 409 last_change 409714 lfor 0/336241 flags hashpspool stripe_width 0 application rgw
pool 10 'default.rgw.buckets.data' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 409 pgp_num 409 last_change 409715 lfor 0/336244 flags hashpspool stripe_width 0 application rgw
pool 11 'default.rgw.buckets.non-ec' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 409 pgp_num 409 last_change 409716 lfor 0/336247 flags hashpspool stripe_width 0 application rgw
[cluster] root at dashboard:~# ceph osd lspools
0 rbd,3 data,4 metadata,5 .rgw.root,6 default.rgw.control,7 default.rgw.meta,8 default.rgw.log,9 default.rgw.buckets.index,10 default.rgw.buckets.data,11 default.rgw.buckets.non-ec,
[cluster] root at dashboard:~# rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR
.rgw.root 1.09KiB 4 0 12 0 0 0 12 8KiB 0 0B
data 29.4GiB 7918 0 23754 0 0 0 1414777 3.74TiB 3524833 4.54TiB
default.rgw.buckets.data 0B 0 0 0 0 0 0 0 0B 0 0B
default.rgw.buckets.index 0B 0 0 0 0 0 0 0 0B 0 0B
default.rgw.buckets.non-ec 0B 0 0 0 0 0 0 0 0B 0 0B
default.rgw.control 0B 8 0 24 0 0 0 0 0B 0 0B
default.rgw.log 0B 207 0 621 0 0 0 21644149 20.6GiB 14422618 0B
default.rgw.meta 0B 0 0 0 0 0 0 0 0B 0 0B
metadata 57.2MiB 95 0 285 0 0 0 780 189MiB 86885 476MiB
rbd 7.09TiB 3053998 1539909 9161994 0 0 0 23432304830 1.07PiB 11174458128 232TiB
total_objects 3062230
total_used 35.0TiB
total_avail 27.8TiB
total_space 62.8TiB
[cluster] root at dashboard:~# for pool in `rados lspools`; do echo $pool; ceph osd pool get $pool size; echo; done
rbd
size: 3
data
size: 3
metadata
size: 3
.rgw.root
size: 3
default.rgw.control
size: 3
default.rgw.meta
size: 3
default.rgw.log
size: 3
default.rgw.buckets.index
size: 3
default.rgw.buckets.data
size: 3
default.rgw.buckets.non-ec
size: 3
Your rbd pool have clones. Lookup to rbd snapshots.
k
|
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
