Re: Ceph capacity versus pool replicated size discrepancy?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hey guys, this is probably a really silly question, but I’m trying to reconcile where all of my space has gone in one cluster that I am responsible for.

The cluster is made up of 36 2TB SSDs across 3 nodes (12 OSDs per node), all using FileStore on XFS.  We are running Ceph Luminous 12.2.8 on this particular cluster. The only pool where data is heavily stored is the “rbd” pool, of which 7.09TiB is consumed.  With a replication of “3”, I would expect that the raw used to be close to 21TiB, but it’s actually closer to 35TiB.  Some additional details are below.  Any thoughts?

[cluster] root at dashboard:~# ceph df
GLOBAL:
    SIZE        AVAIL       RAW USED     %RAW USED
    62.8TiB     27.8TiB      35.1TiB         55.81
POOLS:
    NAME                           ID     USED        %USED     MAX AVAIL     OBJECTS
    rbd                            0      7.09TiB     53.76       6.10TiB     3056783
    data                           3      29.4GiB      0.47       6.10TiB        7918
    metadata                       4      57.2MiB         0       6.10TiB          95
    .rgw.root                      5      1.09KiB         0       6.10TiB           4
    default.rgw.control            6           0B         0       6.10TiB           8
    default.rgw.meta               7           0B         0       6.10TiB           0
    default.rgw.log                8           0B         0       6.10TiB         207
    default.rgw.buckets.index      9           0B         0       6.10TiB           0
    default.rgw.buckets.data       10          0B         0       6.10TiB           0
    default.rgw.buckets.non-ec     11          0B         0       6.10TiB           0

[cluster] root at dashboard:~# ceph --version
ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)

[cluster] root at dashboard:~# ceph osd dump | grep 'replicated size'
pool 0 'rbd' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 682 pgp_num 682 last_change 414873 flags hashpspool min_write_recency_for_promote 1 stripe_width 0 application rbd
pool 3 'data' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 682 pgp_num 682 last_change 409614 flags hashpspool crash_replay_interval 45 min_write_recency_for_promote 1 stripe_width 0 application cephfs
pool 4 'metadata' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 682 pgp_num 682 last_change 409617 flags hashpspool min_write_recency_for_promote 1 stripe_width 0 application cephfs
pool 5 '.rgw.root' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 409 pgp_num 409 last_change 409710 lfor 0/336229 flags hashpspool stripe_width 0 application rgw
pool 6 'default.rgw.control' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 409 pgp_num 409 last_change 409711 lfor 0/336232 flags hashpspool stripe_width 0 application rgw
pool 7 'default.rgw.meta' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 409 pgp_num 409 last_change 409713 lfor 0/336235 flags hashpspool stripe_width 0 application rgw
pool 8 'default.rgw.log' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 409 pgp_num 409 last_change 409712 lfor 0/336238 flags hashpspool stripe_width 0 application rgw
pool 9 'default.rgw.buckets.index' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 409 pgp_num 409 last_change 409714 lfor 0/336241 flags hashpspool stripe_width 0 application rgw
pool 10 'default.rgw.buckets.data' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 409 pgp_num 409 last_change 409715 lfor 0/336244 flags hashpspool stripe_width 0 application rgw
pool 11 'default.rgw.buckets.non-ec' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 409 pgp_num 409 last_change 409716 lfor 0/336247 flags hashpspool stripe_width 0 application rgw

[cluster] root at dashboard:~# ceph osd lspools
0 rbd,3 data,4 metadata,5 .rgw.root,6 default.rgw.control,7 default.rgw.meta,8 default.rgw.log,9 default.rgw.buckets.index,10 default.rgw.buckets.data,11 default.rgw.buckets.non-ec,

[cluster] root at dashboard:~# rados df
POOL_NAME                  USED    OBJECTS CLONES  COPIES  MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS      RD      WR_OPS      WR
.rgw.root                  1.09KiB       4       0      12                  0       0        0          12    8KiB           0      0B
data                       29.4GiB    7918       0   23754                  0       0        0     1414777 3.74TiB     3524833 4.54TiB
default.rgw.buckets.data        0B       0       0       0                  0       0        0           0      0B           0      0B
default.rgw.buckets.index       0B       0       0       0                  0       0        0           0      0B           0      0B
default.rgw.buckets.non-ec      0B       0       0       0                  0       0        0           0      0B           0      0B
default.rgw.control             0B       8       0      24                  0       0        0           0      0B           0      0B
default.rgw.log                 0B     207       0     621                  0       0        0    21644149 20.6GiB    14422618      0B
default.rgw.meta                0B       0       0       0                  0       0        0           0      0B           0      0B
metadata                   57.2MiB      95       0     285                  0       0        0         780  189MiB       86885  476MiB
rbd                        7.09TiB 3053998 1539909 9161994                  0       0        0 23432304830 1.07PiB 11174458128  232TiB

total_objects    3062230
total_used       35.0TiB
total_avail      27.8TiB
total_space      62.8TiB

[cluster] root at dashboard:~# for pool in `rados lspools`; do echo $pool; ceph osd pool get $pool size; echo; done
rbd
size: 3
data
size: 3
metadata
size: 3
.rgw.root
size: 3
default.rgw.control
size: 3
default.rgw.meta
size: 3
default.rgw.log
size: 3
default.rgw.buckets.index
size: 3
default.rgw.buckets.data
size: 3
default.rgw.buckets.non-ec
size: 3


Your rbd pool have clones. Lookup to rbd snapshots.



k

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux