Re: ceph df (octopus) shows USED is 7 times higher than STORED in erasure coded pool

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hey Wladimir,

That output looks like it's from Nautilus or later. My understanding is
that the USED column is in raw bytes, whereas STORED is "user" bytes. If
you're using EC 2:1 for all of those pools, I would expect USED to be at
least 1.5x STORED, which looks to be the case for jerasure21. Perhaps your
libvirt pool is 3x replicated, in which case the numbers add up as well.

Josh

On Tue, Jul 6, 2021 at 5:51 AM Wladimir Mutel <mwg@xxxxxxxxx> wrote:

>         I started my experimental 1-host/8-HDDs setup in 2018 with
> Luminous,
>         and I read
> https://ceph.io/community/new-luminous-erasure-coding-rbd-cephfs/ ,
>         which had interested me in using Bluestore and rewriteable EC
> pools for RBD data.
>         I have about 22 TiB or raw storage, and ceph df shows this :
>
> --- RAW STORAGE ---
> CLASS    SIZE    AVAIL    USED  RAW USED  %RAW USED
> hdd    22 TiB  2.7 TiB  19 TiB    19 TiB      87.78
> TOTAL  22 TiB  2.7 TiB  19 TiB    19 TiB      87.78
>
> --- POOLS ---
> POOL                   ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
> jerasure21              1  256  9.0 TiB    2.32M   13 TiB  97.06    276 GiB
> libvirt                 2  128  1.5 TiB  413.60k  4.5 TiB  91.77    140 GiB
> rbd                     3   32  798 KiB        5  2.7 MiB      0    138 GiB
> iso                     4   32  2.3 MiB       10  8.0 MiB      0    138 GiB
> device_health_metrics   5    1   31 MiB        9   94 MiB   0.02    138 GiB
>
>         If I add USED for libvirt and jerasure21 , I get 17.5 TiB, and 2.7
> TiB is shown at RAW STORAGE/AVAIL
>         Sum of POOLS/MAX AVAIL is about 840 GiB, where are my other
> 2.7-0.840 =~ 1.86 TiB ???
>         Or in different words, where are my (RAW STORAGE/RAW
> USED)-(SUM(POOLS/USED)) = 19-17.5 = 1.5 TiB ?
>
>         As it does not seem I would get any more hosts for this setup,
>         I am seriously thinking of bringing down this Ceph
>         and setting up instead a Btrfs storing qcow2 images served over
> iSCSI
>         which looks simpler to me for single-host situation.
>
> Josh Baergen wrote:
> > Hey Wladimir,
> >
> > I actually don't know where this is referenced in the docs, if anywhere.
> Googling around shows many people discovering this overhead the hard way on
> ceph-users.
> >
> > I also don't know the rbd journaling mechanism in enough depth to
> comment on whether it could be causing this issue for you. Are you seeing a
> high
> > allocated:stored ratio on your cluster?
> >
> > Josh
> >
> > On Sun, Jul 4, 2021 at 6:52 AM Wladimir Mutel <mwg@xxxxxxxxx <mailto:
> mwg@xxxxxxxxx>> wrote:
> >
> >     Dear Mr Baergen,
> >
> >     thanks a lot for your very concise explanation,
> >     however I would like to learn more why default Bluestore alloc.size
> causes such a big storage overhead,
> >     and where in the Ceph docs it is explained how and what to watch for
> to avoid hitting this phenomenon again and again.
> >     I have a feeling this is what I get on my experimental Ceph setup
> with simplest JErasure 2+1 data pool.
> >     Could it be caused by journaled RBD writes to EC data-pool ?
> >
> >     Josh Baergen wrote:
> >      > Hey Arkadiy,
> >      >
> >      > If the OSDs are on HDDs and were created with the default
> >      > bluestore_min_alloc_size_hdd, which is still 64KiB in Octopus,
> then in
> >      > effect data will be allocated from the pool in 640KiB chunks
> (64KiB *
> >      > (k+m)). 5.36M objects taking up 501GiB is an average object size
> of 98KiB
> >      > which results in a ratio of 6.53:1 allocated:stored, which is
> pretty close
> >      > to the 7:1 observed.
> >      >
> >      > If my assumption about your configuration is correct, then the
> only way to
> >      > fix this is to adjust bluestore_min_alloc_size_hdd and recreate
> all your
> >      > OSDs, which will take a while...
> >      >
> >      > Josh
> >      >
> >      > On Tue, Jun 29, 2021 at 3:07 PM Arkadiy Kulev <eth@xxxxxxxxxxxx
> <mailto:eth@xxxxxxxxxxxx>> wrote:
> >      >
> >      >> The pool *default.rgw.buckets.data* has *501 GiB* stored, but
> USED shows
> >      >> *3.5
> >      >> TiB *(7 times higher!)*:*
> >      >>
> >      >> root@ceph-01:~# ceph df
> >      >> --- RAW STORAGE ---
> >      >> CLASS  SIZE     AVAIL    USED     RAW USED  %RAW USED
> >      >> hdd    196 TiB  193 TiB  3.5 TiB   3.6 TiB       1.85
> >      >> TOTAL  196 TiB  193 TiB  3.5 TiB   3.6 TiB       1.85
> >      >>
> >      >> --- POOLS ---
> >      >> POOL                       ID  PGS  STORED   OBJECTS  USED
>  %USED  MAX
> >      >> AVAIL
> >      >> device_health_metrics       1    1   19 KiB       12   56 KiB
>   0
> >      >>   61 TiB
> >      >> .rgw.root                   2   32  2.6 KiB        6  1.1 MiB
>   0
> >      >>   61 TiB
> >      >> default.rgw.log             3   32  168 KiB      210   13 MiB
>   0
> >      >>   61 TiB
> >      >> default.rgw.control         4   32      0 B        8      0 B
>   0
> >      >>   61 TiB
> >      >> default.rgw.meta            5    8  4.8 KiB       11  1.9 MiB
>   0
> >      >>   61 TiB
> >      >> default.rgw.buckets.index   6    8  1.6 GiB      211  4.7 GiB
>   0
> >      >>   61 TiB
> >      >>
> >      >> default.rgw.buckets.data   10  128  501 GiB    5.36M  3.5 TiB
>  1.90
> >      >> 110 TiB
> >      >>
> >      >> The *default.rgw.buckets.data* pool is using erasure coding:
> >      >>
> >      >> root@ceph-01:~# ceph osd erasure-code-profile get EC_RGW_HOST
> >      >> crush-device-class=hdd
> >      >> crush-failure-domain=host
> >      >> crush-root=default
> >      >> jerasure-per-chunk-alignment=false
> >      >> k=6
> >      >> m=4
> >      >> plugin=jerasure
> >      >> technique=reed_sol_van
> >      >> w=8
> >      >>
> >      >> If anyone could help explain why it's using up 7 times more
> space, it would
> >      >> help a lot. Versioning is disabled. ceph version 15.2.13
> (octopus stable).
> >      >>
> >      >> Sincerely,
> >      >> Ark.
> >      >> _______________________________________________
> >      >> ceph-users mailing list -- ceph-users@xxxxxxx <mailto:
> ceph-users@xxxxxxx>
> >      >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> <mailto:ceph-users-leave@xxxxxxx>
> >      >>
> >      > _______________________________________________
> >      > ceph-users mailing list -- ceph-users@xxxxxxx <mailto:
> ceph-users@xxxxxxx>
> >      > To unsubscribe send an email to ceph-users-leave@xxxxxxx <mailto:
> ceph-users-leave@xxxxxxx>
> >      >
> >
>
>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux