Re: ceph df (octopus) shows USED is 7 times higher than STORED in erasure coded pool

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Ceph on a single host makes little to no sense. You're better of running
something like ZFS

On Tue, 6 Jul 2021 at 23:52, Wladimir Mutel <mwg@xxxxxxxxx> wrote:

>         I started my experimental 1-host/8-HDDs setup in 2018 with
> Luminous,
>         and I read
> https://ceph.io/community/new-luminous-erasure-coding-rbd-cephfs/ ,
>         which had interested me in using Bluestore and rewriteable EC
> pools for RBD data.
>         I have about 22 TiB or raw storage, and ceph df shows this :
>
> --- RAW STORAGE ---
> CLASS    SIZE    AVAIL    USED  RAW USED  %RAW USED
> hdd    22 TiB  2.7 TiB  19 TiB    19 TiB      87.78
> TOTAL  22 TiB  2.7 TiB  19 TiB    19 TiB      87.78
>
> --- POOLS ---
> POOL                   ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
> jerasure21              1  256  9.0 TiB    2.32M   13 TiB  97.06    276 GiB
> libvirt                 2  128  1.5 TiB  413.60k  4.5 TiB  91.77    140 GiB
> rbd                     3   32  798 KiB        5  2.7 MiB      0    138 GiB
> iso                     4   32  2.3 MiB       10  8.0 MiB      0    138 GiB
> device_health_metrics   5    1   31 MiB        9   94 MiB   0.02    138 GiB
>
>         If I add USED for libvirt and jerasure21 , I get 17.5 TiB, and 2.7
> TiB is shown at RAW STORAGE/AVAIL
>         Sum of POOLS/MAX AVAIL is about 840 GiB, where are my other
> 2.7-0.840 =~ 1.86 TiB ???
>         Or in different words, where are my (RAW STORAGE/RAW
> USED)-(SUM(POOLS/USED)) = 19-17.5 = 1.5 TiB ?
>
>         As it does not seem I would get any more hosts for this setup,
>         I am seriously thinking of bringing down this Ceph
>         and setting up instead a Btrfs storing qcow2 images served over
> iSCSI
>         which looks simpler to me for single-host situation.
>
> Josh Baergen wrote:
> > Hey Wladimir,
> >
> > I actually don't know where this is referenced in the docs, if anywhere.
> Googling around shows many people discovering this overhead the hard way on
> ceph-users.
> >
> > I also don't know the rbd journaling mechanism in enough depth to
> comment on whether it could be causing this issue for you. Are you seeing a
> high
> > allocated:stored ratio on your cluster?
> >
> > Josh
> >
> > On Sun, Jul 4, 2021 at 6:52 AM Wladimir Mutel <mwg@xxxxxxxxx <mailto:
> mwg@xxxxxxxxx>> wrote:
> >
> >     Dear Mr Baergen,
> >
> >     thanks a lot for your very concise explanation,
> >     however I would like to learn more why default Bluestore alloc.size
> causes such a big storage overhead,
> >     and where in the Ceph docs it is explained how and what to watch for
> to avoid hitting this phenomenon again and again.
> >     I have a feeling this is what I get on my experimental Ceph setup
> with simplest JErasure 2+1 data pool.
> >     Could it be caused by journaled RBD writes to EC data-pool ?
> >
> >     Josh Baergen wrote:
> >      > Hey Arkadiy,
> >      >
> >      > If the OSDs are on HDDs and were created with the default
> >      > bluestore_min_alloc_size_hdd, which is still 64KiB in Octopus,
> then in
> >      > effect data will be allocated from the pool in 640KiB chunks
> (64KiB *
> >      > (k+m)). 5.36M objects taking up 501GiB is an average object size
> of 98KiB
> >      > which results in a ratio of 6.53:1 allocated:stored, which is
> pretty close
> >      > to the 7:1 observed.
> >      >
> >      > If my assumption about your configuration is correct, then the
> only way to
> >      > fix this is to adjust bluestore_min_alloc_size_hdd and recreate
> all your
> >      > OSDs, which will take a while...
> >      >
> >      > Josh
> >      >
> >      > On Tue, Jun 29, 2021 at 3:07 PM Arkadiy Kulev <eth@xxxxxxxxxxxx
> <mailto:eth@xxxxxxxxxxxx>> wrote:
> >      >
> >      >> The pool *default.rgw.buckets.data* has *501 GiB* stored, but
> USED shows
> >      >> *3.5
> >      >> TiB *(7 times higher!)*:*
> >      >>
> >      >> root@ceph-01:~# ceph df
> >      >> --- RAW STORAGE ---
> >      >> CLASS  SIZE     AVAIL    USED     RAW USED  %RAW USED
> >      >> hdd    196 TiB  193 TiB  3.5 TiB   3.6 TiB       1.85
> >      >> TOTAL  196 TiB  193 TiB  3.5 TiB   3.6 TiB       1.85
> >      >>
> >      >> --- POOLS ---
> >      >> POOL                       ID  PGS  STORED   OBJECTS  USED
>  %USED  MAX
> >      >> AVAIL
> >      >> device_health_metrics       1    1   19 KiB       12   56 KiB
>   0
> >      >>   61 TiB
> >      >> .rgw.root                   2   32  2.6 KiB        6  1.1 MiB
>   0
> >      >>   61 TiB
> >      >> default.rgw.log             3   32  168 KiB      210   13 MiB
>   0
> >      >>   61 TiB
> >      >> default.rgw.control         4   32      0 B        8      0 B
>   0
> >      >>   61 TiB
> >      >> default.rgw.meta            5    8  4.8 KiB       11  1.9 MiB
>   0
> >      >>   61 TiB
> >      >> default.rgw.buckets.index   6    8  1.6 GiB      211  4.7 GiB
>   0
> >      >>   61 TiB
> >      >>
> >      >> default.rgw.buckets.data   10  128  501 GiB    5.36M  3.5 TiB
>  1.90
> >      >> 110 TiB
> >      >>
> >      >> The *default.rgw.buckets.data* pool is using erasure coding:
> >      >>
> >      >> root@ceph-01:~# ceph osd erasure-code-profile get EC_RGW_HOST
> >      >> crush-device-class=hdd
> >      >> crush-failure-domain=host
> >      >> crush-root=default
> >      >> jerasure-per-chunk-alignment=false
> >      >> k=6
> >      >> m=4
> >      >> plugin=jerasure
> >      >> technique=reed_sol_van
> >      >> w=8
> >      >>
> >      >> If anyone could help explain why it's using up 7 times more
> space, it would
> >      >> help a lot. Versioning is disabled. ceph version 15.2.13
> (octopus stable).
> >      >>
> >      >> Sincerely,
> >      >> Ark.
> >      >> _______________________________________________
> >      >> ceph-users mailing list -- ceph-users@xxxxxxx <mailto:
> ceph-users@xxxxxxx>
> >      >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> <mailto:ceph-users-leave@xxxxxxx>
> >      >>
> >      > _______________________________________________
> >      > ceph-users mailing list -- ceph-users@xxxxxxx <mailto:
> ceph-users@xxxxxxx>
> >      > To unsubscribe send an email to ceph-users-leave@xxxxxxx <mailto:
> ceph-users-leave@xxxxxxx>
> >      >
> >
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux