Re: Ceph stretch mode / POOL_BACKFILLFULL

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Fri, Apr 21, 2023 at 7:26 AM Kilian Ries <mail@xxxxxxxxxxxxxx> wrote:
>
> Still didn't find out what will happen when the pool is full - but tried a little bit in our testing environment and i were not able to get the pool full before an OSD got full. So in first place one OSD reached the full ratio (pool not quite full, about 98%) and IO stopped (like expected when an OSD reaches full ratio).

I *think* pool full doesn't actually matter if you haven't set quotas,
but those properties have seen some code changes recently. CCing RADOS
people.
We do have a proposed fix but it seems to have languished. :(
-Greg

> I were able to re-balance the OSDs by manually doing reweights. Now, the cluster is much more balanced and even the pool shows more free space (about 75% used).
>
> Also the pg-autoscaler does not really play well with the stretch crush rule ... had to increase / adjust the PGs manually to get a better distribution.
>
> Regards,
> Kilian
> ________________________________
> Von: Kilian Ries <mail@xxxxxxxxxxxxxx>
> Gesendet: Mittwoch, 19. April 2023 12:18:06
> An: ceph-users
> Betreff:  Ceph stretch mode / POOL_BACKFILLFULL
>
> Hi,
>
>
> we run a ceph cluster in stretch mode with one pool. We know about this bug:
>
>
> https://tracker.ceph.com/issues/56650
>
> https://github.com/ceph/ceph/pull/47189
>
>
> Can anyone tell me what happens when a pool gets to 100% full? At the moment raw OSD usage is about 54% but ceph throws me a "POOL_BACKFILLFULL" error:
>
>
> $ ceph df
>
> --- RAW STORAGE ---
>
> CLASS    SIZE   AVAIL    USED  RAW USED  %RAW USED
>
> ssd    63 TiB  29 TiB  34 TiB    34 TiB      54.19
>
> TOTAL  63 TiB  29 TiB  34 TiB    34 TiB      54.19
>
>
>
> --- POOLS ---
>
> POOL             ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
>
> .mgr              1    1  415 MiB      105  1.2 GiB   0.04    1.1 TiB
>
> vm_stretch_live   2   64   15 TiB    4.02M   34 TiB  95.53    406 GiB
>
>
>
> So the pool warning / calculation is just a bug, because it thinks its 50% of the total size. I know ceph will stop IO / set OSDs to read only if the hit a "backfillfull_ratio" ... but what will happen if the pool gets to 100% full ?
>
>
> Will IO still be possible?
>
>
> No limits / quotas are set on the pool ...
>
>
> Thanks
>
> Regards,
>
> Kilian
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux