On Fri, Apr 21, 2023 at 7:26 AM Kilian Ries <mail@xxxxxxxxxxxxxx> wrote: > > Still didn't find out what will happen when the pool is full - but tried a little bit in our testing environment and i were not able to get the pool full before an OSD got full. So in first place one OSD reached the full ratio (pool not quite full, about 98%) and IO stopped (like expected when an OSD reaches full ratio). I *think* pool full doesn't actually matter if you haven't set quotas, but those properties have seen some code changes recently. CCing RADOS people. We do have a proposed fix but it seems to have languished. :( -Greg > I were able to re-balance the OSDs by manually doing reweights. Now, the cluster is much more balanced and even the pool shows more free space (about 75% used). > > Also the pg-autoscaler does not really play well with the stretch crush rule ... had to increase / adjust the PGs manually to get a better distribution. > > Regards, > Kilian > ________________________________ > Von: Kilian Ries <mail@xxxxxxxxxxxxxx> > Gesendet: Mittwoch, 19. April 2023 12:18:06 > An: ceph-users > Betreff: Ceph stretch mode / POOL_BACKFILLFULL > > Hi, > > > we run a ceph cluster in stretch mode with one pool. We know about this bug: > > > https://tracker.ceph.com/issues/56650 > > https://github.com/ceph/ceph/pull/47189 > > > Can anyone tell me what happens when a pool gets to 100% full? At the moment raw OSD usage is about 54% but ceph throws me a "POOL_BACKFILLFULL" error: > > > $ ceph df > > --- RAW STORAGE --- > > CLASS SIZE AVAIL USED RAW USED %RAW USED > > ssd 63 TiB 29 TiB 34 TiB 34 TiB 54.19 > > TOTAL 63 TiB 29 TiB 34 TiB 34 TiB 54.19 > > > > --- POOLS --- > > POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL > > .mgr 1 1 415 MiB 105 1.2 GiB 0.04 1.1 TiB > > vm_stretch_live 2 64 15 TiB 4.02M 34 TiB 95.53 406 GiB > > > > So the pool warning / calculation is just a bug, because it thinks its 50% of the total size. I know ceph will stop IO / set OSDs to read only if the hit a "backfillfull_ratio" ... but what will happen if the pool gets to 100% full ? > > > Will IO still be possible? > > > No limits / quotas are set on the pool ... > > > Thanks > > Regards, > > Kilian > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |