Still didn't find out what will happen when the pool is full - but tried a little bit in our testing environment and i were not able to get the pool full before an OSD got full. So in first place one OSD reached the full ratio (pool not quite full, about 98%) and IO stopped (like expected when an OSD reaches full ratio). I were able to re-balance the OSDs by manually doing reweights. Now, the cluster is much more balanced and even the pool shows more free space (about 75% used). Also the pg-autoscaler does not really play well with the stretch crush rule ... had to increase / adjust the PGs manually to get a better distribution. Regards, Kilian ________________________________ Von: Kilian Ries <mail@xxxxxxxxxxxxxx> Gesendet: Mittwoch, 19. April 2023 12:18:06 An: ceph-users Betreff: Ceph stretch mode / POOL_BACKFILLFULL Hi, we run a ceph cluster in stretch mode with one pool. We know about this bug: https://tracker.ceph.com/issues/56650 https://github.com/ceph/ceph/pull/47189 Can anyone tell me what happens when a pool gets to 100% full? At the moment raw OSD usage is about 54% but ceph throws me a "POOL_BACKFILLFULL" error: $ ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 63 TiB 29 TiB 34 TiB 34 TiB 54.19 TOTAL 63 TiB 29 TiB 34 TiB 34 TiB 54.19 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 415 MiB 105 1.2 GiB 0.04 1.1 TiB vm_stretch_live 2 64 15 TiB 4.02M 34 TiB 95.53 406 GiB So the pool warning / calculation is just a bug, because it thinks its 50% of the total size. I know ceph will stop IO / set OSDs to read only if the hit a "backfillfull_ratio" ... but what will happen if the pool gets to 100% full ? Will IO still be possible? No limits / quotas are set on the pool ... Thanks Regards, Kilian _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |