Den fre 5 feb. 2021 kl 07:38 skrev Pascal Ehlert <pascal@xxxxxxxxxxxx>: > Sorry to jump in here, but would you care to explain why the total disk > usage should stay under 60%? > This is not something I have heard before and a quick Google search > didn't return anything useful. > If you have 3 hosts with 3 drives each and repl=3, then each replica will (at least by default) want to end up on a separate host. If one disk dies, ALL the pgs will want to end up on the last drive of that host, and if it doesn't fit, the recovery will stall. (the cluster will serve IO but in a degraded mode) Same for having 4 hosts with X drives, and if one host dies, all the replicas on that host needs to fit into the 3 remaining hosts, while still ending up below 85-90% when backfill_toofull starts occurring. So if you are to survive planned or unplanned downtime on any single host while the cluster doesn't get full you need extra room, plus the 10-15% margin ceph wants. Of course it will differ a bit if you have 100s of hosts or 50 drives in a host or something extreme like that, but you should have certain levels so that by 50% full you start planning expansion, by 60+% full you execute the plan so that it can get installed before becoming 70% full or something to that effect. Ceph is somewhat like an old raid6 box with a slot or two for hotspares, except you actively use the hotspares because-why-not? So you would still need to have as much space free in your ceph as the hotspares would hold but you use the extra IOPS the hotspares bring in the mean time. -- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |