Hi, Is ceph still backfilling? What is the actual output of ceph -s? If not backfilling, it is strange that you only have 84 pgs on osd.11 but 93.59 percent in use... Are you able to find a pg on 11 which is too big? Perhaps pg query will help to find. Otherwise you should lower the weight of the osd... I had a equal case where i "put" a really big file via rados and got a really big pg. Hth Mehmet Am 3. September 2022 11:41:41 MESZ schrieb Oebele Drijfhout <oebele.drijfhout@xxxxxxxxx>: >Hello Stefan, > >Thank you for your answer. > >On Fri, Sep 2, 2022 at 5:27 PM Stefan Kooman <stefan@xxxxxx> wrote: > >> On 9/2/22 15:55, Oebele Drijfhout wrote: >> > Hello, >> > >> > I'm new to Ceph and I recently inherited a 4 node cluster with 32 OSDs >> and >> > about 116TB raw space, which shows low available space, which I'm trying >> to >> > increase by enabling the balancer and lowering priority for the most-used >> > OSDs. My questions are: is what I did correct with the current state of >> the >> > cluster and can I do more to speed up rebalancing and will we actually >> make >> > more space available this way? >> >> Yes. When it's perfectly balanced the average OSD utilization should >> approach %RAW USED. >> >> >The variance between OSD utilization has been going down during the night, >but I'm worried that we will soon hit 95% full on osd.11. %USE is steadily >going up. What can I do to 1. prevent more data being written to this OSD >and 2. force data off this OSD? > > [truiten@xxxxxxxxxx ~]$ sudo ceph --cluster eun osd df >ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL >%USE VAR PGS STATUS > 0 hdd 3.63869 1.00000 3.6 TiB 1.4 TiB 1.4 TiB 907 MiB 3.3 GiB 2.2 TiB >39.56 0.67 127 up > 1 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 581 MiB 2.5 GiB 2.6 TiB >29.61 0.50 126 up > 2 hdd 3.63869 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 701 MiB 4.2 GiB 1.5 TiB >59.56 1.01 125 up > 3 hdd 3.63869 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 672 MiB 5.5 GiB 1.1 TiB >68.95 1.16 131 up > 4 hdd 3.63869 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 524 MiB 5.7 GiB 1.1 TiB >68.88 1.16 116 up > 5 hdd 3.63869 0.76984 3.6 TiB 2.7 TiB 2.7 TiB 901 MiB 4.9 GiB 976 GiB >73.81 1.25 105 up > 6 hdd 3.63869 0.76984 3.6 TiB 2.7 TiB 2.7 TiB 473 MiB 5.0 GiB 972 GiB >73.90 1.25 99 up > 7 hdd 3.63869 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 647 MiB 3.5 GiB 1.8 TiB >49.27 0.83 125 up > 8 hdd 3.63869 1.00000 3.6 TiB 1.6 TiB 1.6 TiB 624 MiB 3.1 GiB 2.0 TiB >44.21 0.75 124 up > 9 hdd 3.63869 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 934 MiB 4.8 GiB 1.3 TiB >64.76 1.09 121 up >10 hdd 3.63869 1.00000 3.6 TiB 2.2 TiB 2.1 TiB 525 MiB 4.0 GiB 1.5 TiB >59.12 1.00 127 up >11 hdd 3.63869 0.76984 3.6 TiB 3.4 TiB 3.4 TiB 431 MiB 6.2 GiB 239 GiB >93.59 1.58 84 up <--- >12 hdd 3.63869 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 777 MiB 4.2 GiB 1.5 TiB >59.02 1.00 124 up >13 hdd 3.63869 1.00000 3.6 TiB 1.4 TiB 1.4 TiB 738 MiB 3.2 GiB 2.2 TiB >39.46 0.67 125 up >14 hdd 3.63869 1.00000 3.6 TiB 2.2 TiB 2.1 TiB 560 MiB 6.2 GiB 1.5 TiB >59.11 1.00 122 up >15 hdd 3.63869 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 575 MiB 4.4 GiB 1.5 TiB >59.06 1.00 123 up >16 hdd 3.63869 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 625 MiB 3.4 GiB 1.8 TiB >49.24 0.83 124 up >17 hdd 3.63869 0.76984 3.6 TiB 2.7 TiB 2.7 TiB 696 MiB 5.1 GiB 958 GiB >74.28 1.25 93 up >18 hdd 3.63869 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 210 MiB 3.6 GiB 1.6 TiB >54.94 0.93 125 up >19 hdd 3.63869 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 504 MiB 5.2 GiB 1.5 TiB >59.08 1.00 124 up >20 hdd 3.63869 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 796 MiB 5.0 GiB 1.1 TiB >69.14 1.17 121 up >21 hdd 3.63869 1.00000 3.6 TiB 1.6 TiB 1.6 TiB 679 MiB 3.6 GiB 2.0 TiB >44.25 0.75 123 up >22 hdd 3.63869 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 682 MiB 4.9 GiB 1.1 TiB >68.86 1.16 125 up >23 hdd 3.63869 1.00000 3.6 TiB 1.6 TiB 1.6 TiB 575 MiB 3.1 GiB 2.0 TiB >44.83 0.76 124 up >24 hdd 3.63869 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 517 MiB 3.9 GiB 1.5 TiB >59.00 1.00 125 up >25 hdd 3.63869 1.00000 3.6 TiB 2.2 TiB 2.1 TiB 836 MiB 4.5 GiB 1.5 TiB >59.12 1.00 121 up >26 hdd 3.63869 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 520 MiB 5.0 GiB 1.1 TiB >69.31 1.17 109 up >27 hdd 3.63869 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 861 MiB 3.8 GiB 1.7 TiB >54.13 0.91 126 up >28 hdd 3.63869 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 256 MiB 4.3 GiB 1.8 TiB >49.21 0.83 122 up >29 hdd 3.63869 1.00000 3.6 TiB 2.7 TiB 2.7 TiB 998 MiB 5.1 GiB 980 GiB >73.69 1.24 126 up >30 hdd 3.63869 1.00000 3.6 TiB 2.2 TiB 2.1 TiB 1.1 GiB 4.1 GiB 1.5 TiB >59.16 1.00 123 up >31 hdd 3.63869 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 689 MiB 4.4 GiB 1.3 TiB >64.02 1.08 123 up > TOTAL 116 TiB 69 TiB 69 TiB 21 GiB 140 GiB 48 TiB >59.19 >MIN/MAX VAR: 0.50/1.58 STDDEV: 12.44 >_______________________________________________ >ceph-users mailing list -- ceph-users@xxxxxxx >To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |