Hi Mohamed, Changing weights is no longer a good practice. The balancer is supposed to do the job. The number of pg per osd is really tight on your infrastructure. Can you display the ceph osd tree command? ________________________________________________________ Cordialement, *David CASIER* *Ligne directe: +33(0) 9 72 61 98 29* ________________________________________________________ Le lun. 11 déc. 2023 à 11:06, Mohamed LAMDAOUAR <mohamed.lamdaouar@xxxxxxx> a écrit : > Hello the team, > > We initially had a cluster of 3 machines with 4 osd on each machine, we > added 4 machines in the cluster (each machine with 4 osd) > We launched the balancing but it never finished, still in progress. But the > big issue: we have an osd full and all the pools on this osd are read only. > > *ceph osd df *: > > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META > AVAIL %USE VAR PGS STATUS > 20 hdd 9.09569 1.00000 9.1 TiB 580 GiB 576 GiB 1.2 GiB 3.1 GiB > 8.5 TiB 6.23 0.32 169 up > 21 hdd 9.09569 1.00000 9.1 TiB 1.5 TiB 1.5 TiB 252 MiB 7.7 GiB > 7.6 TiB 16.08 0.82 247 up > 22 hdd 9.09569 1.00000 9.1 TiB 671 GiB 667 GiB 204 MiB 4.1 GiB > 8.4 TiB 7.21 0.37 136 up > 23 hdd 9.09569 1.00000 9.1 TiB 665 GiB 660 GiB 270 MiB 4.5 GiB > 8.4 TiB 7.14 0.37 124 up > 0 hdd 9.09569 1.00000 9.1 TiB 1.2 TiB 1.2 TiB 87 MiB 6.0 GiB > 7.9 TiB 13.30 0.68 230 up > 1 hdd 9.09569 1.00000 9.1 TiB 1.3 TiB 1.3 TiB 347 MiB 6.6 GiB > 7.8 TiB 14.01 0.72 153 up > 2 hdd 9.09569 0.65009 9.1 TiB 1.8 TiB 1.8 TiB 443 MiB 7.3 GiB > 7.3 TiB 20.00 1.03 147 up > 3 hdd 9.09569 1.00000 9.1 TiB 617 GiB 611 GiB 220 MiB 5.8 GiB > 8.5 TiB 6.62 0.34 101 up > 4 hdd 9.09569 0.80005 9.1 TiB 2.0 TiB 2.0 TiB 293 MiB 8.2 GiB > 7.1 TiB 22.12 1.13 137 up > 5 hdd 9.09569 1.00000 9.1 TiB 857 GiB 852 GiB 157 MiB 4.9 GiB > 8.3 TiB 9.20 0.47 155 up > 6 hdd 9.09569 1.00000 9.1 TiB 580 GiB 575 GiB 678 MiB 4.4 GiB > 8.5 TiB 6.23 0.32 114 up > 7 hdd 9.09569 0.50000 9.1 TiB 7.7 TiB 7.7 TiB 103 MiB 16 GiB > 1.4 TiB 85.03 4.36 201 up > 24 hdd 9.09569 1.00000 9.1 TiB 1.2 TiB 1.2 TiB 133 MiB 6.2 GiB > 7.9 TiB 13.11 0.67 225 up > 25 hdd 9.09569 0.34999 9.1 TiB 8.3 TiB 8.2 TiB 101 MiB 17 GiB > 860 GiB 90.77 4.66 159 up > 26 hdd 9.09569 1.00000 9.1 TiB 665 GiB 661 GiB 292 MiB 3.8 GiB > 8.4 TiB 7.14 0.37 107 up > 27 hdd 9.09569 1.00000 9.1 TiB 427 GiB 423 GiB 241 MiB 3.4 GiB > 8.7 TiB 4.58 0.24 103 up > 8 hdd 9.09569 1.00000 9.1 TiB 845 GiB 839 GiB 831 MiB 5.9 GiB > 8.3 TiB 9.07 0.47 163 up > 9 hdd 9.09569 1.00000 9.1 TiB 727 GiB 722 GiB 162 MiB 4.8 GiB > 8.4 TiB 7.80 0.40 169 up > 10 hdd 9.09569 0.80005 9.1 TiB 1.9 TiB 1.9 TiB 742 MiB 7.5 GiB > 7.2 TiB 21.01 1.08 136 up > 11 hdd 9.09569 1.00000 9.1 TiB 733 GiB 727 GiB 498 MiB 5.2 GiB > 8.4 TiB 7.87 0.40 163 up > 12 hdd 9.09569 1.00000 9.1 TiB 892 GiB 886 GiB 318 MiB 5.6 GiB > 8.2 TiB 9.58 0.49 254 up > 13 hdd 9.09569 1.00000 9.1 TiB 759 GiB 755 GiB 37 MiB 4.0 GiB > 8.4 TiB 8.15 0.42 134 up > 14 hdd 9.09569 0.85004 9.1 TiB 2.3 TiB 2.3 TiB 245 MiB 7.7 GiB > 6.8 TiB 24.96 1.28 142 up > 15 hdd 9.09569 1.00000 9.1 TiB 7.3 TiB 7.3 TiB 435 MiB 16 GiB > 1.8 TiB 80.17 4.11 213 up > 16 hdd 9.09569 1.00000 9.1 TiB 784 GiB 781 GiB 104 MiB 3.6 GiB > 8.3 TiB 8.42 0.43 247 up > 17 hdd 9.09569 1.00000 9.1 TiB 861 GiB 856 GiB 269 MiB 5.1 GiB > 8.3 TiB 9.25 0.47 102 up > 18 hdd 9.09569 1.00000 9.1 TiB 1.9 TiB 1.9 TiB 962 MiB 8.2 GiB > 7.2 TiB 21.15 1.09 283 up > 19 hdd 9.09569 1.00000 9.1 TiB 893 GiB 888 GiB 291 MiB 4.6 GiB > 8.2 TiB 9.59 0.49 148 up > TOTAL 255 TiB 50 TiB 49 TiB 9.7 GiB 187 GiB > 205 TiB 19.49 > MIN/MAX VAR: 0.24/4.66 STDDEV: 19.63 > > > > > *ceph health detail |grep -i wrn* > [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set > [WRN] OSD_NEARFULL: 2 nearfull osd(s) > [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this > doesn't resolve itself): 16 pgs backfill_toofull > [WRN] PG_NOT_DEEP_SCRUBBED: 1360 pgs not deep-scrubbed in time > [WRN] PG_NOT_SCRUBBED: 53 pgs not scrubbed in time > [WRN] POOL_NEARFULL: 36 pool(s) nearfull > > > Thanks the team ;) > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |