Hello the team,
We initially had a cluster of 3 machines with 4 osd on each machine, we
added 4 machines in the cluster (each machine with 4 osd)
We launched the balancing but it never finished, still in progress. But the
big issue: we have an osd full and all the pools on this osd are read only.
*ceph osd df *:
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS
20 hdd 9.09569 1.00000 9.1 TiB 580 GiB 576 GiB 1.2 GiB 3.1 GiB
8.5 TiB 6.23 0.32 169 up
21 hdd 9.09569 1.00000 9.1 TiB 1.5 TiB 1.5 TiB 252 MiB 7.7 GiB
7.6 TiB 16.08 0.82 247 up
22 hdd 9.09569 1.00000 9.1 TiB 671 GiB 667 GiB 204 MiB 4.1 GiB
8.4 TiB 7.21 0.37 136 up
23 hdd 9.09569 1.00000 9.1 TiB 665 GiB 660 GiB 270 MiB 4.5 GiB
8.4 TiB 7.14 0.37 124 up
0 hdd 9.09569 1.00000 9.1 TiB 1.2 TiB 1.2 TiB 87 MiB 6.0 GiB
7.9 TiB 13.30 0.68 230 up
1 hdd 9.09569 1.00000 9.1 TiB 1.3 TiB 1.3 TiB 347 MiB 6.6 GiB
7.8 TiB 14.01 0.72 153 up
2 hdd 9.09569 0.65009 9.1 TiB 1.8 TiB 1.8 TiB 443 MiB 7.3 GiB
7.3 TiB 20.00 1.03 147 up
3 hdd 9.09569 1.00000 9.1 TiB 617 GiB 611 GiB 220 MiB 5.8 GiB
8.5 TiB 6.62 0.34 101 up
4 hdd 9.09569 0.80005 9.1 TiB 2.0 TiB 2.0 TiB 293 MiB 8.2 GiB
7.1 TiB 22.12 1.13 137 up
5 hdd 9.09569 1.00000 9.1 TiB 857 GiB 852 GiB 157 MiB 4.9 GiB
8.3 TiB 9.20 0.47 155 up
6 hdd 9.09569 1.00000 9.1 TiB 580 GiB 575 GiB 678 MiB 4.4 GiB
8.5 TiB 6.23 0.32 114 up
7 hdd 9.09569 0.50000 9.1 TiB 7.7 TiB 7.7 TiB 103 MiB 16 GiB
1.4 TiB 85.03 4.36 201 up
24 hdd 9.09569 1.00000 9.1 TiB 1.2 TiB 1.2 TiB 133 MiB 6.2 GiB
7.9 TiB 13.11 0.67 225 up
25 hdd 9.09569 0.34999 9.1 TiB 8.3 TiB 8.2 TiB 101 MiB 17 GiB
860 GiB 90.77 4.66 159 up
26 hdd 9.09569 1.00000 9.1 TiB 665 GiB 661 GiB 292 MiB 3.8 GiB
8.4 TiB 7.14 0.37 107 up
27 hdd 9.09569 1.00000 9.1 TiB 427 GiB 423 GiB 241 MiB 3.4 GiB
8.7 TiB 4.58 0.24 103 up
8 hdd 9.09569 1.00000 9.1 TiB 845 GiB 839 GiB 831 MiB 5.9 GiB
8.3 TiB 9.07 0.47 163 up
9 hdd 9.09569 1.00000 9.1 TiB 727 GiB 722 GiB 162 MiB 4.8 GiB
8.4 TiB 7.80 0.40 169 up
10 hdd 9.09569 0.80005 9.1 TiB 1.9 TiB 1.9 TiB 742 MiB 7.5 GiB
7.2 TiB 21.01 1.08 136 up
11 hdd 9.09569 1.00000 9.1 TiB 733 GiB 727 GiB 498 MiB 5.2 GiB
8.4 TiB 7.87 0.40 163 up
12 hdd 9.09569 1.00000 9.1 TiB 892 GiB 886 GiB 318 MiB 5.6 GiB
8.2 TiB 9.58 0.49 254 up
13 hdd 9.09569 1.00000 9.1 TiB 759 GiB 755 GiB 37 MiB 4.0 GiB
8.4 TiB 8.15 0.42 134 up
14 hdd 9.09569 0.85004 9.1 TiB 2.3 TiB 2.3 TiB 245 MiB 7.7 GiB
6.8 TiB 24.96 1.28 142 up
15 hdd 9.09569 1.00000 9.1 TiB 7.3 TiB 7.3 TiB 435 MiB 16 GiB
1.8 TiB 80.17 4.11 213 up
16 hdd 9.09569 1.00000 9.1 TiB 784 GiB 781 GiB 104 MiB 3.6 GiB
8.3 TiB 8.42 0.43 247 up
17 hdd 9.09569 1.00000 9.1 TiB 861 GiB 856 GiB 269 MiB 5.1 GiB
8.3 TiB 9.25 0.47 102 up
18 hdd 9.09569 1.00000 9.1 TiB 1.9 TiB 1.9 TiB 962 MiB 8.2 GiB
7.2 TiB 21.15 1.09 283 up
19 hdd 9.09569 1.00000 9.1 TiB 893 GiB 888 GiB 291 MiB 4.6 GiB
8.2 TiB 9.59 0.49 148 up
TOTAL 255 TiB 50 TiB 49 TiB 9.7 GiB 187 GiB
205 TiB 19.49
MIN/MAX VAR: 0.24/4.66 STDDEV: 19.63
*ceph health detail |grep -i wrn*
[WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set
[WRN] OSD_NEARFULL: 2 nearfull osd(s)
[WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this
doesn't resolve itself): 16 pgs backfill_toofull
[WRN] PG_NOT_DEEP_SCRUBBED: 1360 pgs not deep-scrubbed in time
[WRN] PG_NOT_SCRUBBED: 53 pgs not scrubbed in time
[WRN] POOL_NEARFULL: 36 pool(s) nearfull
Thanks the team ;)
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
