Re: Osd full

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Mohamed,

Changing weights is no longer a good practice.

The balancer is supposed to do the job.

The number of pg per osd is really tight on your infrastructure.

Can you display the ceph osd tree command?
________________________________________________________

Cordialement,

*David CASIER*




*Ligne directe: +33(0) 9 72 61 98 29*
________________________________________________________



Le lun. 11 déc. 2023 à 11:06, Mohamed LAMDAOUAR <mohamed.lamdaouar@xxxxxxx>
a écrit :

> Hello the team,
>
> We initially had a cluster of 3 machines with 4 osd on each machine, we
> added 4 machines in the cluster (each machine with 4 osd)
> We launched the balancing but it never finished, still in progress. But the
> big issue: we have an osd full and all the pools on this osd are read only.
>
> *ceph osd df *:
>
> ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP     META
>  AVAIL    %USE   VAR   PGS  STATUS
> 20    hdd  9.09569   1.00000  9.1 TiB  580 GiB  576 GiB  1.2 GiB  3.1 GiB
> 8.5 TiB   6.23  0.32  169      up
> 21    hdd  9.09569   1.00000  9.1 TiB  1.5 TiB  1.5 TiB  252 MiB  7.7 GiB
> 7.6 TiB  16.08  0.82  247      up
> 22    hdd  9.09569   1.00000  9.1 TiB  671 GiB  667 GiB  204 MiB  4.1 GiB
> 8.4 TiB   7.21  0.37  136      up
> 23    hdd  9.09569   1.00000  9.1 TiB  665 GiB  660 GiB  270 MiB  4.5 GiB
> 8.4 TiB   7.14  0.37  124      up
>  0    hdd  9.09569   1.00000  9.1 TiB  1.2 TiB  1.2 TiB   87 MiB  6.0 GiB
> 7.9 TiB  13.30  0.68  230      up
>  1    hdd  9.09569   1.00000  9.1 TiB  1.3 TiB  1.3 TiB  347 MiB  6.6 GiB
> 7.8 TiB  14.01  0.72  153      up
>  2    hdd  9.09569   0.65009  9.1 TiB  1.8 TiB  1.8 TiB  443 MiB  7.3 GiB
> 7.3 TiB  20.00  1.03  147      up
>  3    hdd  9.09569   1.00000  9.1 TiB  617 GiB  611 GiB  220 MiB  5.8 GiB
> 8.5 TiB   6.62  0.34  101      up
>  4    hdd  9.09569   0.80005  9.1 TiB  2.0 TiB  2.0 TiB  293 MiB  8.2 GiB
> 7.1 TiB  22.12  1.13  137      up
>  5    hdd  9.09569   1.00000  9.1 TiB  857 GiB  852 GiB  157 MiB  4.9 GiB
> 8.3 TiB   9.20  0.47  155      up
>  6    hdd  9.09569   1.00000  9.1 TiB  580 GiB  575 GiB  678 MiB  4.4 GiB
> 8.5 TiB   6.23  0.32  114      up
>  7    hdd  9.09569   0.50000  9.1 TiB  7.7 TiB  7.7 TiB  103 MiB   16 GiB
> 1.4 TiB  85.03  4.36  201      up
> 24    hdd  9.09569   1.00000  9.1 TiB  1.2 TiB  1.2 TiB  133 MiB  6.2 GiB
> 7.9 TiB  13.11  0.67  225      up
> 25    hdd  9.09569   0.34999  9.1 TiB  8.3 TiB  8.2 TiB  101 MiB   17 GiB
> 860 GiB  90.77  4.66  159      up
> 26    hdd  9.09569   1.00000  9.1 TiB  665 GiB  661 GiB  292 MiB  3.8 GiB
> 8.4 TiB   7.14  0.37  107      up
> 27    hdd  9.09569   1.00000  9.1 TiB  427 GiB  423 GiB  241 MiB  3.4 GiB
> 8.7 TiB   4.58  0.24  103      up
>  8    hdd  9.09569   1.00000  9.1 TiB  845 GiB  839 GiB  831 MiB  5.9 GiB
> 8.3 TiB   9.07  0.47  163      up
>  9    hdd  9.09569   1.00000  9.1 TiB  727 GiB  722 GiB  162 MiB  4.8 GiB
> 8.4 TiB   7.80  0.40  169      up
> 10    hdd  9.09569   0.80005  9.1 TiB  1.9 TiB  1.9 TiB  742 MiB  7.5 GiB
> 7.2 TiB  21.01  1.08  136      up
> 11    hdd  9.09569   1.00000  9.1 TiB  733 GiB  727 GiB  498 MiB  5.2 GiB
> 8.4 TiB   7.87  0.40  163      up
> 12    hdd  9.09569   1.00000  9.1 TiB  892 GiB  886 GiB  318 MiB  5.6 GiB
> 8.2 TiB   9.58  0.49  254      up
> 13    hdd  9.09569   1.00000  9.1 TiB  759 GiB  755 GiB   37 MiB  4.0 GiB
> 8.4 TiB   8.15  0.42  134      up
> 14    hdd  9.09569   0.85004  9.1 TiB  2.3 TiB  2.3 TiB  245 MiB  7.7 GiB
> 6.8 TiB  24.96  1.28  142      up
> 15    hdd  9.09569   1.00000  9.1 TiB  7.3 TiB  7.3 TiB  435 MiB   16 GiB
> 1.8 TiB  80.17  4.11  213      up
> 16    hdd  9.09569   1.00000  9.1 TiB  784 GiB  781 GiB  104 MiB  3.6 GiB
> 8.3 TiB   8.42  0.43  247      up
> 17    hdd  9.09569   1.00000  9.1 TiB  861 GiB  856 GiB  269 MiB  5.1 GiB
> 8.3 TiB   9.25  0.47  102      up
> 18    hdd  9.09569   1.00000  9.1 TiB  1.9 TiB  1.9 TiB  962 MiB  8.2 GiB
> 7.2 TiB  21.15  1.09  283      up
> 19    hdd  9.09569   1.00000  9.1 TiB  893 GiB  888 GiB  291 MiB  4.6 GiB
> 8.2 TiB   9.59  0.49  148      up
>                        TOTAL  255 TiB   50 TiB   49 TiB  9.7 GiB  187 GiB
> 205 TiB  19.49
> MIN/MAX VAR: 0.24/4.66  STDDEV: 19.63
>
>
>
>
> *ceph health detail |grep -i wrn*
> [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set
> [WRN] OSD_NEARFULL: 2 nearfull osd(s)
> [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this
> doesn't resolve itself): 16 pgs backfill_toofull
> [WRN] PG_NOT_DEEP_SCRUBBED: 1360 pgs not deep-scrubbed in time
> [WRN] PG_NOT_SCRUBBED: 53 pgs not scrubbed in time
> [WRN] POOL_NEARFULL: 36 pool(s) nearfull
>
>
> Thanks  the team ;)
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux