Eccessive occupation of small OSDs

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Dear Ceph users,

my cluster is made up of 10 old machines, with uneven number of disks and disk size. Essentially I have just one big data pool (6+2 erasure code, with host failure domain) for which I am currently experiencing a very poor available space (88 TB of which 40 TB occupied, as reported by df -h on hosts mounting the cephfs) compared to the raw one (196.5 TB). I have a total of 104 OSDs and 512 PGs for the pool; I cannot increment the PG number since the machines are old and with very low amount of RAM, and some of them are already overloaded.

In this situation I'm seeing a high occupation of small OSDs (500 MB) with respect to bigger ones (2 and 4 TB) even if the weight is set equal to disk capacity (see below for ceph osd tree). For example OSD 9 is at 62% occupancy even with weight 0.5 and reweight 0.75, while the highest occupancy for 2 TB OSDs is 41% (OSD 18) and 4 TB OSDs is 23% (OSD 79). I guess this high occupancy for 500 MB OSDs combined with erasure code size and host failure domain might be the cause of the poor available space, could this be true? The upmap balancer is currently running but I don't know if and how much it could improve the situation.
Any hint is greatly appreciated, thanks.

Nicola

# ceph osd tree       
ID   CLASS  WEIGHT     TYPE NAME         STATUS  REWEIGHT  PRI-AFF
 -1         196.47754  root default                               
 -7          14.55518      host aka                               
  4    hdd    1.81940          osd.4         up   1.00000  1.00000
 11    hdd    1.81940          osd.11        up   1.00000  1.00000
 18    hdd    1.81940          osd.18        up   1.00000  1.00000
 26    hdd    1.81940          osd.26        up   1.00000  1.00000
 32    hdd    1.81940          osd.32        up   1.00000  1.00000
 41    hdd    1.81940          osd.41        up   1.00000  1.00000
 48    hdd    1.81940          osd.48        up   1.00000  1.00000
 55    hdd    1.81940          osd.55        up   1.00000  1.00000
 -3          14.55518      host balin                             
  0    hdd    1.81940          osd.0         up   1.00000  1.00000
  8    hdd    1.81940          osd.8         up   1.00000  1.00000
 15    hdd    1.81940          osd.15        up   1.00000  1.00000
 22    hdd    1.81940          osd.22        up   1.00000  1.00000
 29    hdd    1.81940          osd.29        up   1.00000  1.00000
 34    hdd    1.81940          osd.34        up   1.00000  1.00000
 43    hdd    1.81940          osd.43        up   1.00000  1.00000
 49    hdd    1.81940          osd.49        up   1.00000  1.00000
-13          29.10950      host bifur                             
  3    hdd    3.63869          osd.3         up   1.00000  1.00000
 14    hdd    3.63869          osd.14        up   1.00000  1.00000
 27    hdd    3.63869          osd.27        up   1.00000  1.00000
 37    hdd    3.63869          osd.37        up   1.00000  1.00000
 50    hdd    3.63869          osd.50        up   1.00000  1.00000
 59    hdd    3.63869          osd.59        up   1.00000  1.00000
 64    hdd    3.63869          osd.64        up   1.00000  1.00000
 69    hdd    3.63869          osd.69        up   1.00000  1.00000
-17          29.10950      host bofur                             
  2    hdd    3.63869          osd.2         up   1.00000  1.00000
 21    hdd    3.63869          osd.21        up   1.00000  1.00000
 39    hdd    3.63869          osd.39        up   1.00000  1.00000
 57    hdd    3.63869          osd.57        up   1.00000  1.00000
 66    hdd    3.63869          osd.66        up   1.00000  1.00000
 72    hdd    3.63869          osd.72        up   1.00000  1.00000
 76    hdd    3.63869          osd.76        up   1.00000  1.00000
 79    hdd    3.63869          osd.79        up   1.00000  1.00000
-21          29.10376      host dwalin                            
 88    hdd    1.81898          osd.88        up   1.00000  1.00000
 89    hdd    1.81898          osd.89        up   1.00000  1.00000
 90    hdd    1.81898          osd.90        up   1.00000  1.00000
 91    hdd    1.81898          osd.91        up   1.00000  1.00000
 92    hdd    1.81898          osd.92        up   1.00000  1.00000
 93    hdd    1.81898          osd.93        up   1.00000  1.00000
 94    hdd    1.81898          osd.94        up   1.00000  1.00000
 95    hdd    1.81898          osd.95        up   1.00000  1.00000
 96    hdd    1.81898          osd.96        up   1.00000  1.00000
 97    hdd    1.81898          osd.97        up   1.00000  1.00000
 98    hdd    1.81898          osd.98        up   1.00000  1.00000
 99    hdd    1.81898          osd.99        up   1.00000  1.00000
100    hdd    1.81898          osd.100       up   1.00000  1.00000
101    hdd    1.81898          osd.101       up   1.00000  1.00000
102    hdd    1.81898          osd.102       up   1.00000  1.00000
103    hdd    1.81898          osd.103       up   1.00000  1.00000
 -9          14.55518      host ogion                             
  7    hdd    1.81940          osd.7         up   1.00000  1.00000
 16    hdd    1.81940          osd.16        up   1.00000  1.00000
 23    hdd    1.81940          osd.23        up   1.00000  1.00000
 33    hdd    1.81940          osd.33        up   1.00000  1.00000
 40    hdd    1.81940          osd.40        up   1.00000  1.00000
 47    hdd    1.81940          osd.47        up   1.00000  1.00000
 54    hdd    1.81940          osd.54        up   1.00000  1.00000
 61    hdd    1.81940          osd.61        up   1.00000  1.00000
-19          14.55518      host prestno                           
 81    hdd    1.81940          osd.81        up   1.00000  1.00000
 82    hdd    1.81940          osd.82        up   1.00000  1.00000
 83    hdd    1.81940          osd.83        up   1.00000  1.00000
 84    hdd    1.81940          osd.84        up   1.00000  1.00000
 85    hdd    1.81940          osd.85        up   1.00000  1.00000
 86    hdd    1.81940          osd.86        up   1.00000  1.00000
 87    hdd    1.81940          osd.87        up   1.00000  1.00000
104    hdd    1.81940          osd.104       up   1.00000  1.00000
-15          29.10376      host remolo                            
  6    hdd    1.81897          osd.6         up   1.00000  1.00000
 12    hdd    1.81897          osd.12        up   1.00000  1.00000
 19    hdd    1.81897          osd.19        up   1.00000  1.00000
 28    hdd    1.81897          osd.28        up   1.00000  1.00000
 35    hdd    1.81897          osd.35        up   1.00000  1.00000
 44    hdd    1.81897          osd.44        up   1.00000  1.00000
 52    hdd    1.81897          osd.52        up   1.00000  1.00000
 58    hdd    1.81897          osd.58        up   1.00000  1.00000
 63    hdd    1.81897          osd.63        up   1.00000  1.00000
 67    hdd    1.81897          osd.67        up   1.00000  1.00000
 71    hdd    1.81897          osd.71        up   1.00000  1.00000
 73    hdd    1.81897          osd.73        up   1.00000  1.00000
 74    hdd    1.81897          osd.74        up   1.00000  1.00000
 75    hdd    1.81897          osd.75        up   1.00000  1.00000
 77    hdd    1.81897          osd.77        up   1.00000  1.00000
 78    hdd    1.81897          osd.78        up   1.00000  1.00000
 -5          14.55518      host rokanan                           
  1    hdd    1.81940          osd.1         up   1.00000  1.00000
 10    hdd    1.81940          osd.10        up   1.00000  1.00000
 17    hdd    1.81940          osd.17        up   1.00000  1.00000
 24    hdd    1.81940          osd.24        up   1.00000  1.00000
 31    hdd    1.81940          osd.31        up   1.00000  1.00000
 38    hdd    1.81940          osd.38        up   1.00000  1.00000
 46    hdd    1.81940          osd.46        up   1.00000  1.00000
 53    hdd    1.81940          osd.53        up   1.00000  1.00000
-11           7.27515      host romolo                            
  5    hdd    0.45470          osd.5         up   1.00000  1.00000
  9    hdd    0.45470          osd.9         up   0.75000  1.00000
 13    hdd    0.45470          osd.13        up   1.00000  1.00000
 20    hdd    0.45470          osd.20        up   0.95000  1.00000
 25    hdd    0.45470          osd.25        up   0.75000  1.00000
 30    hdd    0.45470          osd.30        up   1.00000  1.00000
 36    hdd    0.45470          osd.36        up   1.00000  1.00000
 42    hdd    0.45470          osd.42        up   1.00000  1.00000
 45    hdd    0.45470          osd.45        up   0.85004  1.00000
 51    hdd    0.45470          osd.51        up   0.89999  1.00000
 56    hdd    0.45470          osd.56        up   1.00000  1.00000
 60    hdd    0.45470          osd.60        up   1.00000  1.00000
 62    hdd    0.45470          osd.62        up   1.00000  1.00000
 65    hdd    0.45470          osd.65        up   0.85004  1.00000
 68    hdd    0.45470          osd.68        up   1.00000  1.00000
 70    hdd    0.45470          osd.70        up   1.00000  1.00000
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux