Slow cluster / misplaced objects - Ceph 15.2.9

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

We've got an interesting issue we're running into on Ceph 15.2.9. We're
experiencing VERY slow performance from the cluster, and extremely slow
misplaced object correction, with very little cpu/disk/network utilization
(almost idle) across all nodes in the cluster.

We have 7 servers in this cluster, 24 rotational OSDs, and two NVMEs with
12 OSD's worth of DB/WAL files on them. The OSDs are all equal weighted, so
the tree is pretty straightforward:

root@ceph01:~# ceph osd tree

Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492

Inferring config
/var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config

Using recent ceph image
docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5

ID   CLASS  WEIGHT      TYPE NAME            STATUS  REWEIGHT  PRI-AFF

 -1         2149.39062  root default

 -2         2149.39062      rack rack1

 -5          307.05579          host ceph01

  0    hdd    12.79399              osd.0        up   1.00000  1.00000

  1    hdd    12.79399              osd.1        up   1.00000  1.00000

  2    hdd    12.79399              osd.2        up   1.00000  1.00000

  3    hdd    12.79399              osd.3        up   1.00000  1.00000

  4    hdd    12.79399              osd.4        up   1.00000  1.00000

  5    hdd    12.79399              osd.5        up   1.00000  1.00000

  6    hdd    12.79399              osd.6        up   1.00000  1.00000

  7    hdd    12.79399              osd.7        up   1.00000  1.00000

  8    hdd    12.79399              osd.8        up   1.00000  1.00000

  9    hdd    12.79399              osd.9        up   1.00000  1.00000

 10    hdd    12.79399              osd.10       up   1.00000  1.00000

 11    hdd    12.79399              osd.11       up   1.00000  1.00000

 12    hdd    12.79399              osd.12       up   1.00000  1.00000

 13    hdd    12.79399              osd.13       up   1.00000  1.00000

 14    hdd    12.79399              osd.14       up   1.00000  1.00000

 15    hdd    12.79399              osd.15       up   1.00000  1.00000

 16    hdd    12.79399              osd.16       up   1.00000  1.00000

 17    hdd    12.79399              osd.17       up   1.00000  1.00000

 18    hdd    12.79399              osd.18       up   1.00000  1.00000

 19    hdd    12.79399              osd.19       up   1.00000  1.00000

 20    hdd    12.79399              osd.20       up   1.00000  1.00000

 21    hdd    12.79399              osd.21       up   1.00000  1.00000

 22    hdd    12.79399              osd.22       up   1.00000  1.00000

 23    hdd    12.79399              osd.23       up   1.00000  1.00000

 -7          307.05579          host ceph02

 24    hdd    12.79399              osd.24       up   1.00000  1.00000

 25    hdd    12.79399              osd.25       up   1.00000  1.00000

 26    hdd    12.79399              osd.26       up   1.00000  1.00000

 27    hdd    12.79399              osd.27       up   1.00000  1.00000

 28    hdd    12.79399              osd.28       up   1.00000  1.00000

 29    hdd    12.79399              osd.29       up   1.00000  1.00000

 30    hdd    12.79399              osd.30       up   1.00000  1.00000

 31    hdd    12.79399              osd.31       up   1.00000  1.00000

 32    hdd    12.79399              osd.32       up   1.00000  1.00000

 33    hdd    12.79399              osd.33       up   1.00000  1.00000

 34    hdd    12.79399              osd.34       up   1.00000  1.00000

 35    hdd    12.79399              osd.35       up   1.00000  1.00000

 36    hdd    12.79399              osd.36       up   1.00000  1.00000

 37    hdd    12.79399              osd.37       up   1.00000  1.00000

 38    hdd    12.79399              osd.38       up   1.00000  1.00000

 39    hdd    12.79399              osd.39       up   1.00000  1.00000

 40    hdd    12.79399              osd.40       up   1.00000  1.00000

 41    hdd    12.79399              osd.41       up   1.00000  1.00000

 42    hdd    12.79399              osd.42       up   1.00000  1.00000

 43    hdd    12.79399              osd.43       up   1.00000  1.00000

 44    hdd    12.79399              osd.44       up   1.00000  1.00000

 45    hdd    12.79399              osd.45       up   1.00000  1.00000

 46    hdd    12.79399              osd.46       up   1.00000  1.00000

 47    hdd    12.79399              osd.47       up   1.00000  1.00000

 -9          307.05579          host ceph03

 48    hdd    12.79399              osd.48       up   1.00000  1.00000

 49    hdd    12.79399              osd.49       up   1.00000  1.00000

 50    hdd    12.79399              osd.50       up   1.00000  1.00000

 51    hdd    12.79399              osd.51       up   1.00000  1.00000

 52    hdd    12.79399              osd.52       up   1.00000  1.00000

 53    hdd    12.79399              osd.53       up   1.00000  1.00000

 54    hdd    12.79399              osd.54       up   1.00000  1.00000

 55    hdd    12.79399              osd.55       up   1.00000  1.00000

 56    hdd    12.79399              osd.56       up   1.00000  1.00000

 57    hdd    12.79399              osd.57       up   1.00000  1.00000

 58    hdd    12.79399              osd.58       up   1.00000  1.00000

 59    hdd    12.79399              osd.59       up   1.00000  1.00000

 60    hdd    12.79399              osd.60       up   1.00000  1.00000

 61    hdd    12.79399              osd.61       up   1.00000  1.00000

 62    hdd    12.79399              osd.62       up   1.00000  1.00000

 63    hdd    12.79399              osd.63       up   1.00000  1.00000

 64    hdd    12.79399              osd.64       up   1.00000  1.00000

 65    hdd    12.79399              osd.65       up   1.00000  1.00000

 66    hdd    12.79399              osd.66       up   1.00000  1.00000

 67    hdd    12.79399              osd.67       up   1.00000  1.00000

 68    hdd    12.79399              osd.68       up   1.00000  1.00000

 69    hdd    12.79399              osd.69       up   1.00000  1.00000

 70    hdd    12.79399              osd.70       up   1.00000  1.00000

 71    hdd    12.79399              osd.71       up   1.00000  1.00000

-11          307.05579          host ceph04

 72    hdd    12.79399              osd.72       up   1.00000  1.00000

 73    hdd    12.79399              osd.73       up   1.00000  1.00000

 74    hdd    12.79399              osd.74       up   1.00000  1.00000

 75    hdd    12.79399              osd.75       up   1.00000  1.00000

 76    hdd    12.79399              osd.76       up   1.00000  1.00000

 77    hdd    12.79399              osd.77       up   1.00000  1.00000

 78    hdd    12.79399              osd.78       up   1.00000  1.00000

 79    hdd    12.79399              osd.79       up   1.00000  1.00000

 80    hdd    12.79399              osd.80       up   1.00000  1.00000

 81    hdd    12.79399              osd.81       up   1.00000  1.00000

 82    hdd    12.79399              osd.82       up   1.00000  1.00000

 83    hdd    12.79399              osd.83       up   1.00000  1.00000

 84    hdd    12.79399              osd.84       up   1.00000  1.00000

 85    hdd    12.79399              osd.85       up   1.00000  1.00000

 86    hdd    12.79399              osd.86       up   1.00000  1.00000

 87    hdd    12.79399              osd.87       up   1.00000  1.00000

 88    hdd    12.79399              osd.88       up   1.00000  1.00000

 89    hdd    12.79399              osd.89       up   1.00000  1.00000

 90    hdd    12.79399              osd.90       up   1.00000  1.00000

 91    hdd    12.79399              osd.91       up   1.00000  1.00000

 92    hdd    12.79399              osd.92       up   1.00000  1.00000

 93    hdd    12.79399              osd.93       up   1.00000  1.00000

 94    hdd    12.79399              osd.94       up   1.00000  1.00000

 95    hdd    12.79399              osd.95       up   1.00000  1.00000

-13          307.05579          host ceph05

 96    hdd    12.79399              osd.96       up   1.00000  1.00000

 97    hdd    12.79399              osd.97       up   1.00000  1.00000

 98    hdd    12.79399              osd.98       up   1.00000  1.00000

 99    hdd    12.79399              osd.99       up   1.00000  1.00000

100    hdd    12.79399              osd.100      up   1.00000  1.00000

101    hdd    12.79399              osd.101      up   1.00000  1.00000

102    hdd    12.79399              osd.102      up   1.00000  1.00000

103    hdd    12.79399              osd.103      up   1.00000  1.00000

104    hdd    12.79399              osd.104      up   1.00000  1.00000

105    hdd    12.79399              osd.105      up   1.00000  1.00000

106    hdd    12.79399              osd.106      up   1.00000  1.00000

107    hdd    12.79399              osd.107      up   1.00000  1.00000

108    hdd    12.79399              osd.108      up   1.00000  1.00000

109    hdd    12.79399              osd.109      up   1.00000  1.00000

110    hdd    12.79399              osd.110      up   1.00000  1.00000

111    hdd    12.79399              osd.111      up   1.00000  1.00000

112    hdd    12.79399              osd.112      up   1.00000  1.00000

113    hdd    12.79399              osd.113      up   1.00000  1.00000

114    hdd    12.79399              osd.114      up   1.00000  1.00000

115    hdd    12.79399              osd.115      up   1.00000  1.00000

116    hdd    12.79399              osd.116      up   1.00000  1.00000

117    hdd    12.79399              osd.117      up   1.00000  1.00000

118    hdd    12.79399              osd.118      up   1.00000  1.00000

119    hdd    12.79399              osd.119      up   1.00000  1.00000

-15          307.05579          host ceph06

120    hdd    12.79399              osd.120      up   1.00000  1.00000

121    hdd    12.79399              osd.121      up   1.00000  1.00000

122    hdd    12.79399              osd.122      up   1.00000  1.00000

123    hdd    12.79399              osd.123      up   1.00000  1.00000

124    hdd    12.79399              osd.124      up   1.00000  1.00000

125    hdd    12.79399              osd.125      up   1.00000  1.00000

126    hdd    12.79399              osd.126      up   1.00000  1.00000

127    hdd    12.79399              osd.127      up   1.00000  1.00000

128    hdd    12.79399              osd.128      up   1.00000  1.00000

129    hdd    12.79399              osd.129      up   1.00000  1.00000

130    hdd    12.79399              osd.130      up   1.00000  1.00000

131    hdd    12.79399              osd.131      up   1.00000  1.00000

132    hdd    12.79399              osd.132      up   1.00000  1.00000

133    hdd    12.79399              osd.133      up   1.00000  1.00000

134    hdd    12.79399              osd.134      up   1.00000  1.00000

135    hdd    12.79399              osd.135      up   1.00000  1.00000

136    hdd    12.79399              osd.136      up   1.00000  1.00000

137    hdd    12.79399              osd.137      up   1.00000  1.00000

138    hdd    12.79399              osd.138      up   1.00000  1.00000

139    hdd    12.79399              osd.139      up   1.00000  1.00000

140    hdd    12.79399              osd.140      up   1.00000  1.00000

141    hdd    12.79399              osd.141      up   1.00000  1.00000

142    hdd    12.79399              osd.142      up   1.00000  1.00000

143    hdd    12.79399              osd.143      up   1.00000  1.00000

-17          307.05579          host ceph07

144    hdd    12.79399              osd.144      up   1.00000  1.00000

145    hdd    12.79399              osd.145      up   1.00000  1.00000

146    hdd    12.79399              osd.146      up   1.00000  1.00000

147    hdd    12.79399              osd.147      up   1.00000  1.00000

148    hdd    12.79399              osd.148      up   1.00000  1.00000

149    hdd    12.79399              osd.149      up   1.00000  1.00000

150    hdd    12.79399              osd.150      up   1.00000  1.00000

151    hdd    12.79399              osd.151      up   1.00000  1.00000

152    hdd    12.79399              osd.152      up   1.00000  1.00000

153    hdd    12.79399              osd.153      up   1.00000  1.00000

154    hdd    12.79399              osd.154      up   1.00000  1.00000

155    hdd    12.79399              osd.155      up   1.00000  1.00000

156    hdd    12.79399              osd.156      up   1.00000  1.00000

157    hdd    12.79399              osd.157      up   1.00000  1.00000

158    hdd    12.79399              osd.158      up   1.00000  1.00000

159    hdd    12.79399              osd.159      up   1.00000  1.00000

160    hdd    12.79399              osd.160      up   1.00000  1.00000

161    hdd    12.79399              osd.161      up   1.00000  1.00000

162    hdd    12.79399              osd.162      up   1.00000  1.00000

163    hdd    12.79399              osd.163      up   1.00000  1.00000

164    hdd    12.79399              osd.164      up   1.00000  1.00000

165    hdd    12.79399              osd.165      up   1.00000  1.00000

166    hdd    12.79399              osd.166      up   1.00000  1.00000

167    hdd    12.79399              osd.167      up   1.00000  1.00000

root@ceph01:~#

The data distribution looks relatively even (we use upmap as the balancer):

root@ceph01:~# ceph osd df

Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492

Inferring config
/var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config

Using recent ceph image
docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5

ID   CLASS  WEIGHT    REWEIGHT  SIZE     RAW USE  DATA      OMAP      META
    AVAIL    %USE   VAR   PGS  STATUS

  0    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   5.0 MiB   14
GiB  7.1 TiB  44.73  0.93   35      up

  1    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   1.8 MiB   13
GiB  7.2 TiB  43.46  0.90   33      up

  2    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   4.4 MiB   16
GiB  6.2 TiB  51.27  1.07   39      up

  3    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   4.9 MiB   15
GiB  6.7 TiB  47.38  0.99   36      up

  4    hdd  12.79399   1.00000   13 TiB  5.5 TiB   5.5 TiB   812 KiB   13
GiB  7.2 TiB  43.37  0.90   35      up

  5    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB    31 KiB   16
GiB  6.2 TiB  51.27  1.07   40      up

  6    hdd  12.79399   1.00000   13 TiB  5.5 TiB   5.5 TiB   506 KiB   13
GiB  7.3 TiB  43.32  0.90   34      up

  7    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   904 KiB   15
GiB  6.4 TiB  49.93  1.04   38      up

  8    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   3.6 MiB   14
GiB  6.7 TiB  47.35  0.98   37      up

  9    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   5.5 MiB   16
GiB  6.1 TiB  52.58  1.09   40      up

 10    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.6 TiB    75 MiB   16
GiB  6.1 TiB  52.42  1.09   41      up

 11    hdd  12.79399   1.00000   13 TiB  5.5 TiB   5.5 TiB   1.4 MiB   13
GiB  7.2 TiB  43.36  0.90   34      up

 12    hdd  12.79399   1.00000   13 TiB  5.5 TiB   5.5 TiB   586 KiB   13
GiB  7.3 TiB  43.31  0.90   33      up

 13    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   222 KiB   14
GiB  7.1 TiB  44.71  0.93   35      up

 14    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   1.0 MiB   16
GiB  6.1 TiB  52.58  1.09   41      up

 15    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.2 MiB   14
GiB  7.1 TiB  44.73  0.93   34      up

 16    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   1.0 MiB   16
GiB  6.1 TiB  52.53  1.09   40      up

 17    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   3.3 MiB   15
GiB  6.4 TiB  49.92  1.04   38      up

 18    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   1.0 MiB   16
GiB  6.1 TiB  52.61  1.09   45      up

 19    hdd  12.79399   1.00000   13 TiB  5.4 TiB   5.3 TiB   4.0 MiB   13
GiB  7.4 TiB  42.15  0.88   34      up

 20    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   705 KiB   16
GiB  6.1 TiB  52.53  1.09   40      up

 21    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   2.5 MiB   16
GiB  6.1 TiB  52.58  1.09   40      up

 22    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   4.7 MiB   16
GiB  6.1 TiB  52.59  1.09   40      up

 23    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   4.9 MiB   16
GiB  6.1 TiB  52.58  1.09   41      up

 24    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   3.0 MiB   14
GiB  6.7 TiB  47.35  0.98   36      up

 25    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   4.1 MiB   16
GiB  6.1 TiB  52.56  1.09   41      up

 26    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   592 KiB   14
GiB  7.1 TiB  44.75  0.93   34      up

 27    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   2.8 MiB   14
GiB  6.9 TiB  46.07  0.96   36      up

 28    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   5.4 MiB   16
GiB  6.1 TiB  52.56  1.09   40      up

 29    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   340 KiB   15
GiB  6.4 TiB  49.92  1.04   39      up

 30    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB    93 KiB   14
GiB  6.9 TiB  46.03  0.96   35      up

 31    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   2.1 MiB   13
GiB  7.2 TiB  43.57  0.91   35      up

 32    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   5.0 MiB   15
GiB  6.6 TiB  48.68  1.01   38      up

 33    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   175 KiB   13
GiB  7.2 TiB  43.44  0.90   33      up

 34    hdd  12.79399   1.00000   13 TiB  6.9 TiB   6.8 TiB    77 MiB   16
GiB  5.9 TiB  53.86  1.12   43      up

 35    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   2.3 MiB   15
GiB  6.6 TiB  48.68  1.01   37      up

 36    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   1.2 MiB   16
GiB  6.2 TiB  51.25  1.07   41      up

 37    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   5.0 MiB   14
GiB  6.7 TiB  47.37  0.99   36      up

 38    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   3.7 MiB   13
GiB  7.2 TiB  43.44  0.90   35      up

 39    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   341 KiB   15
GiB  6.7 TiB  47.38  0.99   37      up

 40    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.1 MiB   14
GiB  7.1 TiB  44.73  0.93   34      up

 41    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB    22 KiB   16
GiB  6.1 TiB  52.59  1.09   40      up

 42    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   1.9 MiB   14
GiB  6.7 TiB  47.33  0.98   36      up

 43    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   485 KiB   16
GiB  6.1 TiB  52.59  1.09   40      up

 44    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   2.1 MiB   16
GiB  6.1 TiB  52.56  1.09   40      up

 45    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   2.3 MiB   16
GiB  6.1 TiB  52.53  1.09   40      up

 46    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   124 KiB   14
GiB  6.9 TiB  46.08  0.96   37      up

 47    hdd  12.79399   1.00000   13 TiB  5.5 TiB   5.5 TiB   1.2 MiB   13
GiB  7.2 TiB  43.37  0.90   33      up

 48    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   141 KiB   16
GiB  6.1 TiB  52.58  1.09   40      up

 49    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   1.9 MiB   13
GiB  7.2 TiB  43.64  0.91   33      up

 50    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   745 KiB   16
GiB  6.1 TiB  52.52  1.09   41      up

 51    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   186 KiB   16
GiB  6.1 TiB  52.55  1.09   41      up

 52    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   4.0 MiB   16
GiB  6.2 TiB  51.24  1.07   41      up

 53    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   737 KiB   14
GiB  6.7 TiB  47.33  0.98   38      up

 54    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   1.3 MiB   14
GiB  7.1 TiB  44.66  0.93   34      up

 55    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB    97 KiB   14
GiB  7.1 TiB  44.73  0.93   34      up

 56    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   1.2 MiB   14
GiB  7.1 TiB  44.76  0.93   35      up

 57    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   3.5 MiB   14
GiB  7.1 TiB  44.78  0.93   35      up

 58    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB    80 MiB   16
GiB  6.1 TiB  52.56  1.09   41      up

 59    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   4.5 MiB   14
GiB  6.9 TiB  46.08  0.96   36      up

 60    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   903 KiB   14
GiB  7.1 TiB  44.73  0.93   34      up

 61    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB     1 KiB   14
GiB  7.1 TiB  44.65  0.93   34      up

 62    hdd  12.79399   1.00000   13 TiB  6.9 TiB   6.8 TiB   425 KiB   16
GiB  5.9 TiB  53.87  1.12   41      up

 63    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   4.1 MiB   14
GiB  7.2 TiB  43.56  0.91   34      up

 64    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   578 KiB   16
GiB  6.2 TiB  51.27  1.07   39      up

 65    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   3.6 MiB   13
GiB  7.2 TiB  43.44  0.90   33      up

 66    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   288 KiB   13
GiB  7.2 TiB  43.40  0.90   34      up

 67    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   413 KiB   14
GiB  6.7 TiB  47.34  0.98   37      up

 68    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   4.3 MiB   16
GiB  6.1 TiB  52.56  1.09   41      up

 69    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   1.0 MiB   13
GiB  7.2 TiB  43.44  0.90   33      up

 70    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   3.2 MiB   16
GiB  6.1 TiB  52.56  1.09   40      up

 71    hdd  12.79399   1.00000   13 TiB  5.5 TiB   5.5 TiB   1.3 MiB   14
GiB  7.2 TiB  43.36  0.90   35      up

 72    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   4.7 MiB   16
GiB  6.2 TiB  51.29  1.07   39      up

 73    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   1.8 MiB   14
GiB  7.1 TiB  44.73  0.93   34      up

 74    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.6 TiB   1.2 MiB   14
GiB  7.1 TiB  44.64  0.93   34      up

 75    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   1.5 MiB   15
GiB  6.4 TiB  49.95  1.04   38      up

 76    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   3.8 MiB   15
GiB  6.6 TiB  48.68  1.01   37      up

 77    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   5.0 MiB   14
GiB  7.1 TiB  44.65  0.93   34      up

 78    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   1.8 MiB   14
GiB  7.1 TiB  44.68  0.93   35      up

 79    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   984 KiB   14
GiB  6.7 TiB  47.36  0.98   36      up

 80    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   2.6 MiB   13
GiB  7.2 TiB  43.45  0.90   34      up

 81    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   1.2 MiB   16
GiB  6.2 TiB  51.26  1.07   41      up

 82    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   4.9 MiB   15
GiB  6.6 TiB  48.61  1.01   37      up

 83    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   827 KiB   14
GiB  6.7 TiB  47.33  0.98   36      up

 84    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   2.7 MiB   16
GiB  6.2 TiB  51.22  1.07   39      up

 85    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   868 KiB   15
GiB  6.4 TiB  49.94  1.04   40      up

 86    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   4.1 MiB   16
GiB  6.1 TiB  52.57  1.09   42      up

 87    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   702 KiB   16
GiB  6.1 TiB  52.54  1.09   41      up

 88    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB    12 KiB   14
GiB  7.1 TiB  44.76  0.93   35      up

 89    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   2.9 MiB   14
GiB  6.9 TiB  46.07  0.96   35      up

 90    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   1.8 MiB   13
GiB  7.2 TiB  43.48  0.90   34      up

 91    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   836 KiB   16
GiB  6.1 TiB  52.55  1.09   40      up

 92    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   4.0 MiB   13
GiB  7.2 TiB  43.43  0.90   33      up

 93    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   1.5 MiB   16
GiB  6.1 TiB  52.57  1.09   41      up

 94    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   459 KiB   13
GiB  7.2 TiB  43.44  0.90   34      up

 95    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   4.1 MiB   16
GiB  6.2 TiB  51.28  1.07   39      up

 96    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.1 MiB   14
GiB  7.1 TiB  44.75  0.93   35      up

 97    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   662 KiB   14
GiB  7.1 TiB  44.75  0.93   34      up

 98    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   2.9 MiB   14
GiB  6.9 TiB  45.97  0.96   35      up

 99    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   3.8 MiB   16
GiB  6.1 TiB  52.56  1.09   40      up

100    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   1.8 MiB   14
GiB  7.1 TiB  44.76  0.93   36      up

101    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   3.6 MiB   15
GiB  6.4 TiB  49.95  1.04   40      up

102    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.9 MiB   14
GiB  7.1 TiB  44.65  0.93   34      up

103    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   643 KiB   16
GiB  6.1 TiB  52.57  1.09   41      up

104    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   5.5 MiB   15
GiB  6.4 TiB  49.98  1.04   38      up

105    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   830 KiB   13
GiB  7.2 TiB  43.46  0.90   33      up

106    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   1.0 MiB   14
GiB  6.7 TiB  47.34  0.98   36      up

107    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   286 KiB   16
GiB  6.2 TiB  51.23  1.07   39      up

108    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   961 KiB   15
GiB  6.6 TiB  48.68  1.01   37      up

109    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   291 KiB   16
GiB  6.1 TiB  52.56  1.09   42      up

110    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   1.0 MiB   14
GiB  6.9 TiB  46.02  0.96   36      up

111    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   2.7 MiB   16
GiB  6.2 TiB  51.22  1.07   39      up

112    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   631 KiB   15
GiB  6.6 TiB  48.67  1.01   37      up

113    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   4.5 MiB   16
GiB  6.1 TiB  52.56  1.09   41      up

114    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   4.0 MiB   15
GiB  6.4 TiB  49.94  1.04   39      up

115    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   1.5 MiB   14
GiB  6.9 TiB  46.05  0.96   35      up

116    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   4.1 MiB   15
GiB  6.4 TiB  49.93  1.04   38      up

117    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   194 KiB   13
GiB  7.2 TiB  43.44  0.90   35      up

118    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   3.1 MiB   15
GiB  6.4 TiB  49.96  1.04   38      up

119    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   3.9 MiB   14
GiB  7.2 TiB  43.46  0.90   33      up

120    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   1.8 MiB   16
GiB  6.1 TiB  52.55  1.09   40      up

121    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   3.7 MiB   16
GiB  6.1 TiB  52.54  1.09   40      up

122    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   2.4 MiB   15
GiB  6.4 TiB  49.97  1.04   38      up

123    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB  1003 KiB   14
GiB  6.9 TiB  46.02  0.96   35      up

124    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   526 KiB   14
GiB  6.7 TiB  47.35  0.98   36      up

125    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB    92 KiB   15
GiB  6.7 TiB  47.37  0.98   37      up

126    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.9 MiB   14
GiB  7.1 TiB  44.74  0.93   34      up

127    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   5.5 MiB   16
GiB  6.2 TiB  51.26  1.07   39      up

128    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   5.4 MiB   16
GiB  6.1 TiB  52.56  1.09   40      up

129    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   3.6 MiB   15
GiB  6.6 TiB  48.66  1.01   38      up

130    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   2.8 MiB   16
GiB  6.1 TiB  52.56  1.09   41      up

131    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   5.1 MiB   16
GiB  6.2 TiB  51.28  1.07   40      up

132    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   1.7 MiB   16
GiB  6.2 TiB  51.25  1.07   39      up

133    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   230 KiB   16
GiB  6.2 TiB  51.25  1.07   39      up

134    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   2.0 MiB   14
GiB  7.2 TiB  43.45  0.90   34      up

135    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   4.4 MiB   13
GiB  7.2 TiB  43.46  0.90   33      up

136    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   5.6 MiB   16
GiB  6.2 TiB  51.29  1.07   39      up

137    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   3.6 MiB   14
GiB  7.1 TiB  44.75  0.93   36      up

138    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   4.9 MiB   15
GiB  6.4 TiB  49.98  1.04   40      up

139    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB    21 KiB   16
GiB  6.2 TiB  51.23  1.07   39      up

140    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   5.4 MiB   14
GiB  7.1 TiB  44.74  0.93   36      up

141    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   359 KiB   13
GiB  7.2 TiB  43.42  0.90   34      up

142    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB    27 KiB   13
GiB  7.2 TiB  43.46  0.90   33      up

143    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   395 KiB   14
GiB  6.9 TiB  46.07  0.96   35      up

144    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   1.2 MiB   14
GiB  7.1 TiB  44.75  0.93   34      up

145    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   3.3 MiB   14
GiB  7.1 TiB  44.66  0.93   35      up

146    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   4.0 MiB   14
GiB  6.9 TiB  46.06  0.96   35      up

147    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB    46 KiB   16
GiB  6.1 TiB  52.55  1.09   40      up

148    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   4.8 MiB   16
GiB  6.2 TiB  51.23  1.07   39      up

149    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.4 MiB   14
GiB  7.1 TiB  44.70  0.93   35      up

150    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.4 MiB   14
GiB  7.1 TiB  44.65  0.93   35      up

151    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   3.7 MiB   14
GiB  7.1 TiB  44.65  0.93   35      up

152    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   5.3 MiB   15
GiB  6.2 TiB  51.26  1.07   39      up

153    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   2.3 MiB   15
GiB  6.4 TiB  49.94  1.04   39      up

154    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   3.0 MiB   14
GiB  6.7 TiB  47.37  0.99   36      up

155    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   3.0 MiB   14
GiB  6.7 TiB  47.35  0.98   38      up

156    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   4.0 MiB   16
GiB  6.2 TiB  51.26  1.07   41      up

157    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   2.8 MiB   14
GiB  6.9 TiB  46.07  0.96   36      up

158    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   3.0 MiB   16
GiB  6.2 TiB  51.25  1.07   40      up

159    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   3.9 MiB   15
GiB  6.4 TiB  49.93  1.04   38      up

160    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   2.4 MiB   14
GiB  6.9 TiB  46.03  0.96   35      up

161    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   1.1 MiB   16
GiB  6.2 TiB  51.25  1.07   39      up

162    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.6 TiB   4.1 MiB   13
GiB  7.1 TiB  44.64  0.93   34      up

163    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   445 KiB   14
GiB  7.1 TiB  44.69  0.93   34      up

164    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   4.9 MiB   15
GiB  6.4 TiB  49.97  1.04   38      up

165    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   5.5 MiB   16
GiB  6.2 TiB  51.25  1.07   40      up

166    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   2.5 MiB   13
GiB  7.2 TiB  43.45  0.90   33      up

167    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   4.0 MiB   15
GiB  6.4 TiB  49.98  1.04   38      up

                         TOTAL  2.1 PiB  1.0 PiB  1023 TiB   630 MiB  2.4
TiB  1.1 PiB  48.09

MIN/MAX VAR: 0.88/1.12  STDDEV: 3.50

root@ceph01:~#


The cluster status, however, shows we have many misplaced objects and a few
remapped PGs. We had been performing maintenance (upgrade to Ceph 15.2.9),
and initially the misplaced object count was dropping rapidly at a rate of
about 400-500MB/s and 100-200 objects/s. However, once it hit a certain
point, it dropped to 0-4MB/s and 0 objects/s (it's not 0, it is
progressing, but it's below 1 object/s):


root@ceph01:~# ceph -s

Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492

Inferring config
/var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config

Using recent ceph image
docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5

  cluster:

    id:     41bb9256-c3bf-11ea-85b9-9e07b0435492

    health: HEALTH_WARN

            5 slow ops, oldest one blocked for 84 sec, daemons
[osd.141,osd.46] have slow ops.



  services:

    mon: 5 daemons, quorum ceph01,ceph04,ceph02,ceph03,ceph05 (age 25h)

    mgr: ceph05.yropto(active, since 11h), standbys: ceph01.aqkgbl,
ceph03.ytkuyr, ceph04.smbdew, ceph02.ndynmo

    osd: 168 osds: 168 up (since 9h), 168 in (since 6M); 10 remapped pgs



  task status:



  data:

    pools:   3 pools, 1057 pgs

    objects: 166.90M objects, 632 TiB

    usage:   1.0 PiB used, 1.1 PiB / 2.1 PiB avail

    pgs:     583177/1001370417 objects misplaced (0.058%)

             1039 active+clean

             11   active+clean+scrubbing

             5    active+remapped+backfilling

             1    active+clean+scrubbing+deep

             1    active+clean+laggy



  io:

    client:   4.5 MiB/s rd, 31 MiB/s wr, 75 op/s rd, 59 op/s wr

    recovery: 2.7 MiB/s, 0 objects/s



root@ceph01:~#


root@ceph01:~# ceph pg dump |grep remapped

Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492

Inferring config
/var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config

Using recent ceph image
docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5

2.3d1     163805                   0         0          0        0
682273001472            0           0  7462      7462
active+*remapped*+backfilling
2021-02-26T01:28:06.212904+0000    53782'2773494    53782:5593409
[48,122,99,92,158,31]          48   [48,122,99,92,158,34]              48
  50907'2762333  2021-02-23T20:13:50.501757+0000    50907'2762333
2021-02-23T20:13:50.501757+0000              0

2.341     163180                   0         0     136844        0
679698268160            0           0  8569      8569
active+*remapped*+backfilling
2021-02-26T01:27:30.693520+0000    53782'2694466    53782:5497907
[65,125,79,147,45,100]          65   [65,18,79,147,45,100]              65
  51073'2688628  2021-02-24T09:38:41.588350+0000    51073'2688628
2021-02-24T09:38:41.588350+0000              0

2.1e1     162402                   0         0     150624        0
676505321472            0           0  7893      7893
active+*remapped*+backfilling
2021-02-25T15:36:29.121352+0000    53782'3105605    53782:5885140
[112,63,126,0,147,25]         112   [112,50,126,0,147,25]             112
  50907'3086340  2021-02-22T23:14:49.510535+0000    47263'2883486
2021-02-01T15:43:32.089129+0000              0

2.268     163459                   0         0     136459        0
680743612416            0           0  7337      7337
active+*remapped*+backfilling
2021-02-25T15:27:58.957497+0000    53782'2713107    53782:5536140
[93,98,150,49,26,135]          93   [93,98,150,62,26,135]              93
  50907'2698319  2021-02-23T14:16:11.171370+0000    50907'2698319
2021-02-23T14:16:11.171370+0000              0

2.285     163380                   0         0     158866        0
680507670528            0           0  8148      8148
active+*remapped*+backfilling
2021-02-26T01:26:31.442018+0000    53782'3191803    53782:5986426
[67,166,37,90,114,138]          67  [67,147,37,90,114,138]              67
  51073'3177251  2021-02-23T23:51:51.751625+0000    47597'2917541
2021-02-03T20:14:38.762384+0000              0

dumped all

root@ceph01:~#

Even if we stop the scrubs/deep scrubs, things don't improve re: recovery
rate or cluster performance. We've monitored the HDDs/NVME backing these
OSDs and with the scrubs stopped, they are nearly idle. CPU is almost idle.
There's nothing to indicate any problem at all with the underlying hardware
that we can find, at least from a load/failure perspective.

Also, interestingly, rados bench on a newly created **replicated** pool is
also extremely slow, as in < 50MB/s.

Crush rule for our EC (4+2) pool, but we see the performance issues on even
a newly created triple-replicated pool:

root@ceph01:~# ceph osd crush rule dump ecpool

Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492

Inferring config
/var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config

Using recent ceph image
docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5

{

    "rule_id": 1,

    "rule_name": "ecpool",

    "ruleset": 1,

    "type": 3,

    "min_size": 3,

    "max_size": 6,

    "steps": [

        {

            "op": "set_chooseleaf_tries",

            "num": 5

        },

        {

            "op": "set_choose_tries",

            "num": 100

        },

        {

            "op": "take",

            "item": -1,

            "item_name": "default"

        },

        {

            "op": "chooseleaf_indep",

            "num": 0,

            "type": "host"

        },

        {

            "op": "emit"

        }

    ]

}


Cluster/pool space:


root@ceph01:~# ceph df

Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492

Inferring config
/var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config

Using recent ceph image
docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5

--- RAW STORAGE ---

CLASS  SIZE     AVAIL    USED     RAW USED  %RAW USED

hdd    2.1 PiB  1.1 PiB  1.0 PiB   1.0 PiB      48.09

TOTAL  2.1 PiB  1.1 PiB  1.0 PiB   1.0 PiB      48.09



--- POOLS ---

POOL                   ID  PGS   STORED   OBJECTS  USED      %USED  MAX
AVAIL

device_health_metrics   1     1   78 MiB      182   233 MiB      0    295
TiB

ecpool                  2  1024  632 TiB  166.90M  1023 TiB  53.65    589
TiB

rbd                     3    32   59 MiB        5   178 MiB      0    295
TiB

root@ceph01:~#

We're at a loss for why this cluster has suddenly started behaving this
way. With the disks (all of them, on all nodes) effectively being idle with
all the scrubs disabled, plenty of CPU available, and no network issues
(iperf on all paths is full line rate - ~10GB/s - we have a dedicated
public and cluster network), it's befuddling.

We appreciate any assistance you can provide, and are happy to provide any
information that might be useful.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux