Re: Slow cluster / misplaced objects - Ceph 15.2.9

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



We figured this out - it was a leg of an LACP-based interface that was
misbehaving. Once we dropped it, everything went back to normal. Does
anybody know a good way to get a sense of what might be slowing down a
cluster in this regard, with EC? We didn't see any indication of a single
host as a problem until digging into the socket statistics and seeing high
sendqs to that host.

On Thu, Feb 25, 2021 at 7:33 PM David Orman <ormandj@xxxxxxxxxxxx> wrote:

> Hi,
>
> We've got an interesting issue we're running into on Ceph 15.2.9. We're
> experiencing VERY slow performance from the cluster, and extremely slow
> misplaced object correction, with very little cpu/disk/network utilization
> (almost idle) across all nodes in the cluster.
>
> We have 7 servers in this cluster, 24 rotational OSDs, and two NVMEs with
> 12 OSD's worth of DB/WAL files on them. The OSDs are all equal weighted, so
> the tree is pretty straightforward:
>
> root@ceph01:~# ceph osd tree
>
> Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492
>
> Inferring config
> /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config
>
> Using recent ceph image
> docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5
>
> ID   CLASS  WEIGHT      TYPE NAME            STATUS  REWEIGHT  PRI-AFF
>
>  -1         2149.39062  root default
>
>  -2         2149.39062      rack rack1
>
>  -5          307.05579          host ceph01
>
>   0    hdd    12.79399              osd.0        up   1.00000  1.00000
>
>   1    hdd    12.79399              osd.1        up   1.00000  1.00000
>
>   2    hdd    12.79399              osd.2        up   1.00000  1.00000
>
>   3    hdd    12.79399              osd.3        up   1.00000  1.00000
>
>   4    hdd    12.79399              osd.4        up   1.00000  1.00000
>
>   5    hdd    12.79399              osd.5        up   1.00000  1.00000
>
>   6    hdd    12.79399              osd.6        up   1.00000  1.00000
>
>   7    hdd    12.79399              osd.7        up   1.00000  1.00000
>
>   8    hdd    12.79399              osd.8        up   1.00000  1.00000
>
>   9    hdd    12.79399              osd.9        up   1.00000  1.00000
>
>  10    hdd    12.79399              osd.10       up   1.00000  1.00000
>
>  11    hdd    12.79399              osd.11       up   1.00000  1.00000
>
>  12    hdd    12.79399              osd.12       up   1.00000  1.00000
>
>  13    hdd    12.79399              osd.13       up   1.00000  1.00000
>
>  14    hdd    12.79399              osd.14       up   1.00000  1.00000
>
>  15    hdd    12.79399              osd.15       up   1.00000  1.00000
>
>  16    hdd    12.79399              osd.16       up   1.00000  1.00000
>
>  17    hdd    12.79399              osd.17       up   1.00000  1.00000
>
>  18    hdd    12.79399              osd.18       up   1.00000  1.00000
>
>  19    hdd    12.79399              osd.19       up   1.00000  1.00000
>
>  20    hdd    12.79399              osd.20       up   1.00000  1.00000
>
>  21    hdd    12.79399              osd.21       up   1.00000  1.00000
>
>  22    hdd    12.79399              osd.22       up   1.00000  1.00000
>
>  23    hdd    12.79399              osd.23       up   1.00000  1.00000
>
>  -7          307.05579          host ceph02
>
>  24    hdd    12.79399              osd.24       up   1.00000  1.00000
>
>  25    hdd    12.79399              osd.25       up   1.00000  1.00000
>
>  26    hdd    12.79399              osd.26       up   1.00000  1.00000
>
>  27    hdd    12.79399              osd.27       up   1.00000  1.00000
>
>  28    hdd    12.79399              osd.28       up   1.00000  1.00000
>
>  29    hdd    12.79399              osd.29       up   1.00000  1.00000
>
>  30    hdd    12.79399              osd.30       up   1.00000  1.00000
>
>  31    hdd    12.79399              osd.31       up   1.00000  1.00000
>
>  32    hdd    12.79399              osd.32       up   1.00000  1.00000
>
>  33    hdd    12.79399              osd.33       up   1.00000  1.00000
>
>  34    hdd    12.79399              osd.34       up   1.00000  1.00000
>
>  35    hdd    12.79399              osd.35       up   1.00000  1.00000
>
>  36    hdd    12.79399              osd.36       up   1.00000  1.00000
>
>  37    hdd    12.79399              osd.37       up   1.00000  1.00000
>
>  38    hdd    12.79399              osd.38       up   1.00000  1.00000
>
>  39    hdd    12.79399              osd.39       up   1.00000  1.00000
>
>  40    hdd    12.79399              osd.40       up   1.00000  1.00000
>
>  41    hdd    12.79399              osd.41       up   1.00000  1.00000
>
>  42    hdd    12.79399              osd.42       up   1.00000  1.00000
>
>  43    hdd    12.79399              osd.43       up   1.00000  1.00000
>
>  44    hdd    12.79399              osd.44       up   1.00000  1.00000
>
>  45    hdd    12.79399              osd.45       up   1.00000  1.00000
>
>  46    hdd    12.79399              osd.46       up   1.00000  1.00000
>
>  47    hdd    12.79399              osd.47       up   1.00000  1.00000
>
>  -9          307.05579          host ceph03
>
>  48    hdd    12.79399              osd.48       up   1.00000  1.00000
>
>  49    hdd    12.79399              osd.49       up   1.00000  1.00000
>
>  50    hdd    12.79399              osd.50       up   1.00000  1.00000
>
>  51    hdd    12.79399              osd.51       up   1.00000  1.00000
>
>  52    hdd    12.79399              osd.52       up   1.00000  1.00000
>
>  53    hdd    12.79399              osd.53       up   1.00000  1.00000
>
>  54    hdd    12.79399              osd.54       up   1.00000  1.00000
>
>  55    hdd    12.79399              osd.55       up   1.00000  1.00000
>
>  56    hdd    12.79399              osd.56       up   1.00000  1.00000
>
>  57    hdd    12.79399              osd.57       up   1.00000  1.00000
>
>  58    hdd    12.79399              osd.58       up   1.00000  1.00000
>
>  59    hdd    12.79399              osd.59       up   1.00000  1.00000
>
>  60    hdd    12.79399              osd.60       up   1.00000  1.00000
>
>  61    hdd    12.79399              osd.61       up   1.00000  1.00000
>
>  62    hdd    12.79399              osd.62       up   1.00000  1.00000
>
>  63    hdd    12.79399              osd.63       up   1.00000  1.00000
>
>  64    hdd    12.79399              osd.64       up   1.00000  1.00000
>
>  65    hdd    12.79399              osd.65       up   1.00000  1.00000
>
>  66    hdd    12.79399              osd.66       up   1.00000  1.00000
>
>  67    hdd    12.79399              osd.67       up   1.00000  1.00000
>
>  68    hdd    12.79399              osd.68       up   1.00000  1.00000
>
>  69    hdd    12.79399              osd.69       up   1.00000  1.00000
>
>  70    hdd    12.79399              osd.70       up   1.00000  1.00000
>
>  71    hdd    12.79399              osd.71       up   1.00000  1.00000
>
> -11          307.05579          host ceph04
>
>  72    hdd    12.79399              osd.72       up   1.00000  1.00000
>
>  73    hdd    12.79399              osd.73       up   1.00000  1.00000
>
>  74    hdd    12.79399              osd.74       up   1.00000  1.00000
>
>  75    hdd    12.79399              osd.75       up   1.00000  1.00000
>
>  76    hdd    12.79399              osd.76       up   1.00000  1.00000
>
>  77    hdd    12.79399              osd.77       up   1.00000  1.00000
>
>  78    hdd    12.79399              osd.78       up   1.00000  1.00000
>
>  79    hdd    12.79399              osd.79       up   1.00000  1.00000
>
>  80    hdd    12.79399              osd.80       up   1.00000  1.00000
>
>  81    hdd    12.79399              osd.81       up   1.00000  1.00000
>
>  82    hdd    12.79399              osd.82       up   1.00000  1.00000
>
>  83    hdd    12.79399              osd.83       up   1.00000  1.00000
>
>  84    hdd    12.79399              osd.84       up   1.00000  1.00000
>
>  85    hdd    12.79399              osd.85       up   1.00000  1.00000
>
>  86    hdd    12.79399              osd.86       up   1.00000  1.00000
>
>  87    hdd    12.79399              osd.87       up   1.00000  1.00000
>
>  88    hdd    12.79399              osd.88       up   1.00000  1.00000
>
>  89    hdd    12.79399              osd.89       up   1.00000  1.00000
>
>  90    hdd    12.79399              osd.90       up   1.00000  1.00000
>
>  91    hdd    12.79399              osd.91       up   1.00000  1.00000
>
>  92    hdd    12.79399              osd.92       up   1.00000  1.00000
>
>  93    hdd    12.79399              osd.93       up   1.00000  1.00000
>
>  94    hdd    12.79399              osd.94       up   1.00000  1.00000
>
>  95    hdd    12.79399              osd.95       up   1.00000  1.00000
>
> -13          307.05579          host ceph05
>
>  96    hdd    12.79399              osd.96       up   1.00000  1.00000
>
>  97    hdd    12.79399              osd.97       up   1.00000  1.00000
>
>  98    hdd    12.79399              osd.98       up   1.00000  1.00000
>
>  99    hdd    12.79399              osd.99       up   1.00000  1.00000
>
> 100    hdd    12.79399              osd.100      up   1.00000  1.00000
>
> 101    hdd    12.79399              osd.101      up   1.00000  1.00000
>
> 102    hdd    12.79399              osd.102      up   1.00000  1.00000
>
> 103    hdd    12.79399              osd.103      up   1.00000  1.00000
>
> 104    hdd    12.79399              osd.104      up   1.00000  1.00000
>
> 105    hdd    12.79399              osd.105      up   1.00000  1.00000
>
> 106    hdd    12.79399              osd.106      up   1.00000  1.00000
>
> 107    hdd    12.79399              osd.107      up   1.00000  1.00000
>
> 108    hdd    12.79399              osd.108      up   1.00000  1.00000
>
> 109    hdd    12.79399              osd.109      up   1.00000  1.00000
>
> 110    hdd    12.79399              osd.110      up   1.00000  1.00000
>
> 111    hdd    12.79399              osd.111      up   1.00000  1.00000
>
> 112    hdd    12.79399              osd.112      up   1.00000  1.00000
>
> 113    hdd    12.79399              osd.113      up   1.00000  1.00000
>
> 114    hdd    12.79399              osd.114      up   1.00000  1.00000
>
> 115    hdd    12.79399              osd.115      up   1.00000  1.00000
>
> 116    hdd    12.79399              osd.116      up   1.00000  1.00000
>
> 117    hdd    12.79399              osd.117      up   1.00000  1.00000
>
> 118    hdd    12.79399              osd.118      up   1.00000  1.00000
>
> 119    hdd    12.79399              osd.119      up   1.00000  1.00000
>
> -15          307.05579          host ceph06
>
> 120    hdd    12.79399              osd.120      up   1.00000  1.00000
>
> 121    hdd    12.79399              osd.121      up   1.00000  1.00000
>
> 122    hdd    12.79399              osd.122      up   1.00000  1.00000
>
> 123    hdd    12.79399              osd.123      up   1.00000  1.00000
>
> 124    hdd    12.79399              osd.124      up   1.00000  1.00000
>
> 125    hdd    12.79399              osd.125      up   1.00000  1.00000
>
> 126    hdd    12.79399              osd.126      up   1.00000  1.00000
>
> 127    hdd    12.79399              osd.127      up   1.00000  1.00000
>
> 128    hdd    12.79399              osd.128      up   1.00000  1.00000
>
> 129    hdd    12.79399              osd.129      up   1.00000  1.00000
>
> 130    hdd    12.79399              osd.130      up   1.00000  1.00000
>
> 131    hdd    12.79399              osd.131      up   1.00000  1.00000
>
> 132    hdd    12.79399              osd.132      up   1.00000  1.00000
>
> 133    hdd    12.79399              osd.133      up   1.00000  1.00000
>
> 134    hdd    12.79399              osd.134      up   1.00000  1.00000
>
> 135    hdd    12.79399              osd.135      up   1.00000  1.00000
>
> 136    hdd    12.79399              osd.136      up   1.00000  1.00000
>
> 137    hdd    12.79399              osd.137      up   1.00000  1.00000
>
> 138    hdd    12.79399              osd.138      up   1.00000  1.00000
>
> 139    hdd    12.79399              osd.139      up   1.00000  1.00000
>
> 140    hdd    12.79399              osd.140      up   1.00000  1.00000
>
> 141    hdd    12.79399              osd.141      up   1.00000  1.00000
>
> 142    hdd    12.79399              osd.142      up   1.00000  1.00000
>
> 143    hdd    12.79399              osd.143      up   1.00000  1.00000
>
> -17          307.05579          host ceph07
>
> 144    hdd    12.79399              osd.144      up   1.00000  1.00000
>
> 145    hdd    12.79399              osd.145      up   1.00000  1.00000
>
> 146    hdd    12.79399              osd.146      up   1.00000  1.00000
>
> 147    hdd    12.79399              osd.147      up   1.00000  1.00000
>
> 148    hdd    12.79399              osd.148      up   1.00000  1.00000
>
> 149    hdd    12.79399              osd.149      up   1.00000  1.00000
>
> 150    hdd    12.79399              osd.150      up   1.00000  1.00000
>
> 151    hdd    12.79399              osd.151      up   1.00000  1.00000
>
> 152    hdd    12.79399              osd.152      up   1.00000  1.00000
>
> 153    hdd    12.79399              osd.153      up   1.00000  1.00000
>
> 154    hdd    12.79399              osd.154      up   1.00000  1.00000
>
> 155    hdd    12.79399              osd.155      up   1.00000  1.00000
>
> 156    hdd    12.79399              osd.156      up   1.00000  1.00000
>
> 157    hdd    12.79399              osd.157      up   1.00000  1.00000
>
> 158    hdd    12.79399              osd.158      up   1.00000  1.00000
>
> 159    hdd    12.79399              osd.159      up   1.00000  1.00000
>
> 160    hdd    12.79399              osd.160      up   1.00000  1.00000
>
> 161    hdd    12.79399              osd.161      up   1.00000  1.00000
>
> 162    hdd    12.79399              osd.162      up   1.00000  1.00000
>
> 163    hdd    12.79399              osd.163      up   1.00000  1.00000
>
> 164    hdd    12.79399              osd.164      up   1.00000  1.00000
>
> 165    hdd    12.79399              osd.165      up   1.00000  1.00000
>
> 166    hdd    12.79399              osd.166      up   1.00000  1.00000
>
> 167    hdd    12.79399              osd.167      up   1.00000  1.00000
>
> root@ceph01:~#
>
> The data distribution looks relatively even (we use upmap as the balancer):
>
> root@ceph01:~# ceph osd df
>
> Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492
>
> Inferring config
> /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config
>
> Using recent ceph image
> docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5
>
> ID   CLASS  WEIGHT    REWEIGHT  SIZE     RAW USE  DATA      OMAP      META
>     AVAIL    %USE   VAR   PGS  STATUS
>
>   0    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   5.0 MiB   14
> GiB  7.1 TiB  44.73  0.93   35      up
>
>   1    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   1.8 MiB   13
> GiB  7.2 TiB  43.46  0.90   33      up
>
>   2    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   4.4 MiB   16
> GiB  6.2 TiB  51.27  1.07   39      up
>
>   3    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   4.9 MiB   15
> GiB  6.7 TiB  47.38  0.99   36      up
>
>   4    hdd  12.79399   1.00000   13 TiB  5.5 TiB   5.5 TiB   812 KiB   13
> GiB  7.2 TiB  43.37  0.90   35      up
>
>   5    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB    31 KiB   16
> GiB  6.2 TiB  51.27  1.07   40      up
>
>   6    hdd  12.79399   1.00000   13 TiB  5.5 TiB   5.5 TiB   506 KiB   13
> GiB  7.3 TiB  43.32  0.90   34      up
>
>   7    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   904 KiB   15
> GiB  6.4 TiB  49.93  1.04   38      up
>
>   8    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   3.6 MiB   14
> GiB  6.7 TiB  47.35  0.98   37      up
>
>   9    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   5.5 MiB   16
> GiB  6.1 TiB  52.58  1.09   40      up
>
>  10    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.6 TiB    75 MiB   16
> GiB  6.1 TiB  52.42  1.09   41      up
>
>  11    hdd  12.79399   1.00000   13 TiB  5.5 TiB   5.5 TiB   1.4 MiB   13
> GiB  7.2 TiB  43.36  0.90   34      up
>
>  12    hdd  12.79399   1.00000   13 TiB  5.5 TiB   5.5 TiB   586 KiB   13
> GiB  7.3 TiB  43.31  0.90   33      up
>
>  13    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   222 KiB   14
> GiB  7.1 TiB  44.71  0.93   35      up
>
>  14    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   1.0 MiB   16
> GiB  6.1 TiB  52.58  1.09   41      up
>
>  15    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.2 MiB   14
> GiB  7.1 TiB  44.73  0.93   34      up
>
>  16    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   1.0 MiB   16
> GiB  6.1 TiB  52.53  1.09   40      up
>
>  17    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   3.3 MiB   15
> GiB  6.4 TiB  49.92  1.04   38      up
>
>  18    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   1.0 MiB   16
> GiB  6.1 TiB  52.61  1.09   45      up
>
>  19    hdd  12.79399   1.00000   13 TiB  5.4 TiB   5.3 TiB   4.0 MiB   13
> GiB  7.4 TiB  42.15  0.88   34      up
>
>  20    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   705 KiB   16
> GiB  6.1 TiB  52.53  1.09   40      up
>
>  21    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   2.5 MiB   16
> GiB  6.1 TiB  52.58  1.09   40      up
>
>  22    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   4.7 MiB   16
> GiB  6.1 TiB  52.59  1.09   40      up
>
>  23    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   4.9 MiB   16
> GiB  6.1 TiB  52.58  1.09   41      up
>
>  24    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   3.0 MiB   14
> GiB  6.7 TiB  47.35  0.98   36      up
>
>  25    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   4.1 MiB   16
> GiB  6.1 TiB  52.56  1.09   41      up
>
>  26    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   592 KiB   14
> GiB  7.1 TiB  44.75  0.93   34      up
>
>  27    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   2.8 MiB   14
> GiB  6.9 TiB  46.07  0.96   36      up
>
>  28    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   5.4 MiB   16
> GiB  6.1 TiB  52.56  1.09   40      up
>
>  29    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   340 KiB   15
> GiB  6.4 TiB  49.92  1.04   39      up
>
>  30    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB    93 KiB   14
> GiB  6.9 TiB  46.03  0.96   35      up
>
>  31    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   2.1 MiB   13
> GiB  7.2 TiB  43.57  0.91   35      up
>
>  32    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   5.0 MiB   15
> GiB  6.6 TiB  48.68  1.01   38      up
>
>  33    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   175 KiB   13
> GiB  7.2 TiB  43.44  0.90   33      up
>
>  34    hdd  12.79399   1.00000   13 TiB  6.9 TiB   6.8 TiB    77 MiB   16
> GiB  5.9 TiB  53.86  1.12   43      up
>
>  35    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   2.3 MiB   15
> GiB  6.6 TiB  48.68  1.01   37      up
>
>  36    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   1.2 MiB   16
> GiB  6.2 TiB  51.25  1.07   41      up
>
>  37    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   5.0 MiB   14
> GiB  6.7 TiB  47.37  0.99   36      up
>
>  38    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   3.7 MiB   13
> GiB  7.2 TiB  43.44  0.90   35      up
>
>  39    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   341 KiB   15
> GiB  6.7 TiB  47.38  0.99   37      up
>
>  40    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.1 MiB   14
> GiB  7.1 TiB  44.73  0.93   34      up
>
>  41    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB    22 KiB   16
> GiB  6.1 TiB  52.59  1.09   40      up
>
>  42    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   1.9 MiB   14
> GiB  6.7 TiB  47.33  0.98   36      up
>
>  43    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   485 KiB   16
> GiB  6.1 TiB  52.59  1.09   40      up
>
>  44    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   2.1 MiB   16
> GiB  6.1 TiB  52.56  1.09   40      up
>
>  45    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   2.3 MiB   16
> GiB  6.1 TiB  52.53  1.09   40      up
>
>  46    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   124 KiB   14
> GiB  6.9 TiB  46.08  0.96   37      up
>
>  47    hdd  12.79399   1.00000   13 TiB  5.5 TiB   5.5 TiB   1.2 MiB   13
> GiB  7.2 TiB  43.37  0.90   33      up
>
>  48    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   141 KiB   16
> GiB  6.1 TiB  52.58  1.09   40      up
>
>  49    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   1.9 MiB   13
> GiB  7.2 TiB  43.64  0.91   33      up
>
>  50    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   745 KiB   16
> GiB  6.1 TiB  52.52  1.09   41      up
>
>  51    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   186 KiB   16
> GiB  6.1 TiB  52.55  1.09   41      up
>
>  52    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   4.0 MiB   16
> GiB  6.2 TiB  51.24  1.07   41      up
>
>  53    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   737 KiB   14
> GiB  6.7 TiB  47.33  0.98   38      up
>
>  54    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   1.3 MiB   14
> GiB  7.1 TiB  44.66  0.93   34      up
>
>  55    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB    97 KiB   14
> GiB  7.1 TiB  44.73  0.93   34      up
>
>  56    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   1.2 MiB   14
> GiB  7.1 TiB  44.76  0.93   35      up
>
>  57    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   3.5 MiB   14
> GiB  7.1 TiB  44.78  0.93   35      up
>
>  58    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB    80 MiB   16
> GiB  6.1 TiB  52.56  1.09   41      up
>
>  59    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   4.5 MiB   14
> GiB  6.9 TiB  46.08  0.96   36      up
>
>  60    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   903 KiB   14
> GiB  7.1 TiB  44.73  0.93   34      up
>
>  61    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB     1 KiB   14
> GiB  7.1 TiB  44.65  0.93   34      up
>
>  62    hdd  12.79399   1.00000   13 TiB  6.9 TiB   6.8 TiB   425 KiB   16
> GiB  5.9 TiB  53.87  1.12   41      up
>
>  63    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   4.1 MiB   14
> GiB  7.2 TiB  43.56  0.91   34      up
>
>  64    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   578 KiB   16
> GiB  6.2 TiB  51.27  1.07   39      up
>
>  65    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   3.6 MiB   13
> GiB  7.2 TiB  43.44  0.90   33      up
>
>  66    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   288 KiB   13
> GiB  7.2 TiB  43.40  0.90   34      up
>
>  67    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   413 KiB   14
> GiB  6.7 TiB  47.34  0.98   37      up
>
>  68    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   4.3 MiB   16
> GiB  6.1 TiB  52.56  1.09   41      up
>
>  69    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   1.0 MiB   13
> GiB  7.2 TiB  43.44  0.90   33      up
>
>  70    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   3.2 MiB   16
> GiB  6.1 TiB  52.56  1.09   40      up
>
>  71    hdd  12.79399   1.00000   13 TiB  5.5 TiB   5.5 TiB   1.3 MiB   14
> GiB  7.2 TiB  43.36  0.90   35      up
>
>  72    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   4.7 MiB   16
> GiB  6.2 TiB  51.29  1.07   39      up
>
>  73    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   1.8 MiB   14
> GiB  7.1 TiB  44.73  0.93   34      up
>
>  74    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.6 TiB   1.2 MiB   14
> GiB  7.1 TiB  44.64  0.93   34      up
>
>  75    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   1.5 MiB   15
> GiB  6.4 TiB  49.95  1.04   38      up
>
>  76    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   3.8 MiB   15
> GiB  6.6 TiB  48.68  1.01   37      up
>
>  77    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   5.0 MiB   14
> GiB  7.1 TiB  44.65  0.93   34      up
>
>  78    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   1.8 MiB   14
> GiB  7.1 TiB  44.68  0.93   35      up
>
>  79    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   984 KiB   14
> GiB  6.7 TiB  47.36  0.98   36      up
>
>  80    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   2.6 MiB   13
> GiB  7.2 TiB  43.45  0.90   34      up
>
>  81    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   1.2 MiB   16
> GiB  6.2 TiB  51.26  1.07   41      up
>
>  82    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   4.9 MiB   15
> GiB  6.6 TiB  48.61  1.01   37      up
>
>  83    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   827 KiB   14
> GiB  6.7 TiB  47.33  0.98   36      up
>
>  84    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   2.7 MiB   16
> GiB  6.2 TiB  51.22  1.07   39      up
>
>  85    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   868 KiB   15
> GiB  6.4 TiB  49.94  1.04   40      up
>
>  86    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   4.1 MiB   16
> GiB  6.1 TiB  52.57  1.09   42      up
>
>  87    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   702 KiB   16
> GiB  6.1 TiB  52.54  1.09   41      up
>
>  88    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB    12 KiB   14
> GiB  7.1 TiB  44.76  0.93   35      up
>
>  89    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   2.9 MiB   14
> GiB  6.9 TiB  46.07  0.96   35      up
>
>  90    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   1.8 MiB   13
> GiB  7.2 TiB  43.48  0.90   34      up
>
>  91    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   836 KiB   16
> GiB  6.1 TiB  52.55  1.09   40      up
>
>  92    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   4.0 MiB   13
> GiB  7.2 TiB  43.43  0.90   33      up
>
>  93    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   1.5 MiB   16
> GiB  6.1 TiB  52.57  1.09   41      up
>
>  94    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   459 KiB   13
> GiB  7.2 TiB  43.44  0.90   34      up
>
>  95    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   4.1 MiB   16
> GiB  6.2 TiB  51.28  1.07   39      up
>
>  96    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.1 MiB   14
> GiB  7.1 TiB  44.75  0.93   35      up
>
>  97    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   662 KiB   14
> GiB  7.1 TiB  44.75  0.93   34      up
>
>  98    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   2.9 MiB   14
> GiB  6.9 TiB  45.97  0.96   35      up
>
>  99    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   3.8 MiB   16
> GiB  6.1 TiB  52.56  1.09   40      up
>
> 100    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   1.8 MiB   14
> GiB  7.1 TiB  44.76  0.93   36      up
>
> 101    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   3.6 MiB   15
> GiB  6.4 TiB  49.95  1.04   40      up
>
> 102    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.9 MiB   14
> GiB  7.1 TiB  44.65  0.93   34      up
>
> 103    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   643 KiB   16
> GiB  6.1 TiB  52.57  1.09   41      up
>
> 104    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   5.5 MiB   15
> GiB  6.4 TiB  49.98  1.04   38      up
>
> 105    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   830 KiB   13
> GiB  7.2 TiB  43.46  0.90   33      up
>
> 106    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   1.0 MiB   14
> GiB  6.7 TiB  47.34  0.98   36      up
>
> 107    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   286 KiB   16
> GiB  6.2 TiB  51.23  1.07   39      up
>
> 108    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   961 KiB   15
> GiB  6.6 TiB  48.68  1.01   37      up
>
> 109    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   291 KiB   16
> GiB  6.1 TiB  52.56  1.09   42      up
>
> 110    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   1.0 MiB   14
> GiB  6.9 TiB  46.02  0.96   36      up
>
> 111    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   2.7 MiB   16
> GiB  6.2 TiB  51.22  1.07   39      up
>
> 112    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   631 KiB   15
> GiB  6.6 TiB  48.67  1.01   37      up
>
> 113    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   4.5 MiB   16
> GiB  6.1 TiB  52.56  1.09   41      up
>
> 114    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   4.0 MiB   15
> GiB  6.4 TiB  49.94  1.04   39      up
>
> 115    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   1.5 MiB   14
> GiB  6.9 TiB  46.05  0.96   35      up
>
> 116    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   4.1 MiB   15
> GiB  6.4 TiB  49.93  1.04   38      up
>
> 117    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   194 KiB   13
> GiB  7.2 TiB  43.44  0.90   35      up
>
> 118    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   3.1 MiB   15
> GiB  6.4 TiB  49.96  1.04   38      up
>
> 119    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   3.9 MiB   14
> GiB  7.2 TiB  43.46  0.90   33      up
>
> 120    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   1.8 MiB   16
> GiB  6.1 TiB  52.55  1.09   40      up
>
> 121    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   3.7 MiB   16
> GiB  6.1 TiB  52.54  1.09   40      up
>
> 122    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   2.4 MiB   15
> GiB  6.4 TiB  49.97  1.04   38      up
>
> 123    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB  1003 KiB   14
> GiB  6.9 TiB  46.02  0.96   35      up
>
> 124    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   526 KiB   14
> GiB  6.7 TiB  47.35  0.98   36      up
>
> 125    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB    92 KiB   15
> GiB  6.7 TiB  47.37  0.98   37      up
>
> 126    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.9 MiB   14
> GiB  7.1 TiB  44.74  0.93   34      up
>
> 127    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   5.5 MiB   16
> GiB  6.2 TiB  51.26  1.07   39      up
>
> 128    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   5.4 MiB   16
> GiB  6.1 TiB  52.56  1.09   40      up
>
> 129    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   3.6 MiB   15
> GiB  6.6 TiB  48.66  1.01   38      up
>
> 130    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   2.8 MiB   16
> GiB  6.1 TiB  52.56  1.09   41      up
>
> 131    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   5.1 MiB   16
> GiB  6.2 TiB  51.28  1.07   40      up
>
> 132    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   1.7 MiB   16
> GiB  6.2 TiB  51.25  1.07   39      up
>
> 133    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   230 KiB   16
> GiB  6.2 TiB  51.25  1.07   39      up
>
> 134    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   2.0 MiB   14
> GiB  7.2 TiB  43.45  0.90   34      up
>
> 135    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   4.4 MiB   13
> GiB  7.2 TiB  43.46  0.90   33      up
>
> 136    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   5.6 MiB   16
> GiB  6.2 TiB  51.29  1.07   39      up
>
> 137    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   3.6 MiB   14
> GiB  7.1 TiB  44.75  0.93   36      up
>
> 138    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   4.9 MiB   15
> GiB  6.4 TiB  49.98  1.04   40      up
>
> 139    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB    21 KiB   16
> GiB  6.2 TiB  51.23  1.07   39      up
>
> 140    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   5.4 MiB   14
> GiB  7.1 TiB  44.74  0.93   36      up
>
> 141    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   359 KiB   13
> GiB  7.2 TiB  43.42  0.90   34      up
>
> 142    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB    27 KiB   13
> GiB  7.2 TiB  43.46  0.90   33      up
>
> 143    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   395 KiB   14
> GiB  6.9 TiB  46.07  0.96   35      up
>
> 144    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   1.2 MiB   14
> GiB  7.1 TiB  44.75  0.93   34      up
>
> 145    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   3.3 MiB   14
> GiB  7.1 TiB  44.66  0.93   35      up
>
> 146    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   4.0 MiB   14
> GiB  6.9 TiB  46.06  0.96   35      up
>
> 147    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB    46 KiB   16
> GiB  6.1 TiB  52.55  1.09   40      up
>
> 148    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   4.8 MiB   16
> GiB  6.2 TiB  51.23  1.07   39      up
>
> 149    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.4 MiB   14
> GiB  7.1 TiB  44.70  0.93   35      up
>
> 150    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.4 MiB   14
> GiB  7.1 TiB  44.65  0.93   35      up
>
> 151    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   3.7 MiB   14
> GiB  7.1 TiB  44.65  0.93   35      up
>
> 152    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   5.3 MiB   15
> GiB  6.2 TiB  51.26  1.07   39      up
>
> 153    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   2.3 MiB   15
> GiB  6.4 TiB  49.94  1.04   39      up
>
> 154    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   3.0 MiB   14
> GiB  6.7 TiB  47.37  0.99   36      up
>
> 155    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   3.0 MiB   14
> GiB  6.7 TiB  47.35  0.98   38      up
>
> 156    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   4.0 MiB   16
> GiB  6.2 TiB  51.26  1.07   41      up
>
> 157    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   2.8 MiB   14
> GiB  6.9 TiB  46.07  0.96   36      up
>
> 158    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   3.0 MiB   16
> GiB  6.2 TiB  51.25  1.07   40      up
>
> 159    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   3.9 MiB   15
> GiB  6.4 TiB  49.93  1.04   38      up
>
> 160    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   2.4 MiB   14
> GiB  6.9 TiB  46.03  0.96   35      up
>
> 161    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   1.1 MiB   16
> GiB  6.2 TiB  51.25  1.07   39      up
>
> 162    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.6 TiB   4.1 MiB   13
> GiB  7.1 TiB  44.64  0.93   34      up
>
> 163    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   445 KiB   14
> GiB  7.1 TiB  44.69  0.93   34      up
>
> 164    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   4.9 MiB   15
> GiB  6.4 TiB  49.97  1.04   38      up
>
> 165    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   5.5 MiB   16
> GiB  6.2 TiB  51.25  1.07   40      up
>
> 166    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   2.5 MiB   13
> GiB  7.2 TiB  43.45  0.90   33      up
>
> 167    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   4.0 MiB   15
> GiB  6.4 TiB  49.98  1.04   38      up
>
>                          TOTAL  2.1 PiB  1.0 PiB  1023 TiB   630 MiB  2.4
> TiB  1.1 PiB  48.09
>
> MIN/MAX VAR: 0.88/1.12  STDDEV: 3.50
>
> root@ceph01:~#
>
>
> The cluster status, however, shows we have many misplaced objects and a
> few remapped PGs. We had been performing maintenance (upgrade to Ceph
> 15.2.9), and initially the misplaced object count was dropping rapidly at a
> rate of about 400-500MB/s and 100-200 objects/s. However, once it hit a
> certain point, it dropped to 0-4MB/s and 0 objects/s (it's not 0, it is
> progressing, but it's below 1 object/s):
>
>
> root@ceph01:~# ceph -s
>
> Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492
>
> Inferring config
> /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config
>
> Using recent ceph image
> docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5
>
>   cluster:
>
>     id:     41bb9256-c3bf-11ea-85b9-9e07b0435492
>
>     health: HEALTH_WARN
>
>             5 slow ops, oldest one blocked for 84 sec, daemons
> [osd.141,osd.46] have slow ops.
>
>
>
>   services:
>
>     mon: 5 daemons, quorum ceph01,ceph04,ceph02,ceph03,ceph05 (age 25h)
>
>     mgr: ceph05.yropto(active, since 11h), standbys: ceph01.aqkgbl,
> ceph03.ytkuyr, ceph04.smbdew, ceph02.ndynmo
>
>     osd: 168 osds: 168 up (since 9h), 168 in (since 6M); 10 remapped pgs
>
>
>
>   task status:
>
>
>
>   data:
>
>     pools:   3 pools, 1057 pgs
>
>     objects: 166.90M objects, 632 TiB
>
>     usage:   1.0 PiB used, 1.1 PiB / 2.1 PiB avail
>
>     pgs:     583177/1001370417 objects misplaced (0.058%)
>
>              1039 active+clean
>
>              11   active+clean+scrubbing
>
>              5    active+remapped+backfilling
>
>              1    active+clean+scrubbing+deep
>
>              1    active+clean+laggy
>
>
>
>   io:
>
>     client:   4.5 MiB/s rd, 31 MiB/s wr, 75 op/s rd, 59 op/s wr
>
>     recovery: 2.7 MiB/s, 0 objects/s
>
>
>
> root@ceph01:~#
>
>
> root@ceph01:~# ceph pg dump |grep remapped
>
> Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492
>
> Inferring config
> /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config
>
> Using recent ceph image
> docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5
>
> 2.3d1     163805                   0         0          0        0
> 682273001472            0           0  7462      7462  active+*remapped*+backfilling
> 2021-02-26T01:28:06.212904+0000    53782'2773494    53782:5593409
> [48,122,99,92,158,31]          48   [48,122,99,92,158,34]              48
>   50907'2762333  2021-02-23T20:13:50.501757+0000    50907'2762333
> 2021-02-23T20:13:50.501757+0000              0
>
> 2.341     163180                   0         0     136844        0
> 679698268160            0           0  8569      8569  active+*remapped*+backfilling
> 2021-02-26T01:27:30.693520+0000    53782'2694466    53782:5497907
> [65,125,79,147,45,100]          65   [65,18,79,147,45,100]              65
>   51073'2688628  2021-02-24T09:38:41.588350+0000    51073'2688628
> 2021-02-24T09:38:41.588350+0000              0
>
> 2.1e1     162402                   0         0     150624        0
> 676505321472            0           0  7893      7893  active+*remapped*+backfilling
> 2021-02-25T15:36:29.121352+0000    53782'3105605    53782:5885140
> [112,63,126,0,147,25]         112   [112,50,126,0,147,25]             112
>   50907'3086340  2021-02-22T23:14:49.510535+0000    47263'2883486
> 2021-02-01T15:43:32.089129+0000              0
>
> 2.268     163459                   0         0     136459        0
> 680743612416            0           0  7337      7337  active+*remapped*+backfilling
> 2021-02-25T15:27:58.957497+0000    53782'2713107    53782:5536140
> [93,98,150,49,26,135]          93   [93,98,150,62,26,135]              93
>   50907'2698319  2021-02-23T14:16:11.171370+0000    50907'2698319
> 2021-02-23T14:16:11.171370+0000              0
>
> 2.285     163380                   0         0     158866        0
> 680507670528            0           0  8148      8148  active+*remapped*+backfilling
> 2021-02-26T01:26:31.442018+0000    53782'3191803    53782:5986426
> [67,166,37,90,114,138]          67  [67,147,37,90,114,138]              67
>   51073'3177251  2021-02-23T23:51:51.751625+0000    47597'2917541
> 2021-02-03T20:14:38.762384+0000              0
>
> dumped all
>
> root@ceph01:~#
>
> Even if we stop the scrubs/deep scrubs, things don't improve re: recovery
> rate or cluster performance. We've monitored the HDDs/NVME backing these
> OSDs and with the scrubs stopped, they are nearly idle. CPU is almost idle.
> There's nothing to indicate any problem at all with the underlying hardware
> that we can find, at least from a load/failure perspective.
>
> Also, interestingly, rados bench on a newly created **replicated** pool
> is also extremely slow, as in < 50MB/s.
>
> Crush rule for our EC (4+2) pool, but we see the performance issues on
> even a newly created triple-replicated pool:
>
> root@ceph01:~# ceph osd crush rule dump ecpool
>
> Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492
>
> Inferring config
> /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config
>
> Using recent ceph image
> docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5
>
> {
>
>     "rule_id": 1,
>
>     "rule_name": "ecpool",
>
>     "ruleset": 1,
>
>     "type": 3,
>
>     "min_size": 3,
>
>     "max_size": 6,
>
>     "steps": [
>
>         {
>
>             "op": "set_chooseleaf_tries",
>
>             "num": 5
>
>         },
>
>         {
>
>             "op": "set_choose_tries",
>
>             "num": 100
>
>         },
>
>         {
>
>             "op": "take",
>
>             "item": -1,
>
>             "item_name": "default"
>
>         },
>
>         {
>
>             "op": "chooseleaf_indep",
>
>             "num": 0,
>
>             "type": "host"
>
>         },
>
>         {
>
>             "op": "emit"
>
>         }
>
>     ]
>
> }
>
>
> Cluster/pool space:
>
>
> root@ceph01:~# ceph df
>
> Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492
>
> Inferring config
> /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config
>
> Using recent ceph image
> docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5
>
> --- RAW STORAGE ---
>
> CLASS  SIZE     AVAIL    USED     RAW USED  %RAW USED
>
> hdd    2.1 PiB  1.1 PiB  1.0 PiB   1.0 PiB      48.09
>
> TOTAL  2.1 PiB  1.1 PiB  1.0 PiB   1.0 PiB      48.09
>
>
>
> --- POOLS ---
>
> POOL                   ID  PGS   STORED   OBJECTS  USED      %USED  MAX
> AVAIL
>
> device_health_metrics   1     1   78 MiB      182   233 MiB      0    295
> TiB
>
> ecpool                  2  1024  632 TiB  166.90M  1023 TiB  53.65    589
> TiB
>
> rbd                     3    32   59 MiB        5   178 MiB      0    295
> TiB
>
> root@ceph01:~#
>
> We're at a loss for why this cluster has suddenly started behaving this
> way. With the disks (all of them, on all nodes) effectively being idle with
> all the scrubs disabled, plenty of CPU available, and no network issues
> (iperf on all paths is full line rate - ~10GB/s - we have a dedicated
> public and cluster network), it's befuddling.
>
> We appreciate any assistance you can provide, and are happy to provide any
> information that might be useful.
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux