Re: Slow cluster / misplaced objects - Ceph 15.2.9

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello,

within croit, we have a network latency monitoring that would have
shown you the packetlos.
We therefore suggest to install something like a smokeping on your
infrastructure to monitor the quality of your network.

Why does it affect your cluster?

The network is the central component of a Ceph cluster. If this does
not function stably and reliably, Ceph cannot work properly either. It
is practically the backbone of the scale-out cluster and cannot be
replaced by anything. Single packet loss, for example, leads to
retransmits of packets, increased latency and thus reduced data
throughput. This in turn leads to a higher impact during replication
work, which is particularly prevalent in EC. In EC, not only write
accesses but also read accesses must be loaded from several OSDs.

--
Martin Verges
Managing director

Mobile: +49 174 9335695
E-Mail: martin.verges@xxxxxxxx
Chat: https://t.me/MartinVerges

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263

Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx

Am Fr., 26. Feb. 2021 um 20:00 Uhr schrieb David Orman <ormandj@xxxxxxxxxxxx>:
>
> We figured this out - it was a leg of an LACP-based interface that was
> misbehaving. Once we dropped it, everything went back to normal. Does
> anybody know a good way to get a sense of what might be slowing down a
> cluster in this regard, with EC? We didn't see any indication of a single
> host as a problem until digging into the socket statistics and seeing high
> sendqs to that host.
>
> On Thu, Feb 25, 2021 at 7:33 PM David Orman <ormandj@xxxxxxxxxxxx> wrote:
>
> > Hi,
> >
> > We've got an interesting issue we're running into on Ceph 15.2.9. We're
> > experiencing VERY slow performance from the cluster, and extremely slow
> > misplaced object correction, with very little cpu/disk/network utilization
> > (almost idle) across all nodes in the cluster.
> >
> > We have 7 servers in this cluster, 24 rotational OSDs, and two NVMEs with
> > 12 OSD's worth of DB/WAL files on them. The OSDs are all equal weighted, so
> > the tree is pretty straightforward:
> >
> > root@ceph01:~# ceph osd tree
> >
> > Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492
> >
> > Inferring config
> > /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config
> >
> > Using recent ceph image
> > docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5
> >
> > ID   CLASS  WEIGHT      TYPE NAME            STATUS  REWEIGHT  PRI-AFF
> >
> >  -1         2149.39062  root default
> >
> >  -2         2149.39062      rack rack1
> >
> >  -5          307.05579          host ceph01
> >
> >   0    hdd    12.79399              osd.0        up   1.00000  1.00000
> >
> >   1    hdd    12.79399              osd.1        up   1.00000  1.00000
> >
> >   2    hdd    12.79399              osd.2        up   1.00000  1.00000
> >
> >   3    hdd    12.79399              osd.3        up   1.00000  1.00000
> >
> >   4    hdd    12.79399              osd.4        up   1.00000  1.00000
> >
> >   5    hdd    12.79399              osd.5        up   1.00000  1.00000
> >
> >   6    hdd    12.79399              osd.6        up   1.00000  1.00000
> >
> >   7    hdd    12.79399              osd.7        up   1.00000  1.00000
> >
> >   8    hdd    12.79399              osd.8        up   1.00000  1.00000
> >
> >   9    hdd    12.79399              osd.9        up   1.00000  1.00000
> >
> >  10    hdd    12.79399              osd.10       up   1.00000  1.00000
> >
> >  11    hdd    12.79399              osd.11       up   1.00000  1.00000
> >
> >  12    hdd    12.79399              osd.12       up   1.00000  1.00000
> >
> >  13    hdd    12.79399              osd.13       up   1.00000  1.00000
> >
> >  14    hdd    12.79399              osd.14       up   1.00000  1.00000
> >
> >  15    hdd    12.79399              osd.15       up   1.00000  1.00000
> >
> >  16    hdd    12.79399              osd.16       up   1.00000  1.00000
> >
> >  17    hdd    12.79399              osd.17       up   1.00000  1.00000
> >
> >  18    hdd    12.79399              osd.18       up   1.00000  1.00000
> >
> >  19    hdd    12.79399              osd.19       up   1.00000  1.00000
> >
> >  20    hdd    12.79399              osd.20       up   1.00000  1.00000
> >
> >  21    hdd    12.79399              osd.21       up   1.00000  1.00000
> >
> >  22    hdd    12.79399              osd.22       up   1.00000  1.00000
> >
> >  23    hdd    12.79399              osd.23       up   1.00000  1.00000
> >
> >  -7          307.05579          host ceph02
> >
> >  24    hdd    12.79399              osd.24       up   1.00000  1.00000
> >
> >  25    hdd    12.79399              osd.25       up   1.00000  1.00000
> >
> >  26    hdd    12.79399              osd.26       up   1.00000  1.00000
> >
> >  27    hdd    12.79399              osd.27       up   1.00000  1.00000
> >
> >  28    hdd    12.79399              osd.28       up   1.00000  1.00000
> >
> >  29    hdd    12.79399              osd.29       up   1.00000  1.00000
> >
> >  30    hdd    12.79399              osd.30       up   1.00000  1.00000
> >
> >  31    hdd    12.79399              osd.31       up   1.00000  1.00000
> >
> >  32    hdd    12.79399              osd.32       up   1.00000  1.00000
> >
> >  33    hdd    12.79399              osd.33       up   1.00000  1.00000
> >
> >  34    hdd    12.79399              osd.34       up   1.00000  1.00000
> >
> >  35    hdd    12.79399              osd.35       up   1.00000  1.00000
> >
> >  36    hdd    12.79399              osd.36       up   1.00000  1.00000
> >
> >  37    hdd    12.79399              osd.37       up   1.00000  1.00000
> >
> >  38    hdd    12.79399              osd.38       up   1.00000  1.00000
> >
> >  39    hdd    12.79399              osd.39       up   1.00000  1.00000
> >
> >  40    hdd    12.79399              osd.40       up   1.00000  1.00000
> >
> >  41    hdd    12.79399              osd.41       up   1.00000  1.00000
> >
> >  42    hdd    12.79399              osd.42       up   1.00000  1.00000
> >
> >  43    hdd    12.79399              osd.43       up   1.00000  1.00000
> >
> >  44    hdd    12.79399              osd.44       up   1.00000  1.00000
> >
> >  45    hdd    12.79399              osd.45       up   1.00000  1.00000
> >
> >  46    hdd    12.79399              osd.46       up   1.00000  1.00000
> >
> >  47    hdd    12.79399              osd.47       up   1.00000  1.00000
> >
> >  -9          307.05579          host ceph03
> >
> >  48    hdd    12.79399              osd.48       up   1.00000  1.00000
> >
> >  49    hdd    12.79399              osd.49       up   1.00000  1.00000
> >
> >  50    hdd    12.79399              osd.50       up   1.00000  1.00000
> >
> >  51    hdd    12.79399              osd.51       up   1.00000  1.00000
> >
> >  52    hdd    12.79399              osd.52       up   1.00000  1.00000
> >
> >  53    hdd    12.79399              osd.53       up   1.00000  1.00000
> >
> >  54    hdd    12.79399              osd.54       up   1.00000  1.00000
> >
> >  55    hdd    12.79399              osd.55       up   1.00000  1.00000
> >
> >  56    hdd    12.79399              osd.56       up   1.00000  1.00000
> >
> >  57    hdd    12.79399              osd.57       up   1.00000  1.00000
> >
> >  58    hdd    12.79399              osd.58       up   1.00000  1.00000
> >
> >  59    hdd    12.79399              osd.59       up   1.00000  1.00000
> >
> >  60    hdd    12.79399              osd.60       up   1.00000  1.00000
> >
> >  61    hdd    12.79399              osd.61       up   1.00000  1.00000
> >
> >  62    hdd    12.79399              osd.62       up   1.00000  1.00000
> >
> >  63    hdd    12.79399              osd.63       up   1.00000  1.00000
> >
> >  64    hdd    12.79399              osd.64       up   1.00000  1.00000
> >
> >  65    hdd    12.79399              osd.65       up   1.00000  1.00000
> >
> >  66    hdd    12.79399              osd.66       up   1.00000  1.00000
> >
> >  67    hdd    12.79399              osd.67       up   1.00000  1.00000
> >
> >  68    hdd    12.79399              osd.68       up   1.00000  1.00000
> >
> >  69    hdd    12.79399              osd.69       up   1.00000  1.00000
> >
> >  70    hdd    12.79399              osd.70       up   1.00000  1.00000
> >
> >  71    hdd    12.79399              osd.71       up   1.00000  1.00000
> >
> > -11          307.05579          host ceph04
> >
> >  72    hdd    12.79399              osd.72       up   1.00000  1.00000
> >
> >  73    hdd    12.79399              osd.73       up   1.00000  1.00000
> >
> >  74    hdd    12.79399              osd.74       up   1.00000  1.00000
> >
> >  75    hdd    12.79399              osd.75       up   1.00000  1.00000
> >
> >  76    hdd    12.79399              osd.76       up   1.00000  1.00000
> >
> >  77    hdd    12.79399              osd.77       up   1.00000  1.00000
> >
> >  78    hdd    12.79399              osd.78       up   1.00000  1.00000
> >
> >  79    hdd    12.79399              osd.79       up   1.00000  1.00000
> >
> >  80    hdd    12.79399              osd.80       up   1.00000  1.00000
> >
> >  81    hdd    12.79399              osd.81       up   1.00000  1.00000
> >
> >  82    hdd    12.79399              osd.82       up   1.00000  1.00000
> >
> >  83    hdd    12.79399              osd.83       up   1.00000  1.00000
> >
> >  84    hdd    12.79399              osd.84       up   1.00000  1.00000
> >
> >  85    hdd    12.79399              osd.85       up   1.00000  1.00000
> >
> >  86    hdd    12.79399              osd.86       up   1.00000  1.00000
> >
> >  87    hdd    12.79399              osd.87       up   1.00000  1.00000
> >
> >  88    hdd    12.79399              osd.88       up   1.00000  1.00000
> >
> >  89    hdd    12.79399              osd.89       up   1.00000  1.00000
> >
> >  90    hdd    12.79399              osd.90       up   1.00000  1.00000
> >
> >  91    hdd    12.79399              osd.91       up   1.00000  1.00000
> >
> >  92    hdd    12.79399              osd.92       up   1.00000  1.00000
> >
> >  93    hdd    12.79399              osd.93       up   1.00000  1.00000
> >
> >  94    hdd    12.79399              osd.94       up   1.00000  1.00000
> >
> >  95    hdd    12.79399              osd.95       up   1.00000  1.00000
> >
> > -13          307.05579          host ceph05
> >
> >  96    hdd    12.79399              osd.96       up   1.00000  1.00000
> >
> >  97    hdd    12.79399              osd.97       up   1.00000  1.00000
> >
> >  98    hdd    12.79399              osd.98       up   1.00000  1.00000
> >
> >  99    hdd    12.79399              osd.99       up   1.00000  1.00000
> >
> > 100    hdd    12.79399              osd.100      up   1.00000  1.00000
> >
> > 101    hdd    12.79399              osd.101      up   1.00000  1.00000
> >
> > 102    hdd    12.79399              osd.102      up   1.00000  1.00000
> >
> > 103    hdd    12.79399              osd.103      up   1.00000  1.00000
> >
> > 104    hdd    12.79399              osd.104      up   1.00000  1.00000
> >
> > 105    hdd    12.79399              osd.105      up   1.00000  1.00000
> >
> > 106    hdd    12.79399              osd.106      up   1.00000  1.00000
> >
> > 107    hdd    12.79399              osd.107      up   1.00000  1.00000
> >
> > 108    hdd    12.79399              osd.108      up   1.00000  1.00000
> >
> > 109    hdd    12.79399              osd.109      up   1.00000  1.00000
> >
> > 110    hdd    12.79399              osd.110      up   1.00000  1.00000
> >
> > 111    hdd    12.79399              osd.111      up   1.00000  1.00000
> >
> > 112    hdd    12.79399              osd.112      up   1.00000  1.00000
> >
> > 113    hdd    12.79399              osd.113      up   1.00000  1.00000
> >
> > 114    hdd    12.79399              osd.114      up   1.00000  1.00000
> >
> > 115    hdd    12.79399              osd.115      up   1.00000  1.00000
> >
> > 116    hdd    12.79399              osd.116      up   1.00000  1.00000
> >
> > 117    hdd    12.79399              osd.117      up   1.00000  1.00000
> >
> > 118    hdd    12.79399              osd.118      up   1.00000  1.00000
> >
> > 119    hdd    12.79399              osd.119      up   1.00000  1.00000
> >
> > -15          307.05579          host ceph06
> >
> > 120    hdd    12.79399              osd.120      up   1.00000  1.00000
> >
> > 121    hdd    12.79399              osd.121      up   1.00000  1.00000
> >
> > 122    hdd    12.79399              osd.122      up   1.00000  1.00000
> >
> > 123    hdd    12.79399              osd.123      up   1.00000  1.00000
> >
> > 124    hdd    12.79399              osd.124      up   1.00000  1.00000
> >
> > 125    hdd    12.79399              osd.125      up   1.00000  1.00000
> >
> > 126    hdd    12.79399              osd.126      up   1.00000  1.00000
> >
> > 127    hdd    12.79399              osd.127      up   1.00000  1.00000
> >
> > 128    hdd    12.79399              osd.128      up   1.00000  1.00000
> >
> > 129    hdd    12.79399              osd.129      up   1.00000  1.00000
> >
> > 130    hdd    12.79399              osd.130      up   1.00000  1.00000
> >
> > 131    hdd    12.79399              osd.131      up   1.00000  1.00000
> >
> > 132    hdd    12.79399              osd.132      up   1.00000  1.00000
> >
> > 133    hdd    12.79399              osd.133      up   1.00000  1.00000
> >
> > 134    hdd    12.79399              osd.134      up   1.00000  1.00000
> >
> > 135    hdd    12.79399              osd.135      up   1.00000  1.00000
> >
> > 136    hdd    12.79399              osd.136      up   1.00000  1.00000
> >
> > 137    hdd    12.79399              osd.137      up   1.00000  1.00000
> >
> > 138    hdd    12.79399              osd.138      up   1.00000  1.00000
> >
> > 139    hdd    12.79399              osd.139      up   1.00000  1.00000
> >
> > 140    hdd    12.79399              osd.140      up   1.00000  1.00000
> >
> > 141    hdd    12.79399              osd.141      up   1.00000  1.00000
> >
> > 142    hdd    12.79399              osd.142      up   1.00000  1.00000
> >
> > 143    hdd    12.79399              osd.143      up   1.00000  1.00000
> >
> > -17          307.05579          host ceph07
> >
> > 144    hdd    12.79399              osd.144      up   1.00000  1.00000
> >
> > 145    hdd    12.79399              osd.145      up   1.00000  1.00000
> >
> > 146    hdd    12.79399              osd.146      up   1.00000  1.00000
> >
> > 147    hdd    12.79399              osd.147      up   1.00000  1.00000
> >
> > 148    hdd    12.79399              osd.148      up   1.00000  1.00000
> >
> > 149    hdd    12.79399              osd.149      up   1.00000  1.00000
> >
> > 150    hdd    12.79399              osd.150      up   1.00000  1.00000
> >
> > 151    hdd    12.79399              osd.151      up   1.00000  1.00000
> >
> > 152    hdd    12.79399              osd.152      up   1.00000  1.00000
> >
> > 153    hdd    12.79399              osd.153      up   1.00000  1.00000
> >
> > 154    hdd    12.79399              osd.154      up   1.00000  1.00000
> >
> > 155    hdd    12.79399              osd.155      up   1.00000  1.00000
> >
> > 156    hdd    12.79399              osd.156      up   1.00000  1.00000
> >
> > 157    hdd    12.79399              osd.157      up   1.00000  1.00000
> >
> > 158    hdd    12.79399              osd.158      up   1.00000  1.00000
> >
> > 159    hdd    12.79399              osd.159      up   1.00000  1.00000
> >
> > 160    hdd    12.79399              osd.160      up   1.00000  1.00000
> >
> > 161    hdd    12.79399              osd.161      up   1.00000  1.00000
> >
> > 162    hdd    12.79399              osd.162      up   1.00000  1.00000
> >
> > 163    hdd    12.79399              osd.163      up   1.00000  1.00000
> >
> > 164    hdd    12.79399              osd.164      up   1.00000  1.00000
> >
> > 165    hdd    12.79399              osd.165      up   1.00000  1.00000
> >
> > 166    hdd    12.79399              osd.166      up   1.00000  1.00000
> >
> > 167    hdd    12.79399              osd.167      up   1.00000  1.00000
> >
> > root@ceph01:~#
> >
> > The data distribution looks relatively even (we use upmap as the balancer):
> >
> > root@ceph01:~# ceph osd df
> >
> > Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492
> >
> > Inferring config
> > /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config
> >
> > Using recent ceph image
> > docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5
> >
> > ID   CLASS  WEIGHT    REWEIGHT  SIZE     RAW USE  DATA      OMAP      META
> >     AVAIL    %USE   VAR   PGS  STATUS
> >
> >   0    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   5.0 MiB   14
> > GiB  7.1 TiB  44.73  0.93   35      up
> >
> >   1    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   1.8 MiB   13
> > GiB  7.2 TiB  43.46  0.90   33      up
> >
> >   2    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   4.4 MiB   16
> > GiB  6.2 TiB  51.27  1.07   39      up
> >
> >   3    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   4.9 MiB   15
> > GiB  6.7 TiB  47.38  0.99   36      up
> >
> >   4    hdd  12.79399   1.00000   13 TiB  5.5 TiB   5.5 TiB   812 KiB   13
> > GiB  7.2 TiB  43.37  0.90   35      up
> >
> >   5    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB    31 KiB   16
> > GiB  6.2 TiB  51.27  1.07   40      up
> >
> >   6    hdd  12.79399   1.00000   13 TiB  5.5 TiB   5.5 TiB   506 KiB   13
> > GiB  7.3 TiB  43.32  0.90   34      up
> >
> >   7    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   904 KiB   15
> > GiB  6.4 TiB  49.93  1.04   38      up
> >
> >   8    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   3.6 MiB   14
> > GiB  6.7 TiB  47.35  0.98   37      up
> >
> >   9    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   5.5 MiB   16
> > GiB  6.1 TiB  52.58  1.09   40      up
> >
> >  10    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.6 TiB    75 MiB   16
> > GiB  6.1 TiB  52.42  1.09   41      up
> >
> >  11    hdd  12.79399   1.00000   13 TiB  5.5 TiB   5.5 TiB   1.4 MiB   13
> > GiB  7.2 TiB  43.36  0.90   34      up
> >
> >  12    hdd  12.79399   1.00000   13 TiB  5.5 TiB   5.5 TiB   586 KiB   13
> > GiB  7.3 TiB  43.31  0.90   33      up
> >
> >  13    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   222 KiB   14
> > GiB  7.1 TiB  44.71  0.93   35      up
> >
> >  14    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   1.0 MiB   16
> > GiB  6.1 TiB  52.58  1.09   41      up
> >
> >  15    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.2 MiB   14
> > GiB  7.1 TiB  44.73  0.93   34      up
> >
> >  16    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   1.0 MiB   16
> > GiB  6.1 TiB  52.53  1.09   40      up
> >
> >  17    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   3.3 MiB   15
> > GiB  6.4 TiB  49.92  1.04   38      up
> >
> >  18    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   1.0 MiB   16
> > GiB  6.1 TiB  52.61  1.09   45      up
> >
> >  19    hdd  12.79399   1.00000   13 TiB  5.4 TiB   5.3 TiB   4.0 MiB   13
> > GiB  7.4 TiB  42.15  0.88   34      up
> >
> >  20    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   705 KiB   16
> > GiB  6.1 TiB  52.53  1.09   40      up
> >
> >  21    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   2.5 MiB   16
> > GiB  6.1 TiB  52.58  1.09   40      up
> >
> >  22    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   4.7 MiB   16
> > GiB  6.1 TiB  52.59  1.09   40      up
> >
> >  23    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   4.9 MiB   16
> > GiB  6.1 TiB  52.58  1.09   41      up
> >
> >  24    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   3.0 MiB   14
> > GiB  6.7 TiB  47.35  0.98   36      up
> >
> >  25    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   4.1 MiB   16
> > GiB  6.1 TiB  52.56  1.09   41      up
> >
> >  26    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   592 KiB   14
> > GiB  7.1 TiB  44.75  0.93   34      up
> >
> >  27    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   2.8 MiB   14
> > GiB  6.9 TiB  46.07  0.96   36      up
> >
> >  28    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   5.4 MiB   16
> > GiB  6.1 TiB  52.56  1.09   40      up
> >
> >  29    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   340 KiB   15
> > GiB  6.4 TiB  49.92  1.04   39      up
> >
> >  30    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB    93 KiB   14
> > GiB  6.9 TiB  46.03  0.96   35      up
> >
> >  31    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   2.1 MiB   13
> > GiB  7.2 TiB  43.57  0.91   35      up
> >
> >  32    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   5.0 MiB   15
> > GiB  6.6 TiB  48.68  1.01   38      up
> >
> >  33    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   175 KiB   13
> > GiB  7.2 TiB  43.44  0.90   33      up
> >
> >  34    hdd  12.79399   1.00000   13 TiB  6.9 TiB   6.8 TiB    77 MiB   16
> > GiB  5.9 TiB  53.86  1.12   43      up
> >
> >  35    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   2.3 MiB   15
> > GiB  6.6 TiB  48.68  1.01   37      up
> >
> >  36    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   1.2 MiB   16
> > GiB  6.2 TiB  51.25  1.07   41      up
> >
> >  37    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   5.0 MiB   14
> > GiB  6.7 TiB  47.37  0.99   36      up
> >
> >  38    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   3.7 MiB   13
> > GiB  7.2 TiB  43.44  0.90   35      up
> >
> >  39    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   341 KiB   15
> > GiB  6.7 TiB  47.38  0.99   37      up
> >
> >  40    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.1 MiB   14
> > GiB  7.1 TiB  44.73  0.93   34      up
> >
> >  41    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB    22 KiB   16
> > GiB  6.1 TiB  52.59  1.09   40      up
> >
> >  42    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   1.9 MiB   14
> > GiB  6.7 TiB  47.33  0.98   36      up
> >
> >  43    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   485 KiB   16
> > GiB  6.1 TiB  52.59  1.09   40      up
> >
> >  44    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   2.1 MiB   16
> > GiB  6.1 TiB  52.56  1.09   40      up
> >
> >  45    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   2.3 MiB   16
> > GiB  6.1 TiB  52.53  1.09   40      up
> >
> >  46    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   124 KiB   14
> > GiB  6.9 TiB  46.08  0.96   37      up
> >
> >  47    hdd  12.79399   1.00000   13 TiB  5.5 TiB   5.5 TiB   1.2 MiB   13
> > GiB  7.2 TiB  43.37  0.90   33      up
> >
> >  48    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   141 KiB   16
> > GiB  6.1 TiB  52.58  1.09   40      up
> >
> >  49    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   1.9 MiB   13
> > GiB  7.2 TiB  43.64  0.91   33      up
> >
> >  50    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   745 KiB   16
> > GiB  6.1 TiB  52.52  1.09   41      up
> >
> >  51    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   186 KiB   16
> > GiB  6.1 TiB  52.55  1.09   41      up
> >
> >  52    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   4.0 MiB   16
> > GiB  6.2 TiB  51.24  1.07   41      up
> >
> >  53    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   737 KiB   14
> > GiB  6.7 TiB  47.33  0.98   38      up
> >
> >  54    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   1.3 MiB   14
> > GiB  7.1 TiB  44.66  0.93   34      up
> >
> >  55    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB    97 KiB   14
> > GiB  7.1 TiB  44.73  0.93   34      up
> >
> >  56    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   1.2 MiB   14
> > GiB  7.1 TiB  44.76  0.93   35      up
> >
> >  57    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   3.5 MiB   14
> > GiB  7.1 TiB  44.78  0.93   35      up
> >
> >  58    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB    80 MiB   16
> > GiB  6.1 TiB  52.56  1.09   41      up
> >
> >  59    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   4.5 MiB   14
> > GiB  6.9 TiB  46.08  0.96   36      up
> >
> >  60    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   903 KiB   14
> > GiB  7.1 TiB  44.73  0.93   34      up
> >
> >  61    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB     1 KiB   14
> > GiB  7.1 TiB  44.65  0.93   34      up
> >
> >  62    hdd  12.79399   1.00000   13 TiB  6.9 TiB   6.8 TiB   425 KiB   16
> > GiB  5.9 TiB  53.87  1.12   41      up
> >
> >  63    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   4.1 MiB   14
> > GiB  7.2 TiB  43.56  0.91   34      up
> >
> >  64    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   578 KiB   16
> > GiB  6.2 TiB  51.27  1.07   39      up
> >
> >  65    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   3.6 MiB   13
> > GiB  7.2 TiB  43.44  0.90   33      up
> >
> >  66    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   288 KiB   13
> > GiB  7.2 TiB  43.40  0.90   34      up
> >
> >  67    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   413 KiB   14
> > GiB  6.7 TiB  47.34  0.98   37      up
> >
> >  68    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   4.3 MiB   16
> > GiB  6.1 TiB  52.56  1.09   41      up
> >
> >  69    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   1.0 MiB   13
> > GiB  7.2 TiB  43.44  0.90   33      up
> >
> >  70    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   3.2 MiB   16
> > GiB  6.1 TiB  52.56  1.09   40      up
> >
> >  71    hdd  12.79399   1.00000   13 TiB  5.5 TiB   5.5 TiB   1.3 MiB   14
> > GiB  7.2 TiB  43.36  0.90   35      up
> >
> >  72    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   4.7 MiB   16
> > GiB  6.2 TiB  51.29  1.07   39      up
> >
> >  73    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   1.8 MiB   14
> > GiB  7.1 TiB  44.73  0.93   34      up
> >
> >  74    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.6 TiB   1.2 MiB   14
> > GiB  7.1 TiB  44.64  0.93   34      up
> >
> >  75    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   1.5 MiB   15
> > GiB  6.4 TiB  49.95  1.04   38      up
> >
> >  76    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   3.8 MiB   15
> > GiB  6.6 TiB  48.68  1.01   37      up
> >
> >  77    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   5.0 MiB   14
> > GiB  7.1 TiB  44.65  0.93   34      up
> >
> >  78    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   1.8 MiB   14
> > GiB  7.1 TiB  44.68  0.93   35      up
> >
> >  79    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   984 KiB   14
> > GiB  6.7 TiB  47.36  0.98   36      up
> >
> >  80    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   2.6 MiB   13
> > GiB  7.2 TiB  43.45  0.90   34      up
> >
> >  81    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   1.2 MiB   16
> > GiB  6.2 TiB  51.26  1.07   41      up
> >
> >  82    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   4.9 MiB   15
> > GiB  6.6 TiB  48.61  1.01   37      up
> >
> >  83    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   827 KiB   14
> > GiB  6.7 TiB  47.33  0.98   36      up
> >
> >  84    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   2.7 MiB   16
> > GiB  6.2 TiB  51.22  1.07   39      up
> >
> >  85    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   868 KiB   15
> > GiB  6.4 TiB  49.94  1.04   40      up
> >
> >  86    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   4.1 MiB   16
> > GiB  6.1 TiB  52.57  1.09   42      up
> >
> >  87    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   702 KiB   16
> > GiB  6.1 TiB  52.54  1.09   41      up
> >
> >  88    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB    12 KiB   14
> > GiB  7.1 TiB  44.76  0.93   35      up
> >
> >  89    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   2.9 MiB   14
> > GiB  6.9 TiB  46.07  0.96   35      up
> >
> >  90    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   1.8 MiB   13
> > GiB  7.2 TiB  43.48  0.90   34      up
> >
> >  91    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   836 KiB   16
> > GiB  6.1 TiB  52.55  1.09   40      up
> >
> >  92    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   4.0 MiB   13
> > GiB  7.2 TiB  43.43  0.90   33      up
> >
> >  93    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   1.5 MiB   16
> > GiB  6.1 TiB  52.57  1.09   41      up
> >
> >  94    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   459 KiB   13
> > GiB  7.2 TiB  43.44  0.90   34      up
> >
> >  95    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   4.1 MiB   16
> > GiB  6.2 TiB  51.28  1.07   39      up
> >
> >  96    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.1 MiB   14
> > GiB  7.1 TiB  44.75  0.93   35      up
> >
> >  97    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   662 KiB   14
> > GiB  7.1 TiB  44.75  0.93   34      up
> >
> >  98    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   2.9 MiB   14
> > GiB  6.9 TiB  45.97  0.96   35      up
> >
> >  99    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   3.8 MiB   16
> > GiB  6.1 TiB  52.56  1.09   40      up
> >
> > 100    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   1.8 MiB   14
> > GiB  7.1 TiB  44.76  0.93   36      up
> >
> > 101    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   3.6 MiB   15
> > GiB  6.4 TiB  49.95  1.04   40      up
> >
> > 102    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.9 MiB   14
> > GiB  7.1 TiB  44.65  0.93   34      up
> >
> > 103    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   643 KiB   16
> > GiB  6.1 TiB  52.57  1.09   41      up
> >
> > 104    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   5.5 MiB   15
> > GiB  6.4 TiB  49.98  1.04   38      up
> >
> > 105    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   830 KiB   13
> > GiB  7.2 TiB  43.46  0.90   33      up
> >
> > 106    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   1.0 MiB   14
> > GiB  6.7 TiB  47.34  0.98   36      up
> >
> > 107    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   286 KiB   16
> > GiB  6.2 TiB  51.23  1.07   39      up
> >
> > 108    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   961 KiB   15
> > GiB  6.6 TiB  48.68  1.01   37      up
> >
> > 109    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   291 KiB   16
> > GiB  6.1 TiB  52.56  1.09   42      up
> >
> > 110    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   1.0 MiB   14
> > GiB  6.9 TiB  46.02  0.96   36      up
> >
> > 111    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   2.7 MiB   16
> > GiB  6.2 TiB  51.22  1.07   39      up
> >
> > 112    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   631 KiB   15
> > GiB  6.6 TiB  48.67  1.01   37      up
> >
> > 113    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   4.5 MiB   16
> > GiB  6.1 TiB  52.56  1.09   41      up
> >
> > 114    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   4.0 MiB   15
> > GiB  6.4 TiB  49.94  1.04   39      up
> >
> > 115    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   1.5 MiB   14
> > GiB  6.9 TiB  46.05  0.96   35      up
> >
> > 116    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   4.1 MiB   15
> > GiB  6.4 TiB  49.93  1.04   38      up
> >
> > 117    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   194 KiB   13
> > GiB  7.2 TiB  43.44  0.90   35      up
> >
> > 118    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   3.1 MiB   15
> > GiB  6.4 TiB  49.96  1.04   38      up
> >
> > 119    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   3.9 MiB   14
> > GiB  7.2 TiB  43.46  0.90   33      up
> >
> > 120    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   1.8 MiB   16
> > GiB  6.1 TiB  52.55  1.09   40      up
> >
> > 121    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   3.7 MiB   16
> > GiB  6.1 TiB  52.54  1.09   40      up
> >
> > 122    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   2.4 MiB   15
> > GiB  6.4 TiB  49.97  1.04   38      up
> >
> > 123    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB  1003 KiB   14
> > GiB  6.9 TiB  46.02  0.96   35      up
> >
> > 124    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   526 KiB   14
> > GiB  6.7 TiB  47.35  0.98   36      up
> >
> > 125    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB    92 KiB   15
> > GiB  6.7 TiB  47.37  0.98   37      up
> >
> > 126    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.9 MiB   14
> > GiB  7.1 TiB  44.74  0.93   34      up
> >
> > 127    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   5.5 MiB   16
> > GiB  6.2 TiB  51.26  1.07   39      up
> >
> > 128    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   5.4 MiB   16
> > GiB  6.1 TiB  52.56  1.09   40      up
> >
> > 129    hdd  12.79399   1.00000   13 TiB  6.2 TiB   6.2 TiB   3.6 MiB   15
> > GiB  6.6 TiB  48.66  1.01   38      up
> >
> > 130    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB   2.8 MiB   16
> > GiB  6.1 TiB  52.56  1.09   41      up
> >
> > 131    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   5.1 MiB   16
> > GiB  6.2 TiB  51.28  1.07   40      up
> >
> > 132    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   1.7 MiB   16
> > GiB  6.2 TiB  51.25  1.07   39      up
> >
> > 133    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   230 KiB   16
> > GiB  6.2 TiB  51.25  1.07   39      up
> >
> > 134    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   2.0 MiB   14
> > GiB  7.2 TiB  43.45  0.90   34      up
> >
> > 135    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   4.4 MiB   13
> > GiB  7.2 TiB  43.46  0.90   33      up
> >
> > 136    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   5.6 MiB   16
> > GiB  6.2 TiB  51.29  1.07   39      up
> >
> > 137    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   3.6 MiB   14
> > GiB  7.1 TiB  44.75  0.93   36      up
> >
> > 138    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   4.9 MiB   15
> > GiB  6.4 TiB  49.98  1.04   40      up
> >
> > 139    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB    21 KiB   16
> > GiB  6.2 TiB  51.23  1.07   39      up
> >
> > 140    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   5.4 MiB   14
> > GiB  7.1 TiB  44.74  0.93   36      up
> >
> > 141    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   359 KiB   13
> > GiB  7.2 TiB  43.42  0.90   34      up
> >
> > 142    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB    27 KiB   13
> > GiB  7.2 TiB  43.46  0.90   33      up
> >
> > 143    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   395 KiB   14
> > GiB  6.9 TiB  46.07  0.96   35      up
> >
> > 144    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   1.2 MiB   14
> > GiB  7.1 TiB  44.75  0.93   34      up
> >
> > 145    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   3.3 MiB   14
> > GiB  7.1 TiB  44.66  0.93   35      up
> >
> > 146    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   4.0 MiB   14
> > GiB  6.9 TiB  46.06  0.96   35      up
> >
> > 147    hdd  12.79399   1.00000   13 TiB  6.7 TiB   6.7 TiB    46 KiB   16
> > GiB  6.1 TiB  52.55  1.09   40      up
> >
> > 148    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   4.8 MiB   16
> > GiB  6.2 TiB  51.23  1.07   39      up
> >
> > 149    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.4 MiB   14
> > GiB  7.1 TiB  44.70  0.93   35      up
> >
> > 150    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   4.4 MiB   14
> > GiB  7.1 TiB  44.65  0.93   35      up
> >
> > 151    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   3.7 MiB   14
> > GiB  7.1 TiB  44.65  0.93   35      up
> >
> > 152    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   5.3 MiB   15
> > GiB  6.2 TiB  51.26  1.07   39      up
> >
> > 153    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   2.3 MiB   15
> > GiB  6.4 TiB  49.94  1.04   39      up
> >
> > 154    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   3.0 MiB   14
> > GiB  6.7 TiB  47.37  0.99   36      up
> >
> > 155    hdd  12.79399   1.00000   13 TiB  6.1 TiB   6.0 TiB   3.0 MiB   14
> > GiB  6.7 TiB  47.35  0.98   38      up
> >
> > 156    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   4.0 MiB   16
> > GiB  6.2 TiB  51.26  1.07   41      up
> >
> > 157    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   2.8 MiB   14
> > GiB  6.9 TiB  46.07  0.96   36      up
> >
> > 158    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   3.0 MiB   16
> > GiB  6.2 TiB  51.25  1.07   40      up
> >
> > 159    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   3.9 MiB   15
> > GiB  6.4 TiB  49.93  1.04   38      up
> >
> > 160    hdd  12.79399   1.00000   13 TiB  5.9 TiB   5.8 TiB   2.4 MiB   14
> > GiB  6.9 TiB  46.03  0.96   35      up
> >
> > 161    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   1.1 MiB   16
> > GiB  6.2 TiB  51.25  1.07   39      up
> >
> > 162    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.6 TiB   4.1 MiB   13
> > GiB  7.1 TiB  44.64  0.93   34      up
> >
> > 163    hdd  12.79399   1.00000   13 TiB  5.7 TiB   5.7 TiB   445 KiB   14
> > GiB  7.1 TiB  44.69  0.93   34      up
> >
> > 164    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   4.9 MiB   15
> > GiB  6.4 TiB  49.97  1.04   38      up
> >
> > 165    hdd  12.79399   1.00000   13 TiB  6.6 TiB   6.5 TiB   5.5 MiB   16
> > GiB  6.2 TiB  51.25  1.07   40      up
> >
> > 166    hdd  12.79399   1.00000   13 TiB  5.6 TiB   5.5 TiB   2.5 MiB   13
> > GiB  7.2 TiB  43.45  0.90   33      up
> >
> > 167    hdd  12.79399   1.00000   13 TiB  6.4 TiB   6.3 TiB   4.0 MiB   15
> > GiB  6.4 TiB  49.98  1.04   38      up
> >
> >                          TOTAL  2.1 PiB  1.0 PiB  1023 TiB   630 MiB  2.4
> > TiB  1.1 PiB  48.09
> >
> > MIN/MAX VAR: 0.88/1.12  STDDEV: 3.50
> >
> > root@ceph01:~#
> >
> >
> > The cluster status, however, shows we have many misplaced objects and a
> > few remapped PGs. We had been performing maintenance (upgrade to Ceph
> > 15.2.9), and initially the misplaced object count was dropping rapidly at a
> > rate of about 400-500MB/s and 100-200 objects/s. However, once it hit a
> > certain point, it dropped to 0-4MB/s and 0 objects/s (it's not 0, it is
> > progressing, but it's below 1 object/s):
> >
> >
> > root@ceph01:~# ceph -s
> >
> > Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492
> >
> > Inferring config
> > /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config
> >
> > Using recent ceph image
> > docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5
> >
> >   cluster:
> >
> >     id:     41bb9256-c3bf-11ea-85b9-9e07b0435492
> >
> >     health: HEALTH_WARN
> >
> >             5 slow ops, oldest one blocked for 84 sec, daemons
> > [osd.141,osd.46] have slow ops.
> >
> >
> >
> >   services:
> >
> >     mon: 5 daemons, quorum ceph01,ceph04,ceph02,ceph03,ceph05 (age 25h)
> >
> >     mgr: ceph05.yropto(active, since 11h), standbys: ceph01.aqkgbl,
> > ceph03.ytkuyr, ceph04.smbdew, ceph02.ndynmo
> >
> >     osd: 168 osds: 168 up (since 9h), 168 in (since 6M); 10 remapped pgs
> >
> >
> >
> >   task status:
> >
> >
> >
> >   data:
> >
> >     pools:   3 pools, 1057 pgs
> >
> >     objects: 166.90M objects, 632 TiB
> >
> >     usage:   1.0 PiB used, 1.1 PiB / 2.1 PiB avail
> >
> >     pgs:     583177/1001370417 objects misplaced (0.058%)
> >
> >              1039 active+clean
> >
> >              11   active+clean+scrubbing
> >
> >              5    active+remapped+backfilling
> >
> >              1    active+clean+scrubbing+deep
> >
> >              1    active+clean+laggy
> >
> >
> >
> >   io:
> >
> >     client:   4.5 MiB/s rd, 31 MiB/s wr, 75 op/s rd, 59 op/s wr
> >
> >     recovery: 2.7 MiB/s, 0 objects/s
> >
> >
> >
> > root@ceph01:~#
> >
> >
> > root@ceph01:~# ceph pg dump |grep remapped
> >
> > Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492
> >
> > Inferring config
> > /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config
> >
> > Using recent ceph image
> > docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5
> >
> > 2.3d1     163805                   0         0          0        0
> > 682273001472            0           0  7462      7462  active+*remapped*+backfilling
> > 2021-02-26T01:28:06.212904+0000    53782'2773494    53782:5593409
> > [48,122,99,92,158,31]          48   [48,122,99,92,158,34]              48
> >   50907'2762333  2021-02-23T20:13:50.501757+0000    50907'2762333
> > 2021-02-23T20:13:50.501757+0000              0
> >
> > 2.341     163180                   0         0     136844        0
> > 679698268160            0           0  8569      8569  active+*remapped*+backfilling
> > 2021-02-26T01:27:30.693520+0000    53782'2694466    53782:5497907
> > [65,125,79,147,45,100]          65   [65,18,79,147,45,100]              65
> >   51073'2688628  2021-02-24T09:38:41.588350+0000    51073'2688628
> > 2021-02-24T09:38:41.588350+0000              0
> >
> > 2.1e1     162402                   0         0     150624        0
> > 676505321472            0           0  7893      7893  active+*remapped*+backfilling
> > 2021-02-25T15:36:29.121352+0000    53782'3105605    53782:5885140
> > [112,63,126,0,147,25]         112   [112,50,126,0,147,25]             112
> >   50907'3086340  2021-02-22T23:14:49.510535+0000    47263'2883486
> > 2021-02-01T15:43:32.089129+0000              0
> >
> > 2.268     163459                   0         0     136459        0
> > 680743612416            0           0  7337      7337  active+*remapped*+backfilling
> > 2021-02-25T15:27:58.957497+0000    53782'2713107    53782:5536140
> > [93,98,150,49,26,135]          93   [93,98,150,62,26,135]              93
> >   50907'2698319  2021-02-23T14:16:11.171370+0000    50907'2698319
> > 2021-02-23T14:16:11.171370+0000              0
> >
> > 2.285     163380                   0         0     158866        0
> > 680507670528            0           0  8148      8148  active+*remapped*+backfilling
> > 2021-02-26T01:26:31.442018+0000    53782'3191803    53782:5986426
> > [67,166,37,90,114,138]          67  [67,147,37,90,114,138]              67
> >   51073'3177251  2021-02-23T23:51:51.751625+0000    47597'2917541
> > 2021-02-03T20:14:38.762384+0000              0
> >
> > dumped all
> >
> > root@ceph01:~#
> >
> > Even if we stop the scrubs/deep scrubs, things don't improve re: recovery
> > rate or cluster performance. We've monitored the HDDs/NVME backing these
> > OSDs and with the scrubs stopped, they are nearly idle. CPU is almost idle.
> > There's nothing to indicate any problem at all with the underlying hardware
> > that we can find, at least from a load/failure perspective.
> >
> > Also, interestingly, rados bench on a newly created **replicated** pool
> > is also extremely slow, as in < 50MB/s.
> >
> > Crush rule for our EC (4+2) pool, but we see the performance issues on
> > even a newly created triple-replicated pool:
> >
> > root@ceph01:~# ceph osd crush rule dump ecpool
> >
> > Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492
> >
> > Inferring config
> > /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config
> >
> > Using recent ceph image
> > docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5
> >
> > {
> >
> >     "rule_id": 1,
> >
> >     "rule_name": "ecpool",
> >
> >     "ruleset": 1,
> >
> >     "type": 3,
> >
> >     "min_size": 3,
> >
> >     "max_size": 6,
> >
> >     "steps": [
> >
> >         {
> >
> >             "op": "set_chooseleaf_tries",
> >
> >             "num": 5
> >
> >         },
> >
> >         {
> >
> >             "op": "set_choose_tries",
> >
> >             "num": 100
> >
> >         },
> >
> >         {
> >
> >             "op": "take",
> >
> >             "item": -1,
> >
> >             "item_name": "default"
> >
> >         },
> >
> >         {
> >
> >             "op": "chooseleaf_indep",
> >
> >             "num": 0,
> >
> >             "type": "host"
> >
> >         },
> >
> >         {
> >
> >             "op": "emit"
> >
> >         }
> >
> >     ]
> >
> > }
> >
> >
> > Cluster/pool space:
> >
> >
> > root@ceph01:~# ceph df
> >
> > Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492
> >
> > Inferring config
> > /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config
> >
> > Using recent ceph image
> > docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5
> >
> > --- RAW STORAGE ---
> >
> > CLASS  SIZE     AVAIL    USED     RAW USED  %RAW USED
> >
> > hdd    2.1 PiB  1.1 PiB  1.0 PiB   1.0 PiB      48.09
> >
> > TOTAL  2.1 PiB  1.1 PiB  1.0 PiB   1.0 PiB      48.09
> >
> >
> >
> > --- POOLS ---
> >
> > POOL                   ID  PGS   STORED   OBJECTS  USED      %USED  MAX
> > AVAIL
> >
> > device_health_metrics   1     1   78 MiB      182   233 MiB      0    295
> > TiB
> >
> > ecpool                  2  1024  632 TiB  166.90M  1023 TiB  53.65    589
> > TiB
> >
> > rbd                     3    32   59 MiB        5   178 MiB      0    295
> > TiB
> >
> > root@ceph01:~#
> >
> > We're at a loss for why this cluster has suddenly started behaving this
> > way. With the disks (all of them, on all nodes) effectively being idle with
> > all the scrubs disabled, plenty of CPU available, and no network issues
> > (iperf on all paths is full line rate - ~10GB/s - we have a dedicated
> > public and cluster network), it's befuddling.
> >
> > We appreciate any assistance you can provide, and are happy to provide any
> > information that might be useful.
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux