Re: Migration Nautilus to Pacifi : Very high latencies (EC profile)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



We have a newly-built pacific (16.2.7) cluster running 8+3 EC jerasure ~250
OSDS across 21 hosts which has significantly lower than expected IOPS. Only
doing about 30 IOPS per spinning disk (with appropriately sized SSD
bluestore db) around ~100 PGs per OSD. Have around 100 CephFS (ceph fuse
16.2.7) clients using the cluster. Cluster regularly reports slow ops from
the OSDs but the vast majority, 90% plus of the OSDs, are only <50% IOPS
utilized. Plenty of cpu/ram/network left on all cluster nodes. We have
looked for hardware (disk/bond/network/mce) issues across the cluster with
no findings / checked send-qs and received-q's across the cluster to try
and narrow in on an individual failing component but nothing found there.
Slow ops are also spread equally across the servers in the cluster. Does
your cluster report any health warnings (slow ops etc) alongside your
reduced performance?

Respectfully,

*Wes Dillingham*
wes@xxxxxxxxxxxxxxxxx
LinkedIn <http://www.linkedin.com/in/wesleydillingham>


On Mon, May 16, 2022 at 2:00 AM Martin Verges <martin.verges@xxxxxxxx>
wrote:

> Hello,
>
> depending on your workload, drives and OSD allocation size, using the 3+2
> can be way slower than the 4+2. Maybe give it a small benchmark and try if
> you see a huge difference. We had some benchmarks with such and they showed
> quite ugly results in some tests. Best way to deploy EC in our findings is
> in power of 2, like 2+x, 4+x, 8+x, 16+x. Especially when you deploy OSDs
> before the Ceph allocation change patch, you might end up consuming way
> more space if you don't use power of 2. With the 4k allocation size at
> least this has been greatly improved for newer deployed OSDs.
>
> --
> Martin Verges
> Managing director
>
> Mobile: +49 174 9335695  | Chat: https://t.me/MartinVerges
>
> croit GmbH, Freseniusstr. 31h, 81247 Munich
> CEO: Martin Verges - VAT-ID: DE310638492
> Com. register: Amtsgericht Munich HRB 231263
> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
>
>
> On Sun, 15 May 2022 at 20:30, stéphane chalansonnet <schalans@xxxxxxxxx>
> wrote:
>
> > Hi,
> >
> > Thank you for your answer.
> > this is not a good news if you also notice a performance decrease on your
> > side
> > No, as far as we know, you cannot downgrade to Octopus.
> > Going forward seems to be the only way, so Quincy .
> > We have a a qualification cluster so we can try on it (but full virtual
> > configuration)
> >
> >
> > We are using 4+2 and 3+2 profile
> > Are you also on the same profile on your Cluster ?
> > Maybe replicated profile are not be impacted ?
> >
> > Actually, we are trying to recreate one by one the OSD.
> > some parameters can be only set by this way .
> > The first storage Node is almost rebuild, we will see if the latencies on
> > it are below the others ...
> >
> > Wait and see .....
> >
> > Le dim. 15 mai 2022 à 10:16, Martin Verges <martin.verges@xxxxxxxx> a
> > écrit :
> >
> >> Hello,
> >>
> >> what exact EC level do you use?
> >>
> >> I can confirm, that our internal data shows a performance drop when
> using
> >> pacific. So far Octopus is faster and better than pacific but I doubt
> you
> >> can roll back to it. We haven't rerun our benchmarks on Quincy yet, but
> >> according to some presentation it should be faster than pacific. Maybe
> try
> >> to jump away from the pacific release into the unknown!
> >>
> >> --
> >> Martin Verges
> >> Managing director
> >>
> >> Mobile: +49 174 9335695  | Chat: https://t.me/MartinVerges
> >>
> >> croit GmbH, Freseniusstr. 31h, 81247 Munich
> >> CEO: Martin Verges - VAT-ID: DE310638492
> >> Com. register: Amtsgericht Munich HRB 231263
> >> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
> >>
> >>
> >> On Sat, 14 May 2022 at 12:27, stéphane chalansonnet <schalans@xxxxxxxxx
> >
> >> wrote:
> >>
> >>> Hello,
> >>>
> >>> After a successful update from Nautilus to Pacific on Centos8.5, we
> >>> observed some high latencies on our cluster.
> >>>
> >>> We did not find very much thing on community related to latencies post
> >>> migration
> >>>
> >>> Our setup is
> >>> 6x storage Node (256GRAM, 2SSD OSD + 5*6To SATA HDD)
> >>> Erasure coding profile
> >>> We have two EC pool :
> >>> -> Pool1 : Full HDD SAS Drive 6To
> >>> -> Pool2 : Full SSD Drive
> >>>
> >>> Object S3 and RBD block workload
> >>>
> >>> Our performances in nautilus, before the upgrade , are acceptable.
> >>> However , the next day , performance dropped by 3 or 4
> >>> Benchmark showed 15KIOPS on flash drive , before upgrade we had
> >>> almost 80KIOPS
> >>> Also, HDD pool is almost down (too much lantencies
> >>>
> >>> We suspected , maybe, an impact on erasure Coding configuration on
> >>> Pacific
> >>> Anyone observed the same behaviour ? any tuning ?
> >>>
> >>> Thank you for your help.
> >>>
> >>> ceph osd tree
> >>> ID   CLASS  WEIGHT     TYPE NAME                 STATUS  REWEIGHT
> >>> PRI-AFF
> >>>  -1         347.61304  root default
> >>>  -3          56.71570      host cnp31tcephosd01
> >>>   0    hdd    5.63399          osd.0                 up   1.00000
> >>> 1.00000
> >>>   1    hdd    5.63399          osd.1                 up   1.00000
> >>> 1.00000
> >>>   2    hdd    5.63399          osd.2                 up   1.00000
> >>> 1.00000
> >>>   3    hdd    5.63399          osd.3                 up   1.00000
> >>> 1.00000
> >>>   4    hdd    5.63399          osd.4                 up   1.00000
> >>> 1.00000
> >>>   5    hdd    5.63399          osd.5                 up   1.00000
> >>> 1.00000
> >>>   6    hdd    5.63399          osd.6                 up   1.00000
> >>> 1.00000
> >>>   7    hdd    5.63399          osd.7                 up   1.00000
> >>> 1.00000
> >>>  40    ssd    5.82190          osd.40                up   1.00000
> >>> 1.00000
> >>>  48    ssd    5.82190          osd.48                up   1.00000
> >>> 1.00000
> >>>  -5          56.71570      host cnp31tcephosd02
> >>>   8    hdd    5.63399          osd.8                 up   1.00000
> >>> 1.00000
> >>>   9    hdd    5.63399          osd.9               down   1.00000
> >>> 1.00000
> >>>  10    hdd    5.63399          osd.10                up   1.00000
> >>> 1.00000
> >>>  11    hdd    5.63399          osd.11                up   1.00000
> >>> 1.00000
> >>>  12    hdd    5.63399          osd.12                up   1.00000
> >>> 1.00000
> >>>  13    hdd    5.63399          osd.13                up   1.00000
> >>> 1.00000
> >>>  14    hdd    5.63399          osd.14                up   1.00000
> >>> 1.00000
> >>>  15    hdd    5.63399          osd.15                up   1.00000
> >>> 1.00000
> >>>  49    ssd    5.82190          osd.49                up   1.00000
> >>> 1.00000
> >>>  50    ssd    5.82190          osd.50                up   1.00000
> >>> 1.00000
> >>>  -7          56.71570      host cnp31tcephosd03
> >>>  16    hdd    5.63399          osd.16                up   1.00000
> >>> 1.00000
> >>>  17    hdd    5.63399          osd.17                up   1.00000
> >>> 1.00000
> >>>  18    hdd    5.63399          osd.18                up   1.00000
> >>> 1.00000
> >>>  19    hdd    5.63399          osd.19                up   1.00000
> >>> 1.00000
> >>>  20    hdd    5.63399          osd.20                up   1.00000
> >>> 1.00000
> >>>  21    hdd    5.63399          osd.21                up   1.00000
> >>> 1.00000
> >>>  22    hdd    5.63399          osd.22                up   1.00000
> >>> 1.00000
> >>>  23    hdd    5.63399          osd.23                up   1.00000
> >>> 1.00000
> >>>  51    ssd    5.82190          osd.51                up   1.00000
> >>> 1.00000
> >>>  52    ssd    5.82190          osd.52                up   1.00000
> >>> 1.00000
> >>>  -9          56.71570      host cnp31tcephosd04
> >>>  24    hdd    5.63399          osd.24                up   1.00000
> >>> 1.00000
> >>>  25    hdd    5.63399          osd.25                up   1.00000
> >>> 1.00000
> >>>  26    hdd    5.63399          osd.26                up   1.00000
> >>> 1.00000
> >>>  27    hdd    5.63399          osd.27                up   1.00000
> >>> 1.00000
> >>>  28    hdd    5.63399          osd.28                up   1.00000
> >>> 1.00000
> >>>  29    hdd    5.63399          osd.29                up   1.00000
> >>> 1.00000
> >>>  30    hdd    5.63399          osd.30                up   1.00000
> >>> 1.00000
> >>>  31    hdd    5.63399          osd.31                up   1.00000
> >>> 1.00000
> >>>  53    ssd    5.82190          osd.53                up   1.00000
> >>> 1.00000
> >>>  54    ssd    5.82190          osd.54                up   1.00000
> >>> 1.00000
> >>> -11          56.71570      host cnp31tcephosd05
> >>>  32    hdd    5.63399          osd.32                up   1.00000
> >>> 1.00000
> >>>  33    hdd    5.63399          osd.33                up   1.00000
> >>> 1.00000
> >>>  34    hdd    5.63399          osd.34                up   1.00000
> >>> 1.00000
> >>>  35    hdd    5.63399          osd.35                up   1.00000
> >>> 1.00000
> >>>  36    hdd    5.63399          osd.36                up   1.00000
> >>> 1.00000
> >>>  37    hdd    5.63399          osd.37                up   1.00000
> >>> 1.00000
> >>>  38    hdd    5.63399          osd.38                up   1.00000
> >>> 1.00000
> >>>  39    hdd    5.63399          osd.39                up   1.00000
> >>> 1.00000
> >>>  55    ssd    5.82190          osd.55                up   1.00000
> >>> 1.00000
> >>>  56    ssd    5.82190          osd.56                up   1.00000
> >>> 1.00000
> >>> -13          64.03453      host cnp31tcephosd06
> >>>  41    hdd    7.48439          osd.41                up   1.00000
> >>> 1.00000
> >>>  42    hdd    7.48439          osd.42                up   1.00000
> >>> 1.00000
> >>>  43    hdd    7.48439          osd.43                up   1.00000
> >>> 1.00000
> >>>  44    hdd    7.48439          osd.44                up   1.00000
> >>> 1.00000
> >>>  45    hdd    7.48439          osd.45                up   1.00000
> >>> 1.00000
> >>>  46    hdd    7.48439          osd.46                up   1.00000
> >>> 1.00000
> >>>  47    hdd    7.48439          osd.47                up   1.00000
> >>> 1.00000
> >>>  57    ssd    5.82190          osd.57                up   1.00000
> >>> 1.00000
> >>>  58    ssd    5.82190          osd.58                up   1.00000
> >>> 1.00000
> >>> _______________________________________________
> >>> ceph-users mailing list -- ceph-users@xxxxxxx
> >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >>>
> >>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux