Hello, depending on your workload, drives and OSD allocation size, using the 3+2 can be way slower than the 4+2. Maybe give it a small benchmark and try if you see a huge difference. We had some benchmarks with such and they showed quite ugly results in some tests. Best way to deploy EC in our findings is in power of 2, like 2+x, 4+x, 8+x, 16+x. Especially when you deploy OSDs before the Ceph allocation change patch, you might end up consuming way more space if you don't use power of 2. With the 4k allocation size at least this has been greatly improved for newer deployed OSDs. -- Martin Verges Managing director Mobile: +49 174 9335695 | Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx On Sun, 15 May 2022 at 20:30, stéphane chalansonnet <schalans@xxxxxxxxx> wrote: > Hi, > > Thank you for your answer. > this is not a good news if you also notice a performance decrease on your > side > No, as far as we know, you cannot downgrade to Octopus. > Going forward seems to be the only way, so Quincy . > We have a a qualification cluster so we can try on it (but full virtual > configuration) > > > We are using 4+2 and 3+2 profile > Are you also on the same profile on your Cluster ? > Maybe replicated profile are not be impacted ? > > Actually, we are trying to recreate one by one the OSD. > some parameters can be only set by this way . > The first storage Node is almost rebuild, we will see if the latencies on > it are below the others ... > > Wait and see ..... > > Le dim. 15 mai 2022 à 10:16, Martin Verges <martin.verges@xxxxxxxx> a > écrit : > >> Hello, >> >> what exact EC level do you use? >> >> I can confirm, that our internal data shows a performance drop when using >> pacific. So far Octopus is faster and better than pacific but I doubt you >> can roll back to it. We haven't rerun our benchmarks on Quincy yet, but >> according to some presentation it should be faster than pacific. Maybe try >> to jump away from the pacific release into the unknown! >> >> -- >> Martin Verges >> Managing director >> >> Mobile: +49 174 9335695 | Chat: https://t.me/MartinVerges >> >> croit GmbH, Freseniusstr. 31h, 81247 Munich >> CEO: Martin Verges - VAT-ID: DE310638492 >> Com. register: Amtsgericht Munich HRB 231263 >> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx >> >> >> On Sat, 14 May 2022 at 12:27, stéphane chalansonnet <schalans@xxxxxxxxx> >> wrote: >> >>> Hello, >>> >>> After a successful update from Nautilus to Pacific on Centos8.5, we >>> observed some high latencies on our cluster. >>> >>> We did not find very much thing on community related to latencies post >>> migration >>> >>> Our setup is >>> 6x storage Node (256GRAM, 2SSD OSD + 5*6To SATA HDD) >>> Erasure coding profile >>> We have two EC pool : >>> -> Pool1 : Full HDD SAS Drive 6To >>> -> Pool2 : Full SSD Drive >>> >>> Object S3 and RBD block workload >>> >>> Our performances in nautilus, before the upgrade , are acceptable. >>> However , the next day , performance dropped by 3 or 4 >>> Benchmark showed 15KIOPS on flash drive , before upgrade we had >>> almost 80KIOPS >>> Also, HDD pool is almost down (too much lantencies >>> >>> We suspected , maybe, an impact on erasure Coding configuration on >>> Pacific >>> Anyone observed the same behaviour ? any tuning ? >>> >>> Thank you for your help. >>> >>> ceph osd tree >>> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT >>> PRI-AFF >>> -1 347.61304 root default >>> -3 56.71570 host cnp31tcephosd01 >>> 0 hdd 5.63399 osd.0 up 1.00000 >>> 1.00000 >>> 1 hdd 5.63399 osd.1 up 1.00000 >>> 1.00000 >>> 2 hdd 5.63399 osd.2 up 1.00000 >>> 1.00000 >>> 3 hdd 5.63399 osd.3 up 1.00000 >>> 1.00000 >>> 4 hdd 5.63399 osd.4 up 1.00000 >>> 1.00000 >>> 5 hdd 5.63399 osd.5 up 1.00000 >>> 1.00000 >>> 6 hdd 5.63399 osd.6 up 1.00000 >>> 1.00000 >>> 7 hdd 5.63399 osd.7 up 1.00000 >>> 1.00000 >>> 40 ssd 5.82190 osd.40 up 1.00000 >>> 1.00000 >>> 48 ssd 5.82190 osd.48 up 1.00000 >>> 1.00000 >>> -5 56.71570 host cnp31tcephosd02 >>> 8 hdd 5.63399 osd.8 up 1.00000 >>> 1.00000 >>> 9 hdd 5.63399 osd.9 down 1.00000 >>> 1.00000 >>> 10 hdd 5.63399 osd.10 up 1.00000 >>> 1.00000 >>> 11 hdd 5.63399 osd.11 up 1.00000 >>> 1.00000 >>> 12 hdd 5.63399 osd.12 up 1.00000 >>> 1.00000 >>> 13 hdd 5.63399 osd.13 up 1.00000 >>> 1.00000 >>> 14 hdd 5.63399 osd.14 up 1.00000 >>> 1.00000 >>> 15 hdd 5.63399 osd.15 up 1.00000 >>> 1.00000 >>> 49 ssd 5.82190 osd.49 up 1.00000 >>> 1.00000 >>> 50 ssd 5.82190 osd.50 up 1.00000 >>> 1.00000 >>> -7 56.71570 host cnp31tcephosd03 >>> 16 hdd 5.63399 osd.16 up 1.00000 >>> 1.00000 >>> 17 hdd 5.63399 osd.17 up 1.00000 >>> 1.00000 >>> 18 hdd 5.63399 osd.18 up 1.00000 >>> 1.00000 >>> 19 hdd 5.63399 osd.19 up 1.00000 >>> 1.00000 >>> 20 hdd 5.63399 osd.20 up 1.00000 >>> 1.00000 >>> 21 hdd 5.63399 osd.21 up 1.00000 >>> 1.00000 >>> 22 hdd 5.63399 osd.22 up 1.00000 >>> 1.00000 >>> 23 hdd 5.63399 osd.23 up 1.00000 >>> 1.00000 >>> 51 ssd 5.82190 osd.51 up 1.00000 >>> 1.00000 >>> 52 ssd 5.82190 osd.52 up 1.00000 >>> 1.00000 >>> -9 56.71570 host cnp31tcephosd04 >>> 24 hdd 5.63399 osd.24 up 1.00000 >>> 1.00000 >>> 25 hdd 5.63399 osd.25 up 1.00000 >>> 1.00000 >>> 26 hdd 5.63399 osd.26 up 1.00000 >>> 1.00000 >>> 27 hdd 5.63399 osd.27 up 1.00000 >>> 1.00000 >>> 28 hdd 5.63399 osd.28 up 1.00000 >>> 1.00000 >>> 29 hdd 5.63399 osd.29 up 1.00000 >>> 1.00000 >>> 30 hdd 5.63399 osd.30 up 1.00000 >>> 1.00000 >>> 31 hdd 5.63399 osd.31 up 1.00000 >>> 1.00000 >>> 53 ssd 5.82190 osd.53 up 1.00000 >>> 1.00000 >>> 54 ssd 5.82190 osd.54 up 1.00000 >>> 1.00000 >>> -11 56.71570 host cnp31tcephosd05 >>> 32 hdd 5.63399 osd.32 up 1.00000 >>> 1.00000 >>> 33 hdd 5.63399 osd.33 up 1.00000 >>> 1.00000 >>> 34 hdd 5.63399 osd.34 up 1.00000 >>> 1.00000 >>> 35 hdd 5.63399 osd.35 up 1.00000 >>> 1.00000 >>> 36 hdd 5.63399 osd.36 up 1.00000 >>> 1.00000 >>> 37 hdd 5.63399 osd.37 up 1.00000 >>> 1.00000 >>> 38 hdd 5.63399 osd.38 up 1.00000 >>> 1.00000 >>> 39 hdd 5.63399 osd.39 up 1.00000 >>> 1.00000 >>> 55 ssd 5.82190 osd.55 up 1.00000 >>> 1.00000 >>> 56 ssd 5.82190 osd.56 up 1.00000 >>> 1.00000 >>> -13 64.03453 host cnp31tcephosd06 >>> 41 hdd 7.48439 osd.41 up 1.00000 >>> 1.00000 >>> 42 hdd 7.48439 osd.42 up 1.00000 >>> 1.00000 >>> 43 hdd 7.48439 osd.43 up 1.00000 >>> 1.00000 >>> 44 hdd 7.48439 osd.44 up 1.00000 >>> 1.00000 >>> 45 hdd 7.48439 osd.45 up 1.00000 >>> 1.00000 >>> 46 hdd 7.48439 osd.46 up 1.00000 >>> 1.00000 >>> 47 hdd 7.48439 osd.47 up 1.00000 >>> 1.00000 >>> 57 ssd 5.82190 osd.57 up 1.00000 >>> 1.00000 >>> 58 ssd 5.82190 osd.58 up 1.00000 >>> 1.00000 >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@xxxxxxx >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>> >> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |