Hello, what exact EC level do you use? I can confirm, that our internal data shows a performance drop when using pacific. So far Octopus is faster and better than pacific but I doubt you can roll back to it. We haven't rerun our benchmarks on Quincy yet, but according to some presentation it should be faster than pacific. Maybe try to jump away from the pacific release into the unknown! -- Martin Verges Managing director Mobile: +49 174 9335695 | Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx On Sat, 14 May 2022 at 12:27, stéphane chalansonnet <schalans@xxxxxxxxx> wrote: > Hello, > > After a successful update from Nautilus to Pacific on Centos8.5, we > observed some high latencies on our cluster. > > We did not find very much thing on community related to latencies post > migration > > Our setup is > 6x storage Node (256GRAM, 2SSD OSD + 5*6To SATA HDD) > Erasure coding profile > We have two EC pool : > -> Pool1 : Full HDD SAS Drive 6To > -> Pool2 : Full SSD Drive > > Object S3 and RBD block workload > > Our performances in nautilus, before the upgrade , are acceptable. > However , the next day , performance dropped by 3 or 4 > Benchmark showed 15KIOPS on flash drive , before upgrade we had > almost 80KIOPS > Also, HDD pool is almost down (too much lantencies > > We suspected , maybe, an impact on erasure Coding configuration on Pacific > Anyone observed the same behaviour ? any tuning ? > > Thank you for your help. > > ceph osd tree > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > -1 347.61304 root default > -3 56.71570 host cnp31tcephosd01 > 0 hdd 5.63399 osd.0 up 1.00000 1.00000 > 1 hdd 5.63399 osd.1 up 1.00000 1.00000 > 2 hdd 5.63399 osd.2 up 1.00000 1.00000 > 3 hdd 5.63399 osd.3 up 1.00000 1.00000 > 4 hdd 5.63399 osd.4 up 1.00000 1.00000 > 5 hdd 5.63399 osd.5 up 1.00000 1.00000 > 6 hdd 5.63399 osd.6 up 1.00000 1.00000 > 7 hdd 5.63399 osd.7 up 1.00000 1.00000 > 40 ssd 5.82190 osd.40 up 1.00000 1.00000 > 48 ssd 5.82190 osd.48 up 1.00000 1.00000 > -5 56.71570 host cnp31tcephosd02 > 8 hdd 5.63399 osd.8 up 1.00000 1.00000 > 9 hdd 5.63399 osd.9 down 1.00000 1.00000 > 10 hdd 5.63399 osd.10 up 1.00000 1.00000 > 11 hdd 5.63399 osd.11 up 1.00000 1.00000 > 12 hdd 5.63399 osd.12 up 1.00000 1.00000 > 13 hdd 5.63399 osd.13 up 1.00000 1.00000 > 14 hdd 5.63399 osd.14 up 1.00000 1.00000 > 15 hdd 5.63399 osd.15 up 1.00000 1.00000 > 49 ssd 5.82190 osd.49 up 1.00000 1.00000 > 50 ssd 5.82190 osd.50 up 1.00000 1.00000 > -7 56.71570 host cnp31tcephosd03 > 16 hdd 5.63399 osd.16 up 1.00000 1.00000 > 17 hdd 5.63399 osd.17 up 1.00000 1.00000 > 18 hdd 5.63399 osd.18 up 1.00000 1.00000 > 19 hdd 5.63399 osd.19 up 1.00000 1.00000 > 20 hdd 5.63399 osd.20 up 1.00000 1.00000 > 21 hdd 5.63399 osd.21 up 1.00000 1.00000 > 22 hdd 5.63399 osd.22 up 1.00000 1.00000 > 23 hdd 5.63399 osd.23 up 1.00000 1.00000 > 51 ssd 5.82190 osd.51 up 1.00000 1.00000 > 52 ssd 5.82190 osd.52 up 1.00000 1.00000 > -9 56.71570 host cnp31tcephosd04 > 24 hdd 5.63399 osd.24 up 1.00000 1.00000 > 25 hdd 5.63399 osd.25 up 1.00000 1.00000 > 26 hdd 5.63399 osd.26 up 1.00000 1.00000 > 27 hdd 5.63399 osd.27 up 1.00000 1.00000 > 28 hdd 5.63399 osd.28 up 1.00000 1.00000 > 29 hdd 5.63399 osd.29 up 1.00000 1.00000 > 30 hdd 5.63399 osd.30 up 1.00000 1.00000 > 31 hdd 5.63399 osd.31 up 1.00000 1.00000 > 53 ssd 5.82190 osd.53 up 1.00000 1.00000 > 54 ssd 5.82190 osd.54 up 1.00000 1.00000 > -11 56.71570 host cnp31tcephosd05 > 32 hdd 5.63399 osd.32 up 1.00000 1.00000 > 33 hdd 5.63399 osd.33 up 1.00000 1.00000 > 34 hdd 5.63399 osd.34 up 1.00000 1.00000 > 35 hdd 5.63399 osd.35 up 1.00000 1.00000 > 36 hdd 5.63399 osd.36 up 1.00000 1.00000 > 37 hdd 5.63399 osd.37 up 1.00000 1.00000 > 38 hdd 5.63399 osd.38 up 1.00000 1.00000 > 39 hdd 5.63399 osd.39 up 1.00000 1.00000 > 55 ssd 5.82190 osd.55 up 1.00000 1.00000 > 56 ssd 5.82190 osd.56 up 1.00000 1.00000 > -13 64.03453 host cnp31tcephosd06 > 41 hdd 7.48439 osd.41 up 1.00000 1.00000 > 42 hdd 7.48439 osd.42 up 1.00000 1.00000 > 43 hdd 7.48439 osd.43 up 1.00000 1.00000 > 44 hdd 7.48439 osd.44 up 1.00000 1.00000 > 45 hdd 7.48439 osd.45 up 1.00000 1.00000 > 46 hdd 7.48439 osd.46 up 1.00000 1.00000 > 47 hdd 7.48439 osd.47 up 1.00000 1.00000 > 57 ssd 5.82190 osd.57 up 1.00000 1.00000 > 58 ssd 5.82190 osd.58 up 1.00000 1.00000 > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |