Hi, It's a bit weird that you benchmark 1024 bytes -- or is that your realistic use-case? This is smaller than the min alloc unit for even SSDs, so will need a read/modify/write cycle to update, slowing substantially. Anyway, since you didn't mention it, have you disabled the write cache on your drives? See https://docs.ceph.com/en/latest/start/hardware-recommendations/#write-caches for the latest related docs. -- Dan On Mon, Dec 6, 2021 at 5:28 PM <ceph@xxxxxxxxxx> wrote: > > Dear List, > > until we upgraded our cluster 3 weeks ago we had a cute high performing small productive CEPH cluster running Nautilus 14.2.22 on Proxmox 6.4 (Kernel 5.4-143 at this time). Then we started the upgrade to Octopus 15.2.15. Since we did an online upgrade, we stopped the autoconvert with > > ceph config set osd bluestore_fsck_quick_fix_on_mount false > > but followed up the OMAP conversion after the complete upgrade step by step by restarting one OSD after the other. > > Our Setup is > 5 x Storage Node, each : 16 x 2.3GHz, 64GB RAM, 1 x SSD OSD 1.6TB, 1 x 7.68TB (both WD Enterprise, SAS-12), 3 HDD OSD (10TB, SAS-12) with Optane Cache) > 4 x Compute Nodes > 40 GE Storage network (Mellanox Switch + Mellanox CX354 40GE Dual Port Cards, Linux OSS drivers) > 10 GE Cluster/Mgmt Network > > Our performance before the upgrade, Ceph 14.2.22 (about 36k IOPS on the SSD Pool) > > ### SSD Pool on 40GE Switches > # rados bench -p SSD 30 -t 256 -b 1024 write > hints = 1 > Maintaining 256 concurrent writes of 1024 bytes to objects of size 1024 for up to 30 seconds or 0 objects > ... > Total time run: 30.004 > Total writes made: 1094320 > Write size: 1024 > Object size: 1024 > Bandwidth (MB/sec): 35.6177 > Stddev Bandwidth: 4.71909 > Max bandwidth (MB/sec): 40.7314 > Min bandwidth (MB/sec): 21.3037 > Average IOPS: 36472 > Stddev IOPS: 4832.35 > Max IOPS: 41709 > Min IOPS: 21815 > Average Latency(s): 0.00701759 > Stddev Latency(s): 0.00854068 > Max latency(s): 0.445397 > Min latency(s): 0.000909089 > Cleaning up (deleting benchmark objects) > > Our performance after the update CEPH 15.2.15 (drops to max 17k IOPS on the SSD Pool) > # rados bench -p SSD 30 -t 256 -b 1024 write > hints = 1 > Maintaining 256 concurrent writes of 1024 bytes to objects of size 1024 for up to 30 seconds or 0 objects > ... > Total time run: 30.0146 > Total writes made: 468513 > Write size: 1024 > Object size: 1024 > Bandwidth (MB/sec): 15.2437 > Stddev Bandwidth: 0.78677 > Max bandwidth (MB/sec): 16.835 > Min bandwidth (MB/sec): 13.3184 > Average IOPS: 15609 > Stddev IOPS: 805.652 > Max IOPS: 17239 > Min IOPS: 13638 > Average Latency(s): 0.016396 > Stddev Latency(s): 0.00777054 > Max latency(s): 0.140793 > Min latency(s): 0.00106735 > Cleaning up (deleting benchmark objects) > Note : OSD.17 is out on purpose > # ceph osd tree > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > -1 208.94525 root default > -3 41.43977 host xx-ceph01 > 0 hdd 9.17380 osd.0 up 1.00000 1.00000 > 5 hdd 9.17380 osd.5 up 1.00000 1.00000 > 23 hdd 14.65039 osd.23 up 1.00000 1.00000 > 7 ssd 1.45549 osd.7 up 1.00000 1.00000 > 15 ssd 6.98630 osd.15 up 1.00000 1.00000 > -5 41.43977 host xx-ceph02 > 1 hdd 9.17380 osd.1 up 1.00000 1.00000 > 4 hdd 9.17380 osd.4 up 1.00000 1.00000 > 24 hdd 14.65039 osd.24 up 1.00000 1.00000 > 9 ssd 1.45549 osd.9 up 1.00000 1.00000 > 20 ssd 6.98630 osd.20 up 1.00000 1.00000 > -7 41.43977 host xx-ceph03 > 2 hdd 9.17380 osd.2 up 1.00000 1.00000 > 3 hdd 9.17380 osd.3 up 1.00000 1.00000 > 25 hdd 14.65039 osd.25 up 1.00000 1.00000 > 8 ssd 1.45549 osd.8 up 1.00000 1.00000 > 21 ssd 6.98630 osd.21 up 1.00000 1.00000 > -17 41.43977 host xx-ceph04 > 10 hdd 9.17380 osd.10 up 1.00000 1.00000 > 11 hdd 9.17380 osd.11 up 1.00000 1.00000 > 26 hdd 14.65039 osd.26 up 1.00000 1.00000 > 6 ssd 1.45549 osd.6 up 1.00000 1.00000 > 22 ssd 6.98630 osd.22 up 1.00000 1.00000 > -21 43.18616 host xx-ceph05 > 13 hdd 9.17380 osd.13 up 1.00000 1.00000 > 14 hdd 9.17380 osd.14 up 1.00000 1.00000 > 27 hdd 14.65039 osd.27 up 1.00000 1.00000 > 12 ssd 1.45540 osd.12 up 1.00000 1.00000 > 16 ssd 1.74660 osd.16 up 1.00000 1.00000 > 17 ssd 3.49309 osd.17 up 0 1.00000 > 18 ssd 1.74660 osd.18 up 1.00000 1.00000 > 19 ssd 1.74649 osd.19 up 1.00000 1.00000 > > # ceph osd df > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS > 0 hdd 9.17380 1.00000 9.2 TiB 2.5 TiB 2.4 TiB 28 MiB 5.0 GiB 6.6 TiB 27.56 0.96 88 up > 5 hdd 9.17380 1.00000 9.2 TiB 2.6 TiB 2.5 TiB 57 MiB 5.1 GiB 6.6 TiB 27.89 0.98 89 up > 23 hdd 14.65039 1.00000 15 TiB 3.9 TiB 3.8 TiB 40 MiB 7.2 GiB 11 TiB 26.69 0.93 137 up > 7 ssd 1.45549 1.00000 1.5 TiB 634 GiB 633 GiB 33 MiB 1.8 GiB 856 GiB 42.57 1.49 64 up > 15 ssd 6.98630 1.00000 7.0 TiB 2.6 TiB 2.6 TiB 118 MiB 5.9 GiB 4.4 TiB 37.70 1.32 272 up > 1 hdd 9.17380 1.00000 9.2 TiB 2.4 TiB 2.3 TiB 31 MiB 4.7 GiB 6.8 TiB 26.04 0.91 83 up > 4 hdd 9.17380 1.00000 9.2 TiB 2.6 TiB 2.5 TiB 28 MiB 5.2 GiB 6.6 TiB 28.51 1.00 91 up > 24 hdd 14.65039 1.00000 15 TiB 4.0 TiB 3.9 TiB 38 MiB 7.2 GiB 11 TiB 27.06 0.95 139 up > 9 ssd 1.45549 1.00000 1.5 TiB 583 GiB 582 GiB 30 MiB 1.6 GiB 907 GiB 39.13 1.37 59 up > 20 ssd 6.98630 1.00000 7.0 TiB 2.5 TiB 2.5 TiB 81 MiB 7.4 GiB 4.5 TiB 35.45 1.24 260 up > 2 hdd 9.17380 1.00000 9.2 TiB 2.4 TiB 2.3 TiB 26 MiB 4.8 GiB 6.8 TiB 26.01 0.91 83 up > 3 hdd 9.17380 1.00000 9.2 TiB 2.7 TiB 2.6 TiB 29 MiB 5.4 GiB 6.5 TiB 29.38 1.03 94 up > 25 hdd 14.65039 1.00000 15 TiB 4.2 TiB 4.1 TiB 41 MiB 7.7 GiB 10 TiB 28.79 1.01 149 up > 8 ssd 1.45549 1.00000 1.5 TiB 637 GiB 635 GiB 34 MiB 1.7 GiB 854 GiB 42.71 1.49 65 up > 21 ssd 6.98630 1.00000 7.0 TiB 2.5 TiB 2.5 TiB 96 MiB 7.5 GiB 4.5 TiB 35.49 1.24 260 up > 10 hdd 9.17380 1.00000 9.2 TiB 2.2 TiB 2.1 TiB 26 MiB 4.5 GiB 7.0 TiB 24.21 0.85 77 up > 11 hdd 9.17380 1.00000 9.2 TiB 2.5 TiB 2.4 TiB 30 MiB 5.0 GiB 6.7 TiB 27.24 0.95 87 up > 26 hdd 14.65039 1.00000 15 TiB 3.6 TiB 3.5 TiB 37 MiB 6.6 GiB 11 TiB 24.64 0.86 127 up > 6 ssd 1.45549 1.00000 1.5 TiB 572 GiB 570 GiB 29 MiB 1.5 GiB 918 GiB 38.38 1.34 57 up > 22 ssd 6.98630 1.00000 7.0 TiB 2.3 TiB 2.3 TiB 77 MiB 7.0 GiB 4.7 TiB 33.23 1.16 243 up > 13 hdd 9.17380 1.00000 9.2 TiB 2.4 TiB 2.3 TiB 25 MiB 4.8 GiB 6.8 TiB 26.07 0.91 84 up > 14 hdd 9.17380 1.00000 9.2 TiB 2.3 TiB 2.2 TiB 54 MiB 4.6 GiB 6.9 TiB 25.13 0.88 80 up > 27 hdd 14.65039 1.00000 15 TiB 3.7 TiB 3.6 TiB 54 MiB 6.9 GiB 11 TiB 25.55 0.89 131 up > 12 ssd 1.45540 1.00000 1.5 TiB 619 GiB 617 GiB 163 MiB 2.3 GiB 871 GiB 41.53 1.45 63 up > 16 ssd 1.74660 1.00000 1.7 TiB 671 GiB 669 GiB 23 MiB 2.2 GiB 1.1 TiB 37.51 1.31 69 up > 17 ssd 3.49309 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 up > 18 ssd 1.74660 1.00000 1.7 TiB 512 GiB 509 GiB 18 MiB 2.3 GiB 1.2 TiB 28.62 1.00 52 up > 19 ssd 1.74649 1.00000 1.7 TiB 709 GiB 707 GiB 64 MiB 2.0 GiB 1.1 TiB 39.64 1.39 72 up > TOTAL 205 TiB 59 TiB 57 TiB 1.3 GiB 128 GiB 147 TiB 28.60 > MIN/MAX VAR: 0.85/1.49 STDDEV: 6.81 > > > What we have done so far (no success) > > - reformat two of the SSD OSD's (one was still from luminos, non LVM) > - set bluestore_allocator from hybrid back to bitmap > - set osd_memory_target to 6442450944 for some of the SSD OSDs > - cpupower idle-set -D 11 > - bluefs_buffered_io to true > - disabled default firewalls between CEPH nodes (for testing only) > - disabled apparmor > - added memory (runs now on 128GB per Node) > - upgraded OS, runs now on kernel 5.13.19-1 > > What we observe > - HDD Pool has similar behaviour > - load is higher since update, seems like more CPU consumption (see graph1), migration was on 10. Nov, around 10pm > - latency on the "big" 7TB SSD's (i.e. OSD.15) is significantly higher than on the small 1.6TB SSDs (OSD.12), see graph2, must be due to the higher weight though > - load of OSD.15 is 4 times higher than load of OSD.12, must be due to the higher weight though as well > - start of OSD.15 (the 7TB SSD's is significantly slower (~10 sec) compared to the 1.6TB SSDs > - increasing the block size in the benchmark to 4k, 8k or even 16k increases the throughput but keeps the IOPS more or less stable, the drop at 32k is minimal to ~14k IOPS in average > > We already checked the ProxMoxx List without any remedies yet and we are a bit helpless, any suggestions and / or does someone else has similar experiences? > > We are a bit hesitant to upgrade to Pacific, given the current situation. > > Thanks, > > Kai > > > > > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |