Re: 50% IOPS performance drop after upgrade from Nautilus 14.2.22 to Octopus 15.2.15

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

It's a bit weird that you benchmark 1024 bytes -- or is that your
realistic use-case?
This is smaller than the min alloc unit for even SSDs, so will need a
read/modify/write cycle to update, slowing substantially.

Anyway, since you didn't mention it, have you disabled the write cache
on your drives? See
https://docs.ceph.com/en/latest/start/hardware-recommendations/#write-caches
for the latest related docs.

-- Dan





On Mon, Dec 6, 2021 at 5:28 PM <ceph@xxxxxxxxxx> wrote:
>
> Dear List,
>
> until we upgraded our cluster 3 weeks ago we had a cute high performing small productive CEPH cluster running Nautilus 14.2.22 on Proxmox 6.4 (Kernel 5.4-143 at this time). Then we started the upgrade to Octopus 15.2.15. Since we did an online upgrade, we stopped the autoconvert with
>
> ceph config set osd bluestore_fsck_quick_fix_on_mount false
>
> but followed up the OMAP conversion after the complete upgrade step by step by restarting one OSD after the other.
>
> Our Setup is
> 5 x Storage Node, each : 16 x 2.3GHz, 64GB RAM, 1 x SSD OSD 1.6TB, 1 x 7.68TB (both WD Enterprise, SAS-12), 3 HDD OSD (10TB, SAS-12) with Optane Cache)
> 4 x Compute Nodes
> 40 GE Storage network (Mellanox Switch + Mellanox CX354 40GE Dual Port Cards, Linux OSS drivers)
> 10 GE Cluster/Mgmt Network
>
> Our performance before the upgrade, Ceph 14.2.22 (about 36k IOPS on the SSD Pool)
>
> ### SSD Pool on 40GE Switches
> # rados bench -p SSD 30 -t 256 -b 1024 write
> hints = 1
> Maintaining 256 concurrent writes of 1024 bytes to objects of size 1024 for up to 30 seconds or 0 objects
> ...
> Total time run:         30.004
> Total writes made:      1094320
> Write size:             1024
> Object size:            1024
> Bandwidth (MB/sec):     35.6177
> Stddev Bandwidth:       4.71909
> Max bandwidth (MB/sec): 40.7314
> Min bandwidth (MB/sec): 21.3037
> Average IOPS:           36472
> Stddev IOPS:            4832.35
> Max IOPS:               41709
> Min IOPS:               21815
> Average Latency(s):     0.00701759
> Stddev Latency(s):      0.00854068
> Max latency(s):         0.445397
> Min latency(s):         0.000909089
> Cleaning up (deleting benchmark objects)
>
> Our performance after the update CEPH 15.2.15 (drops to max 17k IOPS on the SSD Pool)
> # rados bench -p SSD 30 -t 256 -b 1024 write
> hints = 1
> Maintaining 256 concurrent writes of 1024 bytes to objects of size 1024 for up to 30 seconds or 0 objects
> ...
> Total time run:         30.0146
> Total writes made:      468513
> Write size:             1024
> Object size:            1024
> Bandwidth (MB/sec):     15.2437
> Stddev Bandwidth:       0.78677
> Max bandwidth (MB/sec): 16.835
> Min bandwidth (MB/sec): 13.3184
> Average IOPS:           15609
> Stddev IOPS:            805.652
> Max IOPS:               17239
> Min IOPS:               13638
> Average Latency(s):     0.016396
> Stddev Latency(s):      0.00777054
> Max latency(s):         0.140793
> Min latency(s):         0.00106735
> Cleaning up (deleting benchmark objects)
> Note : OSD.17 is out on purpose
> # ceph osd tree
> ID   CLASS  WEIGHT     TYPE NAME            STATUS  REWEIGHT  PRI-AFF
>  -1         208.94525  root default
>  -3          41.43977      host xx-ceph01
>   0    hdd    9.17380          osd.0            up   1.00000  1.00000
>   5    hdd    9.17380          osd.5            up   1.00000  1.00000
>  23    hdd   14.65039          osd.23           up   1.00000  1.00000
>   7    ssd    1.45549          osd.7            up   1.00000  1.00000
>  15    ssd    6.98630          osd.15           up   1.00000  1.00000
>  -5          41.43977      host xx-ceph02
>   1    hdd    9.17380          osd.1            up   1.00000  1.00000
>   4    hdd    9.17380          osd.4            up   1.00000  1.00000
>  24    hdd   14.65039          osd.24           up   1.00000  1.00000
>   9    ssd    1.45549          osd.9            up   1.00000  1.00000
>  20    ssd    6.98630          osd.20           up   1.00000  1.00000
>  -7          41.43977      host xx-ceph03
>   2    hdd    9.17380          osd.2            up   1.00000  1.00000
>   3    hdd    9.17380          osd.3            up   1.00000  1.00000
>  25    hdd   14.65039          osd.25           up   1.00000  1.00000
>   8    ssd    1.45549          osd.8            up   1.00000  1.00000
>  21    ssd    6.98630          osd.21           up   1.00000  1.00000
> -17          41.43977      host xx-ceph04
>  10    hdd    9.17380          osd.10           up   1.00000  1.00000
>  11    hdd    9.17380          osd.11           up   1.00000  1.00000
>  26    hdd   14.65039          osd.26           up   1.00000  1.00000
>   6    ssd    1.45549          osd.6            up   1.00000  1.00000
>  22    ssd    6.98630          osd.22           up   1.00000  1.00000
> -21          43.18616      host xx-ceph05
>  13    hdd    9.17380          osd.13           up   1.00000  1.00000
>  14    hdd    9.17380          osd.14           up   1.00000  1.00000
>  27    hdd   14.65039          osd.27           up   1.00000  1.00000
>  12    ssd    1.45540          osd.12           up   1.00000  1.00000
>  16    ssd    1.74660          osd.16           up   1.00000  1.00000
>  17    ssd    3.49309          osd.17           up         0  1.00000
>  18    ssd    1.74660          osd.18           up   1.00000  1.00000
>  19    ssd    1.74649          osd.19           up   1.00000  1.00000
>
> # ceph osd df
> ID  CLASS  WEIGHT    REWEIGHT  SIZE     RAW USE  DATA     OMAP     META     AVAIL    %USE   VAR   PGS  STATUS
>  0    hdd   9.17380   1.00000  9.2 TiB  2.5 TiB  2.4 TiB   28 MiB  5.0 GiB  6.6 TiB  27.56  0.96   88      up
>  5    hdd   9.17380   1.00000  9.2 TiB  2.6 TiB  2.5 TiB   57 MiB  5.1 GiB  6.6 TiB  27.89  0.98   89      up
> 23    hdd  14.65039   1.00000   15 TiB  3.9 TiB  3.8 TiB   40 MiB  7.2 GiB   11 TiB  26.69  0.93  137      up
>  7    ssd   1.45549   1.00000  1.5 TiB  634 GiB  633 GiB   33 MiB  1.8 GiB  856 GiB  42.57  1.49   64      up
> 15    ssd   6.98630   1.00000  7.0 TiB  2.6 TiB  2.6 TiB  118 MiB  5.9 GiB  4.4 TiB  37.70  1.32  272      up
>  1    hdd   9.17380   1.00000  9.2 TiB  2.4 TiB  2.3 TiB   31 MiB  4.7 GiB  6.8 TiB  26.04  0.91   83      up
>  4    hdd   9.17380   1.00000  9.2 TiB  2.6 TiB  2.5 TiB   28 MiB  5.2 GiB  6.6 TiB  28.51  1.00   91      up
> 24    hdd  14.65039   1.00000   15 TiB  4.0 TiB  3.9 TiB   38 MiB  7.2 GiB   11 TiB  27.06  0.95  139      up
>  9    ssd   1.45549   1.00000  1.5 TiB  583 GiB  582 GiB   30 MiB  1.6 GiB  907 GiB  39.13  1.37   59      up
> 20    ssd   6.98630   1.00000  7.0 TiB  2.5 TiB  2.5 TiB   81 MiB  7.4 GiB  4.5 TiB  35.45  1.24  260      up
>  2    hdd   9.17380   1.00000  9.2 TiB  2.4 TiB  2.3 TiB   26 MiB  4.8 GiB  6.8 TiB  26.01  0.91   83      up
>  3    hdd   9.17380   1.00000  9.2 TiB  2.7 TiB  2.6 TiB   29 MiB  5.4 GiB  6.5 TiB  29.38  1.03   94      up
> 25    hdd  14.65039   1.00000   15 TiB  4.2 TiB  4.1 TiB   41 MiB  7.7 GiB   10 TiB  28.79  1.01  149      up
>  8    ssd   1.45549   1.00000  1.5 TiB  637 GiB  635 GiB   34 MiB  1.7 GiB  854 GiB  42.71  1.49   65      up
> 21    ssd   6.98630   1.00000  7.0 TiB  2.5 TiB  2.5 TiB   96 MiB  7.5 GiB  4.5 TiB  35.49  1.24  260      up
> 10    hdd   9.17380   1.00000  9.2 TiB  2.2 TiB  2.1 TiB   26 MiB  4.5 GiB  7.0 TiB  24.21  0.85   77      up
> 11    hdd   9.17380   1.00000  9.2 TiB  2.5 TiB  2.4 TiB   30 MiB  5.0 GiB  6.7 TiB  27.24  0.95   87      up
> 26    hdd  14.65039   1.00000   15 TiB  3.6 TiB  3.5 TiB   37 MiB  6.6 GiB   11 TiB  24.64  0.86  127      up
>  6    ssd   1.45549   1.00000  1.5 TiB  572 GiB  570 GiB   29 MiB  1.5 GiB  918 GiB  38.38  1.34   57      up
> 22    ssd   6.98630   1.00000  7.0 TiB  2.3 TiB  2.3 TiB   77 MiB  7.0 GiB  4.7 TiB  33.23  1.16  243      up
> 13    hdd   9.17380   1.00000  9.2 TiB  2.4 TiB  2.3 TiB   25 MiB  4.8 GiB  6.8 TiB  26.07  0.91   84      up
> 14    hdd   9.17380   1.00000  9.2 TiB  2.3 TiB  2.2 TiB   54 MiB  4.6 GiB  6.9 TiB  25.13  0.88   80      up
> 27    hdd  14.65039   1.00000   15 TiB  3.7 TiB  3.6 TiB   54 MiB  6.9 GiB   11 TiB  25.55  0.89  131      up
> 12    ssd   1.45540   1.00000  1.5 TiB  619 GiB  617 GiB  163 MiB  2.3 GiB  871 GiB  41.53  1.45   63      up
> 16    ssd   1.74660   1.00000  1.7 TiB  671 GiB  669 GiB   23 MiB  2.2 GiB  1.1 TiB  37.51  1.31   69      up
> 17    ssd   3.49309         0      0 B      0 B      0 B      0 B      0 B      0 B      0     0    0      up
> 18    ssd   1.74660   1.00000  1.7 TiB  512 GiB  509 GiB   18 MiB  2.3 GiB  1.2 TiB  28.62  1.00   52      up
> 19    ssd   1.74649   1.00000  1.7 TiB  709 GiB  707 GiB   64 MiB  2.0 GiB  1.1 TiB  39.64  1.39   72      up
>                         TOTAL  205 TiB   59 TiB   57 TiB  1.3 GiB  128 GiB  147 TiB  28.60
> MIN/MAX VAR: 0.85/1.49  STDDEV: 6.81
>
>
> What we have done so far (no success)
>
> - reformat two of the SSD OSD's (one was still from luminos, non LVM)
> - set bluestore_allocator from hybrid back to bitmap
> - set osd_memory_target to 6442450944 for some of the SSD OSDs
> - cpupower idle-set -D 11
> - bluefs_buffered_io to true
> - disabled default firewalls between CEPH nodes (for testing only)
> - disabled apparmor
> - added memory (runs now on 128GB per Node)
> - upgraded OS, runs now on kernel 5.13.19-1
>
> What we observe
> - HDD Pool has similar behaviour
> - load is higher since update, seems like more CPU consumption (see graph1), migration was on 10. Nov, around 10pm
> - latency on the "big" 7TB SSD's (i.e. OSD.15) is significantly higher than on the small 1.6TB SSDs (OSD.12), see graph2, must be due to the higher weight though
> - load of OSD.15 is 4 times higher than load of OSD.12, must be due to the higher weight though as well
> - start of OSD.15 (the 7TB SSD's is significantly slower (~10 sec) compared to the 1.6TB SSDs
> - increasing the block size in the benchmark to 4k, 8k or even 16k increases the throughput but keeps the IOPS more or less stable, the drop at 32k is minimal to ~14k IOPS in average
>
> We already checked the ProxMoxx List without any remedies yet and we are a bit helpless, any suggestions and / or does someone else has similar experiences?
>
> We are a bit hesitant to upgrade to Pacific, given the current situation.
>
> Thanks,
>
> Kai
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux