Re: CEPH 16.2.x: disappointing I/O performance

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi!

I can post the crush map tomorrow morning, but it definitely isn't
targeting the NVME drives.

I'm having a performance issue specifically with the HDD-backed pool, where
each OSD is an NVME-backed WAL/DB + HDD-backed storage.

/Z

On Tue, 5 Oct 2021, 22:43 Tor Martin Ølberg, <tmolberg@xxxxxxxxx> wrote:

> Hi Zakhar,
>
> Out of curiosity, what does your crushmap look like? Probably a long shot
> but are you sure your crush map is targeting the NVME's for the rados bench
> you are performing?
>
> Tor Martin Ølberg
>
> On Tue, Oct 5, 2021 at 9:31 PM Christian Wuerdig <
> christian.wuerdig@xxxxxxxxx> wrote:
>
>> Maybe some info is missing but 7k write IOPs at 4k block size seem fairly
>> decent (as you also state) - the bandwidth automatically follows from that
>> so not sure what you're expecting?
>> I am a bit puzzled though - by my math 7k IOPS at 4k should only be
>> 27MiB/sec - not sure how the 120MiB/sec was achieved
>> The read benchmark seems in line with 13k IOPS at 4k making around
>> 52MiB/sec bandwidth which again is expected.
>>
>>
>> On Wed, 6 Oct 2021 at 04:08, Zakhar Kirpichenko <zakhar@xxxxxxxxx> wrote:
>>
>> > Hi,
>> >
>> > I built a CEPH 16.2.x cluster with relatively fast and modern hardware,
>> and
>> > its performance is kind of disappointing. I would very much appreciate
>> an
>> > advice and/or pointers :-)
>> >
>> > The hardware is 3 x Supermicro SSG-6029P nodes, each equipped with:
>> >
>> > 2 x Intel(R) Xeon(R) Gold 5220R CPUs
>> > 384 GB RAM
>> > 2 x boot drives
>> > 2 x 1.6 TB Micron 7300 MTFDHBE1T6TDG drives (DB/WAL)
>> > 2 x 6.4 TB Micron 7300 MTFDHBE6T4TDG drives (storage tier)
>> > 9 x Toshiba MG06SCA10TE 9TB HDDs, write cache off (storage tier)
>> > 2 x Intel XL710 NICs connected to a pair of 40/100GE switches
>> >
>> > All 3 nodes are running Ubuntu 20.04 LTS with the latest 5.4 kernel,
>> > apparmor is disabled, energy-saving features are disabled. The network
>> > between the CEPH nodes is 40G, CEPH access network is 40G, the average
>> > latencies are < 0.15 ms. I've personally tested the network for
>> throughput,
>> > latency and loss, and can tell that it's operating as expected and
>> doesn't
>> > exhibit any issues at idle or under load.
>> >
>> > The CEPH cluster is set up with 2 storage classes, NVME and HDD, with 2
>> > smaller NVME drives in each node used as DB/WAL and each HDD allocated .
>> > ceph osd tree output:
>> >
>> > ID   CLASS  WEIGHT     TYPE NAME                STATUS  REWEIGHT
>> PRI-AFF
>> >  -1         288.37488  root default
>> > -13         288.37488      datacenter ste
>> > -14         288.37488          rack rack01
>> >  -7          96.12495              host ceph01
>> >   0    hdd    9.38680                  osd.0        up   1.00000
>> 1.00000
>> >   1    hdd    9.38680                  osd.1        up   1.00000
>> 1.00000
>> >   2    hdd    9.38680                  osd.2        up   1.00000
>> 1.00000
>> >   3    hdd    9.38680                  osd.3        up   1.00000
>> 1.00000
>> >   4    hdd    9.38680                  osd.4        up   1.00000
>> 1.00000
>> >   5    hdd    9.38680                  osd.5        up   1.00000
>> 1.00000
>> >   6    hdd    9.38680                  osd.6        up   1.00000
>> 1.00000
>> >   7    hdd    9.38680                  osd.7        up   1.00000
>> 1.00000
>> >   8    hdd    9.38680                  osd.8        up   1.00000
>> 1.00000
>> >   9   nvme    5.82190                  osd.9        up   1.00000
>> 1.00000
>> >  10   nvme    5.82190                  osd.10       up   1.00000
>> 1.00000
>> > -10          96.12495              host ceph02
>> >  11    hdd    9.38680                  osd.11       up   1.00000
>> 1.00000
>> >  12    hdd    9.38680                  osd.12       up   1.00000
>> 1.00000
>> >  13    hdd    9.38680                  osd.13       up   1.00000
>> 1.00000
>> >  14    hdd    9.38680                  osd.14       up   1.00000
>> 1.00000
>> >  15    hdd    9.38680                  osd.15       up   1.00000
>> 1.00000
>> >  16    hdd    9.38680                  osd.16       up   1.00000
>> 1.00000
>> >  17    hdd    9.38680                  osd.17       up   1.00000
>> 1.00000
>> >  18    hdd    9.38680                  osd.18       up   1.00000
>> 1.00000
>> >  19    hdd    9.38680                  osd.19       up   1.00000
>> 1.00000
>> >  20   nvme    5.82190                  osd.20       up   1.00000
>> 1.00000
>> >  21   nvme    5.82190                  osd.21       up   1.00000
>> 1.00000
>> >  -3          96.12495              host ceph03
>> >  22    hdd    9.38680                  osd.22       up   1.00000
>> 1.00000
>> >  23    hdd    9.38680                  osd.23       up   1.00000
>> 1.00000
>> >  24    hdd    9.38680                  osd.24       up   1.00000
>> 1.00000
>> >  25    hdd    9.38680                  osd.25       up   1.00000
>> 1.00000
>> >  26    hdd    9.38680                  osd.26       up   1.00000
>> 1.00000
>> >  27    hdd    9.38680                  osd.27       up   1.00000
>> 1.00000
>> >  28    hdd    9.38680                  osd.28       up   1.00000
>> 1.00000
>> >  29    hdd    9.38680                  osd.29       up   1.00000
>> 1.00000
>> >  30    hdd    9.38680                  osd.30       up   1.00000
>> 1.00000
>> >  31   nvme    5.82190                  osd.31       up   1.00000
>> 1.00000
>> >  32   nvme    5.82190                  osd.32       up   1.00000
>> 1.00000
>> >
>> > ceph df:
>> >
>> > --- RAW STORAGE ---
>> > CLASS     SIZE    AVAIL    USED  RAW USED  %RAW USED
>> > hdd    253 TiB  241 TiB  13 TiB    13 TiB       5.00
>> > nvme    35 TiB   35 TiB  82 GiB    82 GiB       0.23
>> > TOTAL  288 TiB  276 TiB  13 TiB    13 TiB       4.42
>> >
>> > --- POOLS ---
>> > POOL                   ID  PGS   STORED  OBJECTS     USED  %USED  MAX
>> AVAIL
>> > images                 12  256   24 GiB    3.15k   73 GiB   0.03     76
>> TiB
>> > volumes                13  256  839 GiB  232.16k  2.5 TiB   1.07     76
>> TiB
>> > backups                14  256   31 GiB    8.56k   94 GiB   0.04     76
>> TiB
>> > vms                    15  256  752 GiB  198.80k  2.2 TiB   0.96     76
>> TiB
>> > device_health_metrics  16   32   35 MiB       39  106 MiB      0     76
>> TiB
>> > volumes-nvme           17  256   28 GiB    7.21k   81 GiB   0.24     11
>> TiB
>> > ec-volumes-meta        18  256   27 KiB        4   92 KiB      0     76
>> TiB
>> > ec-volumes-data        19  256    8 KiB        1   12 KiB      0    152
>> TiB
>> >
>> > Please disregard the ec-pools, as they're not currently in use. All
>> other
>> > pools are configured with min_size=2, size=3. All pools are bound to HDD
>> > storage except for 'volumes-nvme', which is bound to NVME. The number of
>> > PGs was increased recently, as with autoscaler I was getting a very
>> uneven
>> > PG distribution on devices and we're expecting to add 3 more nodes of
>> > exactly the same configuration in the coming weeks. I have to emphasize
>> > that I tested different PG numbers and they didn't have a noticeable
>> impact
>> > on the cluster performance.
>> >
>> > The main issue is that this beautiful cluster isn't very fast. When I
>> test
>> > against the 'volumes' pool, residing on HDD storage class (HDDs with
>> DB/WAL
>> > on NVME), I get unexpectedly low throughput numbers:
>> >
>> > > rados -p volumes bench 30 write --no-cleanup
>> > ...
>> > Total time run:         30.3078
>> > Total writes made:      3731
>> > Write size:             4194304
>> > Object size:            4194304
>> > Bandwidth (MB/sec):     492.415
>> > Stddev Bandwidth:       161.777
>> > Max bandwidth (MB/sec): 820
>> > Min bandwidth (MB/sec): 204
>> > Average IOPS:           123
>> > Stddev IOPS:            40.4442
>> > Max IOPS:               205
>> > Min IOPS:               51
>> > Average Latency(s):     0.129115
>> > Stddev Latency(s):      0.143881
>> > Max latency(s):         1.35669
>> > Min latency(s):         0.0228179
>> >
>> > > rados -p volumes bench 30 seq --no-cleanup
>> > ...
>> > Total time run:       14.7272
>> > Total reads made:     3731
>> > Read size:            4194304
>> > Object size:          4194304
>> > Bandwidth (MB/sec):   1013.36
>> > Average IOPS:         253
>> > Stddev IOPS:          63.8709
>> > Max IOPS:             323
>> > Min IOPS:             91
>> > Average Latency(s):   0.0625202
>> > Max latency(s):       0.551629
>> > Min latency(s):       0.010683
>> >
>> > On average, I get around 550 MB/s writes and 800 MB/s reads with 16
>> threads
>> > and 4MB blocks. The numbers don't look fantastic for this hardware, I
>> can
>> > actually push over 8 GB/s of throughput with fio, 16 threads and 4MB
>> blocks
>> > from an RBD client (KVM Linux VM) connected over a low-latency 40G
>> network,
>> > probably hitting some OSD caches there:
>> >
>> >    READ: bw=8525MiB/s (8939MB/s), 58.8MiB/s-1009MiB/s
>> (61.7MB/s-1058MB/s),
>> > io=501GiB (538GB), run=60001-60153msec
>> > Disk stats (read/write):
>> >   vdc: ios=48163/0, merge=6027/0, ticks=1400509/0, in_queue=1305092,
>> > util=99.48%
>> >
>> > The issue manifests when the same client does something closer to
>> real-life
>> > usage, like a single-thread write or read with 4KB blocks, as if using
>> for
>> > example ext4 file system:
>> >
>> > > fio --name=ttt --ioengine=posixaio --rw=write --bs=4k --numjobs=1
>> > --size=4g --iodepth=1 --runtime=60 --time_based --end_fsync=1
>> > ...
>> > Run status group 0 (all jobs):
>> >   WRITE: bw=120MiB/s (126MB/s), 120MiB/s-120MiB/s (126MB/s-126MB/s),
>> > io=7694MiB (8067MB), run=64079-64079msec
>> > Disk stats (read/write):
>> >   vdc: ios=0/6985, merge=0/406, ticks=0/3062535, in_queue=3048216,
>> > util=77.31%
>> >
>> > > fio --name=ttt --ioengine=posixaio --rw=read --bs=4k --numjobs=1
>> > --size=4g --iodepth=1 --runtime=60 --time_based --end_fsync=1
>> > ...
>> > Run status group 0 (all jobs):
>> >    READ: bw=54.0MiB/s (56.7MB/s), 54.0MiB/s-54.0MiB/s
>> (56.7MB/s-56.7MB/s),
>> > io=3242MiB (3399MB), run=60001-60001msec
>> > Disk stats (read/write):
>> >   vdc: ios=12952/3, merge=0/1, ticks=81706/1, in_queue=56336,
>> util=99.13%
>> >
>> > And this is a total disaster: the IOPS look decent, but the bandwidth is
>> > unexpectedly very very low. I just don't understand why a single RBD
>> client
>> > writes at 120 MB/s (sometimes slower), and 50 MB/s reads look like a bad
>> > joke ¯\_(ツ)_/¯
>> >
>> > When I run these benchmarks, nothing seems to be overloaded, things like
>> > CPU and network are barely utilized, OSD latencies don't show anything
>> > unusual. Thus I am puzzled with these results, as in my opinion SAS HDDs
>> > with DB/WAL on NVME drives should produce better I/O bandwidth, both for
>> > writes and reads. I mean, I can easily get much better performance from
>> a
>> > single HDD shared over network via NFS or iSCSI.
>> >
>> > I am open to suggestions and would very much appreciate comments and/or
>> an
>> > advice on how to improve the cluster performance.
>> >
>> > Best regards,
>> > Zakhar
>> > _______________________________________________
>> > ceph-users mailing list -- ceph-users@xxxxxxx
>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>> >
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@xxxxxxx
>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux