Re: ceph luminous - performance issue

[Steven Vacaroaia <stef97@xxxxxxxxx>]

Thanks for your willingness to help

DELL R620, 1 CPU, 8 cores, 64 GB RAM 

cluster network is using 2 bonded 10 GB NICs ( mode=4), MTU=9000

SSD drives are Enterprise grade  - 400 GB SSD  Toshiba PX04SHB040

HDD drives are  - 10k RPM, 600 GB  Toshiba AL13SEB600

Steven

On 3 January 2018 at 09:41, Brady Deetz <bdeetz@xxxxxxxxx> wrote:

Can you provide more detail regarding the infrastructure backing this environment? What hard drive, ssd, and processor are you using? Also, what is providing networking?

I'm seeing 4k blocksize tests here. Latency is going to destroy you. 

On Jan 3, 2018 8:11 AM, "Steven Vacaroaia" <stef97@xxxxxxxxx> wrote:

Hi,

I am doing a PoC with 3 DELL R620 and 12 OSD , 3 SSD drives ( one on each server), bluestore 

I configured the OSD using the following ( /dev/sda is my SSD drive)

ceph-disk prepare --zap-disk --cluster ceph  --bluestore /dev/sde --block.wal /dev/sda --block.db /dev/sda

Unfortunately both fio and bench tests show much worse performance for the pools than for the individual disks 

Example:

DISKS

fio --filename=/dev/sda --direct=1 --sync=1 --rw=write --bs=4k --numjobs=14 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test

SSD drive

Jobs: 14 (f=14): [W(14)] [100.0% done] [0KB/465.2MB/0KB /s] [0/119K/0 iops] [eta 00m:00s]

HD drive

Jobs: 14 (f=14): [W(14)] [100.0% done] [0KB/179.2MB/0KB /s] [0/45.9K/0 iops] [eta 00m:00s]

POOL

fio write.fio

Jobs: 1 (f=0): [w(1)] [100.0% done] [0KB/51428KB/0KB /s] [0/12.9K/0 iops] 

 cat write.fio

[write-4M]

description="write test with 4k block"

ioengine=rbd

clientname=admin

pool=scbench

rbdname=image01

iodepth=32

runtime=120

rw=randwrite

bs=4k

rados bench -p scbench 12 write

Max bandwidth (MB/sec): 224

Min bandwidth (MB/sec): 0

Average IOPS:           26

Stddev IOPS:            24

Max IOPS:               56

Min IOPS:               0

Average Latency(s):     0.59819

Stddev Latency(s):      1.64017

Max latency(s):         10.8335

Min latency(s):         0.00475139

I must be missing something - any help/suggestions will be greatly appreciated 

Here are some specific info 

ceph -s

  cluster:

    id:     91118dde-f231-4e54-a5f0-a1037f3d5142

    health: HEALTH_OK

  services:

    mon: 1 daemons, quorum mon01

    mgr: mon01(active)

    osd: 12 osds: 12 up, 12 in

  data:

    pools:   4 pools, 484 pgs

    objects: 70082 objects, 273 GB

    usage:   570 GB used, 6138 GB / 6708 GB avail

    pgs:     484 active+clean

  io:

    client:   2558 B/s rd, 2 op/s rd, 0 op/s wr

ceph osd pool ls detail

pool 1 'test-replicated' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 157 flags hashpspool stripe_width 0 application rbd

        removed_snaps [1~3]

pool 2 'test-erasure' erasure size 3 min_size 3 crush_rule 1 object_hash rjenkins pg_num 128 pgp_num 128 last_change 334 flags hashpspool stripe_width 8192 application rbd

        removed_snaps [1~5]

pool 3 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 200 flags hashpspool stripe_width 0 application rbd

        removed_snaps [1~3]

pool 4 'scbench' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 100 pgp_num 100 last_change 330 flags hashpspool stripe_width 0

        removed_snaps [1~3]

[cephuser@ceph ceph-config]$ ceph osd df tree

ID CLASS WEIGHT  REWEIGHT SIZE  USE    AVAIL %USE  VAR  PGS TYPE NAME

-1       6.55128        - 2237G   198G 2038G     0    0   - root default

-7             0        -     0      0     0     0    0   -     host ods03

-3       2.18475        - 2237G   181G 2055G  8.12 0.96   -     host osd01

 3   hdd 0.54619  1.00000  559G 53890M  506G  9.41 1.11  90         osd.3

 4   hdd 0.54619  1.00000  559G 30567M  529G  5.34 0.63  89         osd.4

 5   hdd 0.54619  1.00000  559G 59385M  501G 10.37 1.22  93         osd.5

 6   hdd 0.54619  1.00000  559G 42156M  518G  7.36 0.87  93         osd.6

-5       2.18178        - 2234G   189G 2044G  8.50 1.00   -     host osd02

 0   hdd 0.54520  1.00000  558G 32460M  526G  5.68 0.67  90         osd.0

 1   hdd 0.54520  1.00000  558G 54578M  504G  9.55 1.12  89         osd.1

 2   hdd 0.54520  1.00000  558G 47761M  511G  8.35 0.98  93         osd.2

 7   hdd 0.54619  1.00000  559G 59584M  501G 10.40 1.22  92         osd.7

-9       2.18475        - 2237G   198G 2038G  8.88 1.04   -     host osd03

 8   hdd 0.54619  1.00000  559G 52462M  508G  9.16 1.08  99         osd.8

10   hdd 0.54619  1.00000  559G 35284M  524G  6.16 0.73  88         osd.10

11   hdd 0.54619  1.00000  559G 71739M  489G 12.53 1.47  87         osd.11

12   hdd 0.54619  1.00000  559G 43832M  516G  7.65 0.90  93         osd.12

                    TOTAL 6708G   570G 6138G  8.50

MIN/MAX VAR: 0.63/1.47  STDDEV: 2.06

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com