Re: ceph luminous - performance issue

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Steven.

interesting... 'm quite curious after your post now.

I've migrated our prod. CEPH cluster to 12.2.2 and Bluestore just today and haven't heard back anything "bad" from the applications/users so far. performance tests on our test cluster were good before, but we use S3/RGW only anyhow ;)

there are two things I would like to know/learn... could you try/test and feed back?!

- change all your tests to use >=16k block size, see also BStore comments here (https://www.mail-archive.com/ceph-users@xxxxxxxxxxxxxx/msg43023.html)
- change your "write.fio" file profile from "rw=randwrite" to "rw=write" (or something similar :O ) to compare apples with apples ;)

thanks for your efforts and looking forward for those results ;)

best regards
 Notna

----------------------------------------------------------------------------------

Gesendet: Mittwoch, 03. Januar 2018 um 16:20 Uhr
Von: "Steven Vacaroaia" <stef97@xxxxxxxxx>
An: "Brady Deetz" <bdeetz@xxxxxxxxx>
Cc: ceph-users <ceph-users@xxxxxxxxxxxxxx>
Betreff: Re:  ceph luminous - performance issue

Thanks for your willingness to help
 
DELL R620, 1 CPU, 8 cores, 64 GB RAM 
cluster network is using 2 bonded 10 GB NICs ( mode=4), MTU=9000
 
SSD drives are Enterprise grade  - 400 GB SSD  Toshiba PX04SHB040
HDD drives are  - 10k RPM, 600 GB  Toshiba AL13SEB600
 
Steven

 
On 3 January 2018 at 09:41, Brady Deetz <bdeetz@xxxxxxxxx[mailto:bdeetz@xxxxxxxxx]> wrote:
Can you provide more detail regarding the infrastructure backing this environment? What hard drive, ssd, and processor are you using? Also, what is providing networking?
 
I'm seeing 4k blocksize tests here. Latency is going to destroy you. 
 

On Jan 3, 2018 8:11 AM, "Steven Vacaroaia" <stef97@xxxxxxxxx[mailto:stef97@xxxxxxxxx]> wrote:

Hi,
 
I am doing a PoC with 3 DELL R620 and 12 OSD , 3 SSD drives ( one on each server), bluestore 
 
I configured the OSD using the following ( /dev/sda is my SSD drive)
ceph-disk prepare --zap-disk --cluster ceph  --bluestore /dev/sde --block.wal /dev/sda --block.db /dev/sda
 
Unfortunately both fio and bench tests show much worse performance for the pools than for the individual disks 
 
Example:
DISKS
fio --filename=/dev/sda --direct=1 --sync=1 --rw=write --bs=4k --numjobs=14 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test
 
SSD drive
Jobs: 14 (f=14): [W(14)] [100.0% done] [0KB/465.2MB/0KB /s] [0/119K/0 iops] [eta 00m:00s]
 
HD drive
Jobs: 14 (f=14): [W(14)] [100.0% done] [0KB/179.2MB/0KB /s] [0/45.9K/0 iops] [eta 00m:00s]
 
POOL
 
fio write.fio
Jobs: 1 (f=0): [w(1)] [100.0% done] [0KB/51428KB/0KB /s] [0/12.9K/0 iops] 
 

 cat write.fio
[write-4M]
description="write test with 4k block"
ioengine=rbd
clientname=admin
pool=scbench
rbdname=image01
iodepth=32
runtime=120
rw=randwrite
bs=4k
 
 
rados bench -p scbench 12 write
 

Max bandwidth (MB/sec): 224
Min bandwidth (MB/sec): 0
Average IOPS:           26
Stddev IOPS:            24
Max IOPS:               56
Min IOPS:               0
Average Latency(s):     0.59819
Stddev Latency(s):      1.64017
Max latency(s):         10.8335
Min latency(s):         0.00475139
 
 
 
 
I must be missing something - any help/suggestions will be greatly appreciated 
 
Here are some specific info 
 

ceph -s
  cluster:
    id:     91118dde-f231-4e54-a5f0-a1037f3d5142
    health: HEALTH_OK
 
  services:
    mon: 1 daemons, quorum mon01
    mgr: mon01(active)
    osd: 12 osds: 12 up, 12 in
 
  data:
    pools:   4 pools, 484 pgs
    objects: 70082 objects, 273 GB
    usage:   570 GB used, 6138 GB / 6708 GB avail
    pgs:     484 active+clean
 
  io:
    client:   2558 B/s rd, 2 op/s rd, 0 op/s wr
 

ceph osd pool ls detail
pool 1 'test-replicated' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 157 flags hashpspool stripe_width 0 application rbd
        removed_snaps [1~3]
pool 2 'test-erasure' erasure size 3 min_size 3 crush_rule 1 object_hash rjenkins pg_num 128 pgp_num 128 last_change 334 flags hashpspool stripe_width 8192 application rbd
        removed_snaps [1~5]
pool 3 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 200 flags hashpspool stripe_width 0 application rbd
        removed_snaps [1~3]
pool 4 'scbench' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 100 pgp_num 100 last_change 330 flags hashpspool stripe_width 0
        removed_snaps [1~3]
 
[cephuser@ceph ceph-config]$ ceph osd df tree
ID CLASS WEIGHT  REWEIGHT SIZE  USE    AVAIL %USE  VAR  PGS TYPE NAME
-1       6.55128        - 2237G   198G 2038G     0    0   - root default
-7             0        -     0      0     0     0    0   -     host ods03
-3       2.18475        - 2237G   181G 2055G  8.12 0.96   -     host osd01
 3   hdd 0.54619  1.00000  559G 53890M  506G  9.41 1.11  90         osd.3
 4   hdd 0.54619  1.00000  559G 30567M  529G  5.34 0.63  89         osd.4
 5   hdd 0.54619  1.00000  559G 59385M  501G 10.37 1.22  93         osd.5
 6   hdd 0.54619  1.00000  559G 42156M  518G  7.36 0.87  93         osd.6
-5       2.18178        - 2234G   189G 2044G  8.50 1.00   -     host osd02
 0   hdd 0.54520  1.00000  558G 32460M  526G  5.68 0.67  90         osd.0
 1   hdd 0.54520  1.00000  558G 54578M  504G  9.55 1.12  89         osd.1
 2   hdd 0.54520  1.00000  558G 47761M  511G  8.35 0.98  93         osd.2
 7   hdd 0.54619  1.00000  559G 59584M  501G 10.40 1.22  92         osd.7
-9       2.18475        - 2237G   198G 2038G  8.88 1.04   -     host osd03
 8   hdd 0.54619  1.00000  559G 52462M  508G  9.16 1.08  99         osd.8
10   hdd 0.54619  1.00000  559G 35284M  524G  6.16 0.73  88         osd.10
11   hdd 0.54619  1.00000  559G 71739M  489G 12.53 1.47  87         osd.11
12   hdd 0.54619  1.00000  559G 43832M  516G  7.65 0.90  93         osd.12
                    TOTAL 6708G   570G 6138G  8.50
MIN/MAX VAR: 0.63/1.47  STDDEV: 2.06
 
 _______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx[mailto:ceph-users@xxxxxxxxxxxxxx]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]
 
 
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux