Hi Steven. interesting... 'm quite curious after your post now. I've migrated our prod. CEPH cluster to 12.2.2 and Bluestore just today and haven't heard back anything "bad" from the applications/users so far. performance tests on our test cluster were good before, but we use S3/RGW only anyhow ;) there are two things I would like to know/learn... could you try/test and feed back?! - change all your tests to use >=16k block size, see also BStore comments here (https://www.mail-archive.com/ceph-users@xxxxxxxxxxxxxx/msg43023.html) - change your "write.fio" file profile from "rw=randwrite" to "rw=write" (or something similar :O ) to compare apples with apples ;) thanks for your efforts and looking forward for those results ;) best regards Notna ---------------------------------------------------------------------------------- Gesendet: Mittwoch, 03. Januar 2018 um 16:20 Uhr Von: "Steven Vacaroaia" <stef97@xxxxxxxxx> An: "Brady Deetz" <bdeetz@xxxxxxxxx> Cc: ceph-users <ceph-users@xxxxxxxxxxxxxx> Betreff: Re: ceph luminous - performance issue Thanks for your willingness to help DELL R620, 1 CPU, 8 cores, 64 GB RAM cluster network is using 2 bonded 10 GB NICs ( mode=4), MTU=9000 SSD drives are Enterprise grade - 400 GB SSD Toshiba PX04SHB040 HDD drives are - 10k RPM, 600 GB Toshiba AL13SEB600 Steven On 3 January 2018 at 09:41, Brady Deetz <bdeetz@xxxxxxxxx[mailto:bdeetz@xxxxxxxxx]> wrote: Can you provide more detail regarding the infrastructure backing this environment? What hard drive, ssd, and processor are you using? Also, what is providing networking? I'm seeing 4k blocksize tests here. Latency is going to destroy you. On Jan 3, 2018 8:11 AM, "Steven Vacaroaia" <stef97@xxxxxxxxx[mailto:stef97@xxxxxxxxx]> wrote: Hi, I am doing a PoC with 3 DELL R620 and 12 OSD , 3 SSD drives ( one on each server), bluestore I configured the OSD using the following ( /dev/sda is my SSD drive) ceph-disk prepare --zap-disk --cluster ceph --bluestore /dev/sde --block.wal /dev/sda --block.db /dev/sda Unfortunately both fio and bench tests show much worse performance for the pools than for the individual disks Example: DISKS fio --filename=/dev/sda --direct=1 --sync=1 --rw=write --bs=4k --numjobs=14 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test SSD drive Jobs: 14 (f=14): [W(14)] [100.0% done] [0KB/465.2MB/0KB /s] [0/119K/0 iops] [eta 00m:00s] HD drive Jobs: 14 (f=14): [W(14)] [100.0% done] [0KB/179.2MB/0KB /s] [0/45.9K/0 iops] [eta 00m:00s] POOL fio write.fio Jobs: 1 (f=0): [w(1)] [100.0% done] [0KB/51428KB/0KB /s] [0/12.9K/0 iops] cat write.fio [write-4M] description="write test with 4k block" ioengine=rbd clientname=admin pool=scbench rbdname=image01 iodepth=32 runtime=120 rw=randwrite bs=4k rados bench -p scbench 12 write Max bandwidth (MB/sec): 224 Min bandwidth (MB/sec): 0 Average IOPS: 26 Stddev IOPS: 24 Max IOPS: 56 Min IOPS: 0 Average Latency(s): 0.59819 Stddev Latency(s): 1.64017 Max latency(s): 10.8335 Min latency(s): 0.00475139 I must be missing something - any help/suggestions will be greatly appreciated Here are some specific info ceph -s cluster: id: 91118dde-f231-4e54-a5f0-a1037f3d5142 health: HEALTH_OK services: mon: 1 daemons, quorum mon01 mgr: mon01(active) osd: 12 osds: 12 up, 12 in data: pools: 4 pools, 484 pgs objects: 70082 objects, 273 GB usage: 570 GB used, 6138 GB / 6708 GB avail pgs: 484 active+clean io: client: 2558 B/s rd, 2 op/s rd, 0 op/s wr ceph osd pool ls detail pool 1 'test-replicated' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 157 flags hashpspool stripe_width 0 application rbd removed_snaps [1~3] pool 2 'test-erasure' erasure size 3 min_size 3 crush_rule 1 object_hash rjenkins pg_num 128 pgp_num 128 last_change 334 flags hashpspool stripe_width 8192 application rbd removed_snaps [1~5] pool 3 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 200 flags hashpspool stripe_width 0 application rbd removed_snaps [1~3] pool 4 'scbench' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 100 pgp_num 100 last_change 330 flags hashpspool stripe_width 0 removed_snaps [1~3] [cephuser@ceph ceph-config]$ ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS TYPE NAME -1 6.55128 - 2237G 198G 2038G 0 0 - root default -7 0 - 0 0 0 0 0 - host ods03 -3 2.18475 - 2237G 181G 2055G 8.12 0.96 - host osd01 3 hdd 0.54619 1.00000 559G 53890M 506G 9.41 1.11 90 osd.3 4 hdd 0.54619 1.00000 559G 30567M 529G 5.34 0.63 89 osd.4 5 hdd 0.54619 1.00000 559G 59385M 501G 10.37 1.22 93 osd.5 6 hdd 0.54619 1.00000 559G 42156M 518G 7.36 0.87 93 osd.6 -5 2.18178 - 2234G 189G 2044G 8.50 1.00 - host osd02 0 hdd 0.54520 1.00000 558G 32460M 526G 5.68 0.67 90 osd.0 1 hdd 0.54520 1.00000 558G 54578M 504G 9.55 1.12 89 osd.1 2 hdd 0.54520 1.00000 558G 47761M 511G 8.35 0.98 93 osd.2 7 hdd 0.54619 1.00000 559G 59584M 501G 10.40 1.22 92 osd.7 -9 2.18475 - 2237G 198G 2038G 8.88 1.04 - host osd03 8 hdd 0.54619 1.00000 559G 52462M 508G 9.16 1.08 99 osd.8 10 hdd 0.54619 1.00000 559G 35284M 524G 6.16 0.73 88 osd.10 11 hdd 0.54619 1.00000 559G 71739M 489G 12.53 1.47 87 osd.11 12 hdd 0.54619 1.00000 559G 43832M 516G 7.65 0.90 93 osd.12 TOTAL 6708G 570G 6138G 8.50 MIN/MAX VAR: 0.63/1.47 STDDEV: 2.06 _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx[mailto:ceph-users@xxxxxxxxxxxxxx] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com] _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |