I would say check with rados tool like ceph_smalliobench/rados bench first to see how much performance these tools are reporting. This will help you to isolate
any upstream issues.
Also, check with ‘iostat –xk 1’ for the resource utilization. Hope you are running with powerful enough cpu complex since you are saying network is not a bottleneck.
Thanks & Regards
Somnath
From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx]
On Behalf Of mad Engineer
Sent: Saturday, February 28, 2015 12:29 PM
To: Alexandre DERUMIER
Cc: ceph-users
Subject: Re: [ceph-users] Extreme slowness in SSD cluster with 3 nodes and 9 OSD with 3.16-3 kernel
reinstalled ceph packages and now with memstore backend [osd objectstore =memstore] its giving 400Kbps .No idea where the problem is.
On Sun, Mar 1, 2015 at 12:30 AM, mad Engineer <themadengin33r@xxxxxxxxx> wrote:
tried changing scheduler from deadline to noop also upgraded to Gaint and btrfs filesystem,downgraded kernel to 3.16 from 3.16-3 not much difference
dd if=/dev/zero of=hi bs=4k count=25000 oflag=direct
102400000 bytes (102 MB) copied, 94.691 s, 1.1 MB/s
Earlier on a vmware setup i was getting ~850 KBps and now even on physical server with SSD drives its just over 1MBps.I doubt some serious configuration issues.
Tried iperf between 3 servers all are showing 9 Gbps,tried icmp with different packet size ,no fragmentation.
i also noticed that out of 9 osd 5 are 850 EVO and 4 are 840 EVO.I believe this will not cause this much drop in performance.
On Sat, Feb 28, 2015 at 6:49 PM, Alexandre DERUMIER <aderumier@xxxxxxxxx> wrote:
As optimisation,
try to set ioscheduler to noop,
and also enable rbd_cache=true. (It's really helping for for sequential writes)
but your results seem quite low, 926kb/s with 4k, it's only 200io/s.
check if you don't have any big network latencies, or mtu fragementation problem.
Maybe also try to bench with fio, with more parallel jobs.
----- Mail original -----
De: "mad Engineer" <themadengin33r@xxxxxxxxx>
À: "Philippe Schwarz" <phil@xxxxxxxxxxxxxx>
Cc: "ceph-users" <ceph-users@xxxxxxxxxxxxxx>
Envoyé: Samedi 28 Février 2015 13:06:59
Objet: Re: [ceph-users] Extreme slowness in SSD cluster with 3 nodes and 9 OSD with 3.16-3 kernel
Thanks for the reply Philippe,we were using these disks in our NAS,now
it looks like i am in big trouble :-(
On Sat, Feb 28, 2015 at 5:02 PM, Philippe Schwarz <phil@xxxxxxxxxxxxxx> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Le 28/02/2015 12:19, mad Engineer a écrit :
>> Hello All,
>>
>> I am trying ceph-firefly 0.80.8
>> (69eaad7f8308f21573c604f121956e64679a52a7) with 9 OSD ,all Samsung
>> SSD 850 EVO on 3 servers with 24 G RAM,16 cores @2.27 Ghz Ubuntu
>> 14.04 LTS with 3.16-3 kernel.All are connected to 10G ports with
>> maximum MTU.There are no extra disks for journaling and also there
>> are no separate network for replication and data transfer.All 3
>> nodes are also hosting monitoring process.Operating system runs on
>> SATA disk.
>>
>> When doing a sequential benchmark using "dd" on RBD, mounted on
>> client as ext4 its taking 110s to write 100Mb data at an average
>> speed of 926Kbps.
>>
>> time dd if=/dev/zero of=hello bs=4k count=25000 oflag=direct
>> 25000+0 records in 25000+0 records out 102400000 bytes (102 MB)
>> copied, 110.582 s, 926 kB/s
>>
>> real 1m50.585s user 0m0.106s sys 0m2.233s
>>
>> While doing this directly on ssd mount point shows:
>>
>> time dd if=/dev/zero of=hello bs=4k count=25000 oflag=direct
>> 25000+0 records in 25000+0 records out 102400000 bytes (102 MB)
>> copied, 1.38567 s, 73.9 MB/s
>>
>> OSDs are in XFS with these extra arguments :
>>
>> rw,noatime,inode64,logbsize=256k,delaylog,allocsize=4M
>>
>> ceph.conf
>>
>> [global] fsid = 7d889081-7826-439c-9fe5-d4e57480d9be
>> mon_initial_members = ceph1, ceph2, ceph3 mon_host =
>> 10.99.10.118,10.99.10.119,10.99.10.120 auth_cluster_required =
>> cephx auth_service_required = cephx auth_client_required = cephx
>> filestore_xattr_use_omap = true osd_pool_default_size = 2
>> osd_pool_default_min_size = 2 osd_pool_default_pg_num = 450
>> osd_pool_default_pgp_num = 450 max_open_files = 131072
>>
>> [osd] osd_mkfs_type = xfs osd_op_threads = 8 osd_disk_threads = 4
>> osd_mount_options_xfs =
>> "rw,noatime,inode64,logbsize=256k,delaylog,allocsize=4M"
>>
>>
>> on our traditional storage with Full SAS disk, same "dd" completes
>> in 16s with an average write speed of 6Mbps.
>>
>> Rados bench:
>>
>> rados bench -p rbd 10 write Maintaining 16 concurrent writes of
>> 4194304 bytes for up to 10 seconds or 0 objects Object prefix:
>> benchmark_data_ceph1_2977 sec Cur ops started finished avg MB/s
>> cur MB/s last lat avg lat 0 0 0 0
>> 0 0 - 0 1 16 94 78
>> 311.821 312 0.041228 0.140132 2 16 192 176
>> 351.866 392 0.106294 0.175055 3 16 275 259
>> 345.216 332 0.076795 0.166036 4 16 302 286
>> 285.912 108 0.043888 0.196419 5 16 395 379
>> 303.11 372 0.126033 0.207488 6 16 501 485
>> 323.242 424 0.125972 0.194559 7 16 621 605
>> 345.621 480 0.194155 0.183123 8 16 730 714
>> 356.903 436 0.086678 0.176099 9 16 814 798
>> 354.572 336 0.081567 0.174786 10 16 832
>> 816 326.313 72 0.037431 0.182355 11 16 833
>> 817 297.013 4 0.533326 0.182784 Total time run:
>> 11.489068 Total writes made: 833 Write size:
>> 4194304 Bandwidth (MB/sec): 290.015
>>
>> Stddev Bandwidth: 175.723 Max bandwidth (MB/sec): 480 Min
>> bandwidth (MB/sec): 0 Average Latency: 0.220582 Stddev
>> Latency: 0.343697 Max latency: 2.85104 Min
>> latency: 0.035381
>>
>> Our ultimate aim is to replace existing SAN with ceph,but for that
>> it should meet minimum 8000 iops.Can any one help me with this,OSD
>> are SSD,CPU has good clock speed,backend network is good but still
>> we are not able to extract full capability of SSD disks.
>>
>>
>>
>> Thanks,
>
> Hi, i'm new to ceph so, don't consider my words as holy truth.
>
> It seems that Samsung 840 (so i assume 850) are crappy for ceph :
>
> MTBF :
>
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-November/044258.html
> Bandwidth
> :http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-December/045247.html
>
> And according to a confirmed user of Ceph/ProxmoX, Samsung SSDs should
> be avoided if possible in ceph storage.
>
> Apart from that, it seems there was an limitation in ceph for the use
> of the complete bandwidth available in SSDs; but i think with less
> than 1Mb/s you haven't hit this limit.
>
> I remind you that i'm not a ceph-guru (far from that, indeed), so feel
> free to disagree; i'm on the way to improve my knowledge.
>
> Best regards.
>
>
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
>
> iEYEARECAAYFAlTxp0UACgkQlhqCFkbqHRb5+wCgrXCM3VsnVE6PCbbpOmQXCXbr
> 8u0An2BUgZWismSK0PxbwVDOD5+/UWik
> =0o0v
> -----END PGP SIGNATURE-----
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this
message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy
any and all copies of this message in your possession (whether hard copies or electronically stored copies).
|
