Re: Ceph random read IOPS

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 24-6-2017 05:30, Christian Wuerdig wrote:
> The general advice floating around is that your want CPUs with high
> clock speeds rather than more cores to reduce latency and increase IOPS
> for SSD setups (see also
> http://www.sys-pro.co.uk/ceph-storage-fast-cpus-ssd-performance/) So
> something like a E5-2667V4 might bring better results in that situation.
> Also there was some talk about disabling the processor C states in order
> to bring latency down (something like this should be easy to test:
> https://stackoverflow.com/a/22482722/220986)

I would be very careful to call this a general advice...

Although the article is interesting, it is rather single sided.

The only thing is shows that there is a lineair relation between
clockspeed and write or read speeds???
The article is rather vague on how and what is actually tested.

By just running a single OSD with no replication a lot of the
functionality is left out of the equation.
Nobody is running just 1 osD on a box in a normal cluster host.

Not using a serious SSD is another source of noise on the conclusion.
More Queue depth can/will certainly have impact on concurrency.

I would call this an observation, and nothing more.

--WjW
> 
> On Sat, Jun 24, 2017 at 1:28 AM, Kostas Paraskevopoulos
> <reverend.x3@xxxxxxxxx <mailto:reverend.x3@xxxxxxxxx>> wrote:
> 
>     Hello,
> 
>     We are in the process of evaluating the performance of a testing
>     cluster (3 nodes) with ceph jewel. Our setup consists of:
>     3 monitors (VMs)
>     2 physical servers each connected with 1 JBOD running Ubuntu Server
>     16.04
> 
>     Each server has 32 threads @2.1GHz and 128GB RAM.
>     The disk distribution per server is:
>     38 * HUS726020ALS210 (SAS rotational)
>     2 * HUSMH8010BSS200 (SAS SSD for journals)
>     2 * ST1920FM0043 (SAS SSD for data)
>     1 * INTEL SSDPEDME012T4 (NVME measured with fio ~300K iops)
> 
>     Since we don't currently have a 10Gbit switch, we test the performance
>     with the cluster in a degraded state, the noout flag set and we mount
>     rbd images on the powered on osd node. We confirmed that the network
>     is not saturated during the tests.
> 
>     We ran tests on the NVME disk and the pool created on this disk where
>     we hoped to get the most performance without getting limited by the
>     hardware specs since we have more disks than CPU threads.
> 
>     The nvme disk was at first partitioned with one partition and the
>     journal on the same disk. The performance on random 4K reads was
>     topped at 50K iops. We then removed the osd and partitioned with 4
>     data partitions and 4 journals on the same disk. The performance
>     didn't increase significantly. Also, since we run read tests, the
>     journals shouldn't cause performance issues.
> 
>     We then ran 4 fio processes in parallel on the same rbd mounted image
>     and the total iops reached 100K. More parallel fio processes didn't
>     increase the measured iops.
> 
>     Our ceph.conf is pretty basic (debug is set to 0/0 for everything) and
>     the crushmap just defines the different buckets/rules for the disk
>     separation (rotational, ssd, nvme) in order to create the required
>     pools
> 
>     Is the performance of 100.000 iops for random 4K read normal for a
>     disk that on the same benchmark runs at more than 300K iops on the
>     same hardware or are we missing something?
> 
>     Best regards,
>     Kostas
>     _______________________________________________
>     ceph-users mailing list
>     ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
>     http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>     <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
> 
> 
> 
> 
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux