Re: Is there a setting on Ceph that we can use to fix the minimum read size?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



We configured XFS on our OSDs to use 1M blocks (our use case is RBDs with 1M blocks) due to massive fragmentation in our filestores a while back. We were having to defrag all the time and cluster performance was noticeably degraded. We also create and delete lots of RBD snapshots on a daily basis, so that likely contributed to the fragmentation as well. It’s been MUCH better since we switched XFS to use 1M allocations. Virtually no fragmentation and performance is consistently good.

 

From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Kate Ward
Sent: Tuesday, November 29, 2016 2:02 PM
To: Thomas Bennett <thomas@xxxxxxxxx>
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re: [ceph-users] Is there a setting on Ceph that we can use to fix the minimum read size?

 

I have no experience with XFS, but wouldn't expect poor behaviour with it. I use ZFS myself and know that it would combine writes, but btrfs might be an option.

 

Do you know what block size was used to create the XFS filesystem? It looks like 4k is the default (reasonable) with a max of 64k. Perhaps a larger block size will give better performance for your particular use case. (I use a 1M block size with ZFS.)

 

 

On Tue, Nov 29, 2016 at 10:23 AM Thomas Bennett <thomas@xxxxxxxxx> wrote:

Hi Kate,

 

Thanks for your reply. We currently use xfs as created by ceph-deploy. 

 

What would you recommend we try?

 

Kind regards,

Tom

 

 

On Tue, Nov 29, 2016 at 11:14 AM, Kate Ward <kate.ward@xxxxxxxxxxxxx> wrote:

What filesystem do you use on the OSD? Have you considered a different filesystem that is better at combining requests before they get to the drive?

 

k8

 

On Tue, Nov 29, 2016 at 9:52 AM Thomas Bennett <thomas@xxxxxxxxx> wrote:

Hi,

 

We have a use case where we are reading 128MB objects off spinning disks.

 

We've benchmarked a number of different hard drive and have noticed that for a particular hard drive, we're experiencing slow reads by comparison. 

 

This occurs when we have multiple readers (even just 2) reading objects off the OSD.

 

We've recreated the effect using iozone and have noticed that once the record size drops to 4k, the hard drive miss behaves.

 

Is there a setting on Ceph that we can change to fix the minimum read size when the ceph-osd daemon reads the object of the hard drives, to see if we can overcome the overall slow read rate.

 

Cheers,

Tom


Steve Taylor | Senior Software Engineer | StorageCraft Technology Corporation
380 Data Drive Suite 300 | Draper | Utah | 84020
Office: 801.871.2799 |


If you are not the intended recipient of this message or received it erroneously, please notify the sender and delete it, together with any attachments, and be advised that any dissemination or copying of this message is prohibited.


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



 

--

Thomas Bennett

 

SKA South Africa

Science Processing Team

 

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux