Re: Erasure coding enhancements - design for review

[Sebastian Wagner <sebastian.wagner@xxxxxxxx>]

Hi Bill,

I noticed you mentioned the PR a few moments after I clicked on send 🙈. I really hope we end up with EC being on par with replicated pools without any meaningful performance impact. 

Do you think we can put those design documents into the Ceph Github organization? IIRC, this is the third proposal on this mailing list that I saw just in a few months.

Best,
Sebastian

Am 01.07.24 um 12:01 schrieb Bill Scales:

Hi,

 

Yes, most of the optimizations for partial reads are part of that pull request – it is referenced in the design document.

There is one further enhancement we would like to make which is to reduce the size of the reads to each shard – the pull request currently reads the whole chunk for any shard that needs to be read. This is not an issue with the current default chunk size which is 4K, but if you use a larger chunk size such as 256K then this means that short client reads will read excess data and then discard it.

Currently there are many reasons why using a small chunk size is a good idea, but once both partial reads and partial writes have been implemented performance can be improved further by using a larger chunk size because splitting I/Os into multiple chunks is expensive. There’s more discussion about the chunk size and how to accommodate small sized objects without large storage overheads later in the document.

 

Cheers,

Bill.
bill_scales@xxxxxxxxxx
IBM Distinguished Engineer, IBM Storage 

 

 

From: Sebastian Wagner <sebastian.wagner@xxxxxxxx>
Date: Monday, 1 July 2024 at 09:31
To: Bill Scales <bill_scales@xxxxxxxxxx>, dev@xxxxxxx <dev@xxxxxxx>
Subject: [EXTERNAL] Re: Erasure coding enhancements - design for review

Hi Bill, Is the Partial Reads section the same as https: //github. com/ceph/ceph/pull/55196 ? Best, Sebastian Am 01. 07. 24 um 10: 08 schrieb Bill Scales: Hi, We are planning to enhance the performance of erasure coding, in particular for use with

Hi Bill,

Is the Partial Reads section the same as https://github.com/ceph/ceph/pull/55196 ?

Best,
Sebastian

Am 01.07.24 um 10:08 schrieb Bill Scales:

Hi,

We are planning to enhance the performance of erasure coding, in particular for use with block and file. We've got a design document https://github.com/bill-scales/CephErasureCodingDesign that sets out what we are hoping to achieve. We welcome your feedback, either posting your comments in Slack on #ceph-devel , raising issues in github or getting in contact with myself

 

Cheers,

Bill.
bill_scales@xxxxxxxxxx
IBM Distinguished Engineer, IBM Storage 

 

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

_______________________________________________

Dev mailing list -- dev@xxxxxxx

To unsubscribe send an email to dev-leave@xxxxxxx

--
Head of Software Development
E-Mail: sebastian.wagner@xxxxxxxx

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263

Web | LinkedIn | Youtube | Twitter

TOP 100 Innovator Award Winner by compamedia
Technology Fast50 Award Winner by Deloitte

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

--
Head of Software Development
E-Mail: sebastian.wagner@xxxxxxxx

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263

Web | LinkedIn | Youtube | Twitter

TOP 100 Innovator Award Winner by compamedia
Technology Fast50 Award Winner by Deloitte

_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx