Hey Sage,
Thanks for the reply. I looked at the paper the very interesting PR you
mentioned.
I find the path to be interesting as this scheme actually improves the
core repair algorithm. What I plan to do is actually on scalar RS codes.
Specifically we are trying to implement a (14, 10) RS code with a
different erasure scheme for a single repair. In the generic
implementation, repairing a single node costs 10 bytes in communication.
This scheme brings it down to less than or equal to 8 bytes. I feel this
is interesting because this could very easily be extended to vector
codes. ( I am still doing/verifying the math on that.)
Also, thanks for the info on optimizing building. That and Erwan's
recent thread helped me a lot
Aaditya
On Monday 25 June 2018 05:30 PM, Sage Weil wrote:
On Mon, 25 Jun 2018, Aaditya M Nair wrote:
Hey guys,
Aayush and me are new to the Ceph project and are very interested in
contributing to it.
Great!
In particular, we want to implement a particular example of reed solomon codes
described in https://arxiv.org/abs/1501.06683.
This paper introduces a new repair scheme for reed solomon codes that improves
the network bandwidth usage at the cost of disk usage. We think that it could
be very useful to Ceph as an erasure code plugin.
We would love any and all assistance regarding this project.
I encourage you to look at the existing LRC plugin and at the Clay code
PR that is pending and should merge soon
https://github.com/ceph/ceph/pull/14300
to make sure this new code is adding a useful new point in the design
space. I'm not an EC expert and have trouble identifying the key
properties and tradeoffs between the different coding schemes.
Also, we have some more specific doubts regarding to erasure code plugins
itself. Forgive me if these questions are already answered somewhere.
1. While writing a new plugin, how do you guys load it dynamically when
testing your code? The closest I found was recompiling every time we change
code but I am really hoping that there is a better way.
By default a bunch of files need to be relinked every time the git commit
changes. When developing you can reduce the impact by only building a
minimal set of targets to run vstart (make vstart-base or make vstart) and
you can also disable this behavior entirely by passing
-DENABLE_GIT_VERSION=OFF to cmake or do_cmake.sh.
2. How do you test if the data is split correctly or if it being
reconstructed correctly?
In the simplest case you can simply test creating a pool with your EC code
and then writing and reading back an object. Our integration testing does
this extensively (while mixing in node failures and so on). There are
also a range of unit tests for the erasure codes in src/test/erasure-code
(probably the place to start!), and ceph-erasure-code-corpus, which is
just a pile of EC configurations and data samples driven by scripts in
qa/workunits/erasure-code.
sage
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
