Hi Michael,
I'm trying to reproduce the problem from sources (today's instead of yesterday's but there is no difference that could explain the behaviour you have):
cd src
rm -fr /tmp/dev /tmp/out ; mkdir -p /tmp/dev ; CEPH_DIR=/tmp LC_ALL=C MON=1 OSD=6 bash -x ./vstart.sh -d -n -X -l mon osd
ceph osd erasure-code-profile set profile33 ruleset-failure-domain=osd k=3 m=3
ceph osd crush rule create-erasure ecruleset33 profile33
ceph osd pool create testec-33 20 20 erasure profile33 ecruleset33
./rados --pool testec-33 put SOMETHING /etc/group
but it succeeds. Could you please script a minimal set of commands I could run to repeat the problem you're seeing ?
Cheers
On 30/03/2014 00:26, Michael Nelson wrote:
> I have a small cluster (4 nodes, 15 OSDs, 3-5 OSDs per node) running bits from the firefly branch (0.78-430-gb8ea656).
>
> I am trying out various k/m combinations for EC pools. Certain k/m combinations are causing rados put to fail on the second 4MB chunk. I realize some of these combinations might not make much sense given the size of my cluster, but I am curious if I am seeing errors because of the size of the cluster or due to some other issue (especially k=3/m=3). I have double checked that the failure domain is set to osd and not host (see below).
>
> ++ rados -p testec-22 put xyz sampledata
> ++ rados -p testec-33 put xyz sampledata
> error putting testec-33/xyz: Operation not supported
> ++ rados -p testec-42 put xyz sampledata
> ++ rados -p testec-43 put xyz sampledata
> ++ rados -p testec-52 put xyz sampledata
> error putting testec-52/xyz: Operation not supported
> ++ rados -p testec-53 put xyz sampledata
> error putting testec-53/xyz: Operation not supported
> ++ rados -p testec-62 put xyz sampledata
> error putting testec-62/xyz: Operation not supported
> ++ rados -p testec-63 put xyz sampledata
> error putting testec-63/xyz: Operation not supported
>
> Error from OSD to client:
>
> 2014-03-29 16:08:14.024912 7f6556d13700 1 -- 198.18.32.12:0/1011811 <== osd.3 198.18.32.15:6800/38109 2 ==== osd_op_reply(2 xyz [write 4194304~4194304] v0'0 uv0 ondisk = -95 (Operation not supported)) v6 ==== 170+0+0 (469808551 0 0) 0x7f65300025e0 con 0x2451db0
>
> Configuration for the pools:
>
> ceph osd erasure-code-profile set profile22 ruleset-failure-domain=osd k=2 m=2
> ceph osd crush rule create-erasure ecruleset22 profile22
> ceph osd pool create testec-22 20 20 erasure profile22 ecruleset22
>
> ceph osd erasure-code-profile set profile33 ruleset-failure-domain=osd k=3 m=3
> ceph osd crush rule create-erasure ecruleset33 profile33
> ceph osd pool create testec-33 20 20 erasure profile33 ecruleset33
>
> ceph osd erasure-code-profile set profile42 ruleset-failure-domain=osd k=4 m=2
> ceph osd crush rule create-erasure ecruleset42 profile42
> ceph osd pool create testec-42 20 20 erasure profile42 ecruleset42
>
> ceph osd erasure-code-profile set profile43 ruleset-failure-domain=osd k=4 m=3
> ceph osd crush rule create-erasure ecruleset43 profile43
> ceph osd pool create testec-43 20 20 erasure profile43 ecruleset43
>
> ceph osd erasure-code-profile set profile52 ruleset-failure-domain=osd k=5 m=2
> ceph osd crush rule create-erasure ecruleset52 profile52
> ceph osd pool create testec-52 20 20 erasure profile52 ecruleset52
>
> ceph osd erasure-code-profile set profile53 ruleset-failure-domain=osd k=5 m=3
> ceph osd crush rule create-erasure ecruleset53 profile53
> ceph osd pool create testec-53 20 20 erasure profile53 ecruleset53
>
> ceph osd erasure-code-profile set profile62 ruleset-failure-domain=osd k=6 m=2
> ceph osd crush rule create-erasure ecruleset62 profile62
> ceph osd pool create testec-62 20 20 erasure profile62 ecruleset62
>
> ceph osd erasure-code-profile set profile63 ruleset-failure-domain=osd k=6 m=3
> ceph osd crush rule create-erasure ecruleset63 profile63
> ceph osd pool create testec-63 20 20 erasure profile63 ecruleset63
>
> The CRUSH rulesets all look the same (modulo the name and ids):
>
> { "rule_id": 1,
> "rule_name": "ecruleset22",
> "ruleset": 1,
> "type": 3,
> "min_size": 3,
> "max_size": 20,
> "steps": [
> { "op": "set_chooseleaf_tries",
> "num": 5},
> { "op": "take",
> "item": -1,
> "item_name": "default"},
> { "op": "choose_indep",
> "num": 0,
> "type": "osd"},
> { "op": "emit"}]},
>
> Same with EC profiles:
>
> directory=/usr/lib64/ceph/erasure-code
> k=3
> m=3
> plugin=jerasure
> ruleset-failure-domain=osd
> technique=reed_sol_van
>
> Thanks,
> -mike
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Loïc Dachary, Artisan Logiciel Libre
Attachment:
signature.asc
Description: OpenPGP digital signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
