I have a small cluster (4 nodes, 15 OSDs, 3-5 OSDs per node) running bits
from the firefly branch (0.78-430-gb8ea656).
I am trying out various k/m combinations for EC pools. Certain k/m
combinations are causing rados put to fail on the second 4MB chunk. I
realize some of these combinations might not make much sense given the
size of my cluster, but I am curious if I am seeing errors because of the
size of the cluster or due to some other issue (especially k=3/m=3). I
have double checked that the failure domain is set to osd and not host
(see below).
++ rados -p testec-22 put xyz sampledata
++ rados -p testec-33 put xyz sampledata
error putting testec-33/xyz: Operation not supported
++ rados -p testec-42 put xyz sampledata
++ rados -p testec-43 put xyz sampledata
++ rados -p testec-52 put xyz sampledata
error putting testec-52/xyz: Operation not supported
++ rados -p testec-53 put xyz sampledata
error putting testec-53/xyz: Operation not supported
++ rados -p testec-62 put xyz sampledata
error putting testec-62/xyz: Operation not supported
++ rados -p testec-63 put xyz sampledata
error putting testec-63/xyz: Operation not supported
Error from OSD to client:
2014-03-29 16:08:14.024912 7f6556d13700 1 -- 198.18.32.12:0/1011811 <==
osd.3 198.18.32.15:6800/38109 2 ==== osd_op_reply(2 xyz [write
4194304~4194304] v0'0 uv0 ondisk = -95 (Operation not supported)) v6 ====
170+0+0 (469808551 0 0) 0x7f65300025e0 con 0x2451db0
Configuration for the pools:
ceph osd erasure-code-profile set profile22 ruleset-failure-domain=osd k=2 m=2
ceph osd crush rule create-erasure ecruleset22 profile22
ceph osd pool create testec-22 20 20 erasure profile22 ecruleset22
ceph osd erasure-code-profile set profile33 ruleset-failure-domain=osd k=3 m=3
ceph osd crush rule create-erasure ecruleset33 profile33
ceph osd pool create testec-33 20 20 erasure profile33 ecruleset33
ceph osd erasure-code-profile set profile42 ruleset-failure-domain=osd k=4 m=2
ceph osd crush rule create-erasure ecruleset42 profile42
ceph osd pool create testec-42 20 20 erasure profile42 ecruleset42
ceph osd erasure-code-profile set profile43 ruleset-failure-domain=osd k=4 m=3
ceph osd crush rule create-erasure ecruleset43 profile43
ceph osd pool create testec-43 20 20 erasure profile43 ecruleset43
ceph osd erasure-code-profile set profile52 ruleset-failure-domain=osd k=5 m=2
ceph osd crush rule create-erasure ecruleset52 profile52
ceph osd pool create testec-52 20 20 erasure profile52 ecruleset52
ceph osd erasure-code-profile set profile53 ruleset-failure-domain=osd k=5 m=3
ceph osd crush rule create-erasure ecruleset53 profile53
ceph osd pool create testec-53 20 20 erasure profile53 ecruleset53
ceph osd erasure-code-profile set profile62 ruleset-failure-domain=osd k=6 m=2
ceph osd crush rule create-erasure ecruleset62 profile62
ceph osd pool create testec-62 20 20 erasure profile62 ecruleset62
ceph osd erasure-code-profile set profile63 ruleset-failure-domain=osd k=6 m=3
ceph osd crush rule create-erasure ecruleset63 profile63
ceph osd pool create testec-63 20 20 erasure profile63 ecruleset63
The CRUSH rulesets all look the same (modulo the name and ids):
{ "rule_id": 1,
"rule_name": "ecruleset22",
"ruleset": 1,
"type": 3,
"min_size": 3,
"max_size": 20,
"steps": [
{ "op": "set_chooseleaf_tries",
"num": 5},
{ "op": "take",
"item": -1,
"item_name": "default"},
{ "op": "choose_indep",
"num": 0,
"type": "osd"},
{ "op": "emit"}]},
Same with EC profiles:
directory=/usr/lib64/ceph/erasure-code
k=3
m=3
plugin=jerasure
ruleset-failure-domain=osd
technique=reed_sol_van
Thanks,
-mike
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
