Hi,
I am evaluating ceph cluster for a solution where ceph could be used for provisioning
pools which could be either stored local to a node or replicated across a cluster. This
way ceph could be used as single point of solution for writing both local as well as replicated
data. Local storage helps avoid possible storage cost that comes with replication factor of more
than one and also provide availability as long as the data host is alive.
So I tried an experiment with Ceph cluster where there is one crush rule which replicates data across
nodes and other one only points to a crush bucket that has local ceph osd. Cluster configuration
is pasted below.
Here I observed that if one of the disk is full (95%) entire cluster goes into error state and stops
accepting new writes from/to other nodes. So ceph cluster became unusable even though it’s only
32% full. The writes are blocked even for pools which are not touching the full osd.
I have tried playing around crush hierarchy but it did not help. So is it possible to store data in the above
manner with Ceph ? If yes could we get cluster state in usable state after one of the node is full ?
# ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
134G 94247M 43922M 31.79
# ceph –s
cluster ba658a02-757d-4e3c-7fb3-dc4bf944322f
health HEALTH_ERR
1 full osd(s)
full,sortbitwise,require_jewel_osds flag(s) set
monmap e3: 3 mons at {ip-10-0-9-122=10.0.9.122:6789/0,ip-10-0-9-146=10.0.9.146:6789/0,ip-10-0-9-210=10.0.9.210:6789/0}
election epoch 14, quorum 0,1,2 ip-10-0-9-122,ip-10-0-9-146,ip-10-0-9-210
osdmap e93: 3 osds: 3 up, 3 in
flags full,sortbitwise,require_jewel_osds
pgmap v630: 384 pgs, 6 pools, 43772 MB data, 18640 objects
43922 MB used, 94247 MB / 134 GB avail
384 active+clean
# ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-9 0.04399 rack ip-10-0-9-146-rack
-8 0.04399 host ip-10-0-9-146
2 0.04399 osd.2 up 1.00000 1.00000
-7 0.04399 rack ip-10-0-9-210-rack
-6 0.04399 host ip-10-0-9-210
1 0.04399 osd.1 up 1.00000 1.00000
-5 0.04399 rack ip-10-0-9-122-rack
-3 0.04399 host ip-10-0-9-122
0 0.04399 osd.0 up 1.00000 1.00000
-4 0.13197 rack rep-rack
-3 0.04399 host ip-10-0-9-122
0 0.04399 osd.0 up 1.00000 1.00000
-6 0.04399 host ip-10-0-9-210
1 0.04399 osd.1 up 1.00000 1.00000
-8 0.04399 host ip-10-0-9-146
2 0.04399 osd.2 up 1.00000 1.00000
# ceph osd crush rule list
[
"rep_ruleset",
"ip-10-0-9-122_ruleset",
"ip-10-0-9-210_ruleset",
"ip-10-0-9-146_ruleset"
]
# ceph osd crush rule dump rep_ruleset
{
"rule_id": 0,
"rule_name": "rep_ruleset",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -4,
"item_name": "rep-rack"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
# ceph osd crush rule dump ip-10-0-9-122_ruleset
{
"rule_id": 1,
"rule_name": "ip-10-0-9-122_ruleset",
"ruleset": 1,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -5,
"item_name": "ip-10-0-9-122-rack"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
--
