|
Hi, For example, An OSD (say OSD 1) is holding 100 PGs, especially for some PGs, say PG 1, OSD1 is the primary OSD of PG1, then OSD1 need to peer with all other OSDs in PG1’s acting set
and up set(basically you could think these two sets are other replications for PG1).
So if the cluster with very simple(default) ruleset, it’s possible that an OSD need to peer with all other OSDs.
2.
OSD will randomly select a mon when the OSD boot up , and talking to the mon consistently. It’s the monitor quorum’s job to reach an agreement about the OSD status. See paxos if you want to know more
details in how to reach the agreement.
3.
OSD do ping with Mon, but in reality, the network between monitor and osd likely not the networking between OSD’s . As an instance, Mon <-> OSD is in management network bug OSD<-> OSD is in 10Gb data
network. So only ping with Mon is not enough. Actually there are heartbeat on both public and cluster network, just use to ensure the network connectivity. Xiaoxi From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx]
On Behalf Of Liu, Ming (HPIT-GADSC) Hi, all, I am new to study Ceph. Trying to understand how it works and designs. One basic question for me is about how Ceph OSD perform failure detection. I did some searching but cannot get satisfied answer, so try to ask here and hope someone can kindly help me. The document said, OSD will send heartbeat to other OSD and report failure to Monitor when it detect some other OSD is down. My questions are 1. One particular OSD will heartbeat to how many other OSD? Is it possible for one OSD to do heartbeat to most of other OSDs or even all other OSDs in the cluster? In other words, how OSD decide the list of OSDs
that it needs to check for health? 2. If OSD detect a failure, to which Monitor it report to? This selection is random or has some rule? Or it needs to report to multiple Monitors? 3. Why not each OSD directly do heartbeat with a monitor?
If I get question 1 answered that one OSD maybe need to checking all other OSDs in the cluster, and if this is true for many OSDs in the cluster, it looks like a lot of network traffic and redundant.
Say, OSD-1 check OSD-2, OSD-3, OSD-4, while OSD-2 also check OSD-3, OSD-4, OSD-5. Then both OSD1,2 do redundant health checking for OSD-4,5. If answer of question 1 is: one OSD only need to do heartbeat with very few other OSDs, never has possibility to check most of other OSDs, then I am fine, this will decentralize the monitor for
health checking. But I want to know the rule of how OSD decide which other OSD it needs to check to further understand this. I read almost all articles on internet I can find up to now, but still cannot get very satisfied answer. I don’t want to dive into source code yet, that may take a long time for me. I want to first understand the
principles. Then decide if I really worth to spend huge time to read src code. So really want someone can help me here. Any help will be very appreciated!! Thanks, Ming |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
