Hi,
--
another possibility - the osd's "refusing connections" crashed, there's a window of time where connection attempts will fail with connection refused, in between osd died, the osd being re-started by upstart/systemd, and the OSD gets far enough into it's init process to start listening for new connections.
While your symptoms look the same, there's no guarantee that you're suffering from the same problem, but.. we're currently suffering from ceph-osd v12.2.4 sporadically segfaulting. Either for config reasons or the signal handler fails to do it's thing, we don't get the typical "oops I crashed" reports in the osd log, although journald/systemd did capture stdout which mentions it and there's a kernel log message left behind saying that ceph-osd segfaulted. (http://tracker.ceph.com/issues/23352 ).
-KJ
On Wed, Mar 28, 2018 at 10:50 AM, Andre Goree <andre@xxxxxxxxxx> wrote:
On 2018/03/28 1:39 pm, Subhachandra Chandra wrote:
We have seen similar behavior when there are network issues. AFAIK, the OSD is being reported down by an OSD that cannot reach it. But either another OSD that can reach it or the heartbeat between the OSD and the monitor declares it up. The OSD "boot" message does not seem to indicate an actual OSD restart....
Subhachandra
On Wed, Mar 28, 2018 at 10:30 AM, Andre Goree <andre@xxxxxxxxxx> wrote:
Hello,
I've recently had a minor issue come up where random individual OSDs are failed due to a connection refused on another OSD. I say minor, bc it's not a node-wide issue, and appears to be random nodes -- and besides that, the OSD comes up within less than a second, as if the OSD is sent a "restart," or something.
Great! Thank you! Yes I found it funny that it "restarted" so quickly, and from my readings I remember that it takes more than a single OSD heartbeat failing to produce and _actual_ failure, so as to prevent false positives. Thanks for the insight!
--
Andre Goree
-=-=-=-=-=-
Email - andre at drenet.net
Website - http://blog.drenet.net
PGP key - http://www.drenet.net/pubkey.html
-=-=-=-=-=-
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Kjetil Joergensen <kjetil@xxxxxxxxxxxx>
SRE, Medallia Inc
SRE, Medallia Inc
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
