On Thu, Jan 17, 2019 at 4:42 AM Johan Thomsen <write@xxxxxxxxxx> wrote:
Thanks you for responding!
First thing: I disabled the firewall on all the nodes.
More specifically not firewalld, but the NixOS firewall, since I run NixOS.
I can netcat both udp and tcp traffic on all ports between all nodes
without problems.
Next, I tried raising the mtu to 9000 on the nics where the cluster
network is connected - although I don't see why the mtu should affect
the heartbeat?
I have two bonded nics connected to the cluster network (mtu 9000) and
two separate bonded nics hooked on the public network (mtu 1500).
I've tested traffic and routing on both pairs of nics and traffic gets
through without issues, apparently.
Try 'osd hearbeat min size = 100' in ceph.conf on all osd nodes and restart, we have seen this in some network
configurtion with mtu size mismatch between ports.
None of the above solved the problem :-(
Den tor. 17. jan. 2019 kl. 12.01 skrev Kevin Olbrich <ko@xxxxxxx>:
>
> Are you sure, no service like firewalld is running?
> Did you check that all machines have the same MTU and jumbo frames are
> enabled if needed?
>
> I had this problem when I first started with ceph and forgot to
> disable firewalld.
> Replication worked perfectly fine but the OSD was kicked out every few seconds.
>
> Kevin
>
> Am Do., 17. Jan. 2019 um 11:57 Uhr schrieb Johan Thomsen <write@xxxxxxxxxx>:
> >
> > Hi,
> >
> > I have a sad ceph cluster.
> > All my osds complain about failed reply on heartbeat, like so:
> >
> > osd.10 635 heartbeat_check: no reply from 192.168.160.237:6810 osd.42
> > ever on either front or back, first ping sent 2019-01-16
> > 22:26:07.724336 (cutoff 2019-01-16 22:26:08.225353)
> >
> > .. I've checked the network sanity all I can, and all ceph ports are
> > open between nodes both on the public network and the cluster network,
> > and I have no problems sending traffic back and forth between nodes.
> > I've tried tcpdump'ing and traffic is passing in both directions
> > between the nodes, but unfortunately I don't natively speak the ceph
> > protocol, so I can't figure out what's going wrong in the heartbeat
> > conversation.
> >
> > Still:
> >
> > # ceph health detail
> >
> > HEALTH_WARN nodown,noout flag(s) set; Reduced data availability: 1072
> > pgs inactive, 1072 pgs peering
> > OSDMAP_FLAGS nodown,noout flag(s) set
> > PG_AVAILABILITY Reduced data availability: 1072 pgs inactive, 1072 pgs peering
> > pg 7.3cd is stuck inactive for 245901.560813, current state
> > creating+peering, last acting [13,41,1]
> > pg 7.3ce is stuck peering for 245901.560813, current state
> > creating+peering, last acting [1,40,7]
> > pg 7.3cf is stuck peering for 245901.560813, current state
> > creating+peering, last acting [0,42,9]
> > pg 7.3d0 is stuck peering for 245901.560813, current state
> > creating+peering, last acting [20,8,38]
> > pg 7.3d1 is stuck peering for 245901.560813, current state
> > creating+peering, last acting [10,20,42]
> > (....)
> >
> >
> > I've set "noout" and "nodown" to prevent all osd's from being removed
> > from the cluster. They are all running and marked as "up".
> >
> > # ceph osd tree
> >
> > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
> > -1 249.73434 root default
> > -25 166.48956 datacenter m1
> > -24 83.24478 pod kube1
> > -35 41.62239 rack 10
> > -34 41.62239 host ceph-sto-p102
> > 40 hdd 7.27689 osd.40 up 1.00000 1.00000
> > 41 hdd 7.27689 osd.41 up 1.00000 1.00000
> > 42 hdd 7.27689 osd.42 up 1.00000 1.00000
> > (....)
> >
> > I'm at a point where I don't know which options and what logs to check anymore?
> >
> > Any debug hint would be very much appreciated.
> >
> > btw. I have no important data in the cluster (yet), so if the solution
> > is to drop all osd and recreate them, it's ok for now. But I'd really
> > like to know how the cluster ended in this state.
> >
> > /Johan
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
