Re: pgs stuck in creating+peering state

Bryan Stillwell <bstillwell@xxxxxxxxxxx> · Thu, 17 Jan 2019 16:41:14 +0000

Since you're using jumbo frames, make sure everything between the nodes properly supports them (nics & switches).  I've tested this in the past by using the size option in ping (you need to use  a payload size of 8972 instead of 9000 to
 account for the 28 byte header):

ping -s 8972 192.168.160.237

If that works, then you'll need to pull out tcpdump/wireshark to determine why the packets aren't able to return.

Bryan

From: ceph-users <ceph-users-bounces@xxxxxxxxxxxxxx> on behalf of Johan Thomsen <write@xxxxxxxxxx>

Date: Thursday, January 17, 2019 at 5:42 AM

To: Kevin Olbrich <ko@xxxxxxx>

Cc: ceph-users <ceph-users@xxxxxxxxxxxxxx>

Subject: Re: [ceph-users] pgs stuck in creating+peering state

Thanks you for responding!

First thing: I disabled the firewall on all the nodes.

More specifically not firewalld, but the NixOS firewall, since I run NixOS.

I can netcat both udp and tcp traffic on all ports between all nodes

without problems.

Next, I tried raising the mtu to 9000 on the nics where the cluster

network is connected - although I don't see why the mtu should affect

the heartbeat?

I have two bonded nics connected to the cluster network (mtu 9000) and

two separate bonded nics hooked on the public network (mtu 1500).

I've tested traffic and routing on both pairs of nics and traffic gets

through without issues, apparently.

None of the above solved the problem :-(

Den tor. 17. jan. 2019 kl. 12.01 skrev Kevin Olbrich <ko@xxxxxxx>:

Are you sure, no service like firewalld is running?

Did you check that all machines have the same MTU and jumbo frames are

enabled if needed?

I had this problem when I first started with ceph and forgot to

disable firewalld.

Replication worked perfectly fine but the OSD was kicked out every few seconds.

Kevin

Am Do., 17. Jan. 2019 um 11:57 Uhr schrieb Johan Thomsen <write@xxxxxxxxxx>:

> 

> Hi,

> 

> I have a sad ceph cluster.

> All my osds complain about failed reply on heartbeat, like so:

> 

> osd.10 635 heartbeat_check: no reply from 192.168.160.237:6810 osd.42

> ever on either front or back, first ping sent 2019-01-16

> 22:26:07.724336 (cutoff 2019-01-16 22:26:08.225353)

> 

> .. I've checked the network sanity all I can, and all ceph ports are

> open between nodes both on the public network and the cluster network,

> and I have no problems sending traffic back and forth between nodes.

> I've tried tcpdump'ing and traffic is passing in both directions

> between the nodes, but unfortunately I don't natively speak the ceph

> protocol, so I can't figure out what's going wrong in the heartbeat

> conversation.

> 

> Still:

> 

> # ceph health detail

> 

> HEALTH_WARN nodown,noout flag(s) set; Reduced data availability: 1072

> pgs inactive, 1072 pgs peering

> OSDMAP_FLAGS nodown,noout flag(s) set

> PG_AVAILABILITY Reduced data availability: 1072 pgs inactive, 1072 pgs peering

>     pg 7.3cd is stuck inactive for 245901.560813, current state

> creating+peering, last acting [13,41,1]

>     pg 7.3ce is stuck peering for 245901.560813, current state

> creating+peering, last acting [1,40,7]

>     pg 7.3cf is stuck peering for 245901.560813, current state

> creating+peering, last acting [0,42,9]

>     pg 7.3d0 is stuck peering for 245901.560813, current state

> creating+peering, last acting [20,8,38]

>     pg 7.3d1 is stuck peering for 245901.560813, current state

> creating+peering, last acting [10,20,42]

>    (....)

> 

> 

> I've set "noout" and "nodown" to prevent all osd's from being removed

> from the cluster. They are all running and marked as "up".

> 

> # ceph osd tree

> 

> ID  CLASS WEIGHT    TYPE NAME                          STATUS REWEIGHT PRI-AFF

>  -1       249.73434 root default

> -25       166.48956     datacenter m1

> -24        83.24478         pod kube1

> -35        41.62239             rack 10

> -34        41.62239                 host ceph-sto-p102

>  40   hdd   7.27689                     osd.40             up  1.00000 1.00000

>  41   hdd   7.27689                     osd.41             up  1.00000 1.00000

>  42   hdd   7.27689                     osd.42             up  1.00000 1.00000

>    (....)

> 

> I'm at a point where I don't know which options and what logs to check anymore?

> 

> Any debug hint would be very much appreciated.

> 

> btw. I have no important data in the cluster (yet), so if the solution

> is to drop all osd and recreate them, it's ok for now. But I'd really

> like to know how the cluster ended in this state.

> 

> /Johan

> _______________________________________________

> ceph-users mailing list

> ceph-users@xxxxxxxxxxxxxx

> 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________

ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com