Re: Continuous error: "libceph: monX session lost, hunting for new mon" on one host

["Marco Baldini - H.S. Amiata" <mbaldini@xxxxxxxxxxx>]

Thanks for reply

My ceph.conf:

[global]
         auth client required = none
         auth cluster required = none
         auth service required = none
         bluestore_block_db_size = 64424509440
         cluster network = 10.10.10.0/24
         fsid = 24d5d6bc-0943-4345-b44e-46c19099004b
         keyring = /etc/pve/priv/$cluster.$name.keyring
         mon allow pool delete = true
         osd journal size = 5120
         osd pool default min size = 2
         osd pool default size = 3
         public network = 10.10.10.0/24

[client]
         rbd cache = true
         rbd cache max dirty = 134217728
         rbd cache max dirty age = 2
         rbd cache size = 268435456
         rbd cache target dirty = 67108864
         rbd cache writethrough until flush = true

[osd]
         keyring = /var/lib/ceph/osd/ceph-$id/keyring

[mon.pve-hs-3]
         host = pve-hs-3
         mon addr = 10.10.10.253:6789

[mon.pve-hs-main]
         host = pve-hs-main
         mon addr = 10.10.10.251:6789

[mon.pve-hs-2]
         host = pve-hs-2
         mon addr = 10.10.10.252:6789

Each node has two ethernet cards in LACP bond on network 10.10.10.x

auto bond0
iface bond0 inet static
        address  10.10.10.252
        netmask  255.255.255.0
        slaves enp4s0 enp4s1
        bond_miimon 100
        bond_mode 802.3ad
        bond_xmit_hash_policy layer3+4
#CLUSTER BOND

The LAG on switch (TPLink TL-SG2008) is enabled, I see from "show run"

#
interface gigabitEthernet 1/0/1

  channel-group 4 mode active
#
interface gigabitEthernet 1/0/2

  channel-group 4 mode active
#
interface gigabitEthernet 1/0/3

  channel-group 2 mode active
#
interface gigabitEthernet 1/0/4

  channel-group 2 mode active
#
interface gigabitEthernet 1/0/5

  channel-group 3 mode active
#
interface gigabitEthernet 1/0/6

  channel-group 3 mode active
#
interface gigabitEthernet 1/0/7

#
interface gigabitEthernet 1/0/8

Node 1 is on port 1 and 2, node 2 on port 3 and 4, node 3 on port 5 and 6

Routing table, show with "ip -4 route show  table all"

default via 192.168.2.1 dev vmbr0 onlink
10.10.10.0/24 dev bond0 proto kernel scope link src 10.10.10.252
192.168.1.0/24 dev vmbr1 proto kernel scope link src 192.168.1.252 linkdown
192.168.2.0/24 dev vmbr0 proto kernel scope link src 192.168.2.252
broadcast 10.10.10.0 dev bond0 table local proto kernel scope link src 10.10.10.252
local 10.10.10.252 dev bond0 table local proto kernel scope host src 10.10.10.252
broadcast 10.10.10.255 dev bond0 table local proto kernel scope link src 10.10.10.252
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 192.168.1.0 dev vmbr1 table local proto kernel scope link src 192.168.1.252 linkdown
local 192.168.1.252 dev vmbr1 table local proto kernel scope host src 192.168.1.252
broadcast 192.168.1.255 dev vmbr1 table local proto kernel scope link src 192.168.1.252 linkdown
broadcast 192.168.2.0 dev vmbr0 table local proto kernel scope link src 192.168.2.252
local 192.168.2.252 dev vmbr0 table local proto kernel scope host src 192.168.2.252
broadcast 192.168.2.255 dev vmbr0 table local proto kernel scope link src 192.168.2.252

Network configuration

$ ip -4 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
6: vmbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    inet 192.168.1.252/24 brd 192.168.1.255 scope global vmbr1
       valid_lft forever preferred_lft forever
7: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 10.10.10.252/24 brd 10.10.10.255 scope global bond0
       valid_lft forever preferred_lft forever
8: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.2.252/24 brd 192.168.2.255 scope global vmbr0
       valid_lft forever preferred_lft forever

$ ip -4 link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether 40:8d:5c:b0:2d:fe brd ff:ff:ff:ff:ff:ff
3: enp4s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 98:de:d0:1d:75:4a brd ff:ff:ff:ff:ff:ff
4: enp4s1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 98:de:d0:1d:75:4a brd ff:ff:ff:ff:ff:ff
6: vmbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
7: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 98:de:d0:1d:75:4a brd ff:ff:ff:ff:ff:ff
8: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 40:8d:5c:b0:2d:fe brd ff:ff:ff:ff:ff:ff
9: tap104i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether b2:47:55:9f:d3:0b brd ff:ff:ff:ff:ff:ff
11: veth103i0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether fe:03:27:0d:02:38 brd ff:ff:ff:ff:ff:ff link-netnsid 0
13: veth106i0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether fe:ce:4f:09:24:45 brd ff:ff:ff:ff:ff:ff link-netnsid 1
14: tap109i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 3a:f0:99:3f:6a:75 brd ff:ff:ff:ff:ff:ff
15: tap201i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 16:99:8a:56:6d:7f brd ff:ff:ff:ff:ff:ff

I think it's everything.

Thanks

Il 23/10/2017 15:42, Denes Dolhay ha scritto:

Hi,

Maybe some routing issue?

"CEPH has public and cluster network on 10.10.10.0/24"

This means that the nodes have public and cluster network separately both on 10.10.10.0/24, or that you did not specify a separate cluster network?

Please provide route table, ifconfig, ceph.conf

Regards,

Denes

On 10/23/2017 03:35 PM, Marco Baldini - H.S. Amiata wrote:

Hello

I have a CEPH cluster with 3 nodes, each with 3 OSDs, running Proxmox, CEPH  versions:

{
    "mon": {
        "ceph version 12.2.1 (1a629971a9bcaaae99e5539a3a43f800a297f267) luminous (stable)": 3
    },
    "mgr": {
        "ceph version 12.2.1 (1a629971a9bcaaae99e5539a3a43f800a297f267) luminous (stable)": 3
    },
    "osd": {
        "ceph version 12.2.1 (1a629971a9bcaaae99e5539a3a43f800a297f267) luminous (stable)": 9
    },
    "mds": {},
    "overall": {
        "ceph version 12.2.1 (1a629971a9bcaaae99e5539a3a43f800a297f267) luminous (stable)": 15
    }
}

CEPH has public and cluster network on 10.10.10.0/24, the three nodes are 10.10.10.251, 10.10.10.252, 10.10.10.253 and networking is working good (I kept ping from one of the nodes to the others two running for hours and had 0 packet loss)

On one node with ip 10.10.10.252 I get strange message in dmesg

kern  :info  : [Oct23 14:42] libceph: mon2 10.10.10.253:6789 session lost, hunting for new mon
kern  :info  : [  +0.000391] libceph: mon1 10.10.10.252:6789 session established
kern  :info  : [ +30.721869] libceph: mon1 10.10.10.252:6789 session lost, hunting for new mon
kern  :info  : [  +0.000749] libceph: mon2 10.10.10.253:6789 session established
kern  :info  : [Oct23 14:43] libceph: mon2 10.10.10.253:6789 session lost, hunting for new mon
kern  :info  : [  +0.000312] libceph: mon1 10.10.10.252:6789 session established
kern  :info  : [ +30.721964] libceph: mon1 10.10.10.252:6789 session lost, hunting for new mon
kern  :info  : [  +0.000730] libceph: mon0 10.10.10.251:6789 session established                                                                
kern  :info  : [Oct23 14:44] libceph: mon0 10.10.10.251:6789 session lost, hunting for new mon
kern  :info  : [  +0.000330] libceph: mon1 10.10.10.252:6789 session established                                                                
kern  :info  : [ +30.721899] libceph: mon1 10.10.10.252:6789 session lost, hunting for new mon
kern  :info  : [  +0.000951] libceph: mon0 10.10.10.251:6789 session established                                                                
kern  :info  : [Oct23 14:45] libceph: mon0 10.10.10.251:6789 session lost, hunting for new mon
kern  :info  : [  +0.000733] libceph: mon2 10.10.10.253:6789 session established                                                                
kern  :info  : [ +30.721529] libceph: mon2 10.10.10.253:6789 session lost, hunting for new mon
kern  :info  : [  +0.000328] libceph: mon1 10.10.10.252:6789 session established
kern  :info  : [Oct23 14:46] libceph: mon1 10.10.10.252:6789 session lost, hunting for new mon
kern  :info  : [  +0.001035] libceph: mon0 10.10.10.251:6789 session established
kern  :info  : [ +30.721183] libceph: mon0 10.10.10.251:6789 session lost, hunting for new mon
kern  :info  : [  +0.004221] libceph: mon1 10.10.10.252:6789 session established
kern  :info  : [Oct23 14:47] libceph: mon1 10.10.10.252:6789 session lost, hunting for new mon
kern  :info  : [  +0.000927] libceph: mon0 10.10.10.251:6789 session established
kern  :info  : [ +30.721361] libceph: mon0 10.10.10.251:6789 session lost, hunting for new mon
kern  :info  : [  +0.000524] libceph: mon1 10.10.10.252:6789 session established

and that is going on all the day.

In ceph -w I get

2017-10-23 14:51:57.941131 mon.pve-hs-main [INF] mon.2 10.10.10.253:6789/0
2017-10-23 14:56:57.941433 mon.pve-hs-main [INF] mon.2 10.10.10.253:6789/0
2017-10-23 14:56:58.124457 mon.pve-hs-main [INF] mon.1 10.10.10.252:6789/0
2017-10-23 15:00:00.000184 mon.pve-hs-main [INF] overall HEALTH_OK
2017-10-23 15:01:57.941312 mon.pve-hs-main [INF] mon.1 10.10.10.252:6789/0
2017-10-23 15:01:57.941558 mon.pve-hs-main [INF] mon.2 10.10.10.253:6789/0
2017-10-23 15:06:57.941420 mon.pve-hs-main [INF] mon.1 10.10.10.252:6789/0
2017-10-23 15:06:57.941544 mon.pve-hs-main [INF] mon.2 10.10.10.253:6789/0
2017-10-23 15:11:57.941573 mon.pve-hs-main [INF] mon.1 10.10.10.252:6789/0
2017-10-23 15:11:57.941659 mon.pve-hs-main [INF] mon.2 10.10.10.253:6789/0

pve-hs-main is the host with ip 10.10.10.251

Actually CEPH storage is very low on usage, on average 200 kB/s read or write (as shown with ceph -s) so I don't think it's a problem about load average of the cluster.

The strange is that I see mon1 10.10.10.252:6789 session lost and that's from log of node 10.10.10.252 so it's losing connection with the monitor on the same node, I don't think it's network related.

I already tried with nodes reboot, ceph-mon and ceph-mgr restart, but the problem is still there.

Any ideas?

Thanks

--

Marco Baldini
H.S. Amiata Srl
Ufficio:	0577-779396
Cellulare:	335-8765169
WEB:	www.hsamiata.it
EMAIL:	mbaldini@xxxxxxxxxxx

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

--

Marco Baldini
H.S. Amiata Srl
Ufficio:	0577-779396
Cellulare:	335-8765169
WEB:	www.hsamiata.it
EMAIL:	mbaldini@xxxxxxxxxxx

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com