Standby mgr stopped sending beacons after upgrade to 12.2.8

["Christian Albrecht" <cal@xxxxxxxx>]

Hi all,

after upgrading from 12.2.7 to 12.2.8 the standby mgr instances in my cluster stopped sending beacons.
The service starts and everything seems to work just fine, but after a period of time the mgr disappears.
All of my three mgr daemons are running.


[root@ceph01 ~]# ceph mgr dump
{
    "epoch": 69,
    "active_gid": 1734502,
    "active_name": "ceph01",
    "active_addr": "192.0.2.1:6832/35134",
    "available": true,
    "standbys": [
        {
            "gid": 1734790,
            "name": "ceph02",
            "available_modules": [
                "balancer",
                "dashboard",
                "influx",
                "localpool",
                "prometheus",
                "restful",
                "selftest",
                "status",
                "zabbix"
            ]
        }
    ],
}

Some time later I get this:

[root@ceph01 ~]# ceph mgr dump
{
    "epoch": 69,
    "active_gid": 1734502,
    "active_name": "ceph01",
    "active_addr": "192.0.2.1:6832/35134",
    "available": true,
    "standbys": [],
    ...
}

Here is the log. Before the upgrade the mgr posts a standby beacon every 2 seconds. After the upgrade the message is shown only on startup.

2018-09-13 10:10:19.191188 7f674b736700  1 mgr send_beacon standby
2018-09-13 10:10:21.191565 7f674b736700  1 mgr send_beacon standby
2018-09-13 10:10:23.191952 7f674b736700  1 mgr send_beacon standby
2018-09-13 10:10:25.192320 7f674b736700  1 mgr send_beacon standby
2018-09-13 10:10:27.192695 7f674b736700  1 mgr send_beacon standby
2018-09-13 10:10:29.193071 7f674b736700  1 mgr send_beacon standby
2018-09-13 10:10:29.434679 7f674af35700 -1 received  signal: Terminated from  PID: 1 task name: /usr/lib/systemd/systemd --system --deserialize 22  UID: 0
2018-09-13 10:10:29.434692 7f674af35700 -1 mgr handle_signal *** Got signal Terminated ***
2018-09-13 10:13:12.390340 7f7c56fcb7c0  0 set uid:gid to 167:167 (ceph:ceph)
2018-09-13 10:13:12.390377 7f7c56fcb7c0  0 ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable), process ceph-mgr, pid 2368
2018-09-13 10:13:12.396043 7f7c56fcb7c0  0 pidfile_write: ignore empty --pid-file
2018-09-13 10:13:12.448859 7f7c56fcb7c0  1 mgr send_beacon standby
2018-09-13 10:13:12.466768 7f7c4e11e700  1 mgr init Loading python module 'balancer'
2018-09-13 10:13:12.506115 7f7c4e11e700  1 mgr init Loading python module 'dashboard'
2018-09-13 10:13:12.701486 7f7c4e11e700  1 mgr init Loading python module 'prometheus'
2018-09-13 10:13:12.772187 7f7c4e11e700  1 mgr init Loading python module 'restful'
2018-09-13 10:13:13.282123 7f7c4e11e700  1 mgr init Loading python module 'status'
2018-09-13 10:13:13.390284 7f7c4e11e700  1 mgr load Constructed class from module: dashboard


Let me know I have to provide more information on this.

Thank you for your help.

Best regards,
Christian
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com