ceph-mgr daemon can't be enabled on mimic version

["Liu, Changcheng" <changcheng.liu@xxxxxxxxx>]

Hi all,
     I’m trying to enabling ceph-mgr after upgrading ceph from Jewel to Mimic version.
     However, ceph-mgr daemon can’t be enabled after following http://docs.ceph.com/docs/master/mgr/administrator/.
     Could anyone give some info to debug why the damon can’t be enabled?

     Below is my steps:
      1) create an authentication key for the daemon:
             [wrsroot@controller-0 ~(keystone_admin)]$ ceph auth get-or-create mgr.starlingx mon 'allow profile mgr' osd 'allow *'

      2) export mgr key
             [wrsroot@controller-0 ceph(keystone_admin)]$ mkdir /var/lib/ceph/mgr/ceph-starlingx
             [wrsroot@controller-0 ceph(keystone_admin)]$ sudo ceph auth get mgr.starlingx -o /var/lib/ceph/mgr/ceph-starlingx/keyring

      3) place the key into “mgr data” path.
             [wrsroot@controller-0 ~(keystone_admin)]$ cat /etc/ceph/ceph.conf | grep 'mgr data' -B 1 -A 1
             [mgr]
               mgr data = /var/lib/ceph/mgr/ceph-starlingx

       4) start the ceph-mgr daemon:
             [wrsroot@controller-0 ~(keystone_admin)]$ sudo ceph-mgr -i starlingx

       5) ceph-mgr log:
             [wrsroot@controller-0 ceph(keystone_admin)]$ cat ceph-mgr.starlingx.log 
             2018-12-26 18:33:42.321 7fcc9cb0d340  0 ceph version 128-NOTFOUND (ad8d7d96b05a574b831bbb434c77d83682d0404c) mimic (stable), process ceph-mgr, pid 2035345
             2018-12-26 18:33:42.323 7fcc9cb0d340  0 pidfile_write: ignore empty --pid-file
             2018-12-26 18:33:42.400 7fcc9cb0d340  1 mgr[py] Loading python module 'balancer'
             2018-12-26 18:33:42.465 7fcc9cb0d340  1 mgr[py] Loading python module 'dashboard'
             2018-12-26 18:33:43.413 7fcc9cb0d340  1 mgr[py] Loading python module 'hello'
             2018-12-26 18:33:43.457 7fcc9cb0d340  1 mgr[py] Loading python module 'influx'
             [wrsroot@controller-0 ceph(keystone_admin)]$

        6) check cluster status:
             [wrsroot@controller-0 ceph(keystone_admin)]$ ceph -s
               cluster:
                 id:     99909873-e5ff-4270-a90d-358747158f7f
                 health: HEALTH_WARN
                         1/3 mons down, quorum controller-0,controller-1
             
               services:
                 mon: 3 daemons, quorum controller-0,controller-1, out of quorum: storage-0
                 mgr: no daemons active
                 osd: 0 osds: 0 up, 0 in
             
               data:
                 pools:   0 pools, 0 pgs
                 objects: 0  objects, 0 B
                 usage:   0 B used, 0 B / 0 B avail
                 pgs:     
              
             [wrsroot@controller-0 ceph(keystone_admin)]$

B.R.
Changcheng