It was a firewall issue on the controller nodes.After allowing ceph-mgr port in iptables everything is displaying correctly.Thanks to people on IRC.
Thanks alot,On Thu, Dec 21, 2017 at 5:24 PM, kevin parrikar <kevin.parker092@xxxxxxxxx> wrote:
accidently removed mailing list email
++ceph-users
Thanks a lot JC for looking into this issue. I am really out of ideas.
ceph.conf on mgr node which is also monitor node.
[global]
fsid = 06c5c906-fc43-499f-8a6f-6c8e21807acf
mon_initial_members = node-16 node-30 node-31
mon_host = 172.16.1.9 172.16.1.3 172.16.1.11
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
log_to_syslog_level = info
log_to_syslog = True
osd_pool_default_size = 2
osd_pool_default_min_size = 1
osd_pool_default_pg_num = 64
public_network = 172.16.1.0/24
log_to_syslog_facility = LOG_LOCAL0
osd_journal_size = 2048
auth_supported = cephx
osd_pool_default_pgp_num = 64
osd_mkfs_type = xfs
cluster_network = 172.16.1.0/24
osd_recovery_max_active = 1
osd_max_backfills = 1
mon allow pool delete = true
[client]
rbd_cache_writethrough_until_flush = True
rbd_cache = True
[client.radosgw.gateway]
rgw_keystone_accepted_roles = _member_, Member, admin, swiftoperator
keyring = /etc/ceph/keyring.radosgw.gateway
rgw_frontends = fastcgi socket_port=9000 socket_host=127.0.0.1
rgw_socket_path = /tmp/radosgw.sock
rgw_keystone_revocation_interval = 1000000
rgw_keystone_url = http://192.168.1.3:35357
rgw_keystone_admin_token = jaJSmlTNxgsFp1ttq5SuAT1R
rgw_init_timeout = 360000
host = controller3
rgw_dns_name = *.sapiennetworks.com
rgw_print_continue = True
rgw_keystone_token_cache_size = 10
rgw_data = /var/lib/ceph/radosgw
user = www-dataceph auth list
osd.100
key: AQAtZjpaVZOFBxAAwl0yFLdUOidLzPFjv+HnjA==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
osd.101
key: AQA4ZjpaS4wwGBAABwgoXQRc1J8sav4MUkWceQ==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
osd.102
key: AQBDZjpaBS2tEBAAtFiPKBzh8JGi8Nh3PtAGCg==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
client.admin
key: AQD0yXFYflnYFxAAEz/2XLHO/6RiRXQ5HXRAnw==
caps: [mds] allow *
caps: [mgr] allow *
caps: [mon] allow *
caps: [osd] allow *
client.backups
key: AQC0y3FY4YQNNhAAs5fludq0yvtp/JJt7RT4HA==
caps: [mgr] allow r
caps: [mon] allow r
caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=backups, allow rwx pool=volumes
client.bootstrap-mds
key: AQD5yXFYyIxiFxAAyoqLPnxxqWmUr+zz7S+qVQ==
caps: [mgr] allow r
caps: [mon] allow profile bootstrap-mds
client.bootstrap-mgr
key: AQBmOTpaXqHQDhAAyDXoxlPmG9QovfmmUd8gIg==
caps: [mon] allow profile bootstrap-mgr
client.bootstrap-osd
key: AQD0yXFYuGkSIhAAelSb3TCPuXRFoFJTBh7Vdg==
caps: [mgr] allow r
caps: [mon] allow profile bootstrap-osd
client.bootstrap-rbd
key: AQBnOTpafDS/IRAAnKzuI9AYEF81/6mDVv0QgQ==
caps: [mon] allow profile bootstrap-rbd
client.bootstrap-rgw
key: AQD3yXFYxt1mLRAArxOgRvWmmzT9pmsqTLpXKw==
caps: [mgr] allow r
caps: [mon] allow profile bootstrap-rgw
client.compute
key: AQCbynFYRcNWOBAAPzdAKfP21GvGz1VoHBimGQ==
caps: [mgr] allow r
caps: [mon] allow r
caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images, allow rwx pool=compute
client.images
key: AQCyy3FYSMtlJRAAbJ8/U/R82NXvWBC5LmkPGw==
caps: [mgr] allow r
caps: [mon] allow r
caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=images
client.radosgw.gateway
key: AQA3ynFYAYMSAxAApvfe/booa9KhigpKpLpUOA==
caps: [mgr] allow r
caps: [mon] allow rw
caps: [osd] allow rwx
client.volumes
key: AQCzy3FYa3paKBAA9BlYpQ1PTeR770ghVv1jKQ==
caps: [mgr] allow r
caps: [mon] allow r
caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images
mgr.controller2
key: AQAmVTpaA+9vBhAApD3rMs//Qri+SawjUF4U4Q==
caps: [mds] allow *
caps: [mgr] allow *
caps: [mon] allow *
caps: [osd] allow *
mgr.controller3
key: AQByfDparprIEBAAj7Pxdr/87/v0kmJV49aKpQ==
caps: [mds] allow *
caps: [mgr] allow *
caps: [mon] allow *
caps: [osd] allow *Regards,KevinOn Thu, Dec 21, 2017 at 8:10 AM, kevin parrikar <kevin.parker092@xxxxxxxxx> wrote:KevinRegards,Not sure whats wrong in my setupbut still status is same,also mgr.log is being flooded with below errors.Thanks JC,I tried
ceph auth caps client.admin osd 'allow *' mds 'allow *' mon 'allow *' mgr 'allow *'
2017-12-21 02:39:10.622834 7fb40a22b700 0 Cannot get stat of OSD 140
2017-12-21 02:39:10.622835 7fb40a22b700 0 Cannot get stat of OSD 141On Thu, Dec 21, 2017 at 2:37 AM, Jean-Charles Lopez <jelopez@xxxxxxxxxx> wrote:Hi,make sure client.admin user has an MGR cap using ceph auth list. At some point there was a glitch with the update process that was not adding the MGR cap to the client.admin user.JCOn Dec 20, 2017, at 10:02, kevin parrikar <kevin.parker092@xxxxxxxxx> wrote:______________________________KevinRegards,Can some one help me here pleasei deployed ceph-mgr using ceph-deploy gather-keys && ceph-deploy mgr create ,it was successfull but for some reason ceph -s is not showing correct values.i am able to upload/download glance images but ceph -s shows 0kb used and Available and probably because of that cinder create is failing.hi All,I have upgraded the cluster from Hammer to Jewel and to Luminous .
ceph -s
cluster:
id: 06c5c906-fc43-499f-8a6f-6c8e21807acf
health: HEALTH_WARN
Reduced data availability: 6176 pgs inactive
Degraded data redundancy: 6176 pgs unclean
services:
mon: 3 daemons, quorum controller3,controller2,controller1
mgr: controller3(active)
osd: 71 osds: 71 up, 71 in
rgw: 1 daemon active
data:
pools: 4 pools, 6176 pgs
objects: 0 objects, 0 bytes
usage: 0 kB used, 0 kB / 0 kB avail
pgs: 100.000% pgs unknown
6176 unknown_________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
