This happens (unfortunately) frequently to me. Look for the active mgr (ceph -s), and go restart the mgr service there (systemctl list-units |grep mgr then systemctl restart NAMEOFSERVICE). This normally resolves that error for me. You can look at the journalctl output and you'll likely see errors similar to this issue: https://tracker.ceph.com/issues/45032 I'm hopeful the PR in that issue will resolve the issue for us: https://github.com/ceph/ceph/pull/35022 See if that works - if not - you may want to look at the logs for the mgr (and potentially other) daemons using journalctl. On Tue, May 19, 2020 at 3:45 PM Gencer W. Genç <gencer@xxxxxxxxxxxxx> wrote: > Hi, > > I was browsing dashboard today. Then suddently it stopped working and i > got 502 errors. I checked via root login and see thet ceph health is down > to WARN. > > I can access all rdb devices and CephFS. They work. All OSDs in server-1 > is up. > > > health: HEALTH_WARN > 1 hosts fail cephadm check > failed to probe daemons or devices > > I even restarted server-1. No luck. > > I'm on server-1. cephadm complains it cannot access to server-1. In basic > term, server-1 cannot access server-1 (192.168.0.1) > > server-1: 192.168.0.1 > server-2: 192.168.0.3 > > $ ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get > mgr/cephadm/ssh_identity_key) root@server-1 > > Success. > > cephadm says server-1 is not reachable. Why cephadm/ceph complain about > this and what happend to my dashboard? I also redirected logs to file. I'm > observing for any weird log input but unfortunately nothing helpful. Where > should I look for? What am i missing? > > > So, when I ran this command: > > $ ceph health detail > > Output was: > > HEALTH_WARN 1 hosts fail cephadm check; failed to probe daemons or devices > [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check > host server-1 failed check: Failed to connect to server-1 (server-1). > Check that the host is reachable and accepts > connections using the cephadm SSH key > you may want to run: > > ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get > mgr/cephadm/ssh_identity_key) root@server-1 > [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices > host server-1 scrape failed: Failed to connect to server-1 > (server-1). Check that the host is reachable and accepts connections using > the cephadm SSH key > you may want to run: > > ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get > mgr/cephadm/ssh_identity_key) root@server-1 > host server-1 ceph-volume inventory failed: Failed to connect > to server-1 (server-1). Check that the host is reachable and accepts > connections using the cephadm SSH key > you may want to run: > > ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get > mgr/cephadm/ssh_identity_key) root@server-1 > > Thanks, > Gencer. > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |