If you try shuffling some daemon around on some of the working hosts (e.g. changing the placement of the node-exporter spec so that one of the working hosts is excluded so the node-exporter there should be removed) is cephadm able to actually complete that? Also, does device info for any or all of these hosts show up in `ceph orch device ls`? I know there's been an issue people have run into occasionally where ceph-volume inventory (which cephadm uses to gather device info) was hanging on one of the hosts and it was causing things to get stuck until the host was removed or whatever caused the hang was fixed. There could also be something interesting/useful in the output of `ceph log last 200 cephadm`, `ceph orch host ls` and `ceph orch ls --format yaml`. Some traceback or even just seeing what the last thing logged was could be useful. On Thu, Aug 18, 2022 at 9:31 AM <ceph-mail@xxxxxxxxxxxxxxxx> wrote: > Hi again all, > > I have a new issue with ceph/cephadm/quincy and hopefully someone can > assist. > > I have a cluster of four hosts, that I (finally) managed to bootstrap. I'm > now trying to add several additional hosts. Whether I add the hosts from > the dashboard or CLI, I get the same result - the host is added but no > services are deployed. > > I have tried > > * confirming that the ssh connection works > * enabled debug logging of cephadm and I'm watching the output > * when adding the host, cephadm logs in to the host and runs check-host > * the debug output show error on podman 3.0.1 but it's the same > version as on all the working hosts > * it confirms systemctl, lvcreate, chrony.service, hostname and > concludes Host looks OK > * it outputs Added host > * The host is visible in the list of hosts in the dashboard and ceph > orch host ls, but services are deployed so there is no data under model, > CPUs etc > * I have tried to edit and save the node-exporter and crash services > (both have * for placement) > * I have tried to redeploy the node-exporter service, they just get > redeployed to the existing four hosts > * I have tried ceph orch pause and then ceph orch resume > * I have tried to put the host in maintenance mode and exit the > maintenance mode > * I tried to restart all the mons and mgrs - when restarting the > active mgr, cephadm finally did the inventory of the host I added. When I > added the additional hosts, nothing happened until I restarted the mgr > again. The service size for crash and node-exporter increased to 8, but > still no services are deployed and the running number remains at 4. > > I could see no error anywhere (except the note about podman 3.0.1) and I > didn't have this problem when adding the first three hosts after the > bootstrap. > > Ideas? > > Thanks > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |