Is your network up/up on all nodes? What about your mgr daemon? Try to recover first the daemons, also, can you run a status using the asok file? On Sun, Mar 13, 2022, 11:39 AM GoZippy <gotadvantage@xxxxxxxxx> wrote: > Using 9 node cluster on Proxmox > Ceph was updated automatically with system update > When rebooted nodes I did not set noout and just rebooted nodes - might be > root cause of all the rebalancing and lost connectivity... > > I see monitors active (running) with systemctl status ceph-mon@ > <nodename>.service > > I see my HDD physical disks still assigned as OSD's 1-14 or whatever I have > > Ceph -s hangs > Ceph commands pretty much all hang. Using system control I can turn > monitors off and stop all services - I think... > > sudo systemctl stop ceph\*.service ceph\*.target > > seems to have stopped the services on all nodes... restarting does not > help. > > /var/log/ceph is full of logs as expected (not sure where to start with > them to look at issues) > > Pools and MDS do not show up anymore -maybe got purged or deleted... > > Wondering if there is a way to undelete maps and pool data from debian host > or use ceph tools to make the same name file system, rebuild the map and > not lose data on the ODS stores... > > SEE the following posts I was posting screenshots and logs and trying to > sort it out but got nowhere... came here to list for last resort before I > nuke it all and start over.. would like to save some VM's I have on those > OSD's. > > > https://forum.proxmox.com/threads/ceph-not-working-monitors-and-managers-lost.100672/page-2 > > I read on > > https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/message/K5X6DPSUHMPZ3P7ADV64B4YLPQPWQS5J/ > > it might be possible to just overwrite the name and let the system "heal" > but seems risky and I am not not exactly sure what command to use for > making the same name file system and setting up the same metadata info > too... where can I go about looking in logs or proxmox or otherwise to make > sure I do use the correct names if this is an option - or is there any > better documented way to recover a lost filesystem and restore quorum? > > Home-brew and learning as I go.. so forgive the lack of expertise here - I > am learning. > > Respectfully, > > William Henderson > 316-518-9350 > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |