I’m trying to setup a single-node ceph “cluster”. For my test that would be sufficient. Of course it could be that ceph orch isn’t meant to be used on a single node only. So maybe it’s worth to try out three nodes… root@terraformdemo:~# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 0/1 - 7d count:1 crash 0/0 - 7d * grafana ?:3000 0/1 - 7d count:1 mgr 0/2 - 7d count:2 mon 0/5 - 7d count:5 node-exporter ?:9100 0/0 - 7d * osd.all-available-devices 0/0 - 4d * prometheus ?:9095 0/1 - 7d count:1 root@terraformdemo:~# ceph orch ps No daemons reported I’m not sure what this output means. No daemons running at all? According to ceph status at least I have a running mon and a running mgr – and I can access the ceph dashboard! Again I have the impression that ceph orch is not in sync with the underlying layer… From: Yury Kirsanov <y.kirsanov@xxxxxxxxx> Sent: Dienstag, 9. November 2021 14:46 To: Scharfenberg, Carsten <c.scharfenberg@xxxxxxxxxxxxx> Cc: Сергей Процун <prosergey07@xxxxxxxxx>; Zach Heise <heise@xxxxxxxxxxxx>; ceph-users <ceph-users@xxxxxxx> Subject: Re: Re: fresh pacific installation does not detect available disks By the way, what does: ceph orch ls and ceph orch ps show? On Wed, Nov 10, 2021 at 12:45 AM Yury Kirsanov <y.kirsanov@xxxxxxxxx<mailto:y.kirsanov@xxxxxxxxx>> wrote: Hi, Sorry, but I've never encountered such behaviour, I was first testing my CEPH 16.2.6 in a VMWare environment and it was all working fine there, I was able to add hosts via ceph orch and then add OSDs as usual. If I had to restart the deployment process for some reason - I was just wiping HDDs using dd command and it was picking up these HDDs just fine... Not sure how to help you here, but one thing is confusing me a bit - why do you have just one monitor? Maybe this somehow affects ceph orch? Of course this is highly unlikely but still...If you have nodes that can host OSDs you can deploy MONs to them as well, just use: 'ceph orch apply mon --placement="3 <host1> <host2> <host3>" Regards, Yury. On Wed, Nov 10, 2021 at 12:13 AM Scharfenberg, Carsten <c.scharfenberg@xxxxxxxxxxxxx<mailto:c.scharfenberg@xxxxxxxxxxxxx>> wrote: Thanks Yury, ceph-volume always listed these devices as available. But ceph orch does not. They do not seem to exist for ceph orch. Also adding them manually does not help (I’ve tried that before and now again): root@terraformdemo:~# ceph orch daemon add osd 192.168.72.10:/dev/sdc root@terraformdemo:~# ceph orch daemon add osd 192.168.72.10:/dev/sde root@terraformdemo:~# ceph status cluster: id: 655a7a32-3bbf-11ec-920e-000c29da2e6a health: HEALTH_WARN OSD count 0 < osd_pool_default_size 1 services: mon: 1 daemons, quorum terraformdemo (age 4m) mgr: terraformdemo.aylzbb(active, since 4m) osd: 0 osds: 0 up, 0 in (since 6d) data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: before I did that I rebooted the VM. I’ve also tried with the hostname instead of the IP – no difference… Also it’s quite irritating that there are no error messages… -- Carsten From: Yury Kirsanov <y.kirsanov@xxxxxxxxx<mailto:y.kirsanov@xxxxxxxxx>> Sent: Dienstag, 9. November 2021 14:05 To: Scharfenberg, Carsten <c.scharfenberg@xxxxxxxxxxxxx<mailto:c.scharfenberg@xxxxxxxxxxxxx>> Cc: Сергей Процун <prosergey07@xxxxxxxxx<mailto:prosergey07@xxxxxxxxx>>; Zach Heise <heise@xxxxxxxxxxxx<mailto:heise@xxxxxxxxxxxx>>; ceph-users <ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>> Subject: Re: Re: fresh pacific installation does not detect available disks Try to do: ceph orch daemon add osd *<host>*:/dev/sdc And then ceph orch daemon add osd *<host>*:/dev/sde This should succeed as sdc and sde are both marked as available at the moment. Hope this helps! Regards, Yury. On Wed, Nov 10, 2021 at 12:01 AM Yury Kirsanov <y.kirsanov@xxxxxxxxx<mailto:y.kirsanov@xxxxxxxxx>> wrote: By the way, /dev/sdc is now listed as available: Device Path Size rotates available Model name /dev/sdc 20.00 GB True True VMware Virtual S /dev/sde 20.00 GB True True VMware Virtual S On Tue, Nov 9, 2021 at 11:23 PM Scharfenberg, Carsten <c.scharfenberg@xxxxxxxxxxxxx<mailto:c.scharfenberg@xxxxxxxxxxxxx>> wrote: Thanks for your support, guys. Unfortunately I do not know the tool sgdisk. It’s also not available from the standard Debian package repository. So I’ve tried out Yury’s approach to use dd… without success: root@terraformdemo:~# dd if=/dev/zero of=/dev/sdc bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.05943 s, 1.0 GB/s root@terraformdemo:~# dd if=/dev/zero of=/dev/sde bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.618606 s, 1.7 GB/s root@terraformdemo:~# ceph-volume inventory Device Path Size rotates available Model name /dev/sdc 20.00 GB True True VMware Virtual S /dev/sde 20.00 GB True True VMware Virtual S /dev/sda 20.00 GB True False VMware Virtual S /dev/sdb 20.00 GB True False VMware Virtual S /dev/sdd 20.00 GB True False VMware Virtual S root@terraformdemo:~# ceph orch device ls root@terraformdemo:~# Do you have any other ideas? Could it be that ceph is not usable with this kind of virtual harddisk? -- Carsten From: Сергей Процун <prosergey07@xxxxxxxxx<mailto:prosergey07@xxxxxxxxx>> Sent: Donnerstag, 4. November 2021 21:36 To: Yury Kirsanov <y.kirsanov@xxxxxxxxx<mailto:y.kirsanov@xxxxxxxxx>> Cc: Zach Heise <heise@xxxxxxxxxxxx<mailto:heise@xxxxxxxxxxxx>>; Scharfenberg, Carsten <c.scharfenberg@xxxxxxxxxxxxx<mailto:c.scharfenberg@xxxxxxxxxxxxx>>; ceph-users <ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>> Subject: Re: Re: fresh pacific installation does not detect available disks ACHTUNG: Diese Mail kommt von einem EXTERNEN ABSENDER - bitte VORSICHT beim Öffnen von ANHÄNGEN und im Umgang mit LINKS. Hello, I agree with that point. When ceph creates lvm volumes it adds lvm tags to them. Thats how ceph finds that those they are occupied by ceph. So you should remove lvm volumes and even better clean all data on those lvm volumes. Usually its enough to clean just the head of lvm partition where it stores information of the volumes itself. --- Sergey Protsun чт, 4 лист. 2021, 22:29 користувач Yury Kirsanov <y.kirsanov@xxxxxxxxx<mailto:y.kirsanov@xxxxxxxxx>> пише: Hi, You should erase any partitions or LVM groups on the disks and restart OSD hosts so CEPH would be able to detect drives. I usually just do 'dd if=/dev/zero of=/dev/<sd*> bs=1M count=1024' and then reboot host to make sure it will definitely be clean. Or, alternatively, you can zap the drives, or you can just remove LVM groups using pvremove or remove patitions using fdisk. Regards, Yury. On Fri, 5 Nov 2021, 07:24 Zach Heise, <heise@xxxxxxxxxxxx<mailto:heise@xxxxxxxxxxxx>> wrote: > Hi Carsten, > > When I had problems on my physical hosts (recycled systems that we wanted > to > just use in a test cluster) I found that I needed to use sgdisk --zap-all > /dev/sd{letter} to clean all partition maps off the disks before ceph would > recognize them as available. Worth a shot in your case, even though as > fresh > virtual volumes they shouldn't have anything on them (yet) anyway. > > -----Original Message----- > From: Scharfenberg, Carsten <c.scharfenberg@xxxxxxxxxxxxx<mailto:c.scharfenberg@xxxxxxxxxxxxx>> > Sent: Thursday, November 4, 2021 12:59 PM > To: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> > Subject: fresh pacific installation does not detect available > disks > > Hello everybody, > > as ceph newbie I've tried out setting up ceph pacific according to the > official documentation: https://docs.ceph.com/en/latest/cephadm/install/ > The intention was to setup a single node "cluster" with radosgw to feature > local S3 storage. > This failed because my ceph "cluster" would not detect OSDs. > I started from a Debain 11.1 (bullseye) VM hosted on VMware workstation. Of > course I've added some additional disk images to be used as OSDs. > These are the steps I've performed: > > curl --silent --remote-name --location > https://github.com/ceph/ceph/raw/pacific/src/cephadm/cephadm > chmod +x cephadm > ./cephadm add-repo --release pacific > ./cephadm install > > apt install -y cephadm > > cephadm bootstrap --mon-ip <my_ip> > > cephadm add-repo --release pacific > > cephadm install ceph-common > > ceph orch apply osd --all-available-devices > > > The last command would have no effect. Its sole output is: > > Scheduled osd.all-available-devices update... > > > > Also ceph -s shows that no OSDs were added: > > cluster: > > id: 655a7a32-3bbf-11ec-920e-000c29da2e6a > > health: HEALTH_WARN > > OSD count 0 < osd_pool_default_size 1 > > > > services: > > mon: 1 daemons, quorum terraformdemo (age 2d) > > mgr: terraformdemo.aylzbb(active, since 2d) > > osd: 0 osds: 0 up, 0 in (since 2d) > > > > data: > > pools: 0 pools, 0 pgs > > objects: 0 objects, 0 B > > usage: 0 B used, 0 B / 0 B avail > > pgs: > > > To find out what may be going wrong I've also tried out this: > > cephadm install ceph-osd > > ceph-volume inventory > This results in a list that makes more sense: > > Device Path Size rotates available Model name > > /dev/sdc 20.00 GB True True VMware Virtual S > > /dev/sde 20.00 GB True True VMware Virtual S > > /dev/sda 20.00 GB True False VMware Virtual S > > /dev/sdb 20.00 GB True False VMware Virtual S > > /dev/sdd 20.00 GB True False VMware Virtual S > > > So how can I convince cephadm to use the available devices? > > Regards, > Carsten > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send an email > to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> > To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |