Here is the log from ceph-volume. ``` [2021-02-05 04:03:17,000][ceph_volume.process][INFO ] Running command: /usr/sbin/vgcreate --force --yes ceph-a3886f74-3de9-4e6e-a983-8330eda0bd64 /dev/sdd [2021-02-05 04:03:17,134][ceph_volume.process][INFO ] stdout Physical volume "/dev/sdd" successfully created. [2021-02-05 04:03:17,166][ceph_volume.process][INFO ] stdout Volume group "ceph-a3886f74-3de9-4e6e-a983-8330eda0bd64" successfully created [2021-02-05 04:03:17,189][ceph_volume.process][INFO ] Running command: /usr/sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -S vg_name=ceph-a3886f74-3de9-4e6e-a983-8330eda0bd64 -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size [2021-02-05 04:03:17,229][ceph_volume.process][INFO ] stdout ceph-a3886f74-3de9-4e6e-a983-8330eda0bd64";"1";"0";"wz--n-";"572317";"572317";"4194304 [2021-02-05 04:03:17,229][ceph_volume.api.lvm][DEBUG ] size was passed: 2.18 TB -> 572318 [2021-02-05 04:03:17,235][ceph_volume.process][INFO ] Running command: /usr/sbin/lvcreate --yes -l 572318 -n osd-block-b05c3c90-b7d5-4f13-8a58-f72761c1971b ceph-a3886f74-3de9-4e6e-a983-8330eda0bd64 [2021-02-05 04:03:17,244][ceph_volume.process][INFO ] stderr Volume group "ceph-a3886f74-3de9-4e6e-a983-8330eda0bd64" has insufficient free space (572317 extents): 572318 required. ``` size was passed: 2.18 TB -> 572318 How is this calculated? Thanks! Tony > -----Original Message----- > From: Tony Liu <tonyliu0592@xxxxxxxxxxx> > Sent: Thursday, February 4, 2021 8:34 PM > To: ceph-users@xxxxxxx > Subject: replace OSD failed > > Hi, > > With 15.2.8, run "ceph orch rm osd 12 --replace --force", PGs on osd.12 > are remapped, osd.12 is removed from "ceph osd tree", the daemon is > removed from "ceph orch ps", the device is "available" > in "ceph orch device ls". Everything seems good at this point. > > Then dry-run service spec. > ``` > # cat osd-spec.yaml > service_type: osd > service_id: osd-spec > placement: > hosts: > - ceph-osd-1 > data_devices: > rotational: 1 > db_devices: > rotational: 0 > > # ceph orch apply osd -i osd-spec.yaml --dry-run > +---------+----------+------------+----------+----------+-----+ > |SERVICE |NAME |HOST |DATA |DB |WAL | > +---------+----------+------------+----------+----------+-----+ > |osd |osd-spec |ceph-osd-3 |/dev/sdd |/dev/sdb |- | > +---------+----------+------------+----------+----------+-----+ > ``` > It looks as expected. > > Then "ceph orch apply osd -i osd-spec.yaml". > Here is the log of cephadm. > ``` > /bin/docker:stderr --> relative data size: 1.0 /bin/docker:stderr --> > passed block_db devices: 1 physical, 0 LVM /bin/docker:stderr Running > command: /usr/bin/ceph-authtool --gen-print-key /bin/docker:stderr > Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap- > osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd tree -f json > /bin/docker:stderr Running command: /usr/bin/ceph --cluster ceph --name > client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring > -i - osd new b05c3c90-b7d5-4f13-8a58-f72761c1971b 12 /bin/docker:stderr > Running command: /usr/sbin/vgcreate --force --yes ceph-a3886f74-3de9- > 4e6e-a983-8330eda0bd64 /dev/sdd /bin/docker:stderr stdout: Physical > volume "/dev/sdd" successfully created. > /bin/docker:stderr stdout: Volume group "ceph-a3886f74-3de9-4e6e-a983- > 8330eda0bd64" successfully created /bin/docker:stderr Running command: > /usr/sbin/lvcreate --yes -l 572318 -n osd-block-b05c3c90-b7d5-4f13-8a58- > f72761c1971b ceph-a3886f74-3de9-4e6e-a983-8330eda0bd64 > /bin/docker:stderr stderr: Volume group "ceph-a3886f74-3de9-4e6e-a983- > 8330eda0bd64" has insufficient free space (572317 extents): 572318 > required. > /bin/docker:stderr --> Was unable to complete a new OSD, will rollback > changes ``` Q1, why VG name (ceph-<id>) is different from others (ceph- > block-<id>)? > Q2, where is that 572318 from? Since all HDDs are the same model, VG > "Total PE" of all HDDs is 572317. > Has anyone seen similar issues? Anything I am missing? > > > Thanks! > Tony > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an > email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |