On Thu, Aug 23, 2018 at 11:32 AM, Hervé Ballans <herve.ballans@xxxxxxxxxxxxx> wrote: > Le 23/08/2018 à 16:13, Alfredo Deza a écrit : > > What you mean is that, at this stage, I must directly declare the UUID paths > in value of --block.db (i.e. replace /dev/nvme0n1p1 with its PARTUUID), that > is ? > > No, this all looks correct. How does the ceph-volume.log and > ceph-volume-systemd.log look when you are booting up for the OSDs that > aren't coming up? > > Anything useful in there? > > > ceph-volume.log (extract) > [2018-08-20 11:26:29,430][ceph_volume.process][INFO ] Running command: > systemctl start ceph-osd@1 > [2018-08-20 11:26:32,268][ceph_volume.main][INFO ] Running command: > ceph-volume lvm trigger 1-4a9954ce-0a0f-432b-a91d-eaacb45287d4 > [2018-08-20 11:26:32,269][ceph_volume.process][INFO ] Running command: > /sbin/lvs --noheadings --readonly --separator=";" -o > lv_tags,lv_path,lv_name,vg_name,lv_uuid > [2018-08-20 11:26:32,347][ceph_volume.process][INFO ] stdout > ceph.block_device=/dev/ceph-425a36ca-8b60-471d-80ed-19726714ea3a/osd-block-9380cd27-c0fe-4ede-9ed3-d09eff545037,ceph.block_uuid=kkykJn-EmUa-EvZj-Ffr7-dk06-VUG0-nhnUvB,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=838506b7-e0c6-4022-9e17-2d1cf9458be6,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/nvme0n1p4,ceph.db_uuid=ec4cf738-1627-4ddf-a7ef-91810fb8aaab,ceph.encrypted=0,ceph.osd_fsid=9380cd27-c0fe-4ede-9ed3-d09eff545037,ceph.osd_id=3,ceph.type=block,ceph.vdo=0";"/dev/ceph-425a36ca-8b60-471d-80ed-19726714ea3a/osd-block-9380cd27-c0fe-4ede-9ed3-d09eff545037";"osd-block-9380cd27-c0fe-4ede-9ed3-d09eff545037";"ceph-425a36ca-8b60-471d-80ed-19726714ea3a";"kkykJn-EmUa-EvZj-Ffr7-dk06-VUG0-nhnUvB > [2018-08-20 11:26:32,347][ceph_volume.process][INFO ] stdout > ceph.block_device=/dev/ceph-597bbc7c-33ba-4020-a1c6-fe4297ad0421/osd-block-5d4af2fc-388c-4795-9d1a-53ad8aba56d8,ceph.block_uuid=WdXALj-0bCs-Ndfy-xGIN-q1rF-hcmg-utIiNB,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=838506b7-e0c6-4022-9e17-2d1cf9458be6,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/nvme0n1p8,ceph.db_uuid=9642a688-0991-4a19-b49a-994a22edaf60,ceph.encrypted=0,ceph.osd_fsid=5d4af2fc-388c-4795-9d1a-53ad8aba56d8,ceph.osd_id=7,ceph.type=block,ceph.vdo=0";"/dev/ceph-597bbc7c-33ba-4020-a1c6-fe4297ad0421/osd-block-5d4af2fc-388c-4795-9d1a-53ad8aba56d8";"osd-block-5d4af2fc-388c-4795-9d1a-53ad8aba56d8";"ceph-597bbc7c-33ba-4020-a1c6-fe4297ad0421";"WdXALj-0bCs-Ndfy-xGIN-q1rF-hcmg-utIiNB > [2018-08-20 11:26:32,347][ceph_volume.process][INFO ] stdout > ceph.block_device=/dev/ceph-5dbc3f3a-d781-4d22-9e9c-8c7a74ce39fb/osd-block-b8e82f22-e993-4458-984b-90232b8b3d55,ceph.block_uuid=Q1Aeu5-bm6V-m9yB-O3vX-SFvX-8oHo-SNkrfe,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=838506b7-e0c6-4022-9e17-2d1cf9458be6,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/nvme0n1p3,ceph.db_uuid=c7bb1f33-65f7-4116-ae27-0d73dde82208,ceph.encrypted=0,ceph.osd_fsid=b8e82f22-e993-4458-984b-90232b8b3d55,ceph.osd_id=2,ceph.type=block,ceph.vdo=0";"/dev/ceph-5dbc3f3a-d781-4d22-9e9c-8c7a74ce39fb/osd-block-b8e82f22-e993-4458-984b-90232b8b3d55";"osd-block-b8e82f22-e993-4458-984b-90232b8b3d55";"ceph-5dbc3f3a-d781-4d22-9e9c-8c7a74ce39fb";"Q1Aeu5-bm6V-m9yB-O3vX-SFvX-8oHo-SNkrfe > [2018-08-20 11:26:32,347][ceph_volume.process][INFO ] stdout > ceph.block_device=/dev/ceph-70e7b5f7-408d-4810-9d92-d6679d621db0/osd-block-02540fff-5478-4a67-bf5c-679c72150e8d,ceph.block_uuid=jp7YJ5-vNcj-93at-49LK-bSWT-ke4m-Y0Brzy,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=838506b7-e0c6-4022-9e17-2d1cf9458be6,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/nvme0n1p5,ceph.db_uuid=b5ea7d3a-8c90-4163-a2af-a4dfa045ced6,ceph.encrypted=0,ceph.osd_fsid=02540fff-5478-4a67-bf5c-679c72150e8d,ceph.osd_id=4,ceph.type=block,ceph.vdo=0";"/dev/ceph-70e7b5f7-408d-4810-9d92-d6679d621db0/osd-block-02540fff-5478-4a67-bf5c-679c72150e8d";"osd-block-02540fff-5478-4a67-bf5c-679c72150e8d";"ceph-70e7b5f7-408d-4810-9d92-d6679d621db0";"jp7YJ5-vNcj-93at-49LK-bSWT-ke4m-Y0Brzy > [2018-08-20 11:26:32,347][ceph_volume.process][INFO ] stdout > ceph.block_device=/dev/ceph-766bd78c-ed1a-4e27-8b4d-7adc4c4f2f0d/osd-block-98bfb597-009b-4e88-bc5e-dd22587d21fe,ceph.block_uuid=enriOd-To2e-lLWi-Dc91-keUr-1tf7-4Hvrj2,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=838506b7-e0c6-4022-9e17-2d1cf9458be6,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/nvme0n1p1,ceph.db_uuid=99870da1-b7d1-479d-ad63-14407c593278,ceph.encrypted=0,ceph.osd_fsid=98bfb597-009b-4e88-bc5e-dd22587d21fe,ceph.osd_id=0,ceph.type=block,ceph.vdo=0";"/dev/ceph-766bd78c-ed1a-4e27-8b4d-7adc4c4f2f0d/osd-block-98bfb597-009b-4e88-bc5e-dd22587d21fe";"osd-block-98bfb597-009b-4e88-bc5e-dd22587d21fe";"ceph-766bd78c-ed1a-4e27-8b4d-7adc4c4f2f0d";"enriOd-To2e-lLWi-Dc91-keUr-1tf7-4Hvrj2 > [2018-08-20 11:26:32,347][ceph_volume.process][INFO ] stdout > ceph.block_device=/dev/ceph-8df436c9-1302-4f1f-b185-c060f5e177d5/osd-block-2e5b3463-5904-4aee-9ae1-7d31d8576dc8,ceph.block_uuid=nqgDsS-dz7R-Ddlf-j31m-Q98Y-Qjua-Lmi5vq,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=838506b7-e0c6-4022-9e17-2d1cf9458be6,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/nvme0n1p10,ceph.db_uuid=d7ac0b16-a18c-4e8a-973f-829970e5a5ec,ceph.encrypted=0,ceph.osd_fsid=2e5b3463-5904-4aee-9ae1-7d31d8576dc8,ceph.osd_id=9,ceph.type=block,ceph.vdo=0";"/dev/ceph-8df436c9-1302-4f1f-b185-c060f5e177d5/osd-block-2e5b3463-5904-4aee-9ae1-7d31d8576dc8";"osd-block-2e5b3463-5904-4aee-9ae1-7d31d8576dc8";"ceph-8df436c9-1302-4f1f-b185-c060f5e177d5";"nqgDsS-dz7R-Ddlf-j31m-Q98Y-Qjua-Lmi5vq > [2018-08-20 11:26:32,348][ceph_volume.process][INFO ] stdout > ceph.block_device=/dev/ceph-a23b8ade-25ae-43b2-ae42-8990bae6f389/osd-block-b7100200-9eef-4c85-b855-b5a0a435354c,ceph.block_uuid=xa3cq8-cGm1-1aKI-RC5M-mcqw-zadE-tdcHhK,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=838506b7-e0c6-4022-9e17-2d1cf9458be6,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/nvme0n1p6,ceph.db_uuid=fd88634e-7381-40e4-9aec-1df115ef64cf,ceph.encrypted=0,ceph.osd_fsid=b7100200-9eef-4c85-b855-b5a0a435354c,ceph.osd_id=5,ceph.type=block,ceph.vdo=0";"/dev/ceph-a23b8ade-25ae-43b2-ae42-8990bae6f389/osd-block-b7100200-9eef-4c85-b855-b5a0a435354c";"osd-block-b7100200-9eef-4c85-b855-b5a0a435354c";"ceph-a23b8ade-25ae-43b2-ae42-8990bae6f389";"xa3cq8-cGm1-1aKI-RC5M-mcqw-zadE-tdcHhK > [2018-08-20 11:26:32,348][ceph_volume.process][INFO ] stdout > ceph.block_device=/dev/ceph-cb9bd619-dfd8-4a09-9b60-ecfb50b2f13c/osd-block-bcb9d7e6-44ea-449b-ad97-1aa5f880dfdd,ceph.block_uuid=S51Kgl-r9QD-P7Dn-PYzp-6mXi-TYfD-RGxefs,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=838506b7-e0c6-4022-9e17-2d1cf9458be6,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/nvme0n1p2,ceph.db_uuid=3ee5298a-2cbd-424b-b26d-e9eb04fdbc84,ceph.encrypted=0,ceph.osd_fsid=bcb9d7e6-44ea-449b-ad97-1aa5f880dfdd,ceph.osd_id=1,ceph.type=block,ceph.vdo=0";"/dev/ceph-cb9bd619-dfd8-4a09-9b60-ecfb50b2f13c/osd-block-bcb9d7e6-44ea-449b-ad97-1aa5f880dfdd";"osd-block-bcb9d7e6-44ea-449b-ad97-1aa5f880dfdd";"ceph-cb9bd619-dfd8-4a09-9b60-ecfb50b2f13c";"S51Kgl-r9QD-P7Dn-PYzp-6mXi-TYfD-RGxefs > [2018-08-20 11:26:32,348][ceph_volume.process][INFO ] stdout > ceph.block_device=/dev/ceph-db6fc227-11fb-4510-8a8b-29c16c7d9bfb/osd-block-ba351d69-5c48-418e-a377-4034f503af93,ceph.block_uuid=kpvBCz-Ikuu-52r0-0FQ2-NGR1-hlGP-bHpMi4,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=838506b7-e0c6-4022-9e17-2d1cf9458be6,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/nvme0n1p7,ceph.db_uuid=61bc909c-0930-49b7-90ad-c5e2d27eed9e,ceph.encrypted=0,ceph.osd_fsid=ba351d69-5c48-418e-a377-4034f503af93,ceph.osd_id=6,ceph.type=block,ceph.vdo=0";"/dev/ceph-db6fc227-11fb-4510-8a8b-29c16c7d9bfb/osd-block-ba351d69-5c48-418e-a377-4034f503af93";"osd-block-ba351d69-5c48-418e-a377-4034f503af93";"ceph-db6fc227-11fb-4510-8a8b-29c16c7d9bfb";"kpvBCz-Ikuu-52r0-0FQ2-NGR1-hlGP-bHpMi4 > [2018-08-20 11:26:32,348][ceph_volume.process][INFO ] stdout > ceph.block_device=/dev/ceph-ee343712-359b-4972-bbdf-c66ae5d4bf4a/osd-block-913e65e3-62d9-48f8-a0ef-45315cf64593,ceph.block_uuid=ZctH0P-9zjl-zvTf-xHcd-bIIV-bMGa-vS4wOP,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=838506b7-e0c6-4022-9e17-2d1cf9458be6,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/nvme0n1p9,ceph.db_uuid=70e9e629-2c82-4aaa-bd73-abf22ebc9736,ceph.encrypted=0,ceph.osd_fsid=913e65e3-62d9-48f8-a0ef-45315cf64593,ceph.osd_id=8,ceph.type=block,ceph.vdo=0";"/dev/ceph-ee343712-359b-4972-bbdf-c66ae5d4bf4a/osd-block-913e65e3-62d9-48f8-a0ef-45315cf64593";"osd-block-913e65e3-62d9-48f8-a0ef-45315cf64593";"ceph-ee343712-359b-4972-bbdf-c66ae5d4bf4a";"ZctH0P-9zjl-zvTf-xHcd-bIIV-bMGa-vS4wOP > [2018-08-20 11:26:32,349][ceph_volume][ERROR ] exception caught by decorator > Traceback (most recent call last): > File "/usr/lib/python2.7/dist-packages/ceph_volume/decorators.py", line > 59, in newfunc > return f(*a, **kw) > File "/usr/lib/python2.7/dist-packages/ceph_volume/main.py", line 153, in > main > terminal.dispatch(self.mapper, subcommand_args) > File "/usr/lib/python2.7/dist-packages/ceph_volume/terminal.py", line 182, > in dispatch > instance.main() > File "/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/main.py", > line 38, in main > terminal.dispatch(self.mapper, self.argv) > File "/usr/lib/python2.7/dist-packages/ceph_volume/terminal.py", line 182, > in dispatch > instance.main() > File "/usr/lib/python2.7/dist-packages/ceph_volume/decorators.py", line > 16, in is_root > return func(*a, **kw) > File > "/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/trigger.py", line > 70, in main > Activate(['--auto-detect-objectstore', osd_id, osd_uuid]).main() > File > "/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/activate.py", line > 320, in main > self.activate(args) > File "/usr/lib/python2.7/dist-packages/ceph_volume/decorators.py", line > 16, in is_root > return func(*a, **kw) > File > "/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/activate.py", line > 230, in activate > raise RuntimeError('could not find osd.%s with fsid %s' % (osd_id, > osd_fsid)) > RuntimeError: could not find osd.1 with fsid > 4a9954ce-0a0f-432b-a91d-eaacb45287d4 > > ceph-volume-systemd.log (extract) > [2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received: > lvm-6-ba351d69-5c48-418e-a377-4034f503af93 > [2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received: > lvm-3-9380cd27-c0fe-4ede-9ed3-d09eff545037 > [2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received: > lvm-1-bcb9d7e6-44ea-449b-ad97-1aa5f880dfdd > [2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received: > lvm-4-02540fff-5478-4a67-bf5c-679c72150e8d > [2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received: > lvm-0-98bfb597-009b-4e88-bc5e-dd22587d21fe > [2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received: > lvm-8-913e65e3-62d9-48f8-a0ef-45315cf64593 > [2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received: > lvm-5-b7100200-9eef-4c85-b855-b5a0a435354c > [2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received: > lvm-7-5d4af2fc-388c-4795-9d1a-53ad8aba56d8 > [2018-08-20 11:26:26,386][systemd][INFO ] parsed sub-command: lvm, extra > data: 6-ba351d69-5c48-418e-a377-4034f503af93 > [2018-08-20 11:26:26,386][systemd][INFO ] parsed sub-command: lvm, extra > data: 3-9380cd27-c0fe-4ede-9ed3-d09eff545037 > [2018-08-20 11:26:26,386][systemd][INFO ] parsed sub-command: lvm, extra > data: 1-bcb9d7e6-44ea-449b-ad97-1aa5f880dfdd > [2018-08-20 11:26:26,387][systemd][INFO ] parsed sub-command: lvm, extra > data: 4-02540fff-5478-4a67-bf5c-679c72150e8d > [2018-08-20 11:26:26,387][systemd][INFO ] parsed sub-command: lvm, extra > data: 0-98bfb597-009b-4e88-bc5e-dd22587d21fe > [2018-08-20 11:26:26,387][systemd][INFO ] parsed sub-command: lvm, extra > data: 8-913e65e3-62d9-48f8-a0ef-45315cf64593 > [2018-08-20 11:26:26,387][systemd][INFO ] parsed sub-command: lvm, extra > data: 5-b7100200-9eef-4c85-b855-b5a0a435354c > [2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received: > lvm-9-2e5b3463-5904-4aee-9ae1-7d31d8576dc8 > [2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received: > lvm-2-b8e82f22-e993-4458-984b-90232b8b3d55 > [2018-08-20 11:26:26,387][systemd][INFO ] parsed sub-command: lvm, extra > data: 7-5d4af2fc-388c-4795-9d1a-53ad8aba56d8 > [2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received: > lvm-1-4a9954ce-0a0f-432b-a91d-eaacb45287d4 > [2018-08-20 11:26:26,387][systemd][INFO ] parsed sub-command: lvm, extra > data: 1-4a9954ce-0a0f-432b-a91d-eaacb45287d4 > [2018-08-20 11:26:26,387][systemd][INFO ] parsed sub-command: lvm, extra > data: 9-2e5b3463-5904-4aee-9ae1-7d31d8576dc8 > [2018-08-20 11:26:26,387][systemd][INFO ] parsed sub-command: lvm, extra > data: 2-b8e82f22-e993-4458-984b-90232b8b3d55 > [2018-08-20 11:26:26,458][ceph_volume.process][INFO ] Running command: > ceph-volume lvm trigger 3-9380cd27-c0fe-4ede-9ed3-d09eff545037 > [2018-08-20 11:26:26,458][ceph_volume.process][INFO ] Running command: > ceph-volume lvm trigger 2-b8e82f22-e993-4458-984b-90232b8b3d55 > [2018-08-20 11:26:26,458][ceph_volume.process][INFO ] Running command: > ceph-volume lvm trigger 5-b7100200-9eef-4c85-b855-b5a0a435354c > [2018-08-20 11:26:26,458][ceph_volume.process][INFO ] Running command: > ceph-volume lvm trigger 6-ba351d69-5c48-418e-a377-4034f503af93 > [2018-08-20 11:26:26,458][ceph_volume.process][INFO ] Running command: > ceph-volume lvm trigger 4-02540fff-5478-4a67-bf5c-679c72150e8d > [2018-08-20 11:26:26,459][ceph_volume.process][INFO ] Running command: > ceph-volume lvm trigger 8-913e65e3-62d9-48f8-a0ef-45315cf64593 > [2018-08-20 11:26:26,459][ceph_volume.process][INFO ] Running command: > ceph-volume lvm trigger 0-98bfb597-009b-4e88-bc5e-dd22587d21fe > [2018-08-20 11:26:26,459][ceph_volume.process][INFO ] Running command: > ceph-volume lvm trigger 7-5d4af2fc-388c-4795-9d1a-53ad8aba56d8 > [2018-08-20 11:26:26,459][ceph_volume.process][INFO ] Running command: > ceph-volume lvm trigger 1-4a9954ce-0a0f-432b-a91d-eaacb45287d4 > [2018-08-20 11:26:26,459][ceph_volume.process][INFO ] Running command: > ceph-volume lvm trigger 9-2e5b3463-5904-4aee-9ae1-7d31d8576dc8 > [2018-08-20 11:26:26,459][ceph_volume.process][INFO ] Running command: > ceph-volume lvm trigger 1-bcb9d7e6-44ea-449b-ad97-1aa5f880dfdd > [2018-08-20 11:26:27,068][ceph_volume.process][INFO ] stderr --> > RuntimeError: could not find osd.1 with fsid > 4a9954ce-0a0f-432b-a91d-eaacb45287d4 This is odd: why is osd.1 not found? Do you have an OSD with that ID and FSID? This line means that we have queried all the LVs in the system and we haven't found anything that responds to that ID and FSID > [2018-08-20 11:26:27,086][systemd][WARNING] command returned non-zero exit > status: 1 > [2018-08-20 11:26:27,086][systemd][WARNING] failed activating OSD, retries > left: 30 > > > This sounds very odd. Could you clarify where block and block.db are? > Also useful here would be to take a look at > /var/log/ceph/ceph-volume-systemd.log and ceph-volume.log to > see how ceph-volume is trying to get this OSD up and running. > > Also useful would be to check `ceph-volume lvm list` to verify that > regardless of the name change, it recognizes the correct partition > mapped to the OSD > > Oops ! > > # ceph-volume lvm list > --> KeyError: 'devices' > > Can you re-run this like: > > > CEPH_VOLUME_DEBUG=1 ceph-volume lvm list > > And paste the output? I think this has been fixed since, but want to > double check > > # ceph --version > ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous > (stable) > > # CEPH_VOLUME_DEBUG=1 ceph-volume lvm list > Traceback (most recent call last): > File "/usr/sbin/ceph-volume", line 6, in <module> > main.Volume() > File "/usr/lib/python2.7/dist-packages/ceph_volume/main.py", line 37, in > __init__ > self.main(self.argv) > File "/usr/lib/python2.7/dist-packages/ceph_volume/decorators.py", line > 59, in newfunc > return f(*a, **kw) > File "/usr/lib/python2.7/dist-packages/ceph_volume/main.py", line 153, in > main > terminal.dispatch(self.mapper, subcommand_args) > File "/usr/lib/python2.7/dist-packages/ceph_volume/terminal.py", line 182, > in dispatch > instance.main() > File "/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/main.py", > line 38, in main > terminal.dispatch(self.mapper, self.argv) > File "/usr/lib/python2.7/dist-packages/ceph_volume/terminal.py", line 182, > in dispatch > instance.main() > File > "/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/listing.py", line > 300, in main > self.list(args) > File "/usr/lib/python2.7/dist-packages/ceph_volume/decorators.py", line > 16, in is_root > return func(*a, **kw) > File > "/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/listing.py", line > 120, in list > pretty_report(report) > File > "/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/listing.py", line > 53, in pretty_report > device_metadata_item_template.format(tag_name='devices', > value=','.join(device['devices']))) > KeyError: 'devices' Ok this one was fixed here http://tracker.ceph.com/issues/24952 > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com