Re: ceph_volume.process hangs after reboot with missing osds lockbox.keyring dm-crypt osd luks [solved]

[Jelle de Jong <jelledejong@xxxxxxxxxxxxx>]

Hello everybody,

I solved the issue, there was an network misconfiguration and the some 
of the osd nodes where not able to communicate with the monitoring node.

Kind regards,

Jelle de Jong

On 12/23/22 17:49, Jelle de Jong wrote:
> Hello everybody,
>
> After an reboot of my ceph-osd node non of the osds came back online.
>
> I am suspecting some issue with the ubuntu focal packages in combination 
> with ceph 15.2.16-1focal.
>
> I tried upgrading all ceph packages to 15.2.17-1bionic and downgrading 
> to a few other versions as well but I can not get it back online.
>
> Am I missing something needed for osd teo start like luks crypto keys of 
> sort?
>
> Can someone point me in the right direction on debugging this and 
> getting the osds back up?
>
> I attached some of the logs and system information.
>
> Is there somethign specific with Ubuntu focal and Ceph 15.2 after recent 
> updates?
>
> Kind regards,
>
> Jelle de Jong
>
>
> [2022-12-23 16:29:05,881][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph --cluster ceph --name 
> client.osd-lockbox.2e3c25c2-bfe5-4ecf-b402-5bd75872c568 --keyring 
> /var/lib/ceph/osd/ceph-96/lockbox.keyring config-key get 
> dm-crypt/osd/2e3c25c2-bfe5-4ecf-b402-5bd75872c568/luks
>
> # ls -hal  /var/lib/ceph/osd/*/*
> -rw------- 1 ceph ceph 106 Dec 23 16:29 
> /var/lib/ceph/osd/ceph-100/lockbox.keyring
> -rw------- 1 ceph ceph 106 Dec 23 16:29 
> /var/lib/ceph/osd/ceph-106/lockbox.keyring
> -rw------- 1 ceph ceph 106 Dec 23 16:29 
> /var/lib/ceph/osd/ceph-110/lockbox.keyring
> -rw------- 1 ceph ceph 106 Dec 23 16:29 
> /var/lib/ceph/osd/ceph-114/lockbox.keyring
> -rw------- 1 ceph ceph 106 Dec 23 16:29 
> /var/lib/ceph/osd/ceph-118/lockbox.keyring
> -rw------- 1 ceph ceph 106 Dec 23 16:29 
> /var/lib/ceph/osd/ceph-81/lockbox.keyring
> -rw------- 1 ceph ceph 106 Dec 23 16:29 
> /var/lib/ceph/osd/ceph-85/lockbox.keyring
> -rw------- 1 ceph ceph 106 Dec 23 16:29 
> /var/lib/ceph/osd/ceph-89/lockbox.keyring
> -rw------- 1 ceph ceph 106 Dec 23 16:29 
> /var/lib/ceph/osd/ceph-92/lockbox.keyring
> -rw------- 1 ceph ceph 106 Dec 23 16:29 
> /var/lib/ceph/osd/ceph-96/lockbox.keyring
>
>
> # cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=20.04
> DISTRIB_CODENAME=focal
> DISTRIB_DESCRIPTION="Ubuntu 20.04.5 LTS"
>
> # cat /etc/apt/sources.list.d/eu_ceph_com_debian_octopus.list
> #deb http://eu.ceph.com//debian-octopus focal main
> deb http://eu.ceph.com/debian-15.2.16/ focal main
>
>
> # dpkg -l *ceph*
> Desired=Unknown/Install/Remove/Purge/Hold
> | 
> Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> ||/ Name                          Version        Architecture Description
> +++-=============================-==============-============-=====================================================================
> ii  ceph                          15.2.16-1focal amd64 distributed 
> storage and file system
> ii  ceph-base                     15.2.16-1focal amd64        common 
> ceph daemon libraries and management tools
> un  ceph-client-tools             <none>         <none>       (no 
> description available)
> ii  ceph-common                   15.2.16-1focal amd64        common 
> utilities to mount and interact with a ceph storage cluster
> un  ceph-fs-common                <none>         <none>       (no 
> description available)
> ii  ceph-fuse                     15.2.16-1focal amd64        FUSE-based 
> client for the Ceph distributed file system
> ii  ceph-mds                      15.2.16-1focal amd64        metadata 
> server for the ceph distributed file system
> ii  ceph-mgr                      15.2.16-1focal amd64        manager 
> for the ceph distributed storage system
> ii  ceph-mgr-cephadm              15.2.16-1focal all          cephadm 
> orchestrator module for ceph-mgr
> ii  ceph-mgr-dashboard            15.2.16-1focal all          dashboard 
> module for ceph-mgr
> ii  ceph-mgr-diskprediction-cloud 15.2.16-1focal all 
> diskprediction-cloud module for ceph-mgr
> ii  ceph-mgr-diskprediction-local 15.2.16-1focal all 
> diskprediction-local module for ceph-mgr
> ii  ceph-mgr-k8sevents            15.2.16-1focal all          kubernetes 
> events module for ceph-mgr
> ii  ceph-mgr-modules-core         15.2.16-1focal all          ceph 
> manager modules which are always enabled
> ii  ceph-mon                      15.2.16-1focal amd64        monitor 
> server for the ceph storage system
> ii  ceph-osd                      15.2.16-1focal amd64        OSD server 
> for the ceph storage system
> un  ceph-test                     <none>         <none>       (no 
> description available)
> ii  cephadm                       15.2.16-1focal amd64        cephadm 
> utility to bootstrap ceph daemons with systemd and containers
> un  libceph                       <none>         <none>       (no 
> description available)
> un  libceph1                      <none>         <none>       (no 
> description available)
> un  libcephfs                     <none>         <none>       (no 
> description available)
> ii  libcephfs2                    15.2.16-1focal amd64        Ceph 
> distributed file system client library
> un  python-ceph                   <none>         <none>       (no 
> description available)
> ii  python3-ceph-argparse         15.2.16-1focal all          Python 3 
> utility libraries for Ceph CLI
> ii  python3-ceph-common           15.2.16-1focal all          Python 3 
> utility libraries for Ceph
> ii  python3-cephfs                15.2.16-1focal amd64        Python 3 
> libraries for the Ceph libcephfs library
>
> # dpkg -l *crypt*
> Desired=Unknown/Install/Remove/Purge/Hold
> | 
> Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> ||/ Name                         Version               Architecture 
> Description
> +++-============================-=====================-============-=======================================================================
> un  cryptsetup                   <none>                <none>       (no 
> description available)
> ii  cryptsetup-bin               2:2.2.2-3ubuntu2.4    amd64        disk 
> encryption support - command line tools
> un  cryptsetup-initramfs         <none>                <none>       (no 
> description available)
> un  ecryptfs-utils               <none>                <none>       (no 
> description available)
> ii  libcrypt-dev:amd64           1:4.4.10-10ubuntu4    amd64 libcrypt 
> development files
> ii  libcrypt1:amd64              1:4.4.10-10ubuntu4    amd64 libcrypt 
> shared library
> un  libcrypt1-dev                <none>                <none>       (no 
> description available)
> un  libcrypt2-dev                <none>                <none>       (no 
> description available)
> ii  libcryptsetup12:amd64        2:2.2.2-3ubuntu2.4    amd64        disk 
> encryption support - shared library
> ii  libgcrypt20:amd64            1.8.5-5ubuntu1.1      amd64        LGPL 
> Crypto library - runtime library
> ii  libhcrypto4-heimdal:amd64    7.7.0+dfsg-1ubuntu1.1 amd64 Heimdal 
> Kerberos - crypto library
> ii  libk5crypto3:amd64           1.17-6ubuntu4.1       amd64        MIT 
> Kerberos runtime libraries - Crypto Library
> un  opencryptoki                 <none>                <none>       (no 
> description available)
> un  php-crypt-gpg                <none>                <none>       (no 
> description available)
> un  python-cryptography-doc      <none>                <none>       (no 
> description available)
> ii  python3-bcrypt               3.1.7-2ubuntu1        amd64 password 
> hashing library for Python 3
> un  python3-crypto               <none>                <none>       (no 
> description available)
> ii  python3-cryptography         2.8-3ubuntu0.1        amd64 Python 
> library exposing cryptographic recipes and primitives (Python 3)
> un  python3-cryptography-vectors <none>                <none>       (no 
> description available)
>
> # cat /var/log/ceph/ceph-volume-systemd.log | grep "2022-12-23 16:"
> [2022-12-23 16:29:05,510][systemd][INFO  ] raw systemd input received: 
> lvm-100-a24b3f49-4824-4bab-8857-99b7c6795999
> [2022-12-23 16:29:05,510][systemd][INFO  ] raw systemd input received: 
> lvm-96-2e3c25c2-bfe5-4ecf-b402-5bd75872c568
> [2022-12-23 16:29:05,510][systemd][INFO  ] raw systemd input received: 
> lvm-110-2b7d8995-8924-479f-aded-c0c83d7c6465
> [2022-12-23 16:29:05,510][systemd][INFO  ] raw systemd input received: 
> lvm-114-5b4cb910-15f2-40f0-8666-3f880dfaeb1c
> [2022-12-23 16:29:05,510][systemd][INFO  ] raw systemd input received: 
> lvm-81-48c13ab5-2953-4a06-b588-d6046d345568
> [2022-12-23 16:29:05,510][systemd][INFO  ] parsed sub-command: lvm, 
> extra data: 100-a24b3f49-4824-4bab-8857-99b7c6795999
> [2022-12-23 16:29:05,510][systemd][INFO  ] raw systemd input received: 
> lvm-106-89436dda-f72b-4c65-a9af-972a373336f5
> [2022-12-23 16:29:05,510][systemd][INFO  ] parsed sub-command: lvm, 
> extra data: 96-2e3c25c2-bfe5-4ecf-b402-5bd75872c568
> [2022-12-23 16:29:05,510][systemd][INFO  ] parsed sub-command: lvm, 
> extra data: 110-2b7d8995-8924-479f-aded-c0c83d7c6465
> [2022-12-23 16:29:05,510][systemd][INFO  ] parsed sub-command: lvm, 
> extra data: 114-5b4cb910-15f2-40f0-8666-3f880dfaeb1c
> [2022-12-23 16:29:05,510][systemd][INFO  ] parsed sub-command: lvm, 
> extra data: 81-48c13ab5-2953-4a06-b588-d6046d345568
> [2022-12-23 16:29:05,510][systemd][INFO  ] raw systemd input received: 
> lvm-118-a7f37f45-c48a-47d3-8fe4-1690ea792658
> [2022-12-23 16:29:05,510][systemd][INFO  ] raw systemd input received: 
> lvm-89-5181c3d3-8ed8-4df7-a572-58154ec15482
> [2022-12-23 16:29:05,510][systemd][INFO  ] parsed sub-command: lvm, 
> extra data: 106-89436dda-f72b-4c65-a9af-972a373336f5
> [2022-12-23 16:29:05,510][systemd][INFO  ] parsed sub-command: lvm, 
> extra data: 118-a7f37f45-c48a-47d3-8fe4-1690ea792658
> [2022-12-23 16:29:05,511][systemd][INFO  ] parsed sub-command: lvm, 
> extra data: 89-5181c3d3-8ed8-4df7-a572-58154ec15482
> [2022-12-23 16:29:05,511][ceph_volume.process][INFO  ] Running command: 
> /usr/sbin/ceph-volume lvm trigger 100-a24b3f49-4824-4bab-8857-99b7c6795999
> [2022-12-23 16:29:05,511][ceph_volume.process][INFO  ] Running command: 
> /usr/sbin/ceph-volume lvm trigger 110-2b7d8995-8924-479f-aded-c0c83d7c6465
> [2022-12-23 16:29:05,511][ceph_volume.process][INFO  ] Running command: 
> /usr/sbin/ceph-volume lvm trigger 114-5b4cb910-15f2-40f0-8666-3f880dfaeb1c
> [2022-12-23 16:29:05,511][ceph_volume.process][INFO  ] Running command: 
> /usr/sbin/ceph-volume lvm trigger 81-48c13ab5-2953-4a06-b588-d6046d345568
> [2022-12-23 16:29:05,511][ceph_volume.process][INFO  ] Running command: 
> /usr/sbin/ceph-volume lvm trigger 96-2e3c25c2-bfe5-4ecf-b402-5bd75872c568
> [2022-12-23 16:29:05,511][ceph_volume.process][INFO  ] Running command: 
> /usr/sbin/ceph-volume lvm trigger 106-89436dda-f72b-4c65-a9af-972a373336f5
> [2022-12-23 16:29:05,511][ceph_volume.process][INFO  ] Running command: 
> /usr/sbin/ceph-volume lvm trigger 118-a7f37f45-c48a-47d3-8fe4-1690ea792658
> [2022-12-23 16:29:05,511][ceph_volume.process][INFO  ] Running command: 
> /usr/sbin/ceph-volume lvm trigger 89-5181c3d3-8ed8-4df7-a572-58154ec15482
> [2022-12-23 16:29:05,514][systemd][INFO  ] raw systemd input received: 
> lvm-92-db6b5f4a-563e-48f7-a6a3-92054820c816
> [2022-12-23 16:29:05,514][systemd][INFO  ] parsed sub-command: lvm, 
> extra data: 92-db6b5f4a-563e-48f7-a6a3-92054820c816
> [2022-12-23 16:29:05,514][ceph_volume.process][INFO  ] Running command: 
> /usr/sbin/ceph-volume lvm trigger 92-db6b5f4a-563e-48f7-a6a3-92054820c816
> [2022-12-23 16:29:05,514][systemd][INFO  ] raw systemd input received: 
> lvm-85-7fb5b3c0-3cbf-4ae7-9fbc-a7605c1584ff
> [2022-12-23 16:29:05,514][systemd][INFO  ] parsed sub-command: lvm, 
> extra data: 85-7fb5b3c0-3cbf-4ae7-9fbc-a7605c1584ff
> [2022-12-23 16:29:05,514][ceph_volume.process][INFO  ] Running command: 
> /usr/sbin/ceph-volume lvm trigger 85-7fb5b3c0-3cbf-4ae7-9fbc-a7605c1584ff
>
> # cat /var/log/ceph/ceph-volume.log | grep "2022-12-23 16:"
> [2022-12-23 16:29:05,683][ceph_volume.main][INFO  ] Running command: 
> ceph-volume  lvm trigger 81-48c13ab5-2953-4a06-b588-d6046d345568
> [2022-12-23 16:29:05,683][ceph_volume.main][INFO  ] Running command: 
> ceph-volume  lvm trigger 106-89436dda-f72b-4c65-a9af-972a373336f5
> [2022-12-23 16:29:05,683][ceph_volume.main][INFO  ] Running command: 
> ceph-volume  lvm trigger 118-a7f37f45-c48a-47d3-8fe4-1690ea792658
> [2022-12-23 16:29:05,683][ceph_volume.main][INFO  ] Running command: 
> ceph-volume  lvm trigger 100-a24b3f49-4824-4bab-8857-99b7c6795999
> [2022-12-23 16:29:05,683][ceph_volume.main][INFO  ] Running command: 
> ceph-volume  lvm trigger 92-db6b5f4a-563e-48f7-a6a3-92054820c816
> [2022-12-23 16:29:05,683][ceph_volume.main][INFO  ] Running command: 
> ceph-volume  lvm trigger 114-5b4cb910-15f2-40f0-8666-3f880dfaeb1c
> [2022-12-23 16:29:05,683][ceph_volume.main][INFO  ] Running command: 
> ceph-volume  lvm trigger 89-5181c3d3-8ed8-4df7-a572-58154ec15482
> [2022-12-23 16:29:05,683][ceph_volume.main][INFO  ] Running command: 
> ceph-volume  lvm trigger 96-2e3c25c2-bfe5-4ecf-b402-5bd75872c568
> [2022-12-23 16:29:05,683][ceph_volume.main][INFO  ] Running command: 
> ceph-volume  lvm trigger 110-2b7d8995-8924-479f-aded-c0c83d7c6465
> [2022-12-23 16:29:05,683][ceph_volume.main][INFO  ] Running command: 
> ceph-volume  lvm trigger 85-7fb5b3c0-3cbf-4ae7-9fbc-a7605c1584ff
> [2022-12-23 16:29:05,687][ceph_volume.util.system][WARNING] Executable 
> lvs not found on the host, will return lvs as-is
> [2022-12-23 16:29:05,687][ceph_volume.util.system][WARNING] Executable 
> lvs not found on the host, will return lvs as-is
> [2022-12-23 16:29:05,688][ceph_volume.process][INFO  ] Running command: 
> lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S 
> tags={ceph.osd_id=100,ceph.osd_fsid=a24b3f49-4824-4bab-8857-99b7c6795999} -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size
> [2022-12-23 16:29:05,688][ceph_volume.process][INFO  ] Running command: 
> lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S 
> tags={ceph.osd_id=85,ceph.osd_fsid=7fb5b3c0-3cbf-4ae7-9fbc-a7605c1584ff} 
> -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size
> [2022-12-23 16:29:05,688][ceph_volume.util.system][WARNING] Executable 
> lvs not found on the host, will return lvs as-is
> [2022-12-23 16:29:05,688][ceph_volume.util.system][WARNING] Executable 
> lvs not found on the host, will return lvs as-is
> [2022-12-23 16:29:05,688][ceph_volume.process][INFO  ] Running command: 
> lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S 
> tags={ceph.osd_id=96,ceph.osd_fsid=2e3c25c2-bfe5-4ecf-b402-5bd75872c568} 
> -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size
> [2022-12-23 16:29:05,688][ceph_volume.process][INFO  ] Running command: 
> lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S 
> tags={ceph.osd_id=81,ceph.osd_fsid=48c13ab5-2953-4a06-b588-d6046d345568} 
> -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size
> [2022-12-23 16:29:05,689][ceph_volume.util.system][WARNING] Executable 
> lvs not found on the host, will return lvs as-is
> [2022-12-23 16:29:05,689][ceph_volume.process][INFO  ] Running command: 
> lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S 
> tags={ceph.osd_id=118,ceph.osd_fsid=a7f37f45-c48a-47d3-8fe4-1690ea792658} -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size
> [2022-12-23 16:29:05,689][ceph_volume.util.system][WARNING] Executable 
> lvs not found on the host, will return lvs as-is
> [2022-12-23 16:29:05,689][ceph_volume.util.system][WARNING] Executable 
> lvs not found on the host, will return lvs as-is
> [2022-12-23 16:29:05,689][ceph_volume.util.system][WARNING] Executable 
> lvs not found on the host, will return lvs as-is
> [2022-12-23 16:29:05,690][ceph_volume.process][INFO  ] Running command: 
> lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S 
> tags={ceph.osd_id=110,ceph.osd_fsid=2b7d8995-8924-479f-aded-c0c83d7c6465} -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size
> [2022-12-23 16:29:05,690][ceph_volume.util.system][WARNING] Executable 
> lvs not found on the host, will return lvs as-is
> [2022-12-23 16:29:05,690][ceph_volume.process][INFO  ] Running command: 
> lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S 
> tags={ceph.osd_id=106,ceph.osd_fsid=89436dda-f72b-4c65-a9af-972a373336f5} -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size
> [2022-12-23 16:29:05,690][ceph_volume.process][INFO  ] Running command: 
> lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S 
> tags={ceph.osd_id=92,ceph.osd_fsid=db6b5f4a-563e-48f7-a6a3-92054820c816} 
> -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size
> [2022-12-23 16:29:05,690][ceph_volume.process][INFO  ] Running command: 
> lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S 
> tags={ceph.osd_id=114,ceph.osd_fsid=5b4cb910-15f2-40f0-8666-3f880dfaeb1c} -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size
> [2022-12-23 16:29:05,690][ceph_volume.util.system][WARNING] Executable 
> lvs not found on the host, will return lvs as-is
> [2022-12-23 16:29:05,690][ceph_volume.process][INFO  ] Running command: 
> lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S 
> tags={ceph.osd_id=89,ceph.osd_fsid=5181c3d3-8ed8-4df7-a572-58154ec15482} 
> -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size
> [2022-12-23 16:29:05,805][ceph_volume.process][INFO  ] stdout 
> ceph.block_device=/dev/ceph-868b8d56-4247-42e6-aaa0-60c68d704833/osd-block-2b7d8995-8924-479f-aded-c0c83d7c6465,ceph.block_uuid=Mx2hRl-4J4j-nn4g-pNLd-0hNx-w5Wo-ABkrXD,ceph.cephx_lockbox_secret=AQCXDmhiLa5hJRAA9t44WVnAgu4CddgYnqAWWw==,ceph.cluster_fsid=03fc24b7-762f-4f73-9e0f-c737d7344546,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=1,ceph.osd_fsid=2b7d8995-8924-479f-aded-c0c83d7c6465,ceph.osd_id=110,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-868b8d56-4247-42e6-aaa0-60c68d704833/osd-block-2b7d8995-8924-479f-aded-c0c83d7c6465";"osd-block-2b7d8995-8924-479f-aded-c0c83d7c6465";"ceph-868b8d56-4247-42e6-aaa0-60c68d704833";"Mx2hRl-4J4j-nn4g-pNLd-0hNx-w5Wo-ABkrXD";"3200627245056
> [2022-12-23 16:29:05,805][ceph_volume.devices.lvm.activate][INFO  ] auto 
> detecting objectstore
> [2022-12-23 16:29:05,805][ceph_volume.devices.lvm.activate][INFO  ] 
> unable to find a journal associated with the OSD, assuming bluestore
> [2022-12-23 16:29:05,806][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-110
> [2022-12-23 16:29:05,808][ceph_volume.util.system][WARNING] Executable 
> selinuxenabled not in PATH: 
> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> [2022-12-23 16:29:05,809][ceph_volume.process][INFO  ] Running command: 
> selinuxenabled
> [2022-12-23 16:29:05,809][ceph_volume.util.system][INFO  ] No SELinux 
> found, skipping call to restorecon
> [2022-12-23 16:29:05,810][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-110/lockbox.keyring 
> --create-keyring --name 
> client.osd-lockbox.2b7d8995-8924-479f-aded-c0c83d7c6465 --add-key 
> AQCXDmhiLa5hJRAA9t44WVnAgu4CddgYnqAWWw==
> [2022-12-23 16:29:05,821][ceph_volume.process][INFO  ] stdout 
> ceph.block_device=/dev/ceph-54412af5-2497-4e90-bd4e-998ef60d1cff/osd-block-2e3c25c2-bfe5-4ecf-b402-5bd75872c568,ceph.block_uuid=CdRDTP-1Cgy-hLci-kWj9-FW36-WL74-ppcv3o,ceph.cephx_lockbox_secret=AQBiDmhiI323GRAArw2M/D7YFn82uFm3hZpdnw==,ceph.cluster_fsid=03fc24b7-762f-4f73-9e0f-c737d7344546,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=1,ceph.osd_fsid=2e3c25c2-bfe5-4ecf-b402-5bd75872c568,ceph.osd_id=96,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-54412af5-2497-4e90-bd4e-998ef60d1cff/osd-block-2e3c25c2-bfe5-4ecf-b402-5bd75872c568";"osd-block-2e3c25c2-bfe5-4ecf-b402-5bd75872c568";"ceph-54412af5-2497-4e90-bd4e-998ef60d1cff";"CdRDTP-1Cgy-hLci-kWj9-FW36-WL74-ppcv3o";"3200627245056
> [2022-12-23 16:29:05,821][ceph_volume.process][INFO  ] stdout 
> ceph.block_device=/dev/ceph-ff3a714f-a230-459a-9d31-df7c8706c0ea/osd-block-db6b5f4a-563e-48f7-a6a3-92054820c816,ceph.block_uuid=NflUEr-IxmR-rUho-A5dg-vdLF-LyQ3-IBtza9,ceph.cephx_lockbox_secret=AQBQDmhiPI8fJhAA2zGVUAxLvPhoFzJdVYjGgg==,ceph.cluster_fsid=03fc24b7-762f-4f73-9e0f-c737d7344546,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=1,ceph.osd_fsid=db6b5f4a-563e-48f7-a6a3-92054820c816,ceph.osd_id=92,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-ff3a714f-a230-459a-9d31-df7c8706c0ea/osd-block-db6b5f4a-563e-48f7-a6a3-92054820c816";"osd-block-db6b5f4a-563e-48f7-a6a3-92054820c816";"ceph-ff3a714f-a230-459a-9d31-df7c8706c0ea";"NflUEr-IxmR-rUho-A5dg-vdLF-LyQ3-IBtza9";"3200627245056
> [2022-12-23 16:29:05,821][ceph_volume.process][INFO  ] stdout 
> ceph.block_device=/dev/ceph-bf7d0e68-b999-4187-afe2-85162a0aed52/osd-block-5b4cb910-15f2-40f0-8666-3f880dfaeb1c,ceph.block_uuid=Ifbdmw-AeiL-dvIX-W6m6-cdVk-2GvI-urbdCM,ceph.cephx_lockbox_secret=AQCqDmhi+P/CNRAAafmy4EWla+jRtbWlqk7u/Q==,ceph.cluster_fsid=03fc24b7-762f-4f73-9e0f-c737d7344546,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=1,ceph.osd_fsid=5b4cb910-15f2-40f0-8666-3f880dfaeb1c,ceph.osd_id=114,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-bf7d0e68-b999-4187-afe2-85162a0aed52/osd-block-5b4cb910-15f2-40f0-8666-3f880dfaeb1c";"osd-block-5b4cb910-15f2-40f0-8666-3f880dfaeb1c";"ceph-bf7d0e68-b999-4187-afe2-85162a0aed52";"Ifbdmw-AeiL-dvIX-W6m6-cdVk-2GvI-urbdCM";"3200627245056
> [2022-12-23 16:29:05,821][ceph_volume.process][INFO  ] stdout 
> ceph.block_device=/dev/ceph-5d2cd376-1c01-4c46-ba4c-360ffb5ae060/osd-block-7fb5b3c0-3cbf-4ae7-9fbc-a7605c1584ff,ceph.block_uuid=JJZaYB-pIiG-R2rr-cwXf-jaHZ-8F6g-PznY97,ceph.cephx_lockbox_secret=AQAtDmhiiT4vFBAA43UwmfNYx1ibCL5cZmJ54w==,ceph.cluster_fsid=03fc24b7-762f-4f73-9e0f-c737d7344546,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=1,ceph.osd_fsid=7fb5b3c0-3cbf-4ae7-9fbc-a7605c1584ff,ceph.osd_id=85,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-5d2cd376-1c01-4c46-ba4c-360ffb5ae060/osd-block-7fb5b3c0-3cbf-4ae7-9fbc-a7605c1584ff";"osd-block-7fb5b3c0-3cbf-4ae7-9fbc-a7605c1584ff";"ceph-5d2cd376-1c01-4c46-ba4c-360ffb5ae060";"JJZaYB-pIiG-R2rr-cwXf-jaHZ-8F6g-PznY97";"3200627245056
> [2022-12-23 16:29:05,821][ceph_volume.process][INFO  ] stdout 
> ceph.block_device=/dev/ceph-0d661f1f-0264-44f4-a19e-73a9ffea3e02/osd-block-a7f37f45-c48a-47d3-8fe4-1690ea792658,ceph.block_uuid=O5lOQq-yP6n-aNHs-i5Ya-nTMH-Hnf9-im5vu2,ceph.cephx_lockbox_secret=AQC9Dmhi1wZjOhAAvSVxnH/VrIdqzktAntJkRw==,ceph.cluster_fsid=03fc24b7-762f-4f73-9e0f-c737d7344546,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=1,ceph.osd_fsid=a7f37f45-c48a-47d3-8fe4-1690ea792658,ceph.osd_id=118,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-0d661f1f-0264-44f4-a19e-73a9ffea3e02/osd-block-a7f37f45-c48a-47d3-8fe4-1690ea792658";"osd-block-a7f37f45-c48a-47d3-8fe4-1690ea792658";"ceph-0d661f1f-0264-44f4-a19e-73a9ffea3e02";"O5lOQq-yP6n-aNHs-i5Ya-nTMH-Hnf9-im5vu2";"3200627245056
> [2022-12-23 16:29:05,821][ceph_volume.process][INFO  ] stdout 
> ceph.block_device=/dev/ceph-81c84270-5de5-4498-a60d-a6f5c04d1eab/osd-block-5181c3d3-8ed8-4df7-a572-58154ec15482,ceph.block_uuid=MYR9Lo-Qn4k-1zIv-RWZi-uv1S-mrqS-YxU26A,ceph.cephx_lockbox_secret=AQA/Dmhi/BxFExAAaxcqVkHssaDm/dJ8iyudaQ==,ceph.cluster_fsid=03fc24b7-762f-4f73-9e0f-c737d7344546,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=1,ceph.osd_fsid=5181c3d3-8ed8-4df7-a572-58154ec15482,ceph.osd_id=89,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-81c84270-5de5-4498-a60d-a6f5c04d1eab/osd-block-5181c3d3-8ed8-4df7-a572-58154ec15482";"osd-block-5181c3d3-8ed8-4df7-a572-58154ec15482";"ceph-81c84270-5de5-4498-a60d-a6f5c04d1eab";"MYR9Lo-Qn4k-1zIv-RWZi-uv1S-mrqS-YxU26A";"3200627245056
> [2022-12-23 16:29:05,821][ceph_volume.process][INFO  ] stdout 
> ceph.block_device=/dev/ceph-7dabaa58-cee9-4388-a305-2253e4823347/osd-block-48c13ab5-2953-4a06-b588-d6046d345568,ceph.block_uuid=BFExbF-nA0i-78bJ-0QQp-0Yyw-FqGN-yNz3hd,ceph.cephx_lockbox_secret=AQAaDmhi5qTQMRAANY9Pizy8lLecb7QmZh5g8w==,ceph.cluster_fsid=03fc24b7-762f-4f73-9e0f-c737d7344546,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=1,ceph.osd_fsid=48c13ab5-2953-4a06-b588-d6046d345568,ceph.osd_id=81,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-7dabaa58-cee9-4388-a305-2253e4823347/osd-block-48c13ab5-2953-4a06-b588-d6046d345568";"osd-block-48c13ab5-2953-4a06-b588-d6046d345568";"ceph-7dabaa58-cee9-4388-a305-2253e4823347";"BFExbF-nA0i-78bJ-0QQp-0Yyw-FqGN-yNz3hd";"3200627245056
> [2022-12-23 16:29:05,822][ceph_volume.devices.lvm.activate][INFO  ] auto 
> detecting objectstore
> [2022-12-23 16:29:05,822][ceph_volume.devices.lvm.activate][INFO  ] auto 
> detecting objectstore
> [2022-12-23 16:29:05,822][ceph_volume.devices.lvm.activate][INFO  ] 
> unable to find a journal associated with the OSD, assuming bluestore
> [2022-12-23 16:29:05,822][ceph_volume.devices.lvm.activate][INFO  ] 
> unable to find a journal associated with the OSD, assuming bluestore
> [2022-12-23 16:29:05,822][ceph_volume.devices.lvm.activate][INFO  ] auto 
> detecting objectstore
> [2022-12-23 16:29:05,822][ceph_volume.devices.lvm.activate][INFO  ] 
> unable to find a journal associated with the OSD, assuming bluestore
> [2022-12-23 16:29:05,822][ceph_volume.devices.lvm.activate][INFO  ] auto 
> detecting objectstore
> [2022-12-23 16:29:05,822][ceph_volume.devices.lvm.activate][INFO  ] auto 
> detecting objectstore
> [2022-12-23 16:29:05,822][ceph_volume.devices.lvm.activate][INFO  ] auto 
> detecting objectstore
> [2022-12-23 16:29:05,822][ceph_volume.devices.lvm.activate][INFO  ] auto 
> detecting objectstore
> [2022-12-23 16:29:05,822][ceph_volume.devices.lvm.activate][INFO  ] 
> unable to find a journal associated with the OSD, assuming bluestore
> [2022-12-23 16:29:05,822][ceph_volume.devices.lvm.activate][INFO  ] 
> unable to find a journal associated with the OSD, assuming bluestore
> [2022-12-23 16:29:05,822][ceph_volume.devices.lvm.activate][INFO  ] 
> unable to find a journal associated with the OSD, assuming bluestore
> [2022-12-23 16:29:05,822][ceph_volume.devices.lvm.activate][INFO  ] 
> unable to find a journal associated with the OSD, assuming bluestore
> [2022-12-23 16:29:05,823][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-96
> [2022-12-23 16:29:05,823][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-92
> [2022-12-23 16:29:05,823][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-81
> [2022-12-23 16:29:05,823][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-85
> [2022-12-23 16:29:05,823][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-114
> [2022-12-23 16:29:05,823][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-89
> [2022-12-23 16:29:05,823][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-118
> [2022-12-23 16:29:05,825][ceph_volume.process][INFO  ] stdout 
> ceph.block_device=/dev/ceph-0e0e4578-d3cc-4c55-bda3-444480eaed2c/osd-block-89436dda-f72b-4c65-a9af-972a373336f5,ceph.block_uuid=ydehEc-Jgcu-8L1F-ssUB-sAs3-Zamb-dAKGs4,ceph.cephx_lockbox_secret=AQCGDmhiwDvyJxAArdg1ZZoM49P08PfUuFRIZA==,ceph.cluster_fsid=03fc24b7-762f-4f73-9e0f-c737d7344546,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=1,ceph.osd_fsid=89436dda-f72b-4c65-a9af-972a373336f5,ceph.osd_id=106,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-0e0e4578-d3cc-4c55-bda3-444480eaed2c/osd-block-89436dda-f72b-4c65-a9af-972a373336f5";"osd-block-89436dda-f72b-4c65-a9af-972a373336f5";"ceph-0e0e4578-d3cc-4c55-bda3-444480eaed2c";"ydehEc-Jgcu-8L1F-ssUB-sAs3-Zamb-dAKGs4";"3200627245056
> [2022-12-23 16:29:05,825][ceph_volume.process][INFO  ] stdout 
> ceph.block_device=/dev/ceph-a933012f-d3c7-4cc1-8ab0-b20324996ce8/osd-block-a24b3f49-4824-4bab-8857-99b7c6795999,ceph.block_uuid=DdlPRD-dbyB-sOYn-rt1z-2ZPq-bsnZ-ldGqlZ,ceph.cephx_lockbox_secret=AQBzDmhiDSaaKxAAV7wd1HXf6uy/3NpV3p+erQ==,ceph.cluster_fsid=03fc24b7-762f-4f73-9e0f-c737d7344546,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=1,ceph.osd_fsid=a24b3f49-4824-4bab-8857-99b7c6795999,ceph.osd_id=100,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-a933012f-d3c7-4cc1-8ab0-b20324996ce8/osd-block-a24b3f49-4824-4bab-8857-99b7c6795999";"osd-block-a24b3f49-4824-4bab-8857-99b7c6795999";"ceph-a933012f-d3c7-4cc1-8ab0-b20324996ce8";"DdlPRD-dbyB-sOYn-rt1z-2ZPq-bsnZ-ldGqlZ";"3200627245056
> [2022-12-23 16:29:05,825][ceph_volume.devices.lvm.activate][INFO  ] auto 
> detecting objectstore
> [2022-12-23 16:29:05,825][ceph_volume.util.system][WARNING] Executable 
> selinuxenabled not in PATH: 
> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> [2022-12-23 16:29:05,825][ceph_volume.devices.lvm.activate][INFO  ] 
> unable to find a journal associated with the OSD, assuming bluestore
> [2022-12-23 16:29:05,825][ceph_volume.process][INFO  ] Running command: 
> selinuxenabled
> [2022-12-23 16:29:05,825][ceph_volume.devices.lvm.activate][INFO  ] auto 
> detecting objectstore
> [2022-12-23 16:29:05,825][ceph_volume.devices.lvm.activate][INFO  ] 
> unable to find a journal associated with the OSD, assuming bluestore
> [2022-12-23 16:29:05,825][ceph_volume.util.system][WARNING] Executable 
> selinuxenabled not in PATH: 
> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> [2022-12-23 16:29:05,825][ceph_volume.process][INFO  ] Running command: 
> selinuxenabled
> [2022-12-23 16:29:05,825][ceph_volume.util.system][WARNING] Executable 
> selinuxenabled not in PATH: 
> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> [2022-12-23 16:29:05,825][ceph_volume.process][INFO  ] Running command: 
> selinuxenabled
> [2022-12-23 16:29:05,826][ceph_volume.util.system][WARNING] Executable 
> selinuxenabled not in PATH: 
> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> [2022-12-23 16:29:05,826][ceph_volume.process][INFO  ] Running command: 
> selinuxenabled
> [2022-12-23 16:29:05,826][ceph_volume.util.system][WARNING] Executable 
> selinuxenabled not in PATH: 
> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> [2022-12-23 16:29:05,826][ceph_volume.process][INFO  ] Running command: 
> selinuxenabled
> [2022-12-23 16:29:05,826][ceph_volume.util.system][WARNING] Executable 
> selinuxenabled not in PATH: 
> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> [2022-12-23 16:29:05,826][ceph_volume.process][INFO  ] Running command: 
> selinuxenabled
> [2022-12-23 16:29:05,826][ceph_volume.util.system][WARNING] Executable 
> selinuxenabled not in PATH: 
> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> [2022-12-23 16:29:05,826][ceph_volume.util.system][INFO  ] No SELinux 
> found, skipping call to restorecon
> [2022-12-23 16:29:05,826][ceph_volume.process][INFO  ] Running command: 
> selinuxenabled
> [2022-12-23 16:29:05,826][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-85/lockbox.keyring 
> --create-keyring --name 
> client.osd-lockbox.7fb5b3c0-3cbf-4ae7-9fbc-a7605c1584ff --add-key 
> AQAtDmhiiT4vFBAA43UwmfNYx1ibCL5cZmJ54w==
> [2022-12-23 16:29:05,826][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-106
> [2022-12-23 16:29:05,827][ceph_volume.util.system][INFO  ] No SELinux 
> found, skipping call to restorecon
> [2022-12-23 16:29:05,827][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-100
> [2022-12-23 16:29:05,827][ceph_volume.util.system][INFO  ] No SELinux 
> found, skipping call to restorecon
> [2022-12-23 16:29:05,827][ceph_volume.util.system][INFO  ] No SELinux 
> found, skipping call to restorecon
> [2022-12-23 16:29:05,827][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-81/lockbox.keyring 
> --create-keyring --name 
> client.osd-lockbox.48c13ab5-2953-4a06-b588-d6046d345568 --add-key 
> AQAaDmhi5qTQMRAANY9Pizy8lLecb7QmZh5g8w==
> [2022-12-23 16:29:05,827][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-96/lockbox.keyring 
> --create-keyring --name 
> client.osd-lockbox.2e3c25c2-bfe5-4ecf-b402-5bd75872c568 --add-key 
> AQBiDmhiI323GRAArw2M/D7YFn82uFm3hZpdnw==
> [2022-12-23 16:29:05,827][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-92/lockbox.keyring 
> --create-keyring --name 
> client.osd-lockbox.db6b5f4a-563e-48f7-a6a3-92054820c816 --add-key 
> AQBQDmhiPI8fJhAA2zGVUAxLvPhoFzJdVYjGgg==
> [2022-12-23 16:29:05,827][ceph_volume.util.system][INFO  ] No SELinux 
> found, skipping call to restorecon
> [2022-12-23 16:29:05,827][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-89/lockbox.keyring 
> --create-keyring --name 
> client.osd-lockbox.5181c3d3-8ed8-4df7-a572-58154ec15482 --add-key 
> AQA/Dmhi/BxFExAAaxcqVkHssaDm/dJ8iyudaQ==
> [2022-12-23 16:29:05,827][ceph_volume.util.system][INFO  ] No SELinux 
> found, skipping call to restorecon
> [2022-12-23 16:29:05,827][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-114/lockbox.keyring 
> --create-keyring --name 
> client.osd-lockbox.5b4cb910-15f2-40f0-8666-3f880dfaeb1c --add-key 
> AQCqDmhi+P/CNRAAafmy4EWla+jRtbWlqk7u/Q==
> [2022-12-23 16:29:05,828][ceph_volume.util.system][INFO  ] No SELinux 
> found, skipping call to restorecon
> [2022-12-23 16:29:05,828][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-118/lockbox.keyring 
> --create-keyring --name 
> client.osd-lockbox.a7f37f45-c48a-47d3-8fe4-1690ea792658 --add-key 
> AQC9Dmhi1wZjOhAAvSVxnH/VrIdqzktAntJkRw==
> [2022-12-23 16:29:05,829][ceph_volume.util.system][WARNING] Executable 
> selinuxenabled not in PATH: 
> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> [2022-12-23 16:29:05,829][ceph_volume.util.system][WARNING] Executable 
> selinuxenabled not in PATH: 
> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> [2022-12-23 16:29:05,829][ceph_volume.process][INFO  ] Running command: 
> selinuxenabled
> [2022-12-23 16:29:05,829][ceph_volume.process][INFO  ] Running command: 
> selinuxenabled
> [2022-12-23 16:29:05,830][ceph_volume.util.system][INFO  ] No SELinux 
> found, skipping call to restorecon
> [2022-12-23 16:29:05,830][ceph_volume.util.system][INFO  ] No SELinux 
> found, skipping call to restorecon
> [2022-12-23 16:29:05,830][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-100/lockbox.keyring 
> --create-keyring --name 
> client.osd-lockbox.a24b3f49-4824-4bab-8857-99b7c6795999 --add-key 
> AQBzDmhiDSaaKxAAV7wd1HXf6uy/3NpV3p+erQ==
> [2022-12-23 16:29:05,830][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-106/lockbox.keyring 
> --create-keyring --name 
> client.osd-lockbox.89436dda-f72b-4c65-a9af-972a373336f5 --add-key 
> AQCGDmhiwDvyJxAArdg1ZZoM49P08PfUuFRIZA==
> [2022-12-23 16:29:05,875][ceph_volume.process][INFO  ] stdout creating 
> /var/lib/ceph/osd/ceph-81/lockbox.keyring
> [2022-12-23 16:29:05,875][ceph_volume.process][INFO  ] stdout added 
> entity client.osd-lockbox.48c13ab5-2953-4a06-b588-d6046d345568 
> auth(key=AQAaDmhi5qTQMRAANY9Pizy8lLecb7QmZh5g8w==)
> [2022-12-23 16:29:05,875][ceph_volume.process][INFO  ] stdout creating 
> /var/lib/ceph/osd/ceph-106/lockbox.keyring
> [2022-12-23 16:29:05,875][ceph_volume.process][INFO  ] stdout creating 
> /var/lib/ceph/osd/ceph-92/lockbox.keyring
> [2022-12-23 16:29:05,875][ceph_volume.process][INFO  ] stdout creating 
> /var/lib/ceph/osd/ceph-114/lockbox.keyring
> [2022-12-23 16:29:05,875][ceph_volume.process][INFO  ] stdout creating 
> /var/lib/ceph/osd/ceph-85/lockbox.keyring
> [2022-12-23 16:29:05,875][ceph_volume.process][INFO  ] stdout creating 
> /var/lib/ceph/osd/ceph-100/lockbox.keyring
> [2022-12-23 16:29:05,875][ceph_volume.process][INFO  ] stdout creating 
> /var/lib/ceph/osd/ceph-118/lockbox.keyring
> [2022-12-23 16:29:05,875][ceph_volume.process][INFO  ] stdout creating 
> /var/lib/ceph/osd/ceph-89/lockbox.keyring
> [2022-12-23 16:29:05,877][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-89/lockbox.keyring
> [2022-12-23 16:29:05,878][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-81/lockbox.keyring
> [2022-12-23 16:29:05,878][ceph_volume.process][INFO  ] stdout creating 
> /var/lib/ceph/osd/ceph-96/lockbox.keyring
> [2022-12-23 16:29:05,878][ceph_volume.process][INFO  ] stdout creating 
> /var/lib/ceph/osd/ceph-110/lockbox.keyring
> [2022-12-23 16:29:05,878][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-85/lockbox.keyring
> [2022-12-23 16:29:05,878][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-92/lockbox.keyring
> [2022-12-23 16:29:05,878][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-114/lockbox.keyring
> [2022-12-23 16:29:05,878][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-106/lockbox.keyring
> [2022-12-23 16:29:05,878][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-100/lockbox.keyring
> [2022-12-23 16:29:05,878][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-118/lockbox.keyring
> [2022-12-23 16:29:05,879][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-96/lockbox.keyring
> [2022-12-23 16:29:05,879][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-110/lockbox.keyring
> [2022-12-23 16:29:05,879][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph --cluster ceph --name 
> client.osd-lockbox.5181c3d3-8ed8-4df7-a572-58154ec15482 --keyring 
> /var/lib/ceph/osd/ceph-89/lockbox.keyring config-key get 
> dm-crypt/osd/5181c3d3-8ed8-4df7-a572-58154ec15482/luks
> [2022-12-23 16:29:05,879][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph --cluster ceph --name 
> client.osd-lockbox.7fb5b3c0-3cbf-4ae7-9fbc-a7605c1584ff --keyring 
> /var/lib/ceph/osd/ceph-85/lockbox.keyring config-key get 
> dm-crypt/osd/7fb5b3c0-3cbf-4ae7-9fbc-a7605c1584ff/luks
> [2022-12-23 16:29:05,880][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph --cluster ceph --name 
> client.osd-lockbox.48c13ab5-2953-4a06-b588-d6046d345568 --keyring 
> /var/lib/ceph/osd/ceph-81/lockbox.keyring config-key get 
> dm-crypt/osd/48c13ab5-2953-4a06-b588-d6046d345568/luks
> [2022-12-23 16:29:05,880][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph --cluster ceph --name 
> client.osd-lockbox.5b4cb910-15f2-40f0-8666-3f880dfaeb1c --keyring 
> /var/lib/ceph/osd/ceph-114/lockbox.keyring config-key get 
> dm-crypt/osd/5b4cb910-15f2-40f0-8666-3f880dfaeb1c/luks
> [2022-12-23 16:29:05,880][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph --cluster ceph --name 
> client.osd-lockbox.db6b5f4a-563e-48f7-a6a3-92054820c816 --keyring 
> /var/lib/ceph/osd/ceph-92/lockbox.keyring config-key get 
> dm-crypt/osd/db6b5f4a-563e-48f7-a6a3-92054820c816/luks
> [2022-12-23 16:29:05,880][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph --cluster ceph --name 
> client.osd-lockbox.89436dda-f72b-4c65-a9af-972a373336f5 --keyring 
> /var/lib/ceph/osd/ceph-106/lockbox.keyring config-key get 
> dm-crypt/osd/89436dda-f72b-4c65-a9af-972a373336f5/luks
> [2022-12-23 16:29:05,880][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph --cluster ceph --name 
> client.osd-lockbox.a24b3f49-4824-4bab-8857-99b7c6795999 --keyring 
> /var/lib/ceph/osd/ceph-100/lockbox.keyring config-key get 
> dm-crypt/osd/a24b3f49-4824-4bab-8857-99b7c6795999/luks
> [2022-12-23 16:29:05,880][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph --cluster ceph --name 
> client.osd-lockbox.a7f37f45-c48a-47d3-8fe4-1690ea792658 --keyring 
> /var/lib/ceph/osd/ceph-118/lockbox.keyring config-key get 
> dm-crypt/osd/a7f37f45-c48a-47d3-8fe4-1690ea792658/luks
> [2022-12-23 16:29:05,880][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph --cluster ceph --name 
> client.osd-lockbox.2b7d8995-8924-479f-aded-c0c83d7c6465 --keyring 
> /var/lib/ceph/osd/ceph-110/lockbox.keyring config-key get 
> dm-crypt/osd/2b7d8995-8924-479f-aded-c0c83d7c6465/luks
> [2022-12-23 16:29:05,881][ceph_volume.process][INFO  ] Running command: 
> /usr/bin/ceph --cluster ceph --name 
> client.osd-lockbox.2e3c25c2-bfe5-4ecf-b402-5bd75872c568 --keyring 
> /var/lib/ceph/osd/ceph-96/lockbox.keyring config-key get 
> dm-crypt/osd/2e3c25c2-bfe5-4ecf-b402-5bd75872c568/luks
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx